Extract Inner Feature Selection Archives
Source:R/extract_inner_fselect_archives.R
extract_inner_fselect_archives.RdExtract inner feature selection archives of nested resampling.
Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult.
The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table().
AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.
Arguments
- x
- exclude_columns
(
character())
Exclude columns from result table. Set toNULLif no column should be excluded.
Data structure
The returned data table has the following columns:
experiment(integer(1))
Index, giving the according row number in the original benchmark grid.iteration(integer(1))
Iteration of the outer resampling.One column for each feature of the task.
One column for each performance measure.
runtime_learners(numeric(1))
Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.timestamp(POSIXct)
Time stamp when the evaluation was logged into the archive.batch_nr(integer(1))
Feature sets are evaluated in batches. Each batch has a unique batch number.resample_result(mlr3::ResampleResult)
Resample result of the inner resampling.task_id(character(1)).learner_id(character(1)).resampling_id(character(1)).
Examples
# Nested Resampling on Palmer Penguins Data Set
# create auto fselector
at = auto_fselector(
fselector = fs("random_search"),
learner = lrn("classif.rpart"),
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)
# extract inner archives
extract_inner_fselect_archives(rr)
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> <int> <lgcl> <lgcl> <lgcl> <lgcl> <lgcl> <lgcl>
#> 1: 1 FALSE TRUE FALSE FALSE FALSE FALSE
#> 2: 1 FALSE TRUE FALSE TRUE TRUE FALSE
#> 3: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 4: 1 TRUE TRUE FALSE TRUE TRUE TRUE
#> 5: 1 FALSE TRUE FALSE FALSE TRUE FALSE
#> 6: 1 TRUE FALSE TRUE TRUE FALSE FALSE
#> 7: 1 TRUE FALSE TRUE FALSE FALSE FALSE
#> 8: 1 FALSE FALSE FALSE FALSE TRUE TRUE
#> 9: 1 TRUE FALSE TRUE TRUE TRUE TRUE
#> 10: 1 FALSE FALSE TRUE FALSE TRUE TRUE
#> 11: 2 FALSE TRUE FALSE FALSE FALSE TRUE
#> 12: 2 TRUE FALSE TRUE TRUE TRUE TRUE
#> 13: 2 TRUE FALSE FALSE FALSE TRUE FALSE
#> 14: 2 TRUE FALSE FALSE FALSE TRUE FALSE
#> 15: 2 TRUE TRUE TRUE TRUE TRUE TRUE
#> 16: 2 TRUE FALSE FALSE FALSE FALSE FALSE
#> 17: 2 TRUE FALSE FALSE FALSE FALSE TRUE
#> 18: 2 TRUE TRUE FALSE TRUE FALSE TRUE
#> 19: 2 FALSE TRUE FALSE FALSE FALSE FALSE
#> 20: 2 FALSE FALSE FALSE TRUE FALSE TRUE
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> <lgcl> <num> <num> <POSc> <int> <int>
#> 1: FALSE 0.26315789 0.005 2025-05-22 14:34:57 1 0
#> 2: FALSE 0.10526316 0.005 2025-05-22 14:34:57 1 0
#> 3: TRUE 0.10526316 0.006 2025-05-22 14:34:57 1 0
#> 4: TRUE 0.10526316 0.005 2025-05-22 14:34:57 1 0
#> 5: FALSE 0.07017544 0.004 2025-05-22 14:34:57 1 0
#> 6: FALSE 0.21052632 0.003 2025-05-22 14:34:57 1 0
#> 7: FALSE 0.38596491 0.004 2025-05-22 14:34:57 1 0
#> 8: TRUE 0.29824561 0.005 2025-05-22 14:34:57 1 0
#> 9: TRUE 0.21052632 0.005 2025-05-22 14:34:57 1 0
#> 10: FALSE 0.22807018 0.005 2025-05-22 14:34:57 1 0
#> 11: FALSE 0.21052632 0.006 2025-05-22 14:34:57 1 0
#> 12: TRUE 0.10526316 0.006 2025-05-22 14:34:57 1 0
#> 13: FALSE 0.29824561 0.005 2025-05-22 14:34:57 1 0
#> 14: FALSE 0.29824561 0.004 2025-05-22 14:34:57 1 0
#> 15: TRUE 0.01754386 0.005 2025-05-22 14:34:57 1 0
#> 16: FALSE 0.35087719 0.005 2025-05-22 14:34:57 1 0
#> 17: FALSE 0.35087719 0.005 2025-05-22 14:34:57 1 0
#> 18: TRUE 0.01754386 0.005 2025-05-22 14:34:57 1 0
#> 19: FALSE 0.21052632 0.004 2025-05-22 14:34:57 1 0
#> 20: TRUE 0.15789474 0.005 2025-05-22 14:34:57 1 0
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> errors features
#> <int> <list>
#> 1: 0 bill_length
#> 2: 0 bill_length,flipper_length,island
#> 3: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 4: 0 bill_depth,bill_length,flipper_length,island,sex,year
#> 5: 0 bill_length,island
#> 6: 0 bill_depth,body_mass,flipper_length
#> 7: 0 bill_depth,body_mass
#> 8: 0 island,sex,year
#> 9: 0 bill_depth,body_mass,flipper_length,island,sex,year
#> 10: 0 body_mass,island,sex
#> 11: 0 bill_length,sex
#> 12: 0 bill_depth,body_mass,flipper_length,island,sex,year
#> 13: 0 bill_depth,island
#> 14: 0 bill_depth,island
#> 15: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 16: 0 bill_depth
#> 17: 0 bill_depth,sex
#> 18: 0 bill_depth,bill_length,flipper_length,sex,year
#> 19: 0 bill_length
#> 20: 0 flipper_length,sex,year
#> errors features
#> n_features resample_result task_id learner_id resampling_id
#> <list> <list> <char> <char> <char>
#> 1: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 2: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 3: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 4: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 5: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 6: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 7: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 8: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 9: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 10: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 11: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 12: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 13: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 14: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 15: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 16: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 17: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 18: 5 <ResampleResult> penguins classif.rpart.fselector cv
#> 19: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 20: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> n_features resample_result task_id learner_id resampling_id