Extract Inner Feature Selection Archives
Source:R/extract_inner_fselect_archives.R
extract_inner_fselect_archives.RdExtract inner feature selection archives of nested resampling.
Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult.
The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table().
AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.
Arguments
- x
- exclude_columns
(
character())
Exclude columns from result table. Set toNULLif no column should be excluded.
Data structure
The returned data table has the following columns:
experiment(integer(1))
Index, giving the according row number in the original benchmark grid.iteration(integer(1))
Iteration of the outer resampling.One column for each feature of the task.
One column for each performance measure.
runtime_learners(numeric(1))
Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.timestamp(POSIXct)
Time stamp when the evaluation was logged into the archive.batch_nr(integer(1))
Feature sets are evaluated in batches. Each batch has a unique batch number.resample_result(mlr3::ResampleResult)
Resample result of the inner resampling.task_id(character(1)).learner_id(character(1)).resampling_id(character(1)).
Examples
# Nested Resampling on Palmer Penguins Data Set
# create auto fselector
at = auto_fselector(
fselector = fs("random_search"),
learner = lrn("classif.rpart"),
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)
# extract inner archives
extract_inner_fselect_archives(rr)
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> <int> <lgcl> <lgcl> <lgcl> <lgcl> <lgcl> <lgcl>
#> 1: 1 TRUE TRUE FALSE TRUE TRUE TRUE
#> 2: 1 FALSE TRUE FALSE FALSE FALSE FALSE
#> 3: 1 TRUE TRUE TRUE FALSE TRUE TRUE
#> 4: 1 TRUE TRUE TRUE TRUE FALSE TRUE
#> 5: 1 FALSE FALSE FALSE FALSE TRUE FALSE
#> 6: 1 TRUE TRUE FALSE TRUE TRUE TRUE
#> 7: 1 FALSE TRUE FALSE FALSE TRUE FALSE
#> 8: 1 TRUE FALSE TRUE TRUE TRUE FALSE
#> 9: 1 TRUE FALSE FALSE TRUE FALSE TRUE
#> 10: 1 FALSE FALSE TRUE TRUE FALSE FALSE
#> 11: 2 FALSE FALSE FALSE FALSE FALSE FALSE
#> 12: 2 FALSE FALSE FALSE TRUE FALSE FALSE
#> 13: 2 FALSE FALSE TRUE FALSE FALSE FALSE
#> 14: 2 TRUE TRUE TRUE TRUE TRUE TRUE
#> 15: 2 TRUE FALSE TRUE TRUE TRUE FALSE
#> 16: 2 TRUE TRUE FALSE TRUE TRUE TRUE
#> 17: 2 FALSE TRUE FALSE FALSE FALSE FALSE
#> 18: 2 TRUE TRUE TRUE FALSE TRUE TRUE
#> 19: 2 FALSE FALSE FALSE TRUE TRUE FALSE
#> 20: 2 FALSE FALSE TRUE FALSE FALSE TRUE
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> <lgcl> <num> <num> <POSc> <int> <int>
#> 1: TRUE 0.03508772 0.007 2024-07-24 12:01:03 1 0
#> 2: TRUE 0.19298246 0.005 2024-07-24 12:01:03 1 0
#> 3: FALSE 0.03508772 0.005 2024-07-24 12:01:03 1 0
#> 4: FALSE 0.03508772 0.005 2024-07-24 12:01:03 1 0
#> 5: TRUE 0.29824561 0.004 2024-07-24 12:01:03 1 0
#> 6: TRUE 0.03508772 0.005 2024-07-24 12:01:03 1 0
#> 7: FALSE 0.03508772 0.005 2024-07-24 12:01:03 1 0
#> 8: FALSE 0.10526316 0.005 2024-07-24 12:01:03 1 0
#> 9: FALSE 0.15789474 0.005 2024-07-24 12:01:03 1 0
#> 10: FALSE 0.17543860 0.004 2024-07-24 12:01:03 1 0
#> 11: TRUE 0.57894737 0.005 2024-07-24 12:01:03 1 0
#> 12: FALSE 0.19298246 0.004 2024-07-24 12:01:03 1 0
#> 13: FALSE 0.28070175 0.005 2024-07-24 12:01:03 1 0
#> 14: TRUE 0.14035088 0.005 2024-07-24 12:01:03 1 0
#> 15: FALSE 0.21052632 0.005 2024-07-24 12:01:03 1 0
#> 16: TRUE 0.08771930 0.005 2024-07-24 12:01:03 1 0
#> 17: FALSE 0.33333333 0.005 2024-07-24 12:01:03 1 0
#> 18: FALSE 0.14035088 0.005 2024-07-24 12:01:03 1 0
#> 19: TRUE 0.12280702 0.004 2024-07-24 12:01:03 1 0
#> 20: FALSE 0.31578947 0.004 2024-07-24 12:01:03 1 0
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> errors features
#> <int> <list>
#> 1: 0 bill_depth,bill_length,flipper_length,island,sex,year
#> 2: 0 bill_length,year
#> 3: 0 bill_depth,bill_length,body_mass,island,sex
#> 4: 0 bill_depth,bill_length,body_mass,flipper_length,sex
#> 5: 0 island,year
#> 6: 0 bill_depth,bill_length,flipper_length,island,sex,year
#> 7: 0 bill_length,island
#> 8: 0 bill_depth,body_mass,flipper_length,island
#> 9: 0 bill_depth,flipper_length,sex
#> 10: 0 body_mass,flipper_length
#> 11: 0 year
#> 12: 0 flipper_length
#> 13: 0 body_mass
#> 14: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 15: 0 bill_depth,body_mass,flipper_length,island
#> 16: 0 bill_depth,bill_length,flipper_length,island,sex,year
#> 17: 0 bill_length
#> 18: 0 bill_depth,bill_length,body_mass,island,sex
#> 19: 0 flipper_length,island,year
#> 20: 0 body_mass,sex
#> errors features
#> n_features resample_result task_id learner_id resampling_id
#> <list> <list> <char> <char> <char>
#> 1: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 2: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 3: 5 <ResampleResult> penguins classif.rpart.fselector cv
#> 4: 5 <ResampleResult> penguins classif.rpart.fselector cv
#> 5: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 6: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 7: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 8: 4 <ResampleResult> penguins classif.rpart.fselector cv
#> 9: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 10: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 11: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 12: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 13: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 14: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 15: 4 <ResampleResult> penguins classif.rpart.fselector cv
#> 16: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 17: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 18: 5 <ResampleResult> penguins classif.rpart.fselector cv
#> 19: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 20: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> n_features resample_result task_id learner_id resampling_id