Extract Inner Feature Selection Archives
Source:R/extract_inner_fselect_archives.R
extract_inner_fselect_archives.RdExtract inner feature selection archives of nested resampling.
Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult.
The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table().
AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.
Arguments
- x
- exclude_columns
(
character())
Exclude columns from result table. Set toNULLif no column should be excluded.
Data structure
The returned data table has the following columns:
experiment(integer(1))
Index, giving the according row number in the original benchmark grid.iteration(integer(1))
Iteration of the outer resampling.One column for each feature of the task.
One column for each performance measure.
runtime_learners(numeric(1))
Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.timestamp(POSIXct)
Time stamp when the evaluation was logged into the archive.batch_nr(integer(1))
Feature sets are evaluated in batches. Each batch has a unique batch number.resample_result(mlr3::ResampleResult)
Resample result of the inner resampling.task_id(character(1)).learner_id(character(1)).resampling_id(character(1)).
Examples
# Nested Resampling on Palmer Penguins Data Set
# create auto fselector
at = auto_fselector(
fselector = fs("random_search"),
learner = lrn("classif.rpart"),
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)
# extract inner archives
extract_inner_fselect_archives(rr)
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> <int> <lgcl> <lgcl> <lgcl> <lgcl> <lgcl> <lgcl>
#> 1: 1 FALSE FALSE TRUE TRUE FALSE TRUE
#> 2: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 3: 1 FALSE FALSE FALSE FALSE TRUE FALSE
#> 4: 1 FALSE FALSE FALSE TRUE FALSE FALSE
#> 5: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 6: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 7: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 8: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 9: 1 FALSE TRUE TRUE FALSE FALSE FALSE
#> 10: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 11: 2 TRUE TRUE TRUE FALSE TRUE FALSE
#> 12: 2 FALSE FALSE FALSE FALSE FALSE FALSE
#> 13: 2 TRUE TRUE TRUE TRUE TRUE FALSE
#> 14: 2 TRUE TRUE TRUE TRUE TRUE TRUE
#> 15: 2 FALSE FALSE TRUE FALSE FALSE FALSE
#> 16: 2 TRUE FALSE TRUE FALSE FALSE TRUE
#> 17: 2 FALSE TRUE TRUE TRUE TRUE FALSE
#> 18: 2 TRUE TRUE TRUE TRUE TRUE FALSE
#> 19: 2 FALSE TRUE FALSE FALSE TRUE TRUE
#> 20: 2 FALSE TRUE TRUE TRUE TRUE TRUE
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> <lgcl> <num> <num> <POSc> <int> <int>
#> 1: TRUE 0.26315789 0.005 2024-11-07 21:50:08 1 0
#> 2: TRUE 0.07017544 0.005 2024-11-07 21:50:08 1 0
#> 3: FALSE 0.36842105 0.004 2024-11-07 21:50:08 1 0
#> 4: FALSE 0.26315789 0.004 2024-11-07 21:50:08 1 0
#> 5: TRUE 0.07017544 0.005 2024-11-07 21:50:08 1 0
#> 6: TRUE 0.07017544 0.005 2024-11-07 21:50:08 1 0
#> 7: TRUE 0.07017544 0.005 2024-11-07 21:50:08 1 0
#> 8: FALSE 0.07017544 0.005 2024-11-07 21:50:08 1 0
#> 9: FALSE 0.05263158 0.005 2024-11-07 21:50:08 1 0
#> 10: TRUE 0.07017544 0.006 2024-11-07 21:50:08 1 0
#> 11: TRUE 0.07017544 0.026 2024-11-07 21:50:09 1 0
#> 12: TRUE 0.64912281 0.004 2024-11-07 21:50:09 1 0
#> 13: TRUE 0.05263158 0.005 2024-11-07 21:50:09 1 0
#> 14: FALSE 0.05263158 0.006 2024-11-07 21:50:09 1 0
#> 15: TRUE 0.36842105 0.005 2024-11-07 21:50:09 1 0
#> 16: FALSE 0.40350877 0.005 2024-11-07 21:50:09 1 0
#> 17: TRUE 0.05263158 0.006 2024-11-07 21:50:09 1 0
#> 18: TRUE 0.05263158 0.005 2024-11-07 21:50:09 1 0
#> 19: TRUE 0.07017544 0.005 2024-11-07 21:50:09 1 0
#> 20: TRUE 0.05263158 0.006 2024-11-07 21:50:09 1 0
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> errors features
#> <int> <list>
#> 1: 0 body_mass,flipper_length,sex,year
#> 2: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 3: 0 island
#> 4: 0 flipper_length
#> 5: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 6: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 7: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 8: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex
#> 9: 0 bill_length,body_mass
#> 10: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 11: 0 bill_depth,bill_length,body_mass,island,year
#> 12: 0 year
#> 13: 0 bill_depth,bill_length,body_mass,flipper_length,island,year
#> 14: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex
#> 15: 0 body_mass,year
#> 16: 0 bill_depth,body_mass,sex
#> 17: 0 bill_length,body_mass,flipper_length,island,year
#> 18: 0 bill_depth,bill_length,body_mass,flipper_length,island,year
#> 19: 0 bill_length,island,sex,year
#> 20: 0 bill_length,body_mass,flipper_length,island,sex,year
#> errors features
#> n_features resample_result task_id learner_id resampling_id
#> <list> <list> <char> <char> <char>
#> 1: 4 <ResampleResult> penguins classif.rpart.fselector cv
#> 2: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 3: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 4: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 5: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 6: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 7: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 8: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 9: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 10: 7 <ResampleResult> penguins classif.rpart.fselector cv
#> 11: 5 <ResampleResult> penguins classif.rpart.fselector cv
#> 12: 1 <ResampleResult> penguins classif.rpart.fselector cv
#> 13: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 14: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 15: 2 <ResampleResult> penguins classif.rpart.fselector cv
#> 16: 3 <ResampleResult> penguins classif.rpart.fselector cv
#> 17: 5 <ResampleResult> penguins classif.rpart.fselector cv
#> 18: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> 19: 4 <ResampleResult> penguins classif.rpart.fselector cv
#> 20: 6 <ResampleResult> penguins classif.rpart.fselector cv
#> n_features resample_result task_id learner_id resampling_id