Extract Inner Feature Selection Archives
Source:R/extract_inner_fselect_archives.R
extract_inner_fselect_archives.RdExtract inner feature selection archives of nested resampling.
Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult.
The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table().
AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.
Arguments
- x
- exclude_columns
(
character())
Exclude columns from result table. Set toNULLif no column should be excluded.
Data structure
The returned data table has the following columns:
experiment(integer(1))
Index, giving the according row number in the original benchmark grid.iteration(integer(1))
Iteration of the outer resampling.One column for each feature of the task.
One column for each performance measure.
runtime_learners(numeric(1))
Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.timestamp(POSIXct)
Time stamp when the evaluation was logged into the archive.batch_nr(integer(1))
Feature sets are evaluated in batches. Each batch has a unique batch number.resample_result(mlr3::ResampleResult)
Resample result of the inner resampling.task_id(character(1)).learner_id(character(1)).resampling_id(character(1)).
Examples
# Nested Resampling on Palmer Penguins Data Set
# create auto fselector
at = auto_fselector(
fselector = fs("random_search"),
learner = lrn("classif.rpart"),
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)
# extract inner archives
extract_inner_fselect_archives(rr)
#> iteration bill_depth bill_length body_mass flipper_length island sex
#> 1: 1 TRUE TRUE TRUE FALSE FALSE TRUE
#> 2: 1 FALSE FALSE FALSE FALSE TRUE FALSE
#> 3: 1 FALSE TRUE FALSE FALSE TRUE FALSE
#> 4: 1 FALSE TRUE TRUE TRUE FALSE TRUE
#> 5: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 6: 1 TRUE TRUE FALSE TRUE TRUE TRUE
#> 7: 1 TRUE TRUE TRUE TRUE TRUE TRUE
#> 8: 1 FALSE FALSE FALSE TRUE FALSE FALSE
#> 9: 1 FALSE FALSE FALSE FALSE FALSE TRUE
#> 10: 1 FALSE FALSE FALSE FALSE TRUE FALSE
#> 11: 2 FALSE FALSE FALSE TRUE FALSE FALSE
#> 12: 2 TRUE TRUE FALSE FALSE FALSE TRUE
#> 13: 2 FALSE FALSE TRUE FALSE FALSE FALSE
#> 14: 2 TRUE FALSE TRUE TRUE FALSE TRUE
#> 15: 2 FALSE FALSE FALSE TRUE FALSE FALSE
#> 16: 2 FALSE FALSE TRUE FALSE FALSE FALSE
#> 17: 2 TRUE TRUE FALSE TRUE TRUE TRUE
#> 18: 2 FALSE TRUE FALSE TRUE TRUE FALSE
#> 19: 2 FALSE TRUE FALSE FALSE FALSE FALSE
#> 20: 2 TRUE TRUE TRUE TRUE FALSE FALSE
#> year classif.ce runtime_learners timestamp batch_nr warnings
#> 1: TRUE 0.12280702 0.008 2023-07-05 08:07:12 1 0
#> 2: FALSE 0.29824561 0.007 2023-07-05 08:07:12 1 0
#> 3: FALSE 0.03508772 0.008 2023-07-05 08:07:12 1 0
#> 4: FALSE 0.10526316 0.009 2023-07-05 08:07:12 1 0
#> 5: TRUE 0.08771930 0.009 2023-07-05 08:07:12 1 0
#> 6: TRUE 0.08771930 0.010 2023-07-05 08:07:12 1 0
#> 7: TRUE 0.08771930 0.009 2023-07-05 08:07:12 1 0
#> 8: TRUE 0.22807018 0.008 2023-07-05 08:07:12 1 0
#> 9: FALSE 0.70175439 0.007 2023-07-05 08:07:12 1 0
#> 10: FALSE 0.29824561 0.009 2023-07-05 08:07:12 1 0
#> 11: FALSE 0.33333333 0.009 2023-07-05 08:07:13 1 0
#> 12: TRUE 0.12280702 0.008 2023-07-05 08:07:13 1 0
#> 13: TRUE 0.35087719 0.008 2023-07-05 08:07:13 1 0
#> 14: TRUE 0.33333333 0.011 2023-07-05 08:07:13 1 0
#> 15: FALSE 0.33333333 0.012 2023-07-05 08:07:13 1 0
#> 16: FALSE 0.35087719 0.009 2023-07-05 08:07:13 1 0
#> 17: TRUE 0.08771930 0.033 2023-07-05 08:07:13 1 0
#> 18: FALSE 0.08771930 0.010 2023-07-05 08:07:13 1 0
#> 19: FALSE 0.29824561 0.007 2023-07-05 08:07:13 1 0
#> 20: FALSE 0.08771930 0.009 2023-07-05 08:07:13 1 0
#> errors features
#> 1: 0 bill_depth,bill_length,body_mass,sex,year
#> 2: 0 island
#> 3: 0 bill_length,island
#> 4: 0 bill_length,body_mass,flipper_length,sex
#> 5: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 6: 0 bill_depth,bill_length,flipper_length,island,sex,year
#> 7: 0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 8: 0 flipper_length,year
#> 9: 0 sex
#> 10: 0 island
#> 11: 0 flipper_length
#> 12: 0 bill_depth,bill_length,sex,year
#> 13: 0 body_mass,year
#> 14: 0 bill_depth,body_mass,flipper_length,sex,year
#> 15: 0 flipper_length
#> 16: 0 body_mass
#> 17: 0 bill_depth,bill_length,flipper_length,island,sex,year
#> 18: 0 bill_length,flipper_length,island
#> 19: 0 bill_length
#> 20: 0 bill_depth,bill_length,body_mass,flipper_length
#> resample_result task_id learner_id resampling_id
#> 1: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 2: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 3: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 4: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 5: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 6: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 7: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 8: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 9: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 10: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 11: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 12: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 13: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 14: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 15: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 16: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 17: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 18: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 19: <ResampleResult[21]> penguins classif.rpart.fselector cv
#> 20: <ResampleResult[21]> penguins classif.rpart.fselector cv