Extract Inner Feature Selection Archives — extract_inner_fselect

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

x: (mlr3::ResampleResult | mlr3::BenchmarkResult).
exclude_columns: (character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Value

data.table::data.table().

Data structure

The returned data table has the following columns:

experiment (integer(1))
Index, giving the according row number in the original benchmark grid.
iteration (integer(1))
Iteration of the outer resampling.
One column for each feature of the task.
One column for each performance measure.
runtime_learners (numeric(1))
Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.
timestamp (POSIXct)
Time stamp when the evaluation was logged into the archive.
batch_nr (integer(1))
Feature sets are evaluated in batches. Each batch has a unique batch number.
resample_result (mlr3::ResampleResult)
Resample result of the inner resampling.
task_id (character(1)).
learner_id (character(1)).
resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)
#>    iteration bill_depth bill_length body_mass flipper_length island   sex year
#> 1:         1       TRUE       FALSE     FALSE          FALSE  FALSE FALSE TRUE
#> 2:         1       TRUE        TRUE     FALSE          FALSE  FALSE  TRUE TRUE
#> 3:         1       TRUE        TRUE      TRUE          FALSE  FALSE FALSE TRUE
#> 4:         1       TRUE       FALSE     FALSE           TRUE   TRUE  TRUE TRUE
#> 5:         2       TRUE        TRUE      TRUE           TRUE   TRUE FALSE TRUE
#> 6:         2       TRUE       FALSE     FALSE          FALSE   TRUE FALSE TRUE
#> 7:         2       TRUE       FALSE      TRUE          FALSE  FALSE  TRUE TRUE
#> 8:         2       TRUE        TRUE      TRUE           TRUE   TRUE  TRUE TRUE
#>    classif.ce runtime_learners           timestamp batch_nr warnings errors
#> 1: 0.40350877            0.007 2023-03-02 12:42:20        1        0      0
#> 2: 0.05263158            0.009 2023-03-02 12:42:20        2        0      0
#> 3: 0.05263158            0.009 2023-03-02 12:42:20        3        0      0
#> 4: 0.15789474            0.010 2023-03-02 12:42:20        4        0      0
#> 5: 0.05263158            0.008 2023-03-02 12:42:20        1        0      0
#> 6: 0.12280702            0.008 2023-03-02 12:42:20        2        0      0
#> 7: 0.22807018            0.008 2023-03-02 12:42:20        3        0      0
#> 8: 0.05263158            0.009 2023-03-02 12:42:21        4        0      0
#>                                                          features
#> 1:                                                bill_depth,year
#> 2:                                bill_depth,bill_length,sex,year
#> 3:                          bill_depth,bill_length,body_mass,year
#> 4:                      bill_depth,flipper_length,island,sex,year
#> 5:    bill_depth,bill_length,body_mass,flipper_length,island,year
#> 6:                                         bill_depth,island,year
#> 7:                                  bill_depth,body_mass,sex,year
#> 8: bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>         resample_result  task_id              learner_id resampling_id
#> 1: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 2: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 3: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 4: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 5: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 6: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 7: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 8: <ResampleResult[21]> penguins classif.rpart.fselector            cv