Skip to contents

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

x

(mlr3::ResampleResult | mlr3::BenchmarkResult).

exclude_columns

(character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Data structure

The returned data table has the following columns:

  • experiment (integer(1))
    Index, giving the according row number in the original benchmark grid.

  • iteration (integer(1))
    Iteration of the outer resampling.

  • One column for each feature of the task.

  • One column for each performance measure.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Feature sets are evaluated in batches. Each batch has a unique batch number.

  • resample_result (mlr3::ResampleResult)
    Resample result of the inner resampling.

  • task_id (character(1)).

  • learner_id (character(1)).

  • resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  method = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)
#>    iteration bill_depth bill_length body_mass flipper_length island   sex  year
#> 1:         1       TRUE       FALSE     FALSE          FALSE  FALSE FALSE FALSE
#> 2:         1      FALSE        TRUE     FALSE          FALSE   TRUE  TRUE  TRUE
#> 3:         1      FALSE        TRUE     FALSE          FALSE  FALSE  TRUE FALSE
#> 4:         1      FALSE        TRUE      TRUE          FALSE  FALSE  TRUE FALSE
#> 5:         2       TRUE       FALSE     FALSE          FALSE  FALSE FALSE  TRUE
#> 6:         2      FALSE       FALSE      TRUE          FALSE  FALSE FALSE  TRUE
#> 7:         2      FALSE       FALSE      TRUE           TRUE  FALSE FALSE FALSE
#> 8:         2      FALSE        TRUE     FALSE          FALSE   TRUE FALSE  TRUE
#>    classif.ce runtime_learners           timestamp batch_nr warnings errors
#> 1: 0.26315789            0.020 2023-01-26 18:33:52        1        0      0
#> 2: 0.03508772            0.007 2023-01-26 18:33:52        2        0      0
#> 3: 0.28070175            0.007 2023-01-26 18:33:52        3        0      0
#> 4: 0.10526316            0.008 2023-01-26 18:33:52        4        0      0
#> 5: 0.17543860            0.007 2023-01-26 18:33:53        1        0      0
#> 6: 0.19298246            0.007 2023-01-26 18:33:53        2        0      0
#> 7: 0.10526316            0.007 2023-01-26 18:33:53        3        0      0
#> 8: 0.05263158            0.006 2023-01-26 18:33:53        4        0      0
#>         resample_result  task_id              learner_id resampling_id
#> 1: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 2: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 3: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 4: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 5: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 6: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 7: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 8: <ResampleResult[21]> penguins classif.rpart.fselector            cv