Skip to contents

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

x

(mlr3::ResampleResult | mlr3::BenchmarkResult).

exclude_columns

(character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Data structure

The returned data table has the following columns:

  • experiment (integer(1))
    Index, giving the according row number in the original benchmark grid.

  • iteration (integer(1))
    Iteration of the outer resampling.

  • One column for each feature of the task.

  • One column for each performance measure.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Feature sets are evaluated in batches. Each batch has a unique batch number.

  • resample_result (mlr3::ResampleResult)
    Resample result of the inner resampling.

  • task_id (character(1)).

  • learner_id (character(1)).

  • resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)
#>     iteration bill_depth bill_length body_mass flipper_length island   sex
#>  1:         1       TRUE        TRUE      TRUE          FALSE  FALSE  TRUE
#>  2:         1      FALSE       FALSE     FALSE          FALSE   TRUE FALSE
#>  3:         1      FALSE        TRUE     FALSE          FALSE   TRUE FALSE
#>  4:         1      FALSE        TRUE      TRUE           TRUE  FALSE  TRUE
#>  5:         1       TRUE        TRUE      TRUE           TRUE   TRUE  TRUE
#>  6:         1       TRUE        TRUE     FALSE           TRUE   TRUE  TRUE
#>  7:         1       TRUE        TRUE      TRUE           TRUE   TRUE  TRUE
#>  8:         1      FALSE       FALSE     FALSE           TRUE  FALSE FALSE
#>  9:         1      FALSE       FALSE     FALSE          FALSE  FALSE  TRUE
#> 10:         1      FALSE       FALSE     FALSE          FALSE   TRUE FALSE
#> 11:         2      FALSE       FALSE     FALSE           TRUE  FALSE FALSE
#> 12:         2       TRUE        TRUE     FALSE          FALSE  FALSE  TRUE
#> 13:         2      FALSE       FALSE      TRUE          FALSE  FALSE FALSE
#> 14:         2       TRUE       FALSE      TRUE           TRUE  FALSE  TRUE
#> 15:         2      FALSE       FALSE     FALSE           TRUE  FALSE FALSE
#> 16:         2      FALSE       FALSE      TRUE          FALSE  FALSE FALSE
#> 17:         2       TRUE        TRUE     FALSE           TRUE   TRUE  TRUE
#> 18:         2      FALSE        TRUE     FALSE           TRUE   TRUE FALSE
#> 19:         2      FALSE        TRUE     FALSE          FALSE  FALSE FALSE
#> 20:         2       TRUE        TRUE      TRUE           TRUE  FALSE FALSE
#>      year classif.ce runtime_learners           timestamp batch_nr warnings
#>  1:  TRUE 0.12280702            0.008 2023-07-05 08:07:12        1        0
#>  2: FALSE 0.29824561            0.007 2023-07-05 08:07:12        1        0
#>  3: FALSE 0.03508772            0.008 2023-07-05 08:07:12        1        0
#>  4: FALSE 0.10526316            0.009 2023-07-05 08:07:12        1        0
#>  5:  TRUE 0.08771930            0.009 2023-07-05 08:07:12        1        0
#>  6:  TRUE 0.08771930            0.010 2023-07-05 08:07:12        1        0
#>  7:  TRUE 0.08771930            0.009 2023-07-05 08:07:12        1        0
#>  8:  TRUE 0.22807018            0.008 2023-07-05 08:07:12        1        0
#>  9: FALSE 0.70175439            0.007 2023-07-05 08:07:12        1        0
#> 10: FALSE 0.29824561            0.009 2023-07-05 08:07:12        1        0
#> 11: FALSE 0.33333333            0.009 2023-07-05 08:07:13        1        0
#> 12:  TRUE 0.12280702            0.008 2023-07-05 08:07:13        1        0
#> 13:  TRUE 0.35087719            0.008 2023-07-05 08:07:13        1        0
#> 14:  TRUE 0.33333333            0.011 2023-07-05 08:07:13        1        0
#> 15: FALSE 0.33333333            0.012 2023-07-05 08:07:13        1        0
#> 16: FALSE 0.35087719            0.009 2023-07-05 08:07:13        1        0
#> 17:  TRUE 0.08771930            0.033 2023-07-05 08:07:13        1        0
#> 18: FALSE 0.08771930            0.010 2023-07-05 08:07:13        1        0
#> 19: FALSE 0.29824561            0.007 2023-07-05 08:07:13        1        0
#> 20: FALSE 0.08771930            0.009 2023-07-05 08:07:13        1        0
#>     errors                                                       features
#>  1:      0                      bill_depth,bill_length,body_mass,sex,year
#>  2:      0                                                         island
#>  3:      0                                             bill_length,island
#>  4:      0                       bill_length,body_mass,flipper_length,sex
#>  5:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>  6:      0          bill_depth,bill_length,flipper_length,island,sex,year
#>  7:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>  8:      0                                            flipper_length,year
#>  9:      0                                                            sex
#> 10:      0                                                         island
#> 11:      0                                                 flipper_length
#> 12:      0                                bill_depth,bill_length,sex,year
#> 13:      0                                                 body_mass,year
#> 14:      0                   bill_depth,body_mass,flipper_length,sex,year
#> 15:      0                                                 flipper_length
#> 16:      0                                                      body_mass
#> 17:      0          bill_depth,bill_length,flipper_length,island,sex,year
#> 18:      0                              bill_length,flipper_length,island
#> 19:      0                                                    bill_length
#> 20:      0                bill_depth,bill_length,body_mass,flipper_length
#>          resample_result  task_id              learner_id resampling_id
#>  1: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  2: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  3: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  4: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  5: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  6: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  7: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  8: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#>  9: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 10: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 11: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 12: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 13: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 14: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 15: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 16: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 17: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 18: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 19: <ResampleResult[21]> penguins classif.rpart.fselector            cv
#> 20: <ResampleResult[21]> penguins classif.rpart.fselector            cv