Skip to contents

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

x

(mlr3::ResampleResult | mlr3::BenchmarkResult).

exclude_columns

(character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Data structure

The returned data table has the following columns:

  • experiment (integer(1))
    Index, giving the according row number in the original benchmark grid.

  • iteration (integer(1))
    Iteration of the outer resampling.

  • One column for each feature of the task.

  • One column for each performance measure.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Feature sets are evaluated in batches. Each batch has a unique batch number.

  • resample_result (mlr3::ResampleResult)
    Resample result of the inner resampling.

  • task_id (character(1)).

  • learner_id (character(1)).

  • resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)
#>     iteration bill_depth bill_length body_mass flipper_length island    sex
#>         <int>     <lgcl>      <lgcl>    <lgcl>         <lgcl> <lgcl> <lgcl>
#>  1:         1      FALSE       FALSE     FALSE          FALSE  FALSE  FALSE
#>  2:         1       TRUE       FALSE     FALSE          FALSE  FALSE  FALSE
#>  3:         1      FALSE       FALSE     FALSE          FALSE   TRUE  FALSE
#>  4:         1       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#>  5:         1      FALSE        TRUE     FALSE           TRUE  FALSE   TRUE
#>  6:         1       TRUE       FALSE     FALSE          FALSE   TRUE  FALSE
#>  7:         1      FALSE       FALSE      TRUE           TRUE   TRUE  FALSE
#>  8:         1      FALSE        TRUE     FALSE          FALSE   TRUE  FALSE
#>  9:         1       TRUE        TRUE     FALSE          FALSE   TRUE  FALSE
#> 10:         1       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 11:         2      FALSE       FALSE     FALSE          FALSE  FALSE  FALSE
#> 12:         2      FALSE        TRUE      TRUE          FALSE  FALSE  FALSE
#> 13:         2      FALSE       FALSE      TRUE           TRUE   TRUE  FALSE
#> 14:         2       TRUE        TRUE     FALSE           TRUE  FALSE  FALSE
#> 15:         2      FALSE       FALSE     FALSE          FALSE   TRUE  FALSE
#> 16:         2      FALSE       FALSE     FALSE          FALSE   TRUE  FALSE
#> 17:         2      FALSE       FALSE     FALSE          FALSE   TRUE  FALSE
#> 18:         2       TRUE       FALSE      TRUE          FALSE   TRUE   TRUE
#> 19:         2      FALSE       FALSE     FALSE          FALSE  FALSE   TRUE
#> 20:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#>     iteration bill_depth bill_length body_mass flipper_length island    sex
#>       year classif.ce runtime_learners           timestamp batch_nr warnings
#>     <lgcl>      <num>            <num>              <POSc>    <int>    <int>
#>  1:   TRUE 0.54385965            0.019 2024-03-09 11:48:31        1        0
#>  2:  FALSE 0.31578947            0.003 2024-03-09 11:48:31        1        0
#>  3:  FALSE 0.28070175            0.004 2024-03-09 11:48:31        1        0
#>  4:   TRUE 0.10526316            0.005 2024-03-09 11:48:31        1        0
#>  5:  FALSE 0.10526316            0.005 2024-03-09 11:48:31        1        0
#>  6:  FALSE 0.31578947            0.004 2024-03-09 11:48:31        1        0
#>  7:  FALSE 0.14035088            0.005 2024-03-09 11:48:31        1        0
#>  8:  FALSE 0.07017544            0.005 2024-03-09 11:48:31        1        0
#>  9:   TRUE 0.07017544            0.005 2024-03-09 11:48:31        1        0
#> 10:   TRUE 0.10526316            0.005 2024-03-09 11:48:31        1        0
#> 11:   TRUE 0.73684211            0.005 2024-03-09 11:48:31        1        0
#> 12:  FALSE 0.14035088            0.005 2024-03-09 11:48:31        1        0
#> 13:   TRUE 0.19298246            0.005 2024-03-09 11:48:31        1        0
#> 14:   TRUE 0.08771930            0.005 2024-03-09 11:48:31        1        0
#> 15:   TRUE 0.29824561            0.005 2024-03-09 11:48:31        1        0
#> 16:  FALSE 0.28070175            0.004 2024-03-09 11:48:31        1        0
#> 17:   TRUE 0.29824561            0.019 2024-03-09 11:48:31        1        0
#> 18:  FALSE 0.21052632            0.005 2024-03-09 11:48:31        1        0
#> 19:  FALSE 0.52631579            0.004 2024-03-09 11:48:31        1        0
#> 20:   TRUE 0.07017544            0.004 2024-03-09 11:48:31        1        0
#>       year classif.ce runtime_learners           timestamp batch_nr warnings
#>     errors                                                       features
#>      <int>                                                         <list>
#>  1:      0                                                           year
#>  2:      0                                                     bill_depth
#>  3:      0                                                         island
#>  4:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>  5:      0                                 bill_length,flipper_length,sex
#>  6:      0                                              bill_depth,island
#>  7:      0                                body_mass,flipper_length,island
#>  8:      0                                             bill_length,island
#>  9:      0                             bill_depth,bill_length,island,year
#> 10:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 11:      0                                                           year
#> 12:      0                                          bill_length,body_mass
#> 13:      0                           body_mass,flipper_length,island,year
#> 14:      0                     bill_depth,bill_length,flipper_length,year
#> 15:      0                                                    island,year
#> 16:      0                                                         island
#> 17:      0                                                    island,year
#> 18:      0                                bill_depth,body_mass,island,sex
#> 19:      0                                                            sex
#> 20:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>     errors                                                       features
#>     n_features  resample_result  task_id              learner_id resampling_id
#>         <list>           <list>   <char>                  <char>        <char>
#>  1:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#>  2:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#>  3:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#>  4:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#>  5:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>  6:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#>  7:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>  8:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#>  9:          4 <ResampleResult> penguins classif.rpart.fselector            cv
#> 10:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#> 11:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 12:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 13:          4 <ResampleResult> penguins classif.rpart.fselector            cv
#> 14:          4 <ResampleResult> penguins classif.rpart.fselector            cv
#> 15:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 16:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 17:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 18:          4 <ResampleResult> penguins classif.rpart.fselector            cv
#> 19:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 20:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#>     n_features  resample_result  task_id              learner_id resampling_id