Skip to contents

Extract Inner Feature Selection Archives

Source: R/extract_inner_fselect_archives.R
extract_inner_fselect_archives.Rd

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

x

(mlr3::ResampleResult | mlr3::BenchmarkResult).

exclude_columns

(character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Value

data.table::data.table().

Data structure

The returned data table has the following columns:

  • experiment (integer(1))
    Index, giving the according row number in the original benchmark grid.

  • iteration (integer(1))
    Iteration of the outer resampling.

  • One column for each feature of the task.

  • One column for each performance measure.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Feature sets are evaluated in batches. Each batch has a unique batch number.

  • resample_result (mlr3::ResampleResult)
    Resample result of the inner resampling.

  • task_id (character(1)).

  • learner_id (character(1)).

  • resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)
#>     iteration bill_depth bill_length body_mass flipper_length island    sex
#>         <int>     <lgcl>      <lgcl>    <lgcl>         <lgcl> <lgcl> <lgcl>
#>  1:         1      FALSE        TRUE     FALSE          FALSE  FALSE  FALSE
#>  2:         1      FALSE        TRUE     FALSE           TRUE   TRUE  FALSE
#>  3:         1       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#>  4:         1       TRUE        TRUE     FALSE           TRUE   TRUE   TRUE
#>  5:         1      FALSE        TRUE     FALSE          FALSE   TRUE  FALSE
#>  6:         1       TRUE       FALSE      TRUE           TRUE  FALSE  FALSE
#>  7:         1       TRUE       FALSE      TRUE          FALSE  FALSE  FALSE
#>  8:         1      FALSE       FALSE     FALSE          FALSE   TRUE   TRUE
#>  9:         1       TRUE       FALSE      TRUE           TRUE   TRUE   TRUE
#> 10:         1      FALSE       FALSE      TRUE          FALSE   TRUE   TRUE
#> 11:         2      FALSE        TRUE     FALSE          FALSE  FALSE   TRUE
#> 12:         2       TRUE       FALSE      TRUE           TRUE   TRUE   TRUE
#> 13:         2       TRUE       FALSE     FALSE          FALSE   TRUE  FALSE
#> 14:         2       TRUE       FALSE     FALSE          FALSE   TRUE  FALSE
#> 15:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 16:         2       TRUE       FALSE     FALSE          FALSE  FALSE  FALSE
#> 17:         2       TRUE       FALSE     FALSE          FALSE  FALSE   TRUE
#> 18:         2       TRUE        TRUE     FALSE           TRUE  FALSE   TRUE
#> 19:         2      FALSE        TRUE     FALSE          FALSE  FALSE  FALSE
#> 20:         2      FALSE       FALSE     FALSE           TRUE  FALSE   TRUE
#>     iteration bill_depth bill_length body_mass flipper_length island    sex
#>       year classif.ce runtime_learners           timestamp batch_nr warnings
#>     <lgcl>      <num>            <num>              <POSc>    <int>    <int>
#>  1:  FALSE 0.26315789            0.004 2025-02-12 16:48:59        1        0
#>  2:  FALSE 0.10526316            0.005 2025-02-12 16:48:59        1        0
#>  3:   TRUE 0.10526316            0.005 2025-02-12 16:48:59        1        0
#>  4:   TRUE 0.10526316            0.005 2025-02-12 16:48:59        1        0
#>  5:  FALSE 0.07017544            0.005 2025-02-12 16:48:59        1        0
#>  6:  FALSE 0.21052632            0.004 2025-02-12 16:48:59        1        0
#>  7:  FALSE 0.38596491            0.004 2025-02-12 16:48:59        1        0
#>  8:   TRUE 0.29824561            0.005 2025-02-12 16:48:59        1        0
#>  9:   TRUE 0.21052632            0.005 2025-02-12 16:48:59        1        0
#> 10:  FALSE 0.22807018            0.004 2025-02-12 16:48:59        1        0
#> 11:  FALSE 0.21052632            0.004 2025-02-12 16:48:59        1        0
#> 12:   TRUE 0.10526316            0.004 2025-02-12 16:48:59        1        0
#> 13:  FALSE 0.29824561            0.005 2025-02-12 16:48:59        1        0
#> 14:  FALSE 0.29824561            0.004 2025-02-12 16:48:59        1        0
#> 15:   TRUE 0.01754386            0.005 2025-02-12 16:48:59        1        0
#> 16:  FALSE 0.35087719            0.004 2025-02-12 16:48:59        1        0
#> 17:  FALSE 0.35087719            0.004 2025-02-12 16:48:59        1        0
#> 18:   TRUE 0.01754386            0.005 2025-02-12 16:48:59        1        0
#> 19:  FALSE 0.21052632            0.004 2025-02-12 16:48:59        1        0
#> 20:   TRUE 0.15789474            0.005 2025-02-12 16:48:59        1        0
#>       year classif.ce runtime_learners           timestamp batch_nr warnings
#>     errors                                                       features
#>      <int>                                                         <list>
#>  1:      0                                                    bill_length
#>  2:      0                              bill_length,flipper_length,island
#>  3:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>  4:      0          bill_depth,bill_length,flipper_length,island,sex,year
#>  5:      0                                             bill_length,island
#>  6:      0                            bill_depth,body_mass,flipper_length
#>  7:      0                                           bill_depth,body_mass
#>  8:      0                                                island,sex,year
#>  9:      0            bill_depth,body_mass,flipper_length,island,sex,year
#> 10:      0                                           body_mass,island,sex
#> 11:      0                                                bill_length,sex
#> 12:      0            bill_depth,body_mass,flipper_length,island,sex,year
#> 13:      0                                              bill_depth,island
#> 14:      0                                              bill_depth,island
#> 15:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 16:      0                                                     bill_depth
#> 17:      0                                                 bill_depth,sex
#> 18:      0                 bill_depth,bill_length,flipper_length,sex,year
#> 19:      0                                                    bill_length
#> 20:      0                                        flipper_length,sex,year
#>     errors                                                       features
#>     n_features  resample_result  task_id              learner_id resampling_id
#>         <list>           <list>   <char>                  <char>        <char>
#>  1:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#>  2:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>  3:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#>  4:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#>  5:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#>  6:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>  7:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#>  8:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>  9:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#> 10:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#> 11:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 12:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#> 13:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 14:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 15:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#> 16:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 17:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 18:          5 <ResampleResult> penguins classif.rpart.fselector            cv
#> 19:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 20:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>     n_features  resample_result  task_id              learner_id resampling_id