Skip to contents

Extract Inner Feature Selection Archives

Source: R/extract_inner_fselect_archives.R
extract_inner_fselect_archives.Rd

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

x

(mlr3::ResampleResult | mlr3::BenchmarkResult).

exclude_columns

(character())
Exclude columns from result table. Set to NULL if no column should be excluded.

Value

data.table::data.table().

Data structure

The returned data table has the following columns:

  • experiment (integer(1))
    Index, giving the according row number in the original benchmark grid.

  • iteration (integer(1))
    Iteration of the outer resampling.

  • One column for each feature of the task.

  • One column for each performance measure.

  • runtime_learners (numeric(1))
    Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

  • timestamp (POSIXct)
    Time stamp when the evaluation was logged into the archive.

  • batch_nr (integer(1))
    Feature sets are evaluated in batches. Each batch has a unique batch number.

  • resample_result (mlr3::ResampleResult)
    Resample result of the inner resampling.

  • task_id (character(1)).

  • learner_id (character(1)).

  • resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)
#>     iteration bill_depth bill_length body_mass flipper_length island    sex
#>         <int>     <lgcl>      <lgcl>    <lgcl>         <lgcl> <lgcl> <lgcl>
#>  1:         1       TRUE        TRUE      TRUE          FALSE   TRUE  FALSE
#>  2:         1      FALSE       FALSE     FALSE          FALSE  FALSE  FALSE
#>  3:         1       TRUE        TRUE      TRUE           TRUE   TRUE  FALSE
#>  4:         1       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#>  5:         1      FALSE       FALSE      TRUE          FALSE  FALSE  FALSE
#>  6:         1       TRUE       FALSE      TRUE          FALSE  FALSE   TRUE
#>  7:         1      FALSE        TRUE      TRUE           TRUE   TRUE  FALSE
#>  8:         1       TRUE        TRUE      TRUE           TRUE   TRUE  FALSE
#>  9:         1      FALSE        TRUE     FALSE          FALSE   TRUE   TRUE
#> 10:         1      FALSE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 11:         2      FALSE       FALSE      TRUE           TRUE  FALSE   TRUE
#> 12:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 13:         2      FALSE       FALSE     FALSE          FALSE   TRUE  FALSE
#> 14:         2      FALSE       FALSE     FALSE           TRUE  FALSE  FALSE
#> 15:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 16:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 17:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 18:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#> 19:         2      FALSE        TRUE      TRUE          FALSE  FALSE  FALSE
#> 20:         2       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE
#>     iteration bill_depth bill_length body_mass flipper_length island    sex
#>       year classif.ce runtime_learners           timestamp batch_nr warnings
#>     <lgcl>      <num>            <num>              <POSc>    <int>    <int>
#>  1:   TRUE 0.07017544            0.005 2024-11-07 21:47:32        1        0
#>  2:   TRUE 0.64912281            0.004 2024-11-07 21:47:32        1        0
#>  3:   TRUE 0.05263158            0.006 2024-11-07 21:47:32        1        0
#>  4:  FALSE 0.05263158            0.005 2024-11-07 21:47:32        1        0
#>  5:   TRUE 0.36842105            0.005 2024-11-07 21:47:32        1        0
#>  6:  FALSE 0.40350877            0.006 2024-11-07 21:47:32        1        0
#>  7:   TRUE 0.05263158            0.005 2024-11-07 21:47:32        1        0
#>  8:   TRUE 0.05263158            0.005 2024-11-07 21:47:32        1        0
#>  9:   TRUE 0.07017544            0.006 2024-11-07 21:47:32        1        0
#> 10:   TRUE 0.05263158            0.006 2024-11-07 21:47:32        1        0
#> 11:   TRUE 0.26315789            0.006 2024-11-07 21:47:32        1        0
#> 12:   TRUE 0.07017544            0.005 2024-11-07 21:47:32        1        0
#> 13:  FALSE 0.36842105            0.004 2024-11-07 21:47:32        1        0
#> 14:  FALSE 0.26315789            0.005 2024-11-07 21:47:32        1        0
#> 15:   TRUE 0.07017544            0.007 2024-11-07 21:47:32        1        0
#> 16:   TRUE 0.07017544            0.007 2024-11-07 21:47:32        1        0
#> 17:   TRUE 0.07017544            0.005 2024-11-07 21:47:32        1        0
#> 18:  FALSE 0.07017544            0.005 2024-11-07 21:47:32        1        0
#> 19:  FALSE 0.05263158            0.005 2024-11-07 21:47:32        1        0
#> 20:   TRUE 0.07017544            0.005 2024-11-07 21:47:32        1        0
#>       year classif.ce runtime_learners           timestamp batch_nr warnings
#>     errors                                                       features
#>      <int>                                                         <list>
#>  1:      0                   bill_depth,bill_length,body_mass,island,year
#>  2:      0                                                           year
#>  3:      0    bill_depth,bill_length,body_mass,flipper_length,island,year
#>  4:      0     bill_depth,bill_length,body_mass,flipper_length,island,sex
#>  5:      0                                                 body_mass,year
#>  6:      0                                       bill_depth,body_mass,sex
#>  7:      0               bill_length,body_mass,flipper_length,island,year
#>  8:      0    bill_depth,bill_length,body_mass,flipper_length,island,year
#>  9:      0                                    bill_length,island,sex,year
#> 10:      0           bill_length,body_mass,flipper_length,island,sex,year
#> 11:      0                              body_mass,flipper_length,sex,year
#> 12:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 13:      0                                                         island
#> 14:      0                                                 flipper_length
#> 15:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 16:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 17:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#> 18:      0     bill_depth,bill_length,body_mass,flipper_length,island,sex
#> 19:      0                                          bill_length,body_mass
#> 20:      0 bill_depth,bill_length,body_mass,flipper_length,island,sex,...
#>     errors                                                       features
#>     n_features  resample_result  task_id              learner_id resampling_id
#>         <list>           <list>   <char>                  <char>        <char>
#>  1:          5 <ResampleResult> penguins classif.rpart.fselector            cv
#>  2:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#>  3:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#>  4:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#>  5:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#>  6:          3 <ResampleResult> penguins classif.rpart.fselector            cv
#>  7:          5 <ResampleResult> penguins classif.rpart.fselector            cv
#>  8:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#>  9:          4 <ResampleResult> penguins classif.rpart.fselector            cv
#> 10:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#> 11:          4 <ResampleResult> penguins classif.rpart.fselector            cv
#> 12:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#> 13:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 14:          1 <ResampleResult> penguins classif.rpart.fselector            cv
#> 15:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#> 16:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#> 17:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#> 18:          6 <ResampleResult> penguins classif.rpart.fselector            cv
#> 19:          2 <ResampleResult> penguins classif.rpart.fselector            cv
#> 20:          7 <ResampleResult> penguins classif.rpart.fselector            cv
#>     n_features  resample_result  task_id              learner_id resampling_id