Feature Selection with Recursive Feature Elimination with Cross Validation

Feature selection using the Recursive Feature Elimination with Cross-Validation (RFE-CV) algorithm. See FSelectorRFE for a description of the base algorithm. RFE-CV runs a recursive feature elimination in each iteration of a cross-validation to determine the optimal number of features. Then a recursive feature elimination is run again on the complete dataset with the optimal number of features as the final feature set size. The performance of the optimal feature set is calculated on the complete data set and should not be reported as the performance of the final model. Only works with mlr3::Learners that can calculate importance scores (see the section on optional extractors in mlr3::Learner).

Details

The resampling strategy is changed during the feature selection. The resampling strategy passed to the instance (resampling) is used to determine the optimal number of features. Usually, a cross-validation strategy is used and a recursive feature elimination is run in each iteration of the cross-validation. Internally, mlr3::ResamplingCustom is used to emulate this part of the algorithm. In the final recursive feature elimination run the resampling strategy is changed to mlr3::ResamplingInsample i.e. the complete data set is used for training and testing.

The feature selection terminates itself when the optimal number of features is reached. It is not necessary to set a termination criterion.

Archive

The ArchiveFSelect holds the following additional columns:

• "iteration" (integer(1))
  The resampling iteration in which the feature subset was evaluated.

• "importance" (numeric())
  The importance score vector of the feature subset.

Resources

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

Dictionary

This FSelector can be instantiated with the associated sugar function fs():

fs("rfe")

Control Parameters

n_features

integer(1)
The number of features to select. By default half of the features are selected.

feature_fraction

double(1)
Fraction of features to retain in each iteration. The default 0.5 retrains half of the features.

feature_number

integer(1)
Number of features to remove in each iteration.

subset_sizes

integer()
Vector of number of features to retain in each iteration. Must be sorted in decreasing order.

recursive

logical(1)
If TRUE (default), the feature importance is calculated in each iteration.

The parameter feature_fraction, feature_number and subset_sizes are mutually exclusive.

See also

Other FSelector: mlr_fselectors, mlr_fselectors_design_points, mlr_fselectors_exhaustive_search, mlr_fselectors_genetic_search, mlr_fselectors_random_search, mlr_fselectors_rfe, mlr_fselectors_sequential, mlr_fselectors_shadow_variable_search

Super class

mlr3fselect::FSelector -> FSelectorRFECV

Methods

Public methods

• FSelectorRFECV$new()

• FSelectorRFECV$clone()

Inherited methods

• mlr3fselect::FSelector$format()
• mlr3fselect::FSelector$help()
• mlr3fselect::FSelector$optimize()
• mlr3fselect::FSelector$print()

---

Method new()

Creates a new instance of this R6 class.

Usage

FSelectorRFECV$new()

---

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorRFECV$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# Feature Selection
# \donttest{

# retrieve task and load learner
task = tsk("penguins")
learner = lrn("classif.rpart")

# run feature selection on the Palmer Penguins data set
instance = fselect(
  fselector = fs("rfecv"),
  task = task,
  learner = learner,
  resampling = rsmp("cv", folds = 3),
  measure = msr("classif.ce"),
  store_models = TRUE
)

# best performing feature subset
instance$result
#>    bill_depth bill_length body_mass flipper_length island    sex   year
#>        <lgcl>      <lgcl>    <lgcl>         <lgcl> <lgcl> <lgcl> <lgcl>
#> 1:       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE   TRUE
#>                                                          features n_features
#>                                                            <list>      <int>
#> 1: bill_depth,bill_length,body_mass,flipper_length,island,sex,...          7
#>    classif.ce
#>         <num>
#> 1: 0.03488372

# all evaluated feature subsets
as.data.table(instance$archive)
#>    bill_depth bill_length body_mass flipper_length island    sex   year
#>        <lgcl>      <lgcl>    <lgcl>         <lgcl> <lgcl> <lgcl> <lgcl>
#> 1:       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE   TRUE
#> 2:       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE   TRUE
#> 3:       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE   TRUE
#> 4:       TRUE        TRUE     FALSE           TRUE  FALSE  FALSE  FALSE
#> 5:       TRUE        TRUE     FALSE           TRUE  FALSE  FALSE  FALSE
#> 6:       TRUE        TRUE     FALSE           TRUE  FALSE  FALSE  FALSE
#> 7:       TRUE        TRUE      TRUE           TRUE   TRUE   TRUE   TRUE
#>    classif.ce runtime_learners           timestamp batch_nr warnings errors
#>         <num>            <num>              <POSc>    <int>    <int>  <int>
#> 1: 0.06956522            0.006 2024-03-09 11:48:43        1        0      0
#> 2: 0.06956522            0.006 2024-03-09 11:48:43        1        0      0
#> 3: 0.10526316            0.006 2024-03-09 11:48:43        1        0      0
#> 4: 0.07826087            0.005 2024-03-09 11:48:43        2        0      0
#> 5: 0.06956522            0.005 2024-03-09 11:48:43        2        0      0
#> 6: 0.07017544            0.005 2024-03-09 11:48:43        2        0      0
#> 7: 0.03488372            0.021 2024-03-09 11:48:43        3        0      0
#>                                                         importance iteration
#>                                                             <list>     <int>
#> 1:       86.08828,83.19794,59.17562,55.64275,45.92735, 0.00000,...         1
#> 2:       84.38668,82.13307,65.91505,58.27457,44.85789, 0.00000,...         2
#> 3:       87.48191,80.76197,73.42931,68.07608,57.87605, 0.00000,...         3
#> 4:                                      86.08828,83.19794,59.17562         1
#> 5:                                      84.38668,82.13307,65.91505         2
#> 6:                                      86.76431,80.76197,73.42931         3
#> 7: 124.20793,121.52400,102.74919, 87.26186, 78.61700,  0.00000,...        NA
#>                                                          features n_features
#>                                                            <list>     <list>
#> 1: bill_depth,bill_length,body_mass,flipper_length,island,sex,...          7
#> 2: bill_depth,bill_length,body_mass,flipper_length,island,sex,...          7
#> 3: bill_depth,bill_length,body_mass,flipper_length,island,sex,...          7
#> 4:                          bill_depth,bill_length,flipper_length          3
#> 5:                          bill_depth,bill_length,flipper_length          3
#> 6:                          bill_depth,bill_length,flipper_length          3
#> 7: bill_depth,bill_length,body_mass,flipper_length,island,sex,...          7
#>     resample_result
#>              <list>
#> 1: <ResampleResult>
#> 2: <ResampleResult>
#> 3: <ResampleResult>
#> 4: <ResampleResult>
#> 5: <ResampleResult>
#> 6: <ResampleResult>
#> 7: <ResampleResult>

# subset the task and fit the final model
task$select(instance$result_feature_set)
learner$train(task)
# }