data/knowledge_base/feature_selection.md

Feature Selection (sklearn.feature_selection)

What It Does

sklearn.feature_selection provides methods to select a subset of original features that contribute meaningfully to model performance. Feature selection reduces noise, training time, memory, and overfitting risk, and improves interpretability. It is distinct from feature extraction (PCA, etc.) — it retains original columns rather than creating new representations.

---

Filter Methods (Univariate, Model-Independent)

Fast and scalable. Score each feature independently; cannot detect interactions.

VarianceThreshold

Removes features with variance below a threshold. Default removes zero-variance (constant) features.

sel = VarianceThreshold(threshold=0.16)  # for boolean: p*(1-p), e.g. 0.8*(1-0.8)

Use as a first-pass cleaning step before any other method.

SelectKBest / SelectPercentile

Select the top k features or top p% by statistical score.

X_new = SelectKBest(f_classif, k=20).fit_transform(X, y)
X_new = SelectPercentile(mutual_info_classif, percentile=20).fit_transform(X, y)

Scoring functions:

Function	Problem Type	Captures	Notes
f_classif	Classification	Linear relationship	Fast; assumes normality
f_regression	Regression	Linear relationship	Fast
chi2	Classification	Any	Non-negative features only (counts/frequencies)
mutual_info_classif	Classification	Any (incl. non-linear)	Slower; needs more samples; set random_state
mutual_info_regression	Regression	Any (incl. non-linear)	Same caveats
r_regression	Regression	Linear (Pearson r)	Fast

Do not use regression scoring functions with a classification target.

GenericUnivariateSelect

Allows configurable strategy (k, percentile, fpr, fdr, fwe) for hyperparameter search across selection modes.

Statistical filter variants: SelectFpr (false positive rate), SelectFdr (false discovery rate), SelectFwe (family-wise error) — for p-value-based cutoffs.

---

Wrapper Methods (Model-Dependent)

Train a model to evaluate feature subsets. Slower but accounts for interactions and model-specific behavior.

RFE (Recursive Feature Elimination)

Fits the estimator, ranks features by coef_ or feature_importances_, removes the least important, repeats.

• n_features_to_select — target number of features to keep.
• step — number (or fraction) of features to remove per iteration. step=1 is exhaustive; step=0.1 removes 10% per round.
• estimator must expose coef_ or feature_importances_.

RFECV

Cross-validated RFE that automatically finds the optimal number of features.

• cv=5 — number of folds.
• scoring='roc_auc' — metric to optimize.
• rfecv.n_features_ — optimal feature count at CV peak.

Limitation: Slow for large feature sets (p > 500). Results depend on the estimator — use the same model class you will train on.

SequentialFeatureSelector (SFS)

Greedy forward or backward selection — adds/removes one feature at a time based on cross-validated score.

• direction='forward' — start empty, add best feature each step.
• direction='backward' — start full, remove worst feature each step.
• n_features_to_select — target count.

When to use: Estimator does not expose coef_ or feature_importances_. Forward selection for selecting few features; backward for selecting many.

Caveat: Slower than RFE or SelectFromModel; requires fitting n_features * cv models per step.

---

Embedded Methods (Selection During Training)

SelectFromModel

Meta-transformer that uses a model's coef_ or feature_importances_ to threshold features.

clf = ExtraTreesClassifier(n_estimators=50).fit(X, y)
model = SelectFromModel(clf, prefit=True, threshold="mean")
X_new = model.transform(X)

• threshold — "mean", "median", "0.1*mean", or a numeric value.
• max_features — cap on number of features selected.

L1-based selection: Use Lasso, LogisticRegression(penalty='l1'), or LinearSVC(penalty='l1'). Smaller C or higher alpha = more sparsity = fewer features.

Tree-based selection: Use impurity-based feature_importances_ from RandomForest, ExtraTrees, or gradient boosting.

Caveat: Impurity-based importances are biased toward high-cardinality features. Use permutation importance or SHAP values for less biased estimates.

---

Sparse Data Support

chi2, mutual_info_regression, and mutual_info_classif handle sparse matrices without densification — important for text or high-dimensional one-hot encoded inputs.

---

Feature Selection in Pipelines

Always wrap selection in a Pipeline to prevent data leakage — the selector must be fit only on training folds:

pipe = Pipeline([
    ('selector', SelectKBest(mutual_info_classif, k=20)),
    ('clf', RandomForestClassifier())
])
scores = cross_val_score(pipe, X, y, cv=5, scoring='roc_auc')

Fitting selection outside the CV loop can inflate CV metrics by 0.05–0.15 AUC through selection overfitting.

---

When to Use Each Method

Situation	Method
First pass — remove constants and near-constants	VarianceThreshold
Fast baseline ranking, linear relationships	SelectKBest(f_classif) or SelectKBest(f_regression)
Non-linear or non-monotonic relationships	SelectKBest(mutual_info_classif)
Count/frequency features, classification	SelectKBest(chi2)
Model-based, estimator has coef_/importances_	SelectFromModel
Want auto-selected feature count	RFECV
Estimator has no coef_/importances_	SequentialFeatureSelector
Multi-objective or correlated features	SHAP-based selection (external)

---

Caveats

• Unsupervised selection: SelectPercentile and SelectKBest support unsupervised mode with an appropriate score_func.
• L1 instability: Among correlated features, L1 arbitrarily selects one. ElasticNet is more stable for group selection.
• Hyperparameter tuning for selection: Combine with cross-validation; treat k, threshold, or n_features_to_select as a hyperparameter to tune.
• Impurity importance bias: High-cardinality features appear more important due to more possible split points. Prefer permutation importance or SHAP for final selection decisions.
• Model mismatch: Filter methods optimal for linear models may not be optimal for tree-based models — match the selection strategy to the model family.