Feature selection is one of the most impactful things you can do to improve a machine learning model. Removing irrelevant or redundant features reduces overfitting, speeds up training, and often improves accuracy. This guide covers all three categories of feature selection methods with practical Python code.
Why Feature Selection Matters
More features are not always better. Irrelevant features add noise, making it harder for the model to find the true signal. Correlated features waste model capacity on redundant information. High-dimensional datasets also train more slowly and require more data to generalise. Feature selection addresses all of these. Studies consistently show that well-selected feature subsets outperform all-features baselines, especially on small-to-medium datasets.
Category 1 – Filter Methods
Filter methods rank features based on statistical properties, independently of any model. They’re fast and model-agnostic.
Correlation with Target
import pandas as pd
import numpy as np
corr = df.corr()['target'].abs().sort_values(ascending=False)
print(corr)
# Keep features with |correlation| > 0.1
selected = corr[corr > 0.1].index.tolist()
Variance Threshold
from sklearn.feature_selection import VarianceThreshold
# Remove features with variance < 0.01
selector = VarianceThreshold(threshold=0.01)
X_filtered = selector.fit_transform(X)
print(f"Features kept: {X_filtered.shape[1]} of {X.shape[1]}")
Mutual Information
from sklearn.feature_selection import mutual_info_classif, SelectKBest
# Select top 10 features by mutual information
selector = SelectKBest(score_func=mutual_info_classif, k=10)
X_selected = selector.fit_transform(X, y)
selected_features = X.columns[selector.get_support()]
print(selected_features.tolist())
Mutual information captures non-linear relationships that correlation misses — useful when features have complex relationships with the target.
Category 2 – Wrapper Methods
Wrapper methods train a model on subsets of features and evaluate performance. They're more powerful than filter methods but computationally expensive.
Recursive Feature Elimination (RFE)
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=1000)
rfe = RFE(estimator=model, n_features_to_select=10)
rfe.fit(X_train, y_train)
selected = X.columns[rfe.support_].tolist()
print("Selected features:", selected)
print("Rankings:", rfe.ranking_)
Sequential Feature Selection
from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.ensemble import RandomForestClassifier
sfs = SequentialFeatureSelector(
RandomForestClassifier(n_estimators=100, random_state=42),
n_features_to_select=10,
direction='forward',
cv=5)
sfs.fit(X_train, y_train)
print("Selected:", X.columns[sfs.get_support()].tolist())
Category 3 – Embedded Methods
Embedded methods perform feature selection as part of model training. They're usually the best balance of speed and accuracy.
LASSO (L1 Regularization)
from sklearn.linear_model import LassoCV
lasso = LassoCV(cv=5, random_state=42)
lasso.fit(X_train, y_train)
# Features with non-zero coefficients
selected_mask = lasso.coef_ != 0
selected = X.columns[selected_mask].tolist()
print(f"LASSO selected {len(selected)} features:", selected)
LASSO automatically sets unimportant feature coefficients to exactly zero, effectively removing them. LassoCV finds the optimal regularisation strength automatically.
Tree-Based Feature Importance
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt
rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X_train, y_train)
importance_df = pd.DataFrame({
'feature': X.columns,
'importance': rf.feature_importances_
}).sort_values('importance', ascending=False)
# Plot top 15
importance_df.head(15).plot.barh(x='feature', y='importance', figsize=(10, 6))
plt.title("Random Forest Feature Importance")
plt.tight_layout()
plt.show()
SHAP Values for Robust Importance
pip install shap
import shap
explainer = shap.TreeExplainer(rf)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values[1], X_test, plot_type="bar")
SHAP provides more reliable importance scores than the built-in feature_importances_ because it's based on game theory and accounts for feature interactions.
Removing Correlated Features
# Remove features with correlation > 0.95 (redundant)
corr_matrix = X.corr().abs()
upper_tri = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
to_drop = [col for col in upper_tri.columns if any(upper_tri[col] > 0.95)]
X_reduced = X.drop(columns=to_drop)
print(f"Dropped {len(to_drop)} correlated features")
Recommended Workflow
Start with variance threshold to remove near-constant features, then remove highly correlated features (>0.95). Next apply mutual information or a filter method to get a shortlist of candidates. Then use LASSO or tree importance to narrow down further. Finally, validate the selected feature set with cross-validation — don't just compare training accuracy. The best feature selection is always validated on held-out data.
Conclusion
Feature selection is not optional for serious ML work — it's a critical step between raw data and a production model. Use filter methods for speed and initial screening, LASSO for linear models, and tree importance or SHAP for non-linear models. The right feature subset almost always outperforms throwing all features at a model and hoping for the best.
