Overfitting is the most common failure mode in machine learning — your model learns the training data perfectly, including its noise, and then performs poorly on new data. Regularization is the primary technique for preventing this. By adding a penalty term to the loss function, regularization constrains the model’s weights, forcing it to find simpler solutions that generalise better. Understanding Ridge, Lasso, and Elastic Net is fundamental to building reliable ML models.
The Overfitting Problem and Why Regularization Works
When a model has many features or high polynomial degree, it can memorise the training set — fitting every quirk and noise point — while failing completely on unseen data. This shows up as high training accuracy but low validation accuracy. The gap between these two is your overfitting signal.
Regularization adds a penalty proportional to the magnitude of the model’s coefficients to the loss function. The model is forced to achieve good predictions AND keep its weights small. Larger weights are “expensive,” so the optimiser only uses them when they genuinely improve predictions enough to justify the cost. The result: simpler models that don’t chase noise.
Ridge Regression (L2 Regularization)
Ridge adds the sum of squared coefficients to the loss function, scaled by a hyperparameter alpha (also called lambda). The squared penalty shrinks all coefficients toward zero but never to exactly zero — every feature stays in the model. Ridge is ideal when you believe all features are relevant but want to reduce their individual impact:
from sklearn.linear_model import Ridge, LinearRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression
import numpy as np
import matplotlib.pyplot as plt
# Generate dataset with some noise
X, y = make_regression(n_samples=200, n_features=50, noise=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Always scale features before regularization
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
# Compare: no regularization vs Ridge
lr = LinearRegression().fit(X_train_s, y_train)
ridge = Ridge(alpha=1.0).fit(X_train_s, y_train)
print(f"Linear R2 (train): {lr.score(X_train_s, y_train):.3f}")
print(f"Linear R2 (test): {lr.score(X_test_s, y_test):.3f}")
print(f"Ridge R2 (train): {ridge.score(X_train_s, y_train):.3f}")
print(f"Ridge R2 (test): {ridge.score(X_test_s, y_test):.3f}")
# Find best alpha with cross-validation
from sklearn.linear_model import RidgeCV
ridge_cv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100], cv=5)
ridge_cv.fit(X_train_s, y_train)
print(f"Best alpha: {ridge_cv.alpha_}")
Lasso Regression (L1 Regularization)
Lasso adds the sum of absolute values of coefficients to the loss. This seemingly small difference has a profound effect: Lasso can shrink coefficients to exactly zero, performing automatic feature selection. If you have 100 features but only 20 are truly relevant, Lasso will zero out the other 80 and give you a sparse model that’s easier to interpret:
from sklearn.linear_model import Lasso, LassoCV
# Lasso for feature selection
lasso = Lasso(alpha=0.1, max_iter=10000)
lasso.fit(X_train_s, y_train)
# Count non-zero coefficients (selected features)
n_selected = np.sum(lasso.coef_ != 0)
print(f"Features selected by Lasso: {n_selected} out of {X_train_s.shape[1]}")
print(f"Lasso R2 (test): {lasso.score(X_test_s, y_test):.3f}")
# Automatic alpha selection via cross-validation
lasso_cv = LassoCV(cv=5, max_iter=10000, random_state=42)
lasso_cv.fit(X_train_s, y_train)
print(f"Best alpha: {lasso_cv.alpha_:.4f}")
print(f"Features after CV Lasso: {np.sum(lasso_cv.coef_ != 0)}")
Use Lasso when you suspect only a subset of features matter and you want interpretability. Use Ridge when you believe all features contribute and you simply want to shrink their influence.
Elastic Net: Combining Ridge and Lasso
Elastic Net combines both penalties — L1 for sparsity and L2 for handling correlated features. Lasso has a weakness: when features are highly correlated, it tends to pick one arbitrarily and zero out the rest. Ridge handles correlated features better by shrinking them together. Elastic Net gets both benefits via the l1_ratio parameter (0 = pure Ridge, 1 = pure Lasso):
from sklearn.linear_model import ElasticNet, ElasticNetCV
# Elastic Net with cross-validated hyperparameters
enet_cv = ElasticNetCV(
l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 1.0],
alphas=[0.001, 0.01, 0.1, 1.0],
cv=5,
max_iter=10000,
random_state=42
)
enet_cv.fit(X_train_s, y_train)
print(f"Best alpha: {enet_cv.alpha_:.4f}")
print(f"Best l1_ratio: {enet_cv.l1_ratio_:.2f}")
print(f"Elastic Net R2 (test): {enet_cv.score(X_test_s, y_test):.3f}")
print(f"Features selected: {np.sum(enet_cv.coef_ != 0)}")
Regularization applies beyond linear models. Logistic regression has C=1/alpha parameter (smaller C = more regularization). Neural networks use weight decay (L2) and dropout. Decision trees and ensemble methods use max_depth and min_samples_leaf as structural regularization.
Frequently Asked Questions
When should I use Ridge vs Lasso?
Use Ridge when all features are likely relevant and correlated — for example, predicting house prices from many correlated property features. Use Lasso when you suspect only a few features matter and want automatic selection — for example, identifying which genes predict a disease outcome from thousands of candidates.
Do I need to scale features before regularization?
Yes, always. Regularization penalises large coefficients. Without scaling, features measured in large units (e.g., income in dollars) will have small coefficients while features in small units (e.g., age in years) will have large ones — regularization will incorrectly penalise some features more than others. Use StandardScaler before fitting any regularised model.
How do I choose the alpha hyperparameter?
Use cross-validation. Sklearn provides RidgeCV, LassoCV, and ElasticNetCV which do this automatically. Alternatively, use GridSearchCV with a range of alpha values. Never choose alpha by looking at test set performance — that leaks information.
