Regularisation Techniques in Machine Learning – L1, L2, Dropout, Early Stopping and Beyond

Overfitting — building a model that memorises training data instead of learning general patterns — is the central challenge in machine learning. Regularisation is the collection of techniques that constrain model complexity to improve generalisation. Understanding regularisation deeply, not just knowing the names of the techniques, is a core competency for any senior data scientist and a frequent topic in machine learning interviews.

This guide connects directly to our Model Evaluation and Hyperparameter Tuning guide (which covers the bias-variance tradeoff that regularisation addresses), our Machine Learning Interview Q&A (which includes regularisation questions for tree-based models), our Deep Learning Interview Q&A (dropout, batch normalisation), and our Feature Engineering guide (feature selection as a form of regularisation). For regularisation in the context of fine-tuning large models, see our Neural Network Architectures guide.

The Overfitting Problem — What Regularisation Solves

A model overfits when it fits the training data too closely, capturing noise rather than signal. The symptoms: training loss continues to decrease while validation loss plateaus or increases (the generalisation gap); the model performs dramatically better on training data than held-out test data; the model’s decision boundaries are overly complex relative to the data structure.

Root causes of overfitting: too many parameters relative to training examples (a neural network with 1M parameters trained on 100 examples has an astronomical number of solutions that perfectly fit the data); model complexity exceeds the complexity of the underlying pattern (a degree-15 polynomial for a linear relationship); insufficient training data; or noisy labels that the model memorises.

Regularisation addresses overfitting by: adding a penalty term to the loss function that discourages complex models (L1, L2), randomly disabling parts of the model during training to prevent co-adaptation (dropout), stopping training before overfitting occurs (early stopping), or expanding the effective training dataset (data augmentation).

L1 Regularisation — LASSO

L1 regularisation (Least Absolute Shrinkage and Selection Operator) adds the sum of absolute values of model weights to the loss: Loss_L1 = Loss + λ · Σ|wᵢ|. The parameter λ (also called alpha in scikit-learn) controls regularisation strength — larger λ means more regularisation.

Why L1 produces sparse solutions (automatic feature selection): The L1 penalty has a non-differentiable point at wᵢ=0 — the gradient is -λ for negative weights and +λ for positive weights, creating a constant force pushing weights toward zero. For small weights, this force exceeds the gradient from the data, pushing those weights exactly to zero. L1 does not just shrink weights — it eliminates them entirely. A model regularised with L1 will use only the most important features; all others get weight=0. This makes L1 ideal when you suspect many features are irrelevant (high-dimensional data, many correlated features). Geometrically, the L1 “ball” (feasible region) has corners on the axes — the optimal solution tends to land on a corner where some weights are zero.

L2 Regularisation — Ridge

L2 regularisation (Ridge) adds the sum of squared weights: Loss_L2 = Loss + λ · Σwᵢ². The gradient of the L2 penalty is 2λwᵢ — a force proportional to the weight magnitude. This means: large weights are penalised heavily, small weights are penalised lightly, and weights are never pushed exactly to zero (only asymptotically approach zero as λ → ∞). L2 shrinks all weights but keeps all features. It is preferred when all features are believed to be relevant but some should have smaller influence. Geometrically, the L2 ball is a smooth sphere — the optimal solution tends to land at a non-corner point where all weights are non-zero.

from sklearn.linear_model import Lasso, Ridge, ElasticNet, LassoCV, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

# --- ALWAYS standardise before L1/L2 regularisation ---
# Regularisation penalises weight magnitude; features on different scales
# would be penalised unfairly without standardisation.

lasso_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('lasso',  Lasso(alpha=0.01, max_iter=10_000))
])

# Cross-validated lambda selection (LassoCV, RidgeCV)
lasso_cv = LassoCV(alphas=np.logspace(-4, 1, 50), cv=5, n_jobs=-1)
lasso_cv.fit(X_train, y_train)
print(f'Best alpha: {lasso_cv.alpha_:.6f}')
print(f'Non-zero coefficients: {(lasso_cv.coef_ != 0).sum()} / {X_train.shape[1]}')

# Ridge
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 50), cv=5)
ridge_cv.fit(X_train, y_train)
print(f'Best alpha: {ridge_cv.alpha_:.2f}')

# Elastic Net — l1_ratio=0 is pure L2, l1_ratio=1 is pure L1
enet = ElasticNet(alpha=0.01, l1_ratio=0.5, max_iter=10_000)

Dropout — Regularisation for Neural Networks

Dropout (Srivastava et al., 2014) randomly sets activations to zero during each forward pass with probability p (typically 0.1-0.5 for hidden layers, 0.0-0.1 for input layers). At test time, dropout is disabled and all activations are multiplied by (1-p) to maintain the same expected activation magnitude — or equivalently, weights are scaled by (1-p) at training time (inverted dropout, the modern standard).

Why dropout works as regularisation: Each forward pass uses a different subnetwork (the particular neurons not dropped). The full network cannot rely on any single neuron or co-adapt neurons in a fragile way — it must learn robust features that are useful in the presence of other dropped neurons. This is equivalent to training an exponential ensemble of 2^n subnetworks (where n is the number of neurons) and averaging them at test time. Dropout also acts as a form of noisy input — analogous to Gaussian noise injection, which is a classical regularisation technique.

Where to place dropout: In MLPs: after activation functions in hidden layers. In CNNs: typically not inside conv layers (BN handles regularisation there) but in the fully connected head. In Transformers: after attention weights and after the FFN layer. In RNNs: on inputs and outputs, not recurrent connections (using VariationalDropout for the recurrent path).

import torch.nn as nn

class RegularisedMLP(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim, dropout_rate=0.3):
        super().__init__()
        layers = []
        in_dim = input_dim
        for h in hidden_dims:
            layers += [
                nn.Linear(in_dim, h),
                nn.BatchNorm1d(h),      # BN before activation
                nn.ReLU(),
                nn.Dropout(dropout_rate)  # Dropout after activation
            ]
            in_dim = h
        layers.append(nn.Linear(in_dim, output_dim))
        self.net = nn.Sequential(*layers)

    def forward(self, x): return self.net(x)

model = RegularisedMLP(100, [256, 128, 64], 1, dropout_rate=0.3)

# IMPORTANT: disable dropout at inference
model.train()   # dropout active
model.eval()    # dropout disabled (also affects BatchNorm)

Batch Normalisation — Implicit Regularisation

Batch Normalisation (BN) normalises layer activations within each mini-batch: ẑ = (z – μ_batch) / σ_batch, then applies learnable scale (γ) and shift (β): y = γẑ + β. BN was originally motivated as a solution to internal covariate shift (changing activation distributions during training) — though this explanation has since been questioned. Empirically, BN: enables much higher learning rates (stable training), dramatically reduces sensitivity to weight initialisation, reduces the need for dropout (strong implicit regularisation), and speeds up convergence. BN is the primary regulariser in modern CNNs; Transformer-based models use Layer Normalisation instead (normalises across features, not the batch — works for batch size 1).

Early Stopping and Other Techniques

Early stopping: Monitor validation loss during training; stop when it stops improving (with a patience of N epochs). Effectively regularises by limiting how many gradient descent steps the model takes — preventing it from fitting noise in the training data late in training. Simple to implement and highly effective. The best checkpoint (minimum validation loss) is saved and loaded at the end.

Data augmentation: Artificially increases training set size and teaches invariances. For images: random crop, flip, rotation, colour jitter, MixUp, CutMix (see our Computer Vision Interview Q&A). For text: backtranslation, synonym replacement, random deletion (see our NLP Interview Q&A). For time series: time warping, window slicing, jittering (see our Time Series Interview Q&A). Augmentation directly addresses the root cause of overfitting (insufficient training data) rather than constraining the model.

Weight decay in neural networks: In practice, L2 regularisation in neural networks is implemented as weight decay in the optimiser update: w ← w – lr·∇loss(w) – lr·λ·w. This is the standard for PyTorch’s Adam optimiser — use AdamW (Adam with decoupled weight decay) rather than Adam with L2 in the loss, as these are mathematically different with adaptive optimisers. A typical weight_decay value for AdamW: 0.01-0.1 for fine-tuning, 1e-4 to 1e-2 for training from scratch.

Choosing the right regularisation technique:

Regularisation is not a black box of tricks — each technique has a principled motivation rooted in the bias-variance tradeoff, Bayesian priors (L2 corresponds to a Gaussian prior over weights; L1 corresponds to a Laplace prior), or information-theoretic arguments (dropout as ensemble averaging). For interview questions on regularisation, our ML Interview Q&A covers tree-based regularisation and our Deep Learning Interview Q&A covers dropout, BN, and weight decay questions. The Ensemble Methods guide covers how ensemble learning reduces variance (a complementary regularisation approach). For evaluating whether your regularisation is working correctly, our Model Evaluation guide covers learning curves, nested cross-validation, and the bias-variance diagnostic.