Bayesian Deep Learning – MC Dropout, Deep Ensembles, SWAG and Calibrated Uncertainty

Standard neural networks are overconfident: a well-trained image classifier will assign 99.9% confidence to a cat image it has never seen a variant of, even when a slight perturbation makes the image ambiguous. This overconfidence is not just a theoretical concern — it causes real failures in medical imaging (confident wrong diagnoses), autonomous driving (confident navigation decisions on novel road conditions), and financial risk modelling (confident predictions on market regimes the model was not trained on). Bayesian deep learning addresses this by treating neural network weights as random variables with a posterior distribution rather than fixed point estimates, producing predictive distributions that explicitly represent uncertainty. This guide covers MC Dropout, Deep Ensembles, SWAG, variational inference for neural networks, calibration measurement and temperature scaling, and the practical decision of which uncertainty method to use for which application.

MC Dropout and the Dropout-as-Approximate-Inference Insight

Gal and Ghahramani (2016) showed that a neural network with dropout applied before every weight layer, trained with standard cross-entropy loss, is mathematically equivalent to approximate variational inference in a deep Gaussian process — with the dropout mask playing the role of the variational distribution over weights. The practical implication: by keeping dropout active at test time and running T stochastic forward passes, we draw T approximate samples from the posterior predictive distribution P(y|x, D) ≈ (1/T) Σ_t P(y|x, w_t), w_t ~ q(w). The mean is the prediction; the variance measures uncertainty. For regression, this produces a predictive mean and variance; for classification, it produces a distribution over class probabilities whose entropy measures predictive uncertainty. The connection to our regularisation guide is direct: dropout was originally motivated as a regularisation technique, and MC Dropout reframes it as approximate Bayesian inference — the same operation serves both purposes simultaneously.

MC Dropout is attractive because it requires minimal engineering: any existing model using dropout can be converted to a Bayesian model by simply enabling dropout at inference time. The main limitation is that MC Dropout tends to underestimate uncertainty for out-of-distribution inputs — the dropout samples are all drawn from a distribution anchored near the training data manifold, so the variance between them remains low even when the input is far from anything seen during training. This failure mode (high confidence on OOD inputs) is particularly dangerous for safety-critical applications. The connection to our anomaly detection guide is direct: detecting out-of-distribution inputs is exactly the anomaly detection problem, and MC Dropout uncertainty can serve as a soft anomaly score, complementing the harder OOD detection methods (energy-based OOD scoring, Mahalanobis distance in feature space) covered there.

Deep Ensembles, SWAG and Calibration

Deep Ensembles (Lakshminarayanan et al., 2017) remain the empirical gold standard for neural network uncertainty despite their simplicity. Training M = 5 independently initialised networks on the same data produces ensemble members that have converged to different local minima of the loss landscape — regions of parameter space that all achieve similar training and validation loss but make qualitatively different predictions on out-of-distribution inputs. This functional diversity is the key: while MC Dropout samples all concentrate near a single mode of the posterior, ensemble members span multiple modes, producing much more honest uncertainty on novel inputs. The ensemble predictive distribution for classification is the average of M softmax outputs: p̄(y|x) = (1/M) Σ_m softmax(f_m(x)/T_m), where each model may be separately temperature-scaled. The connection to our ensemble methods guide is direct: deep ensembles apply the same variance-reducing logic (averaging diverse predictors) as bagging and random forests to neural networks, but the source of diversity (random weight initialisation + SGD stochasticity) is different from the data resampling used in bagging.

SWAG (Maddox et al., 2019) approximates the posterior over weights using a Gaussian distribution fit to the trajectory of SGD iterates in a modified training procedure: after standard training, run additional epochs with cyclical learning rates, saving weight snapshots at each cycle peak. Fit a low-rank Gaussian (mean = SWA mean, covariance = rank-20 approximation of the iterate covariance + diagonal variance) to the saved snapshots. At inference, draw weight samples from this Gaussian and average predictions — a cheap but surprisingly accurate approximation to full Bayesian inference. SWAG requires only one training run and adds minimal memory, making it the recommended practical choice when Deep Ensembles’ M× training cost is prohibitive. The connection to our Bayesian statistics guide is direct: SWAG is a form of Laplace approximation (a Gaussian approximation to the posterior centred at the MAP estimate) applied to the SWA weight rather than the maximum likelihood estimate, and its quality depends on how well the posterior is approximated by a Gaussian in the neighbourhood of the SWA solution.

Calibration, Temperature Scaling and Conformal Prediction

A well-calibrated model is one where the predicted probability matches the empirical frequency: among all inputs assigned confidence 0.8, exactly 80% should be correctly classified. Modern neural networks are systematically overconfident (ECE > 0 with accuracy < confidence) because standard cross-entropy training, combined with large model capacity and data augmentation, drives logits to large magnitudes that produce near-degenerate softmax distributions. Temperature scaling fits a single scalar T on a held-out calibration set by minimising NLL: p_calibrated(y=k|x) = exp(z_k/T) / Σ_j exp(z_j/T). T > 1 softens the distribution, reducing overconfidence. Platt scaling and isotonic regression are alternative post-hoc calibration methods when a single temperature is insufficient (e.g., when miscalibration differs across confidence levels). The connection to our model evaluation guide is direct: reliability diagrams (confidence vs. accuracy plots) and ECE are the standard calibration diagnostic tools alongside the discrimination metrics (AUC, F1) covered there.

Conformal prediction provides formal coverage guarantees without any distributional assumptions: the conformal prediction set C_{1-α}(x) = {y : s(x, y) ≤ q̂_{1-α}}, where s is a non-conformity score (e.g., 1 − p̂(y|x) for classification) and q̂_{1-α} is the (1−α) quantile of non-conformity scores on a calibration set, satisfies P(Y ∈ C_{1-α}(X)) ≥ 1−α on exchangeable test data. This guarantee holds for any underlying model — Bayesian or not — and makes conformal prediction the recommended method when the application requires provable coverage (regulatory compliance in clinical decision support, safety certification for autonomous systems) rather than merely well-calibrated probabilities. The combination of Deep Ensembles (for well-calibrated probabilities) and conformal prediction sets (for guaranteed coverage) represents the current best practice for uncertainty quantification in production ML systems.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Regularisation (Dropout as a regulariser and as approximate inference) is in our Regularisation guide. Conformal prediction and guaranteed coverage intervals are in our Conformal Prediction guide. Model evaluation and calibration metrics (ECE, reliability diagrams) are in our Model Evaluation guide. Bayesian statistics and variational inference foundations are in our Bayesian Statistics guide. Anomaly detection and OOD input detection are in our Anomaly Detection guide. Ensemble methods (bagging, boosting) and their connections to deep ensembles are in our Ensemble Methods guide.