📋 KEY INSIGHTS
- Hyperparameter optimisation (HPO) is the outer loop of model training: while gradient descent optimises model parameters (weights) given a fixed configuration, HPO searches over the space of configurations (learning rate, architecture depth, regularisation strength, batch size) to find the one that maximises validation performance. The challenge is that each configuration evaluation requires a full training run — an expensive function evaluation — making HPO an instance of black-box optimisation where the objective function (validation accuracy) is expensive, noisy, and has no accessible gradients. The three dominant HPO algorithm families are: sequential model-based optimisation (Bayesian optimisation with surrogate models), multi-fidelity methods (evaluate cheap proxies first, only invest full compute in promising configurations), and evolutionary/population-based methods (maintain a population of configurations and iteratively improve it).
- Tree-structured Parzen Estimator (TPE, Bergstra et al., 2011) is the algorithm underlying Optuna and HyperOpt, and the most widely used HPO method in practice. TPE models the conditional distribution p(x|y) of configurations given their performance using two kernel density estimators: l(x) models configurations that achieved performance better than a threshold γ (the “good” distribution) and g(x) models configurations that achieved performance worse than γ (the “bad” distribution). The acquisition function is the ratio l(x)/g(x) — a configuration that is likely under l(x) and unlikely under g(x) is both promising (similar to past good configurations) and informative. TPE naturally handles conditional search spaces (e.g. the depth hyperparameter only matters if the architecture type is “deep”) by using tree-structured dependency graphs, hence the name.
- CMA-ES (Covariance Matrix Adaptation Evolution Strategy) is the gold standard for continuous HPO spaces: it maintains a multivariate Gaussian distribution over the hyperparameter space and updates the mean, step size, and full covariance matrix after each generation of evaluations. The covariance adaptation captures correlations between hyperparameters — if high learning rate and low weight decay always co-occur in good configurations, the covariance matrix develops off-diagonal entries that bias sampling toward the correlated region. CMA-ES is particularly effective for hyperparameters with complex interactions (e.g. learning rate and batch size, which have an effective learning rate relationship) and for optimising neural architecture parameters that form a continuous landscape.
- Hyperband (Li et al., 2017) is the foundational multi-fidelity HPO method: allocate a small budget (few training epochs, small data fraction) to many random configurations, then promote the top fraction to a larger budget, iterating until a full-budget winner is found. The key insight: configurations that will be good at full budget tend to be relatively good even at small budget — early stopping on bad configurations frees compute for more configurations, dramatically improving the exploration-exploitation tradeoff compared to full-budget random search. BOHB (Bayesian Optimisation with HyperBand, Falkner et al., ICML 2018) combines Hyperband’s multi-fidelity scheduling with TPE’s model-based sampling: use TPE to generate configurations that are both promising (high l(x)/g(x)) and allocated efficiently across fidelity levels. BOHB is the state-of-the-art on most HPO benchmarks, combining the best of both worlds.
- Population-Based Training (PBT, Jaderberg et al., DeepMind 2017) is a fundamentally different approach that jointly optimises hyperparameters and model weights during training: maintain a population of K models training simultaneously, and periodically exploit (copy weights from better-performing to worse-performing models) and explore (perturb hyperparameters of the copied model). PBT is particularly powerful for hyperparameters that benefit from scheduled adaptation during training — learning rate schedules, curriculum difficulty, data augmentation intensity — because it discovers schedules rather than fixed values. PBT is the dominant HPO method for RL training (where hyperparameter sensitivity is extreme) and was used to train AlphaStar’s StarCraft II agents at DeepMind.
- Multi-objective HPO simultaneously optimises multiple conflicting objectives — validation accuracy and inference latency, accuracy and memory footprint, accuracy and training cost. The result is a Pareto front of configurations where no configuration is dominated (better on all objectives) by another. Optuna’s NSGAIISampler implements multi-objective TPE with NSGA-II (Non-dominated Sorting Genetic Algorithm II) for Pareto front estimation. The connection to our Neural Architecture Search guide is direct: NAS is multi-objective HPO over discrete architecture spaces, optimising accuracy vs. FLOPS or latency — the DARTS and EfficientNet results described there use the same Pareto front framework as multi-objective HPO, applied to the architecture search space.
Every machine learning model has two levels of parameters: the model parameters (weights) learned by gradient descent during training, and the hyperparameters (learning rate, architecture depth, regularisation coefficients, batch size, data augmentation policy) that control the training process itself. While gradient descent provides an efficient, principled algorithm for optimising model weights, no equivalent first-order algorithm exists for hyperparameters — their effect on model performance can only be measured after a full (or partial) training run. This makes HPO a classic expensive black-box optimisation problem. This guide covers the full HPO algorithmic landscape: random search and grid search baselines, TPE and Bayesian optimisation with Optuna, CMA-ES for continuous spaces, Hyperband and BOHB for multi-fidelity HPO, Population-Based Training for schedule optimisation, multi-objective HPO for the accuracy-efficiency tradeoff, and practical guidance on search space design and parallelisation.
TPE, Bayesian Optimisation and Optuna
Random search (Bergstra & Bengio, 2012) is a surprisingly strong HPO baseline: it outperforms grid search because it samples each hyperparameter independently, ensuring that the full range of each hyperparameter is explored even with a small budget — grid search wastes evaluations exploring Cartesian combinations that share the same value of the most important hyperparameter. The empirical rule: random search needs approximately 60 evaluations to find a configuration within 5% of optimal with 95% probability for problems where only 1-2 hyperparameters are truly important. Bayesian optimisation (BO) improves on random search by building a probabilistic surrogate model of the objective function (validation loss as a function of hyperparameters), then using the surrogate to select the next configuration that balances exploitation (high predicted performance) and exploration (high uncertainty). Gaussian Process BO uses a GP as the surrogate: the GP posterior provides a Gaussian predictive distribution at each untested configuration, and acquisition functions (Expected Improvement, Upper Confidence Bound, Probability of Improvement) convert the posterior into a scalar score for configuration ranking. GP BO is data-efficient (finds good configurations in few evaluations) but scales poorly with the number of hyperparameters (GP inference is O(n^3) in the number of past evaluations, limiting it to ~100 dimensions). The connection to our Gaussian Processes guide is direct: Bayesian HPO with GP surrogates is the canonical application of GPs as probabilistic surrogate models for expensive function optimisation, using the same GP posterior update equations described there. TPE replaces the GP surrogate with a non-parametric density ratio estimator that scales better to high-dimensional and conditional search spaces. Optuna implements TPE with several extensions: CMA-ES sampler for continuous spaces, NSGAIISampler for multi-objective problems, and integration with pruning (Hyperband-style early stopping) via the pruners API. Optuna’s define-by-run API makes hyperparameter spaces easy to define and modify: trial.suggest_float(‘lr’, 1e-5, 1e-1, log=True), trial.suggest_int(‘n_layers’, 2, 8), trial.suggest_categorical(‘optimizer’, [‘adam’, ‘sgd’, ‘adamw’]). The connection to our AutoML guide is direct: Optuna is the HPO engine underlying several AutoML frameworks (AutoGluon, FLAML) and is the most widely used standalone HPO library in production ML pipelines.
| Algorithm | Surrogate Model | Multi-Fidelity? | Search Space | Best For |
|---|---|---|---|---|
| Random Search | None | No | Any | Baseline; few hyperparams |
| GP Bayesian Optimisation | Gaussian Process | No | Continuous, low-dim | Few evaluations; <20 dims |
| TPE (Optuna default) | Kernel density estimators | No (+ pruners) | Any incl. conditional | General purpose; widely used |
| CMA-ES | Multivariate Gaussian | No | Continuous | Correlated continuous spaces |
| Hyperband | None | Yes | Any | Fixed total compute budget |
| BOHB | TPE per fidelity | Yes | Any | SOTA on most benchmarks |
| PBT | None (population) | Yes (adaptive) | Continuous | RL; adaptive schedules |
BOHB, Population-Based Training and Multi-Objective HPO
BOHB (Bayesian Optimisation with HyperBand) addresses the key limitation of pure Bayesian optimisation for deep learning HPO: each evaluation (a full training run) is so expensive that standard BO can only afford 20-100 evaluations, which is insufficient to explore the high-dimensional hyperparameter spaces of deep networks. BOHB uses Hyperband’s successive halving schedule to generate many low-fidelity (few epochs) evaluations cheaply, while using TPE to focus evaluations on promising regions of the search space identified from earlier iterations. The successive halving bracket: start with n configurations at minimum budget b_min, keep the top 1/η fraction and train them to η×b_min budget, repeat until one configuration remains at maximum budget b_max. BOHB runs multiple interleaved brackets with different n and b_min values (the HyperBand schedule), sharing the TPE model across all brackets. This achieves 10-100× wall-clock speedup over standard BO on deep learning benchmarks (HPOBench, JAHS-Bench-201) while matching or beating its final performance. The HpBandSter Python library implements BOHB with distributed worker support for multi-GPU HPO. Population-Based Training differs from all other HPO methods in a critical way: it adapts hyperparameters during training, discovering training schedules rather than fixed values. The PBT algorithm: initialise K workers with random hyperparameters and start training in parallel; every T steps, each worker checks whether it is in the bottom 20% of the population by current performance; if so, copy the weights and hyperparameters of a randomly selected top-20% worker, then perturb the copied hyperparameters (multiply by a random factor in [0.8, 1.2] for continuous hyperparameters); continue training with the updated weights and hyperparameters. PBT naturally discovers warm restart learning rate schedules, progressive data augmentation curricula, and loss function weight schedules that fixed HPO methods cannot represent. The connection to our RL for real-world applications guide is direct: PBT was developed specifically for RL training at DeepMind and is the standard HPO method for large-scale RL — it was used to tune AlphaStar, AlphaCode, and other DeepMind systems where the sensitivity of RL training to hyperparameter schedules is extreme. Multi-objective HPO is increasingly important as ML models are deployed to resource-constrained environments (mobile devices, edge hardware, real-time APIs): a model that achieves 92% accuracy at 100ms latency may be preferable to one that achieves 94% accuracy at 500ms latency, depending on the application. Optuna’s multi-objective interface accepts a list of return values from the objective function: return accuracy, latency and sets directions=[‘maximize’, ‘minimize’]. The resulting Pareto front study.best_trials contains all non-dominated configurations, visualised as a scatter plot where the practitioner can select their preferred tradeoff point. The connection to our model compression guide is direct: the accuracy-latency Pareto front from multi-objective HPO is the same optimisation target as model compression (pruning, quantisation, knowledge distillation) — both seek to move the Pareto frontier by finding configurations that achieve a better accuracy-efficiency tradeoff.
✦ SUMMARIZE THIS ARTICLE WITH AI
AutoML (Optuna and BOHB as the HPO engines in AutoML frameworks) is in our AutoML guide. Gaussian Processes (GP surrogate models for Bayesian HPO) are in our Gaussian Processes guide. Neural Architecture Search (multi-objective HPO over discrete architecture spaces) is in our NAS guide. Model compression (accuracy-efficiency Pareto front optimisation) is in our Model Compression guide. RL for real-world applications (PBT for RL training schedule optimisation) is in our RL guide. Model evaluation (cross-validation and train/val/test split strategy for HPO) is in our Model Evaluation guide. MLflow (experiment tracking for HPO runs) is in our MLflow guide.



