Hyperparameter Tuning: How to Optimise Any ML Model (2026 Guide)

You trained a Random Forest. Accuracy is 83%. Your colleague trains the same Random Forest on the same data and gets 91%. The difference is not the algorithm, not the data, and not luck. The difference is hyperparameter tuning — a systematic process for finding the configuration that makes an algorithm perform at its best on your specific problem.

Hyperparameter tuning is one of the most high-leverage skills in a data scientist’s toolkit. A well-tuned model often outperforms a more complex but poorly tuned one. And unlike feature engineering, which requires deep domain knowledge, tuning is a systematic process you can apply to any model on any dataset. This guide walks through every major strategy, from brute-force grid search to intelligent Bayesian optimisation, with complete Python code for each.

Parameters vs Hyperparameters: The Core Distinction

Every machine learning model has two types of values that affect its predictions. Parameters are learned automatically from the data during training — the weights in a neural network, the split thresholds in a decision tree, the coefficients in a linear regression. You do not set these; the optimisation algorithm finds them by minimising the loss function on training data.

Hyperparameters control how the learning process itself works — they govern the structure of the model and the learning algorithm, not the learned values. How many trees should the Random Forest grow? How deep can each tree be? What fraction of features should be considered at each split? These settings shape what the model can learn and how efficiently it learns, but they are not updated during training. You set them before training starts, and they remain fixed throughout.

The reason tuning matters so much is that no default hyperparameter values are universally optimal. The defaults in scikit-learn are reasonable starting points, but “reasonable” rarely means “best for your specific dataset.” The relationship between hyperparameter values and model performance is often non-linear, non-obvious, and depends heavily on the size, noise level, and structure of your particular data.

Strategy 1: Grid Search — Exhaustive and Safe

Grid search is the most straightforward approach: define a grid of hyperparameter values to test, train and evaluate the model on every combination using cross-validation, and pick the combination that scored highest. It is slow, but thorough within the bounds you define.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
import pandas as pd

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth':    [None, 5, 10, 15],
    'min_samples_leaf': [1, 5, 10],
    'max_features': ['sqrt', 'log2']
}
# 3 * 4 * 3 * 2 = 72 combinations * 5 folds = 360 model fits

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,     # Use all CPU cores in parallel
    verbose=1,
    return_train_score=True
)
grid_search.fit(X_train, y_train)

print('Best hyperparameters:')
for param, val in grid_search.best_params_.items():
    print(f'  {param}: {val}')
print(f'
Best CV AUC: {grid_search.best_score_:.4f}')

# Examine top 10 configurations
results = pd.DataFrame(grid_search.cv_results_)
top10 = results.nlargest(10, 'mean_test_score')[
    ['params', 'mean_test_score', 'std_test_score', 'mean_train_score']
]
print('
Top 10 configurations:')
print(top10.to_string(index=False))

Strategy 2: Random Search — Often Better, Always Faster

A key insight from research by Bergstra and Bengio: in most machine learning problems, only a small number of hyperparameters have a large effect on performance. The rest barely matter. Grid search wastes most of its computation on unimportant parameters. Random search, by sampling randomly from the parameter space, is much more likely to hit the important dimensions with useful values.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_distributions = {
    'n_estimators':     randint(50, 600),     # Uniform integer from 50 to 600
    'max_depth':        [None, 3, 5, 7, 10, 15, 20],
    'min_samples_leaf': randint(1, 30),
    'max_features':     uniform(0.3, 0.7),    # Continuous: 30% to 100% of features
    'bootstrap':        [True, False]
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_distributions,
    n_iter=100,        # Test 100 random combinations
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    random_state=42,
    verbose=1
)
random_search.fit(X_train, y_train)

print('Best hyperparameters (Random Search):')
for param, val in random_search.best_params_.items():
    print(f'  {param}: {val}')
print(f'
Best CV AUC: {random_search.best_score_:.4f}')

Strategy 3: Optuna — Bayesian Optimisation (Best for Production)

Both grid and random search are unintelligent — they do not learn from previous trials. Optuna uses Bayesian optimisation: it builds a probabilistic model of which hyperparameter regions are likely to produce good results, then samples more densely from promising regions. This makes it substantially more sample-efficient — it finds better configurations with fewer model fits.

pip install optuna

import optuna
from sklearn.model_selection import cross_val_score

optuna.logging.set_verbosity(optuna.logging.WARNING)  # Suppress verbose output

def objective(trial):
    params = {
        'n_estimators':     trial.suggest_int('n_estimators', 50, 600),
        'max_depth':        trial.suggest_int('max_depth', 3, 25),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 25),
        'max_features':     trial.suggest_float('max_features', 0.3, 1.0),
        'bootstrap':        trial.suggest_categorical('bootstrap', [True, False])
    }
    model  = RandomForestClassifier(**params, random_state=42, n_jobs=-1)
    scores = cross_val_score(model, X_train, y_train, cv=5,
                             scoring='roc_auc', n_jobs=1)
    return scores.mean()

study = optuna.create_study(direction='maximize',
                            sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=150, show_progress_bar=True)

print(f'
Best AUC: {study.best_value:.4f}')
print('Best hyperparameters:')
for param, val in study.best_params.items():
    print(f'  {param}: {val}')

Optuna also provides excellent analysis tools to understand which hyperparameters matter most:

from optuna.visualization import plot_param_importances, plot_optimization_history

# Which hyperparameters had the most impact?
plot_param_importances(study).show()

# How did performance improve over trials?
plot_optimization_history(study).show()

Applying Tuned Hyperparameters to the Test Set

best_params = study.best_params
final_model = RandomForestClassifier(**best_params, random_state=42, n_jobs=-1)
final_model.fit(X_train, y_train)

from sklearn.metrics import classification_report, roc_auc_score
y_pred = final_model.predict(X_test)
y_prob = final_model.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))
print(f'Test ROC-AUC: {roc_auc_score(y_test, y_prob):.4f}')

Frequently Asked Questions

What are the most important hyperparameters to tune for Random Forest?

In order of typical impact: n_estimators (more trees generally improve performance up to a point of diminishing returns — try values from 100 to 500); max_depth (controls overfitting — None means trees grow until pure, which can overfit; values of 5-20 are typical); min_samples_leaf (ensures leaf nodes have enough samples to represent real patterns, not noise; values of 5-20 are common); max_features (fraction of features considered at each split — lower values add more randomness and reduce correlation between trees). For most datasets, tuning these four with 100 random search trials will recover most of the achievable performance improvement.

How do I avoid overfitting to the validation set during hyperparameter tuning?

This is a subtle but important problem. When you run 200 tuning trials, you will inevitably find hyperparameter values that score well on your validation data partly by chance — they happened to work well for this particular random split, not because they are genuinely better. The more trials you run, the worse this problem gets. The solution is a three-way data split: train set for fitting models, validation set for tuning, and test set that you touch exactly once at the very end to report final performance. Never use test set performance to make tuning decisions. If you used cross-validation for tuning, your validation performance estimate is already more robust than a single split, but you still need a held-out test set for final evaluation.

How many trials should I run in Random Search or Optuna?

For Random Search, 50-100 trials typically captures most of the achievable improvement, especially if you are tuning 3-5 hyperparameters. For Optuna, 100-200 trials works well for most problems because Bayesian sampling is more efficient than random sampling. Beyond that, you hit diminishing returns — the 200th trial is rarely meaningfully better than the 150th. The practical limit is often your time budget: each trial fits the model with 5-fold cross-validation, so 200 trials means 1,000 model fits. For a fast model like logistic regression, this takes seconds. For XGBoost on a large dataset, each trial might take a minute, making 200 trials a 3-hour run.

Should I tune hyperparameters before or after feature engineering?

Feature engineering first, always. Good features can lift performance by 10-30% on their own — far more than hyperparameter tuning typically achieves (3-8% in most cases). Tuning hyperparameters on a model trained on poor features optimises noise. Get your features right first: handle missing values and outliers appropriately, encode categorical variables, create interaction terms and domain-specific features where you have expertise. Once your feature set is stable, tune hyperparameters as the final step to squeeze out remaining performance.