Gradient Descent Explained: How Machine Learning Models Actually Learn (2026)

Gradient descent is the engine that powers nearly every machine learning model you’ve ever used. It’s the algorithm that makes neural networks learn from data, that trains logistic regression, and that optimises the parameters of almost every model in production today. Yet most data scientists treat it as a black box. Understanding gradient descent — really understanding it — makes you dramatically better at diagnosing training problems, tuning hyperparameters, and designing models that actually converge.

The Core Idea: Finding the Minimum of a Loss Function

Think of your model’s loss function as a landscape of hills and valleys. Every point in this landscape corresponds to a particular set of model weights, and the elevation represents the loss (error) at those weights. Your goal is to find the lowest point — the weights that produce the smallest loss on your training data.

Gradient descent does this by computing the gradient (slope) of the loss function at the current point, then taking a step in the downhill direction. It repeats this process until it reaches a minimum. The gradient tells you which direction is uphill; the negative gradient is downhill. Here’s the update rule for a single parameter:

import numpy as np
import matplotlib.pyplot as plt

# Simple gradient descent from scratch on a quadratic loss
# Loss function: L(w) = (w - 3)^2  (minimum at w=3)
def loss(w):
    return (w - 3) ** 2

def gradient(w):
    return 2 * (w - 3)

# Gradient descent loop
w = 0.0          # starting weight
lr = 0.1         # learning rate
history = [w]

for step in range(30):
    grad = gradient(w)
    w = w - lr * grad    # step downhill
    history.append(w)

print(f"Final w: {w:.6f} (true minimum: 3.0)")

# Plot convergence
plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
w_range = np.linspace(-1, 7, 100)
plt.plot(w_range, loss(w_range), 'b-', label='Loss L(w)')
plt.scatter(history, [loss(h) for h in history], c='red', s=30, zorder=5)
plt.xlabel('w'); plt.ylabel('Loss'); plt.title('Gradient Descent on Loss Surface')
plt.legend()

plt.subplot(1, 2, 2)
plt.plot([loss(h) for h in history], 'r-o', markersize=4)
plt.xlabel('Step'); plt.ylabel('Loss'); plt.title('Loss vs Training Step')
plt.tight_layout()
plt.show()

Batch, Stochastic, and Mini-Batch Gradient Descent

Batch gradient descent computes the gradient using the entire training dataset at each step. This gives an accurate gradient estimate but is extremely slow for large datasets — you need one full pass over all data just to take a single step.

Stochastic gradient descent (SGD) computes the gradient using just one randomly chosen training example per step. This is much faster and can escape local minima (due to noisy gradients), but converges erratically.

Mini-batch gradient descent is the practical middle ground used in almost every modern ML system. It uses a small batch of examples (typically 32-256) per step, giving a reasonably accurate gradient estimate while being fast enough to train on large datasets:

import numpy as np

def mini_batch_gradient_descent(X, y, lr=0.01, epochs=100, batch_size=32):
    # Mini-batch gradient descent for linear regression
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0
    loss_history = []

    for epoch in range(epochs):
        # Shuffle data at start of each epoch
        indices = np.random.permutation(n_samples)
        X_shuffled = X[indices]
        y_shuffled = y[indices]

        epoch_loss = 0
        for start in range(0, n_samples, batch_size):
            X_batch = X_shuffled[start:start + batch_size]
            y_batch = y_shuffled[start:start + batch_size]
            batch_n = len(X_batch)

            # Forward pass
            y_pred = X_batch @ weights + bias

            # Compute gradients (MSE loss)
            error = y_pred - y_batch
            dw = (2 / batch_n) * X_batch.T @ error
            db = (2 / batch_n) * np.sum(error)
            epoch_loss += np.mean(error ** 2)

            # Update parameters
            weights -= lr * dw
            bias    -= lr * db

        loss_history.append(epoch_loss / (n_samples // batch_size))

    return weights, bias, loss_history

Learning Rate: The Most Important Hyperparameter

The learning rate controls how big a step you take at each iteration. Too large and you’ll overshoot the minimum — the loss will oscillate or diverge. Too small and training will take forever. This sensitivity is why learning rate scheduling (reducing the learning rate as training progresses) is standard practice:

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=1000, n_features=20, noise=10, random_state=42)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# sklearn's SGD with learning rate schedule
sgd = SGDRegressor(
    loss='squared_error',
    learning_rate='invscaling',  # lr reduces as: lr0 / (t^power_t)
    eta0=0.01,                   # initial learning rate
    max_iter=1000,
    random_state=42
)
sgd.fit(X_scaled, y)
print(f"R2 score: {sgd.score(X_scaled, y):.3f}")

Advanced Optimisers: Momentum, RMSprop, and Adam

Plain gradient descent can be slow in flat regions and oscillates in narrow valleys. Modern optimisers address this. Momentum accumulates a velocity vector in directions of persistent gradient, accelerating in consistent directions. Adam (Adaptive Moment Estimation) combines momentum with per-parameter learning rates — it’s the default choice for training neural networks and works well without manual learning rate tuning:

import torch
import torch.nn as nn

# PyTorch example: comparing optimisers
model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1))

# Adam is the modern default
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999))

# SGD with momentum
# optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Adam with weight decay (L2 regularization built in)
# optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

Frequently Asked Questions

How do I know if gradient descent is converging?
Plot the training loss versus epoch. It should decrease and eventually plateau. If it oscillates wildly or increases, your learning rate is too large. If it barely moves, the learning rate is too small or your gradient is vanishing (common in deep networks).

What causes a model to get stuck in a local minimum?
For non-convex loss surfaces (like neural networks), there are many local minima. In practice, most local minima in deep networks are “good enough” — they’re nearly as good as the global minimum. Mini-batch noise actually helps escape poor local minima. True saddle points (where gradient is zero but it’s not a minimum) are a bigger concern, and Adam handles them better than vanilla SGD.

Should I always use Adam?
Adam is an excellent default, but SGD with momentum sometimes finds better generalisations in computer vision tasks. For tabular data and most standard models, Adam or its variant AdamW is the practical choice. When in doubt, start with Adam.