Reinforcement Learning Explained – Algorithms, Concepts and Real-World Applications

Reinforcement learning (RL) is the branch of machine learning where an agent learns to make decisions by interacting with an environment — receiving rewards for good actions and penalties for bad ones. Unlike supervised learning, RL requires no labelled dataset; the training signal comes entirely from the environment’s feedback. RL has produced some of the most dramatic AI achievements: AlphaGo defeating world champion Go players, AlphaStar mastering StarCraft II, and ChatGPT’s RLHF alignment layer. This guide explains the theory, algorithms, and applications of reinforcement learning from the ground up.

If you are preparing for ML interviews, also check our Machine Learning Interview Q&A and Deep Learning Interview Q&A for questions specifically covering RL theory.

The Reinforcement Learning Framework

RL is formally defined as a Markov Decision Process (MDP) — a mathematical framework for sequential decision-making under uncertainty. An MDP has five components:

State (S): A representation of the environment at time t. In a chess game, the state is the full board position. In robotics, the state is the robot’s joint angles and velocities, plus sensor readings. The state must satisfy the Markov property — the future is independent of the past given the present state. All information needed to predict future dynamics is encoded in the current state.

Action (A): The set of decisions the agent can make at each time step. Discrete action spaces: move left/right/up/down in a grid world, choose from 18 Atari joystick positions. Continuous action spaces: apply a torque in [-2, 2] Nm to a robotic joint. Discrete actions are handled well by Q-Learning variants; continuous actions require policy gradient methods.

Reward (R): A scalar signal from the environment indicating how good the last action was. Reward design (reward shaping) is one of the hardest parts of RL — an incorrectly designed reward leads to unexpected and often bizarre agent behaviour. A famous example: a simulated boat racing agent given reward for hitting reward rings learned to spin in circles collecting the same rings repeatedly instead of completing the race.

Transition Dynamics P(s’|s,a): The probability distribution over next states given current state and action. Model-based RL learns P explicitly; model-free RL learns a policy directly from experience without modelling P.

Discount Factor (gamma): A value in [0, 1] that weights future rewards. gamma=0: agent is completely myopic. gamma=1: agent weights all future rewards equally. Typical values: 0.95-0.99. The discounted return Gt = Rt + gamma*Rt+1 + gamma^2*Rt+2 + … is what the agent tries to maximise.

Value Functions and the Bellman Equation

The value function V(s) estimates how good it is to be in state s — the expected discounted return from that state following a policy pi. The Bellman equation is recursive: the value of a state equals the immediate reward plus the discounted value of the next state. This self-referential structure is the foundation of all RL algorithms.

The action-value function Q(s, a) estimates the value of taking action a in state s, then following policy pi. The optimal Q-function Q*(s, a) gives the maximum expected return achievable from (s, a). If we know Q*, we can derive the optimal policy: pi*(s) = argmax_a Q*(s, a). Q-Learning directly estimates Q*.

# Bellman update for Q-Learning
# Q(s, a) += alpha * [r + gamma * max Q(s', a') - Q(s, a)]
# alpha = learning rate (0.01 to 0.5)
# TD error = r + gamma*max_Q(s') - Q(s,a)

import numpy as np

class TabularQLearning:
    def __init__(self, n_states, n_actions, alpha=0.1, gamma=0.99, epsilon=0.1):
        self.Q       = np.zeros((n_states, n_actions))
        self.alpha   = alpha
        self.gamma   = gamma
        self.epsilon = epsilon

    def act(self, state):
        if np.random.rand() < self.epsilon:
            return np.random.randint(self.Q.shape[1])   # explore
        return np.argmax(self.Q[state])                 # exploit

    def update(self, s, a, r, s_next, done):
        target = r if done else r + self.gamma * np.max(self.Q[s_next])
        self.Q[s, a] += self.alpha * (target - self.Q[s, a])

Deep Q-Networks (DQN)

DQN (DeepMind, 2013-2015) replaced the Q-table with a neural network Q(s, a; theta). Given an Atari game frame (84x84 grayscale, stacked 4 frames for motion), the network outputs Q-values for all 18 actions simultaneously. This was the first RL algorithm to learn control policies directly from raw pixels, achieving superhuman performance on 49 Atari games.

Experience Replay: Stores experiences (s, a, r, s') in a replay buffer of size 1M. During training, random mini-batches of 32 are sampled. This breaks temporal correlations and allows each experience to be used multiple times.

Target Network: A separate frozen network provides stable TD targets. Updated every C=10,000 steps. Without this, the target moves with every update — like trying to hit a moving bullseye.

import torch
import torch.nn as nn
import random
from collections import deque

class DQN(nn.Module):
    def __init__(self, n_obs, n_actions):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_obs, 128), nn.ReLU(),
            nn.Linear(128, 128),  nn.ReLU(),
            nn.Linear(128, n_actions)
        )
    def forward(self, x): return self.net(x)

class ReplayBuffer:
    def __init__(self, capacity=10_000):
        self.buf = deque(maxlen=capacity)
    def push(self, *args): self.buf.append(args)
    def sample(self, batch_size): return random.sample(self.buf, batch_size)
    def __len__(self): return len(self.buf)

Policy Gradient and PPO

Policy gradient methods directly optimise the policy pi(a|s; theta) using gradient ascent on expected return. The Policy Gradient Theorem: increase the log-probability of actions that lead to high returns, decrease it for bad actions.

Actor-Critic: Combines a policy network (actor) that selects actions with a value network (critic) that estimates V(s). The critic's TD error (advantage) updates the actor. A2C and A3C are standard implementations.

PPO (Proximal Policy Optimisation, OpenAI 2017): The most widely-used policy gradient algorithm. Prevents excessively large policy updates by clipping the importance sampling ratio: L = min(r_t * A_t, clip(r_t, 1-eps, 1+eps) * A_t) where eps=0.2. PPO is the algorithm used in ChatGPT's RLHF stage and most robotics RL systems.

Real-World Applications

LLM Alignment (RLHF): Human raters rank model outputs; a reward model is trained; the LLM is fine-tuned with PPO to maximise the reward model's score. This is how GPT-4, Claude, and Gemini are aligned.

Recommendation Systems: YouTube, Netflix, and Spotify use RL to optimise session-level engagement rather than immediate click-through rate, reducing regretted content.

Robotics: Training arms for manipulation in simulation with RL, then transferring to real hardware (sim-to-real transfer).

Finance: Portfolio management, market making (optimising bid-ask spreads), and order execution minimising market impact.

Healthcare: Personalising treatment dosing, radiotherapy planning. Offline RL (batch RL) learns from historical patient data without exploring on real patients.

Algorithm selection guide: Discrete actions, low-dimensional state → Q-Learning, DQN. Atari-style pixel input → Rainbow DQN. Continuous actions, robotics → SAC or PPO. Offline RL from fixed dataset → CQL, IQL. Understanding RL — including its failure modes like reward hacking and sample inefficiency — is increasingly expected in senior ML interviews. See our Deep Learning Interview Q&A for RLHF-specific questions.