Reinforcement Learning for Real-World Applications – Offline RL, Reward Shaping and Safe RL

Reinforcement learning achieved its most celebrated successes in game environments — AlphaGo defeating world champions, OpenAI Five beating professional Dota 2 players, Atari game-playing agents surpassing human performance. These achievements drove enormous research excitement but obscured a key limitation: game environments provide unlimited cheap simulation, dense rewards, and full state observability — conditions that rarely hold in real-world applications. Real-world RL deployment in robotics, industrial control, recommendation systems, and healthcare requires addressing sample efficiency (each interaction is expensive), safety (exploration must not violate physical constraints), partial observability, and reward misspecification. This guide covers the practical techniques that have enabled real-world RL deployment: offline RL from historical data, reward shaping, model-based RL for sample efficiency, safe RL for constraint satisfaction, and the production deployment considerations for RL systems in recommenders, industrial control, and financial trading.

Offline RL — Learning from Historical Data without Online Exploration

Offline RL (or batch RL) is the paradigm most relevant to data scientists who have access to large historical interaction datasets but cannot or do not want to run live experiments. The standard online RL assumption — that the agent can freely interact with the environment to collect new experience — fails when each interaction costs money (a bidding system where each bid costs ad spend), time (a clinical trial where each decision takes months), or safety (a manufacturing process where exploration could damage equipment). Offline RL trains a policy exclusively from a fixed dataset D = {(s_t, a_t, r_t, s_{t+1})} collected by one or more behavioural policies. Conservative Q-Learning (CQL, Kumar et al., 2020) addresses the Q-value overestimation problem by adding a conservative regularisation to the standard Q-learning objective: L_CQL = L_Bellman + α × (E_{s~D, a~π}[Q(s,a)] − E_{s~D, a~D}[Q(s,a)]), which penalises large Q-values for actions not well-covered by the data distribution. Decision Transformer (Chen et al., 2021) reframes offline RL as a sequence modelling problem: given the history of states, actions, and return-to-go (the sum of future rewards), predict the next action to take. This Transformer-based approach connects directly to our Transformers guide and enables leveraging large pre-trained language model architectures for RL — treating the RL trajectory as a language and generating high-return trajectories autoregressively. The connection to our multi-armed bandits guide is direct: contextual bandits are a special case of offline RL where there is no temporal credit assignment (the reward is immediate and actions have no state transition effect).

Reward Shaping, Safe RL and Industrial Applications

Reward shaping is the art of designing reward functions that guide the agent toward desired behaviour without unintentionally incentivising undesired behaviour (reward hacking). The classic reward hacking example: an RL boat racing agent trained with reward proportional to score learned to spin in circles collecting score multipliers rather than completing the race. Potential-based reward shaping F(s, a, s’) = γΦ(s’) − Φ(s) is the theoretically safe approach: it adds a shaped reward that is exactly zero in expectation along optimal trajectories (the shaping does not change optimal policy), while providing denser feedback that accelerates learning. Φ(s) can be a heuristic value function (distance to goal, domain-specific progress metric), a learned value function from offline data, or the output of a reward model trained from human feedback — the last approach is exactly RLHF. Intrinsic motivation rewards — providing reward proportional to prediction error (curiosity, ICM) or information gain (VIME) — enable exploration in environments with sparse external rewards by incentivising the agent to visit novel, surprising states. The connection to our reinforcement learning guide is direct: reward shaping is an extension of the basic RL framework described there, adding a shaping function to the standard MDP reward signal.

Safe RL constrains the agent’s behaviour to satisfy safety constraints (temperature bounds, equipment limits, collision avoidance) at all times — not just in expectation. The constrained MDP (CMDP) framework adds constraint cost functions C_i(s, a) and requires that the expected cumulative constraint cost satisfies E[Σ_t C_i(s_t, a_t)] ≤ d_i. Lagrangian relaxation solves the CMDP by maintaining Lagrange multipliers λ_i for each constraint and optimising the Lagrangian objective L(π, λ) = J(π) − Σ_i λ_i (E[Σ_t C_i] − d_i), updating λ_i upward when constraints are violated and downward when they are satisfied with slack. This dual gradient descent approach is implemented in Safe-RL libraries (Safety Gym, OmniSafe). For the highest-stakes industrial applications (nuclear reactor control, medical device dosing), Model Predictive Control (MPC) is used as a safety layer that constrains the RL policy’s actions to those that are provably safe under the learned dynamics model. Google DeepMind’s deployment of RL to reduce Google’s data centre cooling energy by 30%+ used this architecture: an RL policy proposes setpoints, a safety layer validates them against physical constraints, and a human operator retains override capability.

RL for Recommenders, Finance and Energy

RL-based recommendation (covering both collaborative filtering extensions and the newer deep recommender systems) moves from myopic click maximisation to long-term engagement optimisation. The Markov Decision Process formulation: state = user history + context, action = recommended item, reward = engagement signal (watch time, purchase, subscription retention), transition = updated user state after seeing the recommendation. The key challenge is non-stationarity: user preferences change over time, making the environment non-stationary from the agent’s perspective. Exploration in production (the explore-exploit tradeoff from our bandits guide) is particularly fraught because exploration means showing users sub-optimal recommendations, which has direct revenue consequences — requiring carefully designed Thompson Sampling or UCB strategies that bound exploration to acceptable business cost. In algorithmic trading, RL agents trained to maximise Sharpe ratio or risk-adjusted return on historical data face the fundamental offline RL challenge: the historical data was generated under a specific market regime, and the RL policy’s actions would themselves change market dynamics if deployed at scale. Conservative offline RL (CQL, IQL) with careful validation on out-of-sample periods is the standard approach. Energy grid dispatch — deciding how to schedule generation from solar, wind, gas, and battery storage to minimise cost while meeting demand — is a natural CMDP: the reward is negative cost, and safety constraints enforce grid frequency stability and generation capacity limits.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Reinforcement learning foundations (MDP, Q-learning, policy gradient, PPO, SAC) are in our Reinforcement Learning guide. RLHF (reward modelling from human feedback) is in our RLHF guide. Multi-armed bandits (the contextual bandit special case of RL) are in our Bandits guide. Deep recommender systems (where RL-based recommendation extends supervised Two-Tower models) are in our Deep Recommenders guide. AI agents that use RL-trained policies for planning are in our AI Agents guide. Causal inference for counterfactual reward estimation in offline RL is in our Causal Inference guide.