📋 KEY INSIGHTS
- Simulation and digital twins are foundational infrastructure for ML systems that must learn from experience in environments where real-world data collection is expensive, slow, dangerous, or ethically problematic. Physical robotics cannot be trained by crashing real robots thousands of times; autonomous vehicle models cannot be validated by driving into pedestrians; drug dosing policies cannot be optimised on real patients; and power grid control policies cannot be tested by deliberately destabilising the grid. Simulators provide unlimited, cheap, safe training data — but at the cost of a reality gap: the simulator is an imperfect model of the real world, and policies trained purely in simulation may fail when deployed in reality. Bridging the sim-to-real gap is therefore one of the central challenges of applied RL and ML for physical systems.
- Physics-based simulators (Isaac Gym, MuJoCo, Bullet, Brax) model rigid body dynamics, contact forces, and actuation with varying degrees of physical realism. IsaacGym (NVIDIA) enables GPU-parallelised rigid body simulation: running 10,000 parallel robot training environments on a single GPU, achieving millions of environment steps per second — training a dexterous manipulation policy in hours rather than weeks on traditional CPU-based simulators. Differentiable simulators (Brax, DiffTaichi, Warp) go further: they compute gradients through the physics simulation, enabling gradient-based policy optimisation rather than gradient-free RL. Differentiable simulation dramatically reduces sample complexity for simple tasks but struggles with contact-rich manipulation (rigid body contacts create discontinuities in the simulation gradient that make optimisation challenging). The connection to our RL for real-world applications guide is direct: IsaacGym is the simulation backend for virtually all state-of-the-art dexterous manipulation results (DexHand, BiDexHands) described there — the ability to parallelise training across 10,000+ environments makes RL for contact-rich manipulation tractable.
- Domain randomisation (Tobin et al., 2017; OpenAI Dactyl) is the standard sim-to-real transfer technique: train in simulation with randomised physical parameters (object mass, friction coefficients, robot link lengths, visual appearance, lighting) across a wide distribution, so the trained policy is robust to the specific parameter values encountered in reality. The key insight: if the real world’s physical parameters lie within the randomisation range, the policy trained over the randomised distribution will work in reality. ADR (Automatic Domain Randomisation, OpenAI 2019) automatically expands the randomisation range during training: start with a narrow distribution, track how often the policy succeeds, and when success rate is high enough, expand the distribution further. ADR produced the Dactyl robot hand that solved a Rubik’s cube one-handed — training entirely in simulation with ADR and deploying to a physical hand without any real robot training data.
- Digital twins are dynamic virtual replicas of physical assets (machines, buildings, cities, electrical grids) that maintain synchrony with their real-world counterparts through continuous data ingestion and model updating. Unlike static simulation models, digital twins evolve in real time: sensor data from the physical asset updates the digital twin’s state, and the twin’s predictions and simulations inform control decisions for the physical asset. The ML layer of a digital twin provides: anomaly detection (compare real sensor readings to twin-predicted readings; large discrepancies indicate real-world faults), predictive maintenance (simulate asset degradation trajectories to predict failure time), control optimisation (run thousands of simulated control scenarios in the twin to find the optimal control action before executing it in reality), and what-if analysis (simulate the effect of proposed changes before physical implementation). The connection to our manufacturing data science guide is direct: digital twin technology is the enabler of the “predictive maintenance” and “quality optimisation” use cases described there — the digital twin of a CNC machine provides the high-fidelity simulation environment for training and validating ML models before deployment to the real machine.
- World models (Ha & Schmidhuber, 2018; Dreamer, Hafner et al., 2020) are learned latent dynamics models that can substitute for real environment interactions: train a neural network to model the environment’s transition function (latent state s_{t+1} = f(s_t, a_t)) and reward function (r_t = g(s_t, a_t)), then train the RL policy entirely within the imagined rollouts of the world model. This approach reduces real environment interactions by 10-100× for visual control tasks (DMControl, Atari). Dreamer-v3 (Hafner et al., 2023) achieves human-level performance or better on 150+ tasks across diverse domains using the same hyperparameters — a remarkable generalisation result for a world model approach. The connection to our structured state space models guide is direct: Dreamer-v3’s recurrent state space model (RSSM) uses a GRU-based latent dynamics model to maintain a compact world state; the SSM perspective (the RSSM as a learned linear dynamical system extended with nonlinear prediction heads) connects world models to the structured state space models described in that guide.
- Sim-to-real transfer for language-conditioned robot manipulation (following natural language instructions in physical environments) is an active frontier, connecting simulation with LLMs. SayCan (Akhlaghi et al., 2022) uses LLM planning to decompose high-level instructions (“make me a snack”) into low-level robot primitives (“pick up the apple”), with each primitive’s feasibility estimated by a value function trained in simulation. This language-grounded planning bridges the LLM’s semantic knowledge with the robot’s physical capabilities, enabling generalisation to novel instructions without task-specific training. The connection to our AI agents guide is direct: SayCan is a physical robot instantiation of the ReAct agent framework described there — the LLM reasons about which tool (robot primitive) to use next, the value function provides affordance estimates, and the robot executes the selected primitive, with perception providing the observation of the current environment state.
The history of AI breakthroughs is inseparable from the history of simulation: DeepMind’s Atari agents trained in an Atari emulator; AlphaGo and AlphaZero trained in a Go game engine; OpenAI’s Dactyl trained in an IsaacGym physics simulator; and the latest autonomous driving models trained in photorealistic game-engine-based simulators. Simulation solves a fundamental bottleneck in ML for physical systems — the speed, cost, and safety of data collection. This guide covers the full simulation-ML stack: physics-based simulators and GPU parallelisation, domain randomisation for sim-to-real transfer, digital twins for industrial ML, world models for model-based RL, neural simulators and learned physics, and language-conditioned robot learning connecting simulation with LLMs.
Physics Simulators, Domain Randomisation and Sim-to-Real Transfer
The fidelity-speed tradeoff in simulation: high-fidelity simulators (finite element analysis for structural mechanics, computational fluid dynamics for aerodynamics) model physical phenomena with near-perfect accuracy but run orders of magnitude slower than real time. Low-fidelity simulators (rigid body physics with simplified contact models like MuJoCo) run faster than real time (10,000× for IsaacGym) but have larger reality gaps. The optimal simulator for a given ML application depends on the sensitivity of the trained policy to the aspects of physics that the simulator models imperfectly: a manipulation policy that must precisely control contact forces needs a high-fidelity contact model; a locomotion policy that mainly deals with periodic leg motion may work well with a simplified rigid body simulator. IsaacGym’s GPU parallelisation exploits the embarrassing parallelism of independent RL environments: each of the 10,000 environments maintains its own simulation state (joint positions, velocities, contact forces) in GPU memory, and all environments step simultaneously in a single CUDA kernel. This parallelism reduces wall-clock time for RL training by 100-1000× compared to CPU-based simulation, making previously infeasible RL problems (dexterous manipulation with 24-DOF robot hands) tractable. The connection to our multi-agent RL guide is direct: IsaacGym enables MARL for multi-robot coordination at scale — training 1000 pairs of collaborative robot arms simultaneously, each pair learning to coordinate on assembly tasks, is feasible with GPU parallelisation but would take years on CPU simulators. Domain randomisation at training time creates a policy that is robust to the entire randomisation distribution, not just any single set of physical parameters. The randomisation range must be calibrated: too narrow (the range excludes the real-world parameters) causes transfer failure; too wide (physically implausible parameter ranges) reduces task performance because the policy must be conservative enough to work under extreme conditions it will never encounter in reality. System identification (measuring the real robot’s physical parameters with a calibration procedure) narrows the required randomisation range, improving both transfer success and policy performance. Sim-to-real for visual policies (policies that operate on camera images) faces an additional domain gap: simulation renders do not look like real cameras (lighting, texture, reflections are approximated). RAN (Randomised + Adapted Networks) and RCAN (Randomised-to-Canonical Adaptation) use image-to-image translation (CycleGAN from our diffusion/generative models family) to either randomise simulation renders to look like diverse real images, or adapt real images to look like canonical simulation renders, reducing the visual domain gap. The connection to our synthetic data guide is direct: domain randomisation for visual sim-to-real transfer is a specialised form of synthetic data augmentation — the randomly rendered simulation images are synthetic training data, and the goal is to design the synthetic data distribution to maximise transfer to the real data distribution, the same challenge as designing task-relevant synthetic data augmentation described in that guide.
| Simulator | Physics Engine | GPU Parallel? | Differentiable? | Key Applications |
|---|---|---|---|---|
| MuJoCo | Rigid body + contact | No (CPU) | Partial (mjx) | Locomotion, manipulation benchmarks |
| IsaacGym / Isaac Lab | PhysX (GPU) | Yes (10k+ envs) | No | Dexterous manipulation, legged robots |
| Brax | Spring-based (JAX) | Yes (JAX XLA) | Yes | Locomotion; differentiable policy opt |
| CARLA | Unreal Engine | No | No | Autonomous driving perception + planning |
| Dreamer-v3 (world model) | Learned RSSM | Yes (imagined rollouts) | Yes (backprop through imagination) | Visual control; 150+ tasks |
Digital Twins, World Models and Language-Conditioned Robot Learning
Industrial digital twins combine three ML components: a physics-based simulator (high-fidelity CAD model of the asset, parameterised with measured physical properties), a data assimilation layer (Kalman filter or particle filter that updates the simulator’s state from real sensor readings), and an ML surrogate model (a fast-running neural network emulator trained on simulator outputs, enabling real-time what-if analysis that the full simulator is too slow for). The surrogate model is typically a neural ODE (from our neural ODEs guide) or a physics-informed neural network (PINN) that embeds known physical equations as constraints, enabling accurate extrapolation to operating conditions not in the training data. The connection to our data science for energy guide is direct: power grid digital twins combine the physics of electrical networks (Kirchhoff’s laws, power flow equations) with ML anomaly detection and RL-based control optimisation — the grid digital twin is the high-fidelity simulation environment for training the grid optimisation policies described in that guide. World models for model-based RL have matured dramatically with Dreamer-v3’s universal hyperparameters: the RSSM learns a compact latent representation of the environment state from pixel observations, predicts future states given actions, and enables policy training entirely within the latent space of the world model (imagination). The imagination rollout: encode the current observation to a latent state; apply the learned dynamics model iteratively to generate a trajectory of future latent states under the learned policy; compute imagined rewards along the trajectory; backpropagate through the trajectory to update the policy (analytic policy gradient through the differentiable world model). This analytic gradient is much more sample-efficient than Monte Carlo policy gradient (REINFORCE) in model-free RL. The connection to our SSM guide is direct: Dreamer’s RSSM (Recurrent State Space Model) uses a GRU for the deterministic state component and a Gaussian distribution for the stochastic state component — the same linear+nonlinear combination as S4 and Mamba SSMs described there, applied to the latent dynamics modelling problem rather than sequence modelling. Language-conditioned robot learning (SayCan, RT-2, PaLM-E) uses vision-language models to interpret natural language instructions and visual observations, grounding LLM semantic knowledge in robot affordances. RT-2 (Robotic Transformer 2, DeepMind 2023) fine-tunes a PaLM-E vision-language model on robot action prediction: the model receives a language instruction and an image of the current environment and outputs a robot action token sequence, treating robot control as a next-token prediction task. Training jointly on web-scale vision-language data and robot demonstration data produces a robot policy that generalises to novel objects and tasks not seen in robot training data, by leveraging the rich semantic knowledge in the vision-language pre-training.
✦ SUMMARIZE THIS ARTICLE WITH AI
RL for real-world applications (IsaacGym training for dexterous manipulation; sim-to-real for robot learning) is in our RL guide. Multi-agent RL (GPU-parallelised MARL for multi-robot training) is in our MARL guide. Neural ODEs and PINNs (surrogate models for digital twins) are in our Neural ODEs guide. Data science for manufacturing (industrial digital twin applications) is in our Manufacturing guide. Synthetic data (domain randomisation as synthetic data augmentation for sim-to-real) is in our Synthetic Data guide. AI agents (SayCan and RT-2 as physical robot ReAct agents) are in our AI Agents guide. Structured state space models (Dreamer’s RSSM as an SSM for latent dynamics) are in our SSM guide.


