📋 KEY INSIGHTS
- Neural Ordinary Differential Equations (Neural ODEs, Chen et al., NeurIPS 2018) define a continuous-depth neural network by replacing the discrete layer-to-layer transformation of a residual network with a continuously-parameterised ODE: dh(t)/dt = f(h(t), t; θ), where f is a neural network, h(t) is the hidden state at continuous “depth” t, and the output is obtained by integrating from t₀ to t₁ using a black-box ODE solver. This reformulation has three important consequences: (1) constant memory cost during training regardless of integration steps (via the adjoint method); (2) adaptive computation (the ODE solver uses more steps for harder inputs); (3) continuous-time dynamics that naturally handle irregularly-sampled time series without discretisation.
- The adjoint sensitivity method is the key computational innovation that makes Neural ODE training tractable: rather than backpropagating through all intermediate ODE solver steps (memory-intensive), it solves a second ODE backwards in time to compute the gradient of the loss with respect to network parameters. The adjoint state a(t) = dL/dh(t) satisfies da(t)/dt = −a(t)^T ∂f/∂h, and the parameter gradient is ∫ a(t)^T ∂f/∂θ dt. This requires only O(1) memory (independent of the number of solver steps) at the cost of an additional backward ODE solve, enabling training of Neural ODEs with the memory footprint of a fixed-depth model.
- Physics-Informed Neural Networks (PINNs, Raissi et al., 2019) incorporate known physical laws (expressed as PDEs — partial differential equations) directly into the neural network training loss, enabling physics-guided learning from sparse observational data. The PINN loss combines: a data loss (MSE on labelled observations), a physics residual loss (the PDE residual evaluated at collocation points — locations where the neural network solution should satisfy the PDE), and optionally boundary/initial condition losses. The physics residual is computed by automatic differentiation: the neural network u(x,t;θ) is a differentiable function, so ∂u/∂t, ∂u/∂x, ∂²u/∂x² etc. are computed by backpropagation through the network.
- Neural ODEs for time series model the dynamics of a latent state using a continuous-time ODE, combined with an encoder (mapping observations to initial latent state) and a decoder (mapping latent state to observations). This Latent ODE architecture naturally handles irregularly-sampled time series — a critical advantage over standard RNN/LSTM models that assume fixed time steps. The ODE dynamics are shared across time, acting as a regulariser that enforces temporal smoothness; the ODE solver automatically handles variable time gaps between observations by integrating for the appropriate time interval. The connection to our time series deep learning guide is direct: Latent ODEs and Neural CDEs (Controlled Differential Equations) are the leading architectures for irregularly-sampled clinical time series, sensor data with missing timestamps, and financial tick data.
- Operator learning (DeepONet, Fourier Neural Operator) extends the PINN paradigm from learning a solution function u(x,t) for a fixed PDE to learning the solution operator that maps input functions (initial conditions, boundary conditions, PDE coefficients) to output functions (solutions). Neural Operators are trained once on a family of PDEs and then evaluate at essentially zero cost for new inputs — amortising the expensive simulation cost across many inference calls. FNO (Fourier Neural Operator, Li et al., 2021) achieves this by learning convolutions in Fourier space, enabling resolution-invariant operators that generalise to grids finer than those seen during training.
- Scientific machine learning (SciML) is the umbrella term for ML methods that incorporate scientific knowledge (physical laws, conservation principles, known symmetries) to achieve data-efficient, interpretable, and physically-consistent models. Universal Differential Equations (UDEs, Rackauckas et al.) combine known mechanistic ODE terms with neural network terms for unknown mechanisms: dX/dt = f_known(X) + NN(X;θ), enabling data-driven discovery of unknown physical terms while respecting known conservation laws. The connection to our climate science guide and healthcare guide is direct: SciML is transforming both domains by enabling physically-consistent surrogates for expensive climate simulations and hybrid mechanistic-statistical models for clinical pharmacokinetics.
Standard deep learning treats neural networks as purely data-driven function approximators, ignoring the vast body of scientific knowledge about the physical, biological, and economic systems that generate our data. Physics-informed and scientific machine learning methods address this gap by incorporating domain knowledge — differential equations, conservation laws, symmetries, and known mechanistic relationships — directly into the learning objective. The result: models that learn from dramatically less data, generalise better out-of-distribution, and produce physically interpretable outputs. This guide covers the foundational methods of scientific ML: Neural ODEs and the adjoint sensitivity method, Physics-Informed Neural Networks (PINNs) for PDE-constrained learning, Latent ODEs for irregular time series, Neural Operators (DeepONet, FNO) for operator learning, Universal Differential Equations for hybrid mechanistic-data-driven modelling, and practical tooling with JAX/Diffrax and PyTorch.
Neural ODEs — Continuous-Depth Networks and the Adjoint Method
The core insight of Neural ODEs is that a residual network h_{t+1} = h_t + f(h_t, t; θ) is an Euler discretisation of the ODE dh/dt = f(h(t), t; θ). Instead of a fixed number of discrete layers, the Neural ODE defines a continuous transformation parameterised by f (a neural network) and evaluated by integrating the ODE from t=0 to t=1 (or t=T for time series). The integration is performed by a black-box ODE solver (Dormand-Prince RK45, implicit methods for stiff ODEs) that adaptively chooses step sizes to satisfy a user-specified error tolerance — harder inputs trigger more steps, providing a form of adaptive computation depth. Training uses the adjoint sensitivity method rather than backpropagating through the solver steps: at the end of the forward pass, the solver state is discarded (saving memory), then the adjoint ODE da/dt = −a(t)^T ∂f(h(t),t;θ)/∂h is solved backwards in time alongside the original ODE (re-running the forward pass backwards), computing gradients with constant memory. The connection to our backpropagation guide is direct: the adjoint method is a continuous-time analogue of backpropagation — both compute parameter gradients via the chain rule applied to a parameterised transformation, but the adjoint method operates on a continuous dynamical system rather than a discrete computational graph. The torchdiffeq (PyTorch) and Diffrax (JAX) libraries implement Neural ODEs with the adjoint method, providing a simple API: odeint(func, z0, t) integrates the ODE defined by func from initial state z0 over time points t.
| Method | Key Idea | Handles Irregular TS? | Physics Knowledge Required? | Primary Application |
|---|---|---|---|---|
| Neural ODE | Continuous-depth residual net | Yes (Latent ODE variant) | No | Latent dynamics; generative models |
| PINN | PDE residual as training loss | Yes (continuous collocation) | Yes (PDE form required) | Scientific simulation surrogates |
| Neural CDE | ODE driven by continuous data path | Yes (by design) | No | Irregular time series classification |
| FNO | Convolution in Fourier space | Yes (resolution-invariant) | No (data-driven operator) | PDE surrogate; weather emulation |
| UDE | Known ODE + neural network unknown term | Via ODE integration | Partial (known terms) | Scientific discovery; pharmacokinetics |
Physics-Informed Neural Networks and Operator Learning
Physics-Informed Neural Networks (PINNs) represent the solution to a PDE as a neural network u(x, t; θ) and enforce the PDE constraint by adding a physics residual term to the training loss. For the heat equation ∂u/∂t = α ∂²u/∂x², the PINN loss is L = λ_data × Σ_{(x_i,t_i,u_i)∈D} (u(x_i,t_i;θ) − u_i)² + λ_phys × Σ_{(x_j,t_j)∈C} (∂u/∂t − α ∂²u/∂x²)², where D is the set of labelled observations and C is the set of unlabelled collocation points (a grid of (x, t) locations where the PDE residual is evaluated). The PDE derivatives ∂u/∂t and ∂²u/∂x² are computed by automatic differentiation through the neural network — differentiating through the forward pass to get the exact partial derivatives at any input point. This eliminates the mesh-discretisation requirement of classical numerical solvers (finite element, finite difference), enabling solutions on complex geometries and inverse problem solving (inferring PDE parameters from sparse observations). The connection to our climate science guide is direct: PINNs and Neural Operators are the primary tools for building fast climate simulation surrogates — GraphCast and Pangu-Weather are essentially large-scale neural operators trained to emulate atmospheric dynamics that satisfy physical conservation laws. Neural Operators (DeepONet, FNO) learn the mapping from input function (initial condition, forcing function) to output function (solution at all points), rather than learning a solution for a fixed input as PINNs do. FNO (Fourier Neural Operator) applies convolution in frequency space using the Fast Fourier Transform: transforming the input function to Fourier space, applying learned linear weights to the Fourier modes, and transforming back — achieving O(n log n) complexity vs O(n²) for spatial convolution, with the critical advantage that the learned Fourier weights are resolution-independent, enabling zero-shot generalisation to finer grids. The connection to our neural network architectures guide is direct: Neural ODEs and PINNs are architecturally extensions of standard MLPs and CNNs with continuous dynamics and physics constraints, building on the same fundamental architectures.
✦ SUMMARIZE THIS ARTICLE WITH AI
ChatGPTTime series deep learning (Neural CDE, irregular TS architectures) is in our Time Series Deep Learning guide. Backpropagation and automatic differentiation (the foundation of PINN PDE residual computation) are in our Calculus for ML guide. Neural network architectures (the MLP building blocks of PINNs and Neural ODEs) are in our Neural Network Architectures guide. Data science for climate science (Neural Operator-based weather forecasting) is in our Climate Science guide. Data science for healthcare (pharmacokinetic UDE models) is in our Healthcare guide. Gaussian Processes (the Bayesian alternative to PINNs for data-efficient scientific regression) are in our Gaussian Processes guide.
