Data Science for Climate Science – ML Weather Forecasting, Downscaling and Emissions Monitoring

Climate change is arguably the most important data science problem of our generation: the quality of climate projections, emissions monitoring, and weather forecasting directly affects policy decisions worth trillions of dollars and human welfare at a global scale. The domain is uniquely data-rich (decades of satellite observations, reanalysis datasets covering the full atmosphere at hourly resolution), physically structured (governed by well-understood PDEs), and computationally intensive (running a single global climate model simulation costs millions of CPU-hours). ML is transforming climate science at every level: replacing slow numerical PDE solvers with fast neural emulators, enabling super-resolution of coarse model output, detecting emissions anomalies from satellite observations, and predicting extreme weather events with unprecedented lead time and spatial precision. This guide covers ML weather forecasting (Pangu-Weather, GraphCast, FourCastNet), ERA5 and climate data infrastructure, statistical downscaling, emissions monitoring from satellite data, and physics-informed ML for climate applications.

ERA5, Climate Data Infrastructure and Spatiotemporal Arrays

ERA5 is the foundational dataset for ML weather and climate modelling, providing a physically consistent 80-year record of the global atmosphere at 0.25° horizontal resolution and 37 pressure levels, updated hourly. Accessing ERA5 requires the Copernicus Climate Data Store (CDS) API; processing it efficiently requires xarray and zarr, which represent multidimensional gridded data (latitude × longitude × pressure level × time) as chunked arrays that can be processed in parallel using Dask. A single year of ERA5 at 0.25° resolution is approximately 50TB; ML training sets typically use a 30–40 year subset at reduced resolution (1.4° for Pangu-Weather, 0.25° for GraphCast). The connection to our big data technologies guide is direct: ERA5 processing at scale requires the same Spark and distributed computing patterns described there, applied to NetCDF/Zarr arrays rather than Parquet tables. The connection to our spatial data science guide is direct: ERA5 is gridded raster data, and the spatial interpolation, coordinate reference system, and zonal statistics methods described there apply directly to climate raster processing.

For practical climate ML projects, the WeatherBench 2 benchmark (Rasp et al., 2024) provides standardised training/validation/test splits of ERA5, evaluation metrics (RMSE, anomaly correlation coefficient, spectral energy density), and baseline scores for all major ML and NWP models. This enables apples-to-apples comparison of new models against established baselines — the climate science equivalent of the ImageNet benchmark. Smaller entry-point datasets: NOAA’s GHCN (Global Historical Climatology Network) for station-level temperature records, ERA5-Land (0.1° resolution land surface), and Copernicus Emergency Management Service data for disaster event detection.

ML Weather Forecasting — Pangu-Weather, GraphCast and FourCastNet

The landmark ML weather forecasting models share a common design: (1) Input: ERA5 surface variables (2m temperature, 10m wind, mean sea level pressure, total precipitation) and pressure-level variables (temperature, wind, geopotential, specific humidity) at the current and previous 6-hour timestep; (2) Architecture: a large deep learning model (Transformer or GNN) that maps current atmospheric state to the state 6 hours later; (3) Autoregressive rollout: to produce a 10-day forecast, the model’s output is fed back as input, repeated 40 times. GraphCast (Lam et al., 2023) uses a multi-mesh GNN that encodes the 0.25° ERA5 grid to a coarser icosahedral mesh, performs 16 rounds of message passing in latent space (capturing long-range spatial dependencies), and decodes back to the 0.25° grid. The GNN on icosahedral mesh naturally handles the irregular geometry of the sphere (avoiding the pole distortions of rectilinear grids) and models long-range atmospheric teleconnections via multi-hop message passing. The connection to our graph neural networks guide and GAT guide is direct: GraphCast is one of the most successful real-world GNN applications, demonstrating that GNNs operating on physically motivated graph structures (spherical mesh edges encoding spatial proximity) outperform both NWP and naive grid-based deep learning models. Pangu-Weather uses a 3D Swin Transformer that treats the ERA5 pressure levels as the “depth” dimension, jointly modelling horizontal and vertical atmospheric structure — the connection to our Transformers guide and neural architectures guide is direct.

Statistical Downscaling, Emissions Monitoring and Physics-Informed ML

Statistical downscaling maps coarse GCM output (50–100 km) to fine-resolution fields (1–10 km) needed for regional impact assessment (flood risk, crop yield, urban heat island projections). Deep learning super-resolution methods — originally developed for image upsampling — transfer directly: train a CNN (SRCNN, ESRGAN) or diffusion model on paired coarse-fine ERA5 samples, then apply it to GCM output at inference time. Diffusion-based downscaling (StormCast, GenCast) achieves particularly high fidelity on precipitation extremes because the diffusion model’s stochastic sampling captures the multimodal distribution of convective precipitation patterns that deterministic super-resolution methods smooth out. The connection to our diffusion models guide is direct: climate downscaling with diffusion models uses exactly the same DDPM/score-based framework described there, with ERA5 high-resolution fields as the target distribution and interpolated coarse fields as the conditioning input.

Satellite-based emissions monitoring is an emerging ML application with direct climate policy relevance. Sentinel-5P TROPOMI measures column-averaged methane (XCH4) and CO₂ concentrations globally at 7km × 5.5km resolution. Point-source detection (identifying individual facility emissions from satellite imagery) uses CNNs trained on pairs of (satellite image, known emission source location) to detect plume signatures. Atmospheric inversion modelling — running the atmospheric transport equation backward from observed concentrations to infer surface fluxes — is the Bayesian inference problem at the heart of emissions monitoring and connects directly to our Bayesian statistics and Gaussian processes guides (the prior over flux distributions is typically a GP; the posterior combines the prior with satellite observations via Bayes’ theorem). Physics-informed neural networks (PINNs) for climate modelling embed the atmospheric governing equations (Navier-Stokes, thermodynamic energy equation) as PDE residual terms in the training loss: L = L_data + λ × L_physics, where L_physics penalises violations of the physical equations. This constrains predictions to physically plausible atmospheric states, improving extrapolation beyond the training distribution — critical for climate projections under novel future forcing scenarios not represented in the historical training data.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Big data processing for ERA5 at petabyte scale is in our Big Data Technologies guide. Spatial data science for gridded raster processing and geospatial joins is in our Spatial Data Science guide. GNNs for GraphCast-style atmospheric mesh modelling are in our Graph Neural Networks guide. Diffusion models for probabilistic downscaling are in our Diffusion Models guide. Bayesian statistics and Gaussian processes for atmospheric inversion are in our Bayesian Statistics and Gaussian Processes guides. Time series deep learning for autoregressive weather forecast rollout is in our Time Series Deep Learning guide.