Tuesday, November 17, 2026
HomeData ScienceCausal Representation Learning – Identifiability, Disentanglement, IRM and OOD Generalisation

Causal Representation Learning – Identifiability, Disentanglement, IRM and OOD Generalisation

Table of Content

📋 KEY INSIGHTS

  • Causal representation learning (CRL) studies how to learn representations whose dimensions correspond to causally meaningful, independent factors of variation in the data — representations that are both disentangled (each latent dimension corresponds to one semantic factor) and causally structured (the latent factors have a known or learnable causal graph relating them). The motivation: standard representation learning learns features optimised for predictive performance on the training distribution but that may bundle together causally distinct factors. A representation entangling lighting and shape in a vision model will fail when lighting shifts (domain shift) but shape stays constant, because the model cannot isolate the causally relevant factor (shape) from the spurious one (lighting). Causal representations are robust to distribution shift by design.
  • The identifiability problem is the central theoretical challenge for CRL: when can we guarantee that the learned representation recovers the true generative factors of the data, up to some equivalence class? Independent Component Analysis (ICA) answers this for linear mixing functions and non-Gaussian, independent sources: given observed data X = As (linear mixture of independent non-Gaussian source signals s), the true sources s are identifiable up to permutation and scaling. Nonlinear ICA (Hyvarinen & Morioka, 2016; Khemakhem et al., 2020) extends identifiability to nonlinear mixing functions with auxiliary conditioning variables: when sources are conditionally independent given an auxiliary variable (task ID, time index, domain label), nonlinear ICA is identifiable under mild regularity conditions. This theoretical result justifies self-supervised learning objectives as identifiable representation learning — contrastive learning can be viewed as nonlinear ICA with the data augmentation view as the auxiliary variable.
  • Disentangled representations learn a latent space where each dimension corresponds to one independent generative factor (shape, colour, size, orientation) and can be intervened on independently. β-VAE (Higgins et al., 2017) achieves disentanglement by increasing the KL penalty in the VAE objective: L_β-VAE = E[log p(x|z)] − β·KL(q(z|x)||p(z)), where β > 1 pushes the posterior q(z|x) toward the standard Gaussian prior and forces independent, axis-aligned structure in the latent space. FactorVAE, Total Correlation VAE (TC-VAE), and DIP-VAE are variants with different disentanglement regularisers. The connection to our dimensionality reduction guide is direct: β-VAE is a nonlinear generalisation of PCA’s independence-encouraging structure, using the information bottleneck objective to find a compressed representation where dimensions are maximally independent.
  • Causal graphs in the latent space extend disentanglement from independence to causality: rather than assuming all latent factors are mutually independent, CRL learns a Directed Acyclic Graph (DAG) over the latent factors encoding their causal relationships. iVAE (Identifiable VAE, Khemakhem et al., 2020) proves identifiability under conditional independence with an auxiliary variable. ILCM (Intervention-based CRL, Lippe et al., 2022) uses interventional data (observations from different causal regimes, e.g. images taken under different controlled interventions on the generative factors) to jointly learn the causal graph and the latent representation. The connection to our causal discovery guide is direct: CRL in the latent space uses the same DAG learning algorithms (PC algorithm, NOTEARS) as causal discovery in the observed variable space, but applied to the learned latent factors rather than directly observed variables.
  • Out-of-distribution (OOD) generalisation is the primary practical motivation for CRL: a model that learns the true causal structure of its domain will generalise to new environments where only the spurious correlations change, not the causal mechanisms. InvariantRiskMinimisation (IRM, Arjovsky et al., 2019) learns a representation φ such that the optimal classifier on top of φ is the same across all training environments — the invariance condition captures the intuition that causal features have the same predictive relationship to the label across all environments, while spurious features have environment-specific relationships. DIRL and DomainBed implement multi-environment training and invariance regularisation. The connection to our causal inference guide is direct: IRM is applying the causal invariance principle — causal mechanisms are invariant across environments, spurious correlations are not — directly as a learning objective.
  • Practical CRL requires either multiple environments/domains (for environment-based invariance learning), interventional data (for graph learning), or strong inductive biases (like object-centric representations for visual scenes). Object-centric representation learning (Slot Attention, GENESIS, MONet) decomposes a scene image into separate object representations — each slot captures one object — providing a natural disentanglement of the scene into independent causal entities. The connection to our self-supervised learning guide is direct: contrastive learning methods (SimCLR, BYOL) can be theoretically grounded as nonlinear ICA under the view-augmentation framework, providing a principled connection between practical self-supervised objectives and the identifiability theory of CRL.

Standard deep learning models are remarkably capable within their training distribution but notoriously brittle under distribution shift: a model that achieves 95% accuracy on the test set from the same distribution as training may drop to 60% when tested on data from a slightly different environment, domain, or time period. The root cause is that deep learning models learn correlations — including spurious correlations that happen to be predictive in the training data but break in new environments. Causal representation learning (CRL) addresses this by seeking representations that capture the true causal structure of the data-generating process rather than just the statistical correlations in the training set. This guide covers the full CRL landscape: the identifiability problem and nonlinear ICA theory, disentangled representations (β-VAE and variants), causal graphs in the latent space, Invariant Risk Minimisation for OOD generalisation, object-centric representations, and the connection between modern self-supervised learning and identifiable representation learning.

Identifiability, Nonlinear ICA and Disentangled Representations

The identifiability question asks: when does learning a generative model of the data guarantee that the learned representation recovers the true underlying factors? Without identifiability, two completely different representations can generate identical data — the learned representation may be an arbitrary rotation or nonlinear mixing of the true factors, providing no interpretability or causal grounding. Linear ICA is identifiable under two conditions: (1) the mixing function is linear and invertible, and (2) at most one source signal is Gaussian (all others are non-Gaussian). These conditions are restrictive but sufficient to guarantee recovery of the true independent sources up to permutation and scaling. Nonlinear ICA (Khemakhem et al., 2020) proves identifiability for nonlinear mixing functions when: (1) the latent sources have an exponential family conditional distribution given an auxiliary variable u (task ID, time index, etc.): p(z_i|u) = h_i(z_i) exp[T_i(z_i)^T λ_i(u) − log Z_i(u)]; and (2) the sufficient statistics T_i(z_i) are linearly independent for different values of u. The key insight: the auxiliary variable u induces variation in the source distribution that breaks the symmetry between different representations and makes the true sources identifiable. This result directly justifies the structure of many self-supervised learning methods: in SimCLR, the two augmented views of the same image (z_1, z_2) can be seen as observations of the same latent sources z under two different augmentation-induced auxiliary conditions u_1 and u_2, and the contrastive objective is learning the conditional distributions p(z|u) that identify the true image factors. The connection to our self-supervised learning guide is direct: von Kugelgen et al. (2021) proved that self-supervised contrastive learning identifies the true content variables (the semantic factors shared across views) under the auxiliary-variable ICA framework. β-VAE achieves approximate disentanglement in practice by over-constraining the posterior — the increased β forces the encoder to use the minimum number of latent dimensions necessary to reconstruct the input, automatically discovering factored representations for datasets with few independent generative factors (dSprites: shape/scale/rotation/x/y; CelebA: face attributes). DCI (Disentanglement, Completeness, Informativeness) and MIG (Mutual Information Gap) are the standard disentanglement metrics: DCI measures how much each latent dimension is explained by a single generative factor (D), how completely each factor is captured by a single dimension (C), and how predictive the representation is of factors (I). The connection to our dimensionality reduction guide is direct: disentanglement in latent space is the deep generative analogue of independent component analysis in the linear setting — both seek axis-aligned, statistically independent feature dimensions corresponding to generative factors.

MethodApproachIdentifiabilityRequiresPrimary Metric
β-VAEIncreased KL penaltyPartial (heuristic)Single domainDCI, MIG
iVAEConditional VAE with aux variableProvable (nonlinear ICA)Auxiliary variable uMCC (mean correlation coefficient)
IRMInvariant risk minimisationCausal feature identificationMultiple environmentsOOD accuracy, worst-domain
ILCMInterventional data + DAG learningProvable (intervention-based)Interventional pairsGraph SHD, latent MCC
Slot AttentionObject-centric decompositionObjects as causal entitiesVisual scenesARI (adjusted rand index)

IRM, OOD Generalisation and Practical CRL Applications

A person in a white cleanroom suit operating machinery in a yellow-lit laboratory
Photo by Vo Huyen on Unsplash

Invariant Risk Minimisation (IRM) formalises the causal invariance principle as a learning objective: across all training environments E = {e_1, e_2, …, e_m} (e.g. different hospitals for clinical data, different countries for purchase data, different lighting conditions for vision data), learn a representation φ such that the optimal linear classifier w* on top of φ achieves the same w* across all environments. The invariance of w* across environments is the signature of causal features: causal features have the same predictive relationship to the label regardless of which environment generated the data, because the causal mechanism p(Y|causes) is invariant. Spurious features (correlated with Y in some environments but not others) violate invariance — IRM penalises representations for which w* differs across environments. Practical IRM implementation: add a gradient penalty term ||∇_{w|w=1} L_e(w·φ)||^2 to the standard ERM loss for each environment e, measuring how much the classifier would need to change w from 1.0 to minimise environment e’s loss — zero gradient means w=1 is already optimal for that environment. The connection to our causal inference guide is direct: IRM operationalises the do-calculus principle that causal mechanisms are invariant under interventions on non-descendants — training environments are viewed as interventions on the spurious correlations, and IRM recovers the invariant causal features. DomainBed (Gulrajani & Lopez-Paz, 2021) benchmarks IRM and 20+ other OOD generalisation methods on standardised vision datasets (PACS, OfficeHome, Terra Incognita, DomainNet) with controlled evaluation, finding that well-tuned ERM (standard empirical risk minimisation with strong data augmentation) often matches or beats more sophisticated invariance methods — a sobering finding suggesting that the practical gap between correlational and causal representations may be smaller than theoretically expected when training distributions are diverse enough. CRL for language: CausalVAE-based approaches for text generation learn latent causal graphs over semantic factors (sentiment, topic, style) enabling controllable generation via interventions on specific latent factors. The connection to our generative AI guide is direct: controllable text generation (changing sentiment while preserving content) requires the kind of disentangled causal representation that CRL methods provide — the ability to intervene on one semantic factor without changing others is precisely the do-calculus intervention in the latent space. Object-centric CRL for robotics: representing a visual scene as a set of object representations (positions, velocities, identities) rather than a monolithic pixel-space representation enables compositional generalisation — a robot trained to manipulate two objects can generalise to novel configurations of three or four objects, because the causal structure (object-action relationships) is preserved in the object-centric representation. The connection to our RL for real-world applications guide is direct: object-centric world models (Dreamer + slot attention) learn causally structured environment representations that enable better sample efficiency and compositional generalisation in model-based RL for robotics.

✦ SUMMARIZE THIS ARTICLE WITH AI

Causal inference (the causal framework underlying IRM and CRL) is in our Causal Inference guide. Causal discovery (DAG learning used in ILCM for latent causal graphs) is in our Causal Discovery guide. Self-supervised learning (contrastive learning as nonlinear ICA) is in our Self-Supervised Learning guide. Dimensionality reduction (β-VAE as nonlinear generalisation of ICA/PCA) is in our Dimensionality Reduction guide. Generative AI (controllable generation via causal latent interventions) is in our Generative AI guide. RL for real-world applications (object-centric world models for robotics) is in our RL for Real-World Applications guide. Federated learning (CRL for learning invariant representations across distributed environments/institutions) is in our Federated Learning guide.

Leave feedback about this

  • Rating

Durgesh Kekare
Durgesh Kekarehttps://www.dataexpertise.in
Durgesh Kekare is a data science educator and founder of DataExpertise.in. With expertise in Python, machine learning, and analytics, he helps 10,000+ learners break into data careers.

Latest Posts

List of Categories