📋 KEY INSIGHTS
- Mechanistic interpretability (MI) is the subfield of AI safety and interpretability that attempts to reverse-engineer the algorithms implemented by neural network weights — to understand not just what a model predicts, but which internal circuits (compositions of attention heads and MLP neurons) compute those predictions, and what concepts each neuron or feature direction represents. Unlike post-hoc attribution methods (SHAP, LIME) that explain individual predictions in terms of input features, MI seeks a complete computational account of the model: the specific sequence of linear and nonlinear operations that transform raw inputs into outputs, expressed in human-interpretable terms.
- Superposition is the central phenomenon that makes mechanistic interpretability hard: neural networks represent far more concepts (features) than they have neurons, by encoding multiple features in overlapping directions in the same activation space. A single neuron can be polysemantic — responding to multiple unrelated concepts (a neuron in GPT-2 that activates for “Barack Obama”, “basketball”, and “pizza” is polysemantic because all three concepts are stored in overlapping linear combinations involving that neuron). The toy model of superposition (Elhage et al., Anthropic 2022) proves that networks will use superposition whenever the number of sparse features exceeds the number of neurons: the network trades off reconstruction fidelity against interference between features, storing more features at the cost of partial cross-activation.
- Sparse Autoencoders (SAEs) are the dominant practical tool for mechanistic interpretability in 2024-2025. The core insight: if a network uses superposition to encode many sparse features in a dense activation space, an autoencoder trained to reconstruct those activations with a sparsity penalty on the hidden layer will learn to disentangle the superposed features into monosemantic (single-concept) directions. The SAE architecture: encode h = ReLU(W_enc(x − b_pre) + b_enc), decode x_hat = W_dec h + b_pre, train with L = ||x − x_hat||^2 + λ||h||_1. The L1 penalty forces most hidden units to zero for any given input, learning to represent each input as a sparse combination of a large dictionary of features. Anthropic’s large-scale SAE work on Claude 3 Sonnet identified millions of interpretable features including abstract concepts like “safety”, “deception”, and emotion-related features, plus the now-famous “Assistant” token feature implicated in sycophancy.
- Circuits are the mechanistic unit of analysis: a circuit is a subgraph of the computational graph (specific attention heads, MLP layers, residual stream directions) that implements a specific subtask. The induction circuit, discovered in GPT-2 Small (Olsson et al., 2022), is the clearest example: two attention heads (a “previous token” head in layer 0 and an “induction” head in layer 1) work together to implement in-context learning of repeated sequences — the previous token head copies the token before each occurrence of a repeated token to the induction head’s key, and the induction head attends back to the matching key to predict the next token. This two-head circuit explains a substantial fraction of GPT-2’s ability to continue repeated sequences and is the prototype for all subsequent circuit analysis.
- Activation patching (also called causal tracing or interchange intervention) is the core experimental method for circuit discovery: given a clean prompt and a corrupted prompt (where a specific fact is changed, e.g. “The Eiffel Tower is in Paris” vs. “The Eiffel Tower is in Berlin”), patch the activations from the clean run into specific positions of the corrupted run and measure how much the output probability of the correct answer recovers. High recovery means that component stores the factual information causally. Causal tracing in GPT models for factual recall (Meng et al., 2022, the ROME paper) found that factual associations are stored primarily in the MLP layers of the middle residual stream layers, not in attention heads — a finding that enabled ROME’s weight editing approach and the broader field of knowledge editing.
- Automated interpretability scales MI beyond what human analysts can do manually: rather than requiring a researcher to manually inspect which inputs activate each SAE feature, automated interpretability pipelines use an LLM (Claude, GPT-4) to generate natural language labels for each feature by presenting it with the top-activating examples and asking for a description. The resulting feature labels are scored for faithfulness by testing whether the label can predict which new examples will activate the feature (simulation score). This automated pipeline has been used to label millions of features in large models, creating interpretable feature atlases that can be browsed, searched, and used to monitor model internals in deployment — connecting to our ML monitoring guide where real-time feature activation monitoring is described as an emerging production interpretability tool.
The dominant paradigm in machine learning interpretability — post-hoc attribution methods that explain individual predictions in terms of input feature importance — leaves a fundamental question unanswered: what algorithm did the neural network learn to implement? SHAP values tell you that “credit score was the most important feature for this loan decision”, but not how the neural network uses credit score internally — which neurons respond to it, what other features it combines with, or whether the network has learned a coherent credit assessment algorithm or a collection of spurious patterns that happen to work on the training distribution. Mechanistic interpretability takes the reverse engineering approach: treat a trained neural network as a compiled program and attempt to decompile it into human-readable algorithms. This guide covers the full mechanistic interpretability toolkit: superposition and polysemanticity, sparse autoencoders for feature extraction, circuit analysis and the induction circuit, activation patching and causal tracing, automated interpretability at scale, and practical tooling with TransformerLens and the SAELens library.
Superposition, Polysemanticity and Sparse Autoencoders
The superposition hypothesis (Elhage et al., 2022) explains why mechanistic interpretability is hard and why the standard approach of associating one concept per neuron fails. Consider a network that needs to represent F features (e.g. 10,000 distinct semantic concepts) in an activation space of dimension N neurons (e.g. 512). If F > N, the network cannot assign each feature to a unique neuron direction. However, if features are sparse (most features are inactive for any given input), the network can store features in directions that are not axis-aligned — using the full N-dimensional space rather than just N axes. The cost: two features stored in non-orthogonal directions will partially activate each other (interference), causing errors. The network trades off storage capacity (more features) against interference (more errors), settling on an equilibrium where common features get near-orthogonal directions (high precision) and rare features share directions with others (acceptable interference). The toy model of superposition demonstrates this with a linear network trained to reconstruct sparse synthetic features: as the number of features exceeds the number of neurons, the network transitions from axis-aligned monosemantic representations (one feature per neuron) to polysemantic representations (multiple features per neuron in superposition). Sparse autoencoders break the superposition by inverting it: given activations h ∈ R^N that are dense mixtures of sparse features, an SAE learns to find the sparse feature decomposition. The encoder W_enc ∈ R^{M×N} maps from the N-dimensional activation space to an M-dimensional feature space (M >> N, typically M = 4N to 32N), and the L1 penalty forces each activation to be explained by at most a few features. The decoder W_dec then reconstructs the activation from the sparse feature vector, with the reconstruction loss ensuring that the features actually explain the activations. The connection to our sparse deep learning guide is direct: SAEs are applying the same sparse representation principle used in sparse attention and sparse MoE routing — the key insight that overparameterised sparse models can store exponentially more patterns than their parameter count suggests. The connection to our model interpretability guide is direct: SHAP and LIME explain model predictions in terms of input features; SAEs explain model computations in terms of internal features — the two approaches are complementary levels of analysis. Anthropic’s December 2023 paper trained SAEs on GPT-2 Small and found 512 interpretable features, including a “beginning of sequence” feature, a “Python code” feature, and features for specific proper nouns. The 2024 work scaled this to Claude 3 Sonnet, finding features for abstract concepts including “deception”, “power”, and “Assistant-role activation” — features that are directly relevant to AI safety monitoring and the alignment work described in our reward modelling guide.
| Tool / Method | What It Does | Level of Analysis | Primary Use |
|---|---|---|---|
| SHAP / LIME | Input feature attribution | Prediction-level | Explaining individual decisions |
| Sparse Autoencoder (SAE) | Decomposes activations into monosemantic features | Representation-level | Feature discovery and labelling |
| Activation Patching | Causal intervention on specific components | Circuit-level | Localising information storage |
| Circuit Analysis | Traces full algorithm across layers | Algorithm-level | Understanding model computations |
| Automated Interpretability | LLM-labelled features at scale | Feature atlas | Monitoring; safety evaluation |
| Logit Lens / Tuned Lens | Reads off predictions at each layer | Layer-level | Understanding information flow |
Circuit Analysis, Activation Patching and Practical MI Tooling
The induction circuit (Olsson et al., 2022) is the best-understood example of a complete mechanistic account of a neural network behaviour. The behaviour: given a sequence containing a repeated pattern [A][B]…[A], a language model should predict [B] after the second [A]. This in-context pattern completion is a form of one-shot learning that emerges from the composition of two attention heads: the previous-token head (layer 0) writes the identity of the token that immediately precedes each token into the residual stream at the current token position; the induction head (layer 1) uses a key that matches “what token came before this position” against the previous-token information in the residual stream, finding positions where the previous token matches the current token (i.e. where [A] was seen before), and then copying the values from the following positions (i.e. copying [B]) to the output. This algorithm — previous-token copying composed with key-value matching — explains GPT-2 Small’s ability to continue repeated sequences, and is identifiable precisely because the two-head composition implements a recognisable algorithm. Activation patching experiments validate circuit hypotheses by measuring causal necessity: patch the activations of each candidate circuit component from the clean run into the corrupted run, one at a time, and measure the output recovery. If a component is causally necessary for the behaviour, patching it in should substantially recover the correct output. The connection to our causal inference guide is direct: activation patching is a do-calculus intervention on the model’s computational graph — do(activation_at_layer_l_position_p = clean_value) and measure the effect on the output. TransformerLens (Neel Nanda, 2022) is the standard Python library for MI research on transformer models: it provides hooks for caching and patching any activation in a transformer’s forward pass, implements logit lens and direct logit attribution, and supports circuit analysis workflows. SAELens provides pre-trained SAEs for GPT-2 and Pythia model families with the associated feature atlases. The connection to our LLM evaluation guide is direct: activation patching and circuit analysis are increasingly used in LLM evaluation to understand not just whether a model gets the right answer but whether it uses a coherent algorithm to get there — an important distinction for safety evaluation where a model might give the right answer for the wrong internal reason.
✦ SUMMARIZE THIS ARTICLE WITH AI
SHAP and LIME (complementary prediction-level interpretability) are in our Model Interpretability guide. Sparse deep learning (the sparse representation principle underlying SAEs) is in our Sparse Deep Learning guide. Reward modelling (AI safety context for MI, feature monitoring) is in our Reward Modelling guide. LLM evaluation (circuit analysis for safety evaluation) is in our LLM Evaluation guide. Causal inference (activation patching as do-calculus intervention) is in our Causal Inference guide. Data science ethics (AI safety motivation for MI) is in our Data Science Ethics guide. ML monitoring (SAE feature activation monitoring in production) is in our ML System Monitoring guide.



