3.5 Causal Methods: Patching & Circuit Discovery

Learn

Looking is not enough

Everything so far has been observational. Logit lens shows you what a layer predicts. A probe shows you that a direction encodes a property. An SAE shows you a feature fires on a concept. None of that establishes that the model uses any of it.

This is not pedantry. You can train a probe to 95% accuracy on a direction the model provably never reads — the information is there, sitting in the residual stream, and nothing downstream touches it. Interpretability's central epistemic discipline is the habit of asking, about every claim: is this correlational or causal?

Key idea

The only way to establish that a component matters is to change it and see what happens. Every method in this module is a variation on one move: replace an activation with something else, rerun the rest of the forward pass, and measure the damage.

The variations differ in what you replace the activation with, and the choice is not innocent.

zero ablation

Set the activation to zero. Simple, and usually wrong: zero is not a neutral value. A model whose activations all sit far from the origin is thrown off-distribution by zeroing anything, and the damage you measure is partly the damage of being somewhere the model has never been.

mean ablation

Replace with the activation's mean over some distribution. Better — you stay near the data manifold, and you delete the component's variation rather than its existence. The choice of distribution to average over is a real modelling decision: mean over all text, or mean over your task's prompts, answer different questions.

resample ablation

Replace with the activation from a different input, sampled from a distribution where the property you care about differs. This keeps everything on-distribution and asks the sharpest question: does this component carry this specific information, or just any plausible activation?

The same intervention site, three replacement values. Each answers a different question, and a component can look essential under one and irrelevant under another.

Careful

An ablation result is always relative to its baseline. “Head 9.9 is essential” is not a fact about the model; it is a fact about the model, the task, the metric, and the thing you replaced head 9.9 with. Papers that do not state their baseline are not reporting a result.

Learn

Activation patching: two directions, two questions

Activation patching is resample ablation with the resampling distribution chosen surgically. You build a clean prompt and a corrupted one that differ in exactly the property you are studying, run both, splice one activation from one run into the other, and measure the output.

Because the two prompts differ in one controlled way, the difference in outcome is attributable to that difference. This is a randomized controlled trial with a population of one, run inside a neural network.

\text{logit diff} \;=\; \mathrm{logit}(\text{correct}) - \mathrm{logit}(\text{wrong})

The metric matters as much as the intervention. Logit difference is the field's default because it is linear in the residual stream — softmax normalization cancels, so a component's contribution adds up the way your intuition wants it to. Probability and accuracy are non-linear and will hide effects at the ceiling.

Key idea

Which run you patch into determines which question you are asking. Denoising — clean activation into a corrupted run — asks whether the component is sufficient to restore the behavior. Noising — corrupted activation into a clean run — asks whether it is necessary. These are different questions, they routinely give different answers, and conflating them is the most common error in the literature.

Heimersheim & Nanda's guide is blunt about the failure mode: patching a component and seeing no damage does not prove the component is unused. Models self-repair. Knock out the heads that do a job and other heads that were sitting idle will step in — you will see this happen for real in the IOI circuit, where ablating all three name mover heads costs only about 5% of the logit difference because backup heads take over.

path patching

Plain patching measures a component's total effect, through every downstream route. Path patching restricts the intervention to one route: patch

h

's contribution only where it feeds into component

r

, and recompute everything else normally. This is how you establish wiring rather than mere importance — it is what showed that S-inhibition heads act entirely through the name movers' queries.

attribution patching

Patching is expensive: one forward pass per component per position. Attribution patching approximates the whole map with a first-order Taylor expansion —

\Delta \mathcal{L} \approx (a_{\text{clean}} - a_{\text{corrupt}}) \cdot \nabla_{a} \mathcal{L}

— which needs two forward passes and one backward pass total, regardless of how many components you score. Neel Nanda's write-up is the standard reference.

When the approximation breaks

A linear approximation is only good for small perturbations, and a clean-versus-corrupt activation difference is not small. Attribution patching is reliable for finding the many components with near-zero effect and unreliable exactly where the effect is large — including getting signs wrong at saturated components. Use it as a cheap filter, then verify the survivors with real patching. That two-stage recipe is what automated circuit discovery methods do.

Learn

The worked example: indirect object identification

Every field needs one problem that everybody has solved. In mechanistic interpretability it is IOI, from Wang, Variengien, Conmy, Shlegeris & Steinhardt (2022).

The task: complete “When John and Mary went to the store, John gave a drink to ___”. The answer is Mary — theindirect object, the name that appears once. GPT-2 small gets this right, with a mean logit difference of 3.56 over 100,000 examples and the right answer preferred 99.3% of the time. It is a real linguistic behavior, small enough to fully reverse-engineer.

Here is the algorithm a person would use:

Identify all previous names in the sentence (Mary, John, John).
Remove the names that are duplicated (John).
Output the remaining name.

The remarkable finding is that GPT-2 small implements almost exactly this, in 26 attention heads across 7 classes — about 1.1% of the model's (head, token position) pairs. Three classes map onto the three steps: duplicate-token heads detect the repeat, S-inhibition heads suppress it, name mover heads copy what is left.

Key idea

The mechanism that does the actual work is almost embarrassingly simple: the name mover heads attend to a name and copy it. All the cleverness is upstream, in arranging for them to attend to the right name. Circuits are usually like this — a trivial output step plus an elaborate addressing scheme.

Two of the seven classes do not fit the story and are the most interesting part of the paper. Negative name mover heads (10.7, 11.10) write against the correct answer, apparently hedging to limit loss when the model is wrong. Backup name mover heads do nothing at all — until you ablate the name movers, at which point they take over the job.

Self-repair breaks naive ablation

Backup heads mean that “I removed this component and performance held up” is compatible with the component being the primary mechanism. The model has redundancy you did not know about. Any claim of the form “X is not necessary” based on a single ablation is unsafe.

How was it found? Backwards, from the logits. Path patch every head to the logits: three heads dominate — the name movers. Path patch every head to the name movers' queries: four heads appear — the S-inhibition heads. Path patch to the S-inhibition heads' values: the duplicate-token and induction heads appear. Each step narrows the target and each step uses path patching rather than plain patching, because the question at each step is about a specific route.

Learn

What “the circuit explains 87% of performance” means

Wang et al. did something unusual: they tried to falsify their own result, with three explicit criteria. Learning to apply them is the most transferable thing in this module.

faithfulness

Does the circuit alone do the task? Mean-ablate everything outside the circuit and measure. For IOI:

|F(M) - F(C)| = 0.46

, which is 13% of the full model's 3.56 logit difference — so the circuit achieves 87% of the model's performance. This is the number everyone quotes.

completeness

Does the circuit contain everything used for the task? For every subset

K

, removing

K

from the circuit and from the whole model should hurt about equally. Faithfulness alone is not enough: backup name movers show that a circuit can score well while omitting components that would step in under intervention.

minimality

Does the circuit contain anything irrelevant? For every node

v

, there should exist some context in which removing

v

matters. Otherwise you have padded your explanation.

The 87% is doing less work than it looks

When Wang et al. ran completeness with a greedy adversarial search for the worst subset, they found subsets with an incompleteness score up to 3.09 — 87% of the original logit difference. Their own criterion, pushed hard, says the circuit is substantially incomplete. They report this. That is what good interpretability work looks like, and it is why “explains X% of performance” should be read as “on the specific distribution, metric and ablation baseline chosen, this subgraph reproduces X% of the measured quantity” — not as “we understand X% of the model.”

Causal scrubbing (Chan et al., Redwood Research, 2022) pushes the idea to its logical end. You state your hypothesis as a mapping from your idealized computational graph onto the model's graph. The hypothesis licenses a set of resamplings: if you claim head 5.5 only carries “is this token duplicated”, then swapping its activation for one from any other prompt with the same duplication structure must not change the output. Scrub every activation your hypothesis says is interchangeable, and see how much performance survives.

Key idea

Causal scrubbing inverts the burden of proof. Instead of collecting evidence for a circuit, you specify the hypothesis precisely enough that it makes a maximal set of interventions harmless — then run them all. What survives is a lower bound on how much your story explains. It is demanding, it usually produces a humbling number, and that is the point.

Safety tie-in

This is the module where interpretability becomes an audit rather than a story. If you want to make a safety claim — “this model has no backdoor trigger”, “this refusal is driven by harm-detection and not by surface style” — you need interventions, not activations, and you need a completeness criterion, not just a faithfulness one. Self-repair is the specific reason to worry: a mechanism you ablated and declared harmless may be the primary one, silently backed up. Assume adversarial subsets exist and go looking for them.

Explore

Feel it: patch a component, read the circuit

First, run the experiment: pick a token position and a component, and see how much of the behavior one patch moves. Then read the answer key — the circuit those patches add up to.

Activation patching sandbox — IOI in GPT-2 small

Pick a token position, pick a direction, then click any component to patch it. The number is the share of the clean logit difference that single patch moves.

Clean prompt

WhenJohnS1andS1+1MaryIOwenttothestore,JohnS2gaveadrinktoEND→Mary

Corrupted prompt

When John and Mary went to the store, Chris gave a drink to → (no duplicate name; the model has no reason to prefer Mary)

Patch at token

Direction

Attention · layer 9 · END

logit difference recovered57.6%

Name Mover Heads — 9.6, 9.9 (and 10.0 above)

The end of the road. These heads sit at the END position, attend to earlier names, and copy whatever they attend to straight into the logits — their copy score is above 95%. Patching them alone recovers most of the logit difference, which is why the circuit was discovered by starting at the logits and walking backwards.

Denoising runs the corrupted prompt and splices in one clean activation. A large number means this component is sufficient, on its own, to restore the behavior.

Effect sizes are hand-authored to reproduce the qualitative findings of Wang et al. (2022) — which components matter, at which positions, and roughly how much. They are not measured values; the notebook problem has you measure the real ones. GPT-2 small's true mean logit difference on this task is 3.56, and it prefers the indirect object 99.3% of the time.

The IOI circuit in GPT-2 small

Wang et al.'s Figure 2, made clickable. Horizontal axis is token position, vertical axis is layer. Step through the algorithm, or click a head class.

26 attention heads in 7 classes — about 1.1% of the (head, token position) pairs in GPT-2 small. Click any class to see what it does and how it was identified.

Name Mover Heads9.9 · 9.6 · 10.0

Active at: END
Attends to: previous names in the sentence — and, thanks to S-inhibition, the IO name specifically
Writes to: the logits, directly

Step 3: output the remaining name. They attend to a name and copy it. That is the entire mechanism — no cleverness beyond “attend to the right name.”

How they were identified

Copy score above 95%: feed a name through the head's OV circuit and the top output logit is that name. Patching them at END recovers most of the logit difference, which is how the circuit was found.

Head indices are layer.head from Wang et al. (2022). The circuit recovers 87% of GPT-2 small's logit difference on the task — and the same paper shows that number does not mean what you would like it to mean.

Things to try: (1) Select attention, layer 9 and switch between END and S2 — the same component is decisive at one position and inert at the other. Components matter at positions, and any analysis that averages over positions throws that away. (2) With END selected, compare layer 9 in denoise versus noise: patching the name movers in recovers far more than knocking them out destroys. That gap is self-repair by backup heads, and it is exactly why sufficiency and necessity need separate experiments. (3) Find the one component with a negative effect (attention, layer 11, at END) and note that any ranking by absolute effect size would have filed it with the name movers instead of against them. (4) In the circuit map, step to “2 · Inhibit it” and follow the dashed arrow: the S-inhibition heads write into queries, not values — they change where attention looks, not what it carries.

Practice

Problem set

The IOI replication is the field's rite of passage and the reason this module is 240 minutes. Do the pencil problems first; they will save you hours of confused debugging.

1.Design the corrupted promptpencil & paper

Clean prompt: “When John and Mary went to the store, John gave a drink to” → Mary. Here are three candidate corruptions:

When John and Mary went to the store, Chris gave a drink to
When John and Mary went to the store, Mary gave a drink to
When John and Mary went to the park, John gave a drink to

For each: what does patching with it isolate, and what confound does it introduce? Which would you use to find the name mover heads, and which to find the duplicate-token heads?

2.Sufficiency, necessity, and a contradictionpencil & paper

You patch attention layer 9 at the END position and find: denoising recovers 58% of the logit difference; noising destroys only 34% of it.

(a) Explain why these numbers are not required to match. (b) Give the specific mechanism in GPT-2 small that produces this gap. (c) A colleague concludes “layer 9 attention is only 34% necessary, so it is not the main mechanism.” What is wrong with that inference, and what experiment settles it?

3.Where the linear approximation diespencil & paper

Attribution patching estimates the effect of replacing activation $a$ with $a'$ as

\Delta \mathcal{L} \;\approx\; (a' - a)^{\top} \nabla_{a} \mathcal{L}

(a) State the assumption this makes. (b) You attribution-patch every head in GPT-2 small on IOI and then verify with real patching. Where should you expect the approximation to be worst, and in which direction will it err? (c) Given that, what is the right way to use attribution patching in a discovery pipeline?

4.Replicate the IOI circuitcode

The rite of passage. In TransformerLens with GPT-2 small, build an IOI dataset of at least 100 prompts over several templates and both name orders, then:

Establish the baseline logit difference between IO and S. You should land near 3.5.
Patch each head's output at each token position (clean → corrupted with a third name) and produce the heatmap. Heads 9.9, 9.6, 10.0 should dominate at END.
Path patch each head into the queries of those three heads. You should recover 7.3, 7.9, 8.6, 8.10.
Plot the attention patterns of the name movers before and after patching the S-inhibition heads, and confirm attention moves off IO onto S1.

Success check: your heatmap reproduces Figure 3 of Wang et al. qualitatively, and your S-inhibition set matches theirs exactly.

5.Attribution patching vs. the real thingcode

Using the same setup, implement attribution patching: cache activations on both clean and corrupted runs, take gradients of the logit difference with respect to the clean activations, and score every (head, position) with $(a_{\text{corrupt}} - a_{\text{clean}}) \cdot \nabla_a$ .

Success check: produce a scatter plot of attribution score against true patching effect over all (head, position) pairs, report the correlation, and identify by name every point where the approximation is badly wrong.

6.Write a causal-scrubbing hypothesispencil & paper

State the IOI hypothesis precisely enough to scrub. For each of these three claims, write down the resampling it licenses — the set of alternative inputs whose activation you could swap in without, on your hypothesis, changing the output:

Duplicate-token head 3.0 at S2 carries only “this token is duplicated, and the earlier copy was at position $p$ ”.
S-inhibition head 8.6 at END carries only the token identity and position of S.
Name mover head 9.9 at END copies whatever name it attends to.

Then say which claim you expect to survive scrubbing least well, and why.

0 of 6 problems marked done

Check

Check yourself

1.

You train a linear probe that reads a sentence's grammatical tense from layer 6 with 96% accuracy. What have you shown?

2.

You want to know whether head 8.6 is necessary for the IOI behavior. Which experiment answers that question?

3.

Why is logit difference the preferred metric for patching experiments rather than the probability of the correct answer?

4.

In the IOI circuit, the S-inhibition heads write into the queries of the name mover heads. What does that mean mechanically?

5.

Ablating all three name mover heads costs only about 5% of the logit difference. The right conclusion is:

6.

When should you reach for path patching instead of plain activation patching?

7.

A paper reports “our circuit explains 87% of the model's performance on the task.” What follow-up question matters most?

Answer all 7 questions to submit.

Submit your answers to complete this check.

Go deeper

Read the methods guide before the paper. It will save you from the three mistakes everyone makes on their first patching experiment.

EssentialHow to use and interpret activation patchingpaper

Stefan Heimersheim & Neel Nanda · 2024 · 1h

Short, practical, and the best possible preparation for the notebook. Read the denoising-versus-noising section twice — that distinction is the one people get wrong for years. Pay attention to the discussion of self-repair and to the advice on choosing corrupted prompts; both save real debugging time.

EssentialInterpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallpaper

Wang, Variengien, Conmy, Shlegeris & Steinhardt · 2022 · 3h, 2 sittings

Sitting 1: §2–3, following Figure 2 as you go — the discovery procedure (start at the logits, path patch backwards) is more valuable than the specific heads. Sitting 2: §4, the validation criteria. Read the completeness discussion carefully: their own adversarial search finds large incompleteness scores, and their willingness to report that is the standard to hold other papers to.

Attribution Patching: Activation Patching At Industrial Scaleblog

Neel Nanda · 2023 · 45 min

The original write-up of the gradient approximation, including honest discussion of where it fails. Read for the intuition, then note the recommended pipeline — cheap filter first, real patching to confirm — which is how every scalable circuit method since has been built.

Causal Scrubbing: a method for rigorously testing interpretability hypothesesblog

Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, et al. (Redwood Research) · 2022 · 1.5h

Long and worth it for the conceptual move: state the hypothesis so precisely that it licenses a maximal set of resamplings, then run them all. Read the first two sections and the induction-head worked example; you can skip the formalism on a first pass. Expect to find the resulting numbers humbling.

Towards Automated Circuit Discovery for Mechanistic Interpretabilitypaper

Conmy, Mavor-Parker, Lynch, Heimersheim & Garriga-Alonso · 2023 · 1h

ACDC automates the backwards search you did by hand on IOI. Read §2–3 for the algorithm and §4 for how it scores against known circuits — it recovers most of IOI, which is both encouraging and a useful calibration on how much of the work was mechanical.

TransformerLens — Exploratory Analysis Demotool

Neel Nanda & the TransformerLens contributors · ongoing · 2h (do-along)

The notebook that walks IOI patching end to end. Do not read it — run it, then delete the analysis cells and rewrite them yourself. The library's caching and hooking API is what you will use for every experiment in Parts 3 to 5, so learning it properly here pays for itself.