Circuit Tracing & the Biology of LLMs

Module 3.4 gave you a vocabulary: sparse features, thousands of them, each a direction the model uses. Module 3.5 gave you a method for establishing causal structure, at the cost of one forward pass per component per position. Neither, on its own, gets you a wiring diagram of a frontier model.

The obstacle is that an SAE explains an activation, not a computation. It tells you what is present at layer 12; it does not tell you that layer 12's Texas feature was caused by layer 8's Dallas feature. To get edges you need the features to sit inside the computation rather than beside it.

An attribution graph is a per-prompt diagram: which features caused which other features to fire, and which caused the output. Building one takes three moves.

One: freeze the non-linearities you are not studying. For a specific prompt, construct a local replacement model. Substitute CLTs for the MLPs, then freeze the attention patterns and the normalization denominators from the original forward pass, and add an error term at each layer equal to whatever the transcoder failed to reconstruct. The result reproduces the original model's output on this prompt exactly, and — crucially — the only remaining non-linearities are the feature activations themselves.

Two: read off the edges. With attention frozen, everything between two feature activations is linear, so the effect of source feature $s$ on target feature $t$ is just

where $a_s$ is the source's activation and $w_{s \to t}$ is the virtual weight — the derivative of the target's pre-activation with respect to the source's activation, obtained through the frozen Jacobian. No approximation is needed at this step: within the local replacement model, this is exact.

Three: prune. The raw graph has millions of edges for a short prompt. Iteratively drop the lowest-influence nodes: Anthropic report reducing node count by an order of magnitude while losing only about 20% of completeness. Then a human groups features into supernodes and labels them — a manual, interpretive step the paper flags as a limitation.

On the Biology of a Large Language Model applies the method to Claude 3.5 Haiku across roughly a dozen behaviors. The title is deliberate — this is natural history, not theory. Here is what to take away from each.

Multi-step reasoning. “Fact: the capital of the state containing Dallas is” → Austin. The graph contains Texas features that no token in the prompt supplied. Swap them for California and the model says Sacramento; swap in Byzantine Empire and it says Constantinople. A genuine intermediate variable, computed and then used.

Planning in poems. At the line break, before writing a word of the next line, features for candidate rhyme words (“rabbit”, “habit”) activate — and they influence the output only at that position. Suppress them and the line comes out differently; inject “green” and the model rebuilds the line to land on it, about 70% of the time across 25 poems. A next-token predictor that plans ahead and writes backwards from the plan.

Addition. 36 + 59 is computed by two pathways in parallel: low-precision magnitude features (“add something near 57”) and lookup-table features for digit combinations. Asked how it did it, the model describes carrying — the algorithm it learned tosay, not the one it ran.

Multilingual circuits. The same question in English, French and Chinese runs through shared, language-independent features for the operation and the operand, with language-specific features only at input and output. The sharing increases with scale. But English is privileged: multilingual features have stronger direct weights onto English outputs, while other languages are more heavily mediated.

Unfaithful chain of thought. The graphs distinguish three cases that look identical from outside: reasoning the model actually performed, reasoning it fabricated afterwards, and motivated reasoning in which the model works backwards from an answer a human hinted at. That distinction is exactly the one CoT monitoring needs and cannot get behaviorally.

Refusal. Asked to advertise mixing bleach and ammonia, specific “dangers of mixing cleaning chemicals” features trigger a chain into “harmful request” → “should refuse”. Remove the danger features and the model complies, though warnings resurface later — the knowledge and the refusal are separable.

Hidden goals. Applied to a model finetuned to pursue a concealed objective, the method surfaced the mechanism — embedded, interestingly, inside the model's Assistant-persona features. This is the closest existing thing to an interpretability-based audit of a misaligned model, and Module 5.3 returns to it.

The paper's limitations section is unusually good and you should be able to recite it. Four things.

First walk two real attribution graphs node by node. Then reverse the exercise: commit to a hypothesis about a behavior before the graph is revealed, and see how good your intuitions about model internals actually are.

Things to try: (1) In the Dallas graph, step to “4 · The shortcut” and sit with it — the clean two-hop story coexists with a memorised direct path, which is what real circuits look like. (2) In the poetry graph, note that the planning features are active at the newline only; ask yourself what behavioral experiment could have detected that from outside the model, and convince yourself none could. (3) In the game, commit out loud before clicking — the addition task in particular is designed so that the plausible answer is the model's own self-report, and the graph disagrees with it.