3.1 The Interp Mindset & Observational Tools

Learn

Reverse-engineering a program nobody wrote

You now know how a transformer computes. That is not the same as knowing what any particular transformer does. Training wrote billions of numbers into the weights; those numbers implement algorithms; nobody chose those algorithms and nobody has read them. Mechanistic interpretability is the attempt to read them — to take a trained network and recover, in human terms, the program it is running.

The framing that started the modern field comes from Olah et al.'s Zoom In: a neural network is more like a compiled binary than like a statistical black box. It has structure. The structure has parts. The parts connect. And, crucially, the parts recur across models — which means there is a stable subject matter to study rather than a fresh mystery per checkpoint.

Three claims that make mech interp a research program rather than a hobby. Each is empirical, each has counterexamples, and the third is the shakiest.

feature

A property of the input that the network represents — “this text is in French”, “the current token is a closing bracket”, “the subject of the sentence is a person”. The working hypothesis is that features are directions in activation space: a feature is present when the activation vector has a large component along some direction.

circuit

A subgraph of the network — specific features connected by specific weights — that implements a human-describable computation. The induction circuit in Module 3.2 is the canonical example: two attention heads in different layers that together implement “repeat what followed this token last time”.

motif / universality

A circuit shape that recurs. Universality is the conjecture that the same motifs appear across architectures, scales, and even across artificial and biological networks. Curve detectors in vision models are the strongest evidence; induction heads are the strongest evidence in language models. It is a conjecture, not a result — and the recent literature reports plenty of model-specific idiosyncrasy.

Key idea

The bet of mech interp is that models are decomposable: that behaviour comes from parts you can name, and that naming the parts lets you predict behaviour you have not observed. If the bet is wrong, interpretability degrades to a very fancy form of correlation hunting. Most of the difficulty in the rest of Part 3 comes from superposition, which is exactly the thing that makes decomposition hard.

It is worth being clear about what interp is for, because there is a competing tool that is cheaper and often better. Behavioural evaluations ask the model questions and grade the answers. They are the workhorse of AI safety today and they scale beautifully. But they only see what the model chooses to show you. Interp is the microscope: slower, narrower, higher resolution, and — this is the point — it looks at the mechanism rather than the output.

Why the microscope earns its cost

Every failure mode that matters most is one where behaviour and mechanism come apart: a model that behaves well because it detects it is being tested, a backdoor that only fires on a trigger you did not think to try, a chain of thought that reads as honest reasoning while the answer was determined elsewhere. Evals cannot distinguish “aligned” from “aligned on the distribution you sampled”. That distinction is the whole reason this course exists, and Part 5 is where it gets cashed out.

Learn

The logit lens: reading an unfinished thought

Here is the cheapest observational tool in the field, and one of the most revealing. Recall from Module 1.3 that the residual stream is a running sum: every block adds to it, and the final unembedding $W_U$ turns the last value into logits. Nothing stops you from applying $W_U$ to the residual stream early.

\text{lens}_\ell = \mathrm{softmax}\big(W_U \cdot \mathrm{LN}_f(x_\ell)\big)

where $x_\ell$ is the residual stream after layer $\ell$ and $\mathrm{LN}_f$ is the model's final layer norm. That is the whole method. It works because the residual stream keeps one basis from start to finish — the vector after layer 4 lives in the same coordinate system as the vector the unembedding was trained on.

The logit lens taps the residual stream at every depth with the model's own output head. Nothing is retrained; you are just asking 'if the model had to answer now, what would it say?'

nostalgebraist reported the result in 2020 and it has held up: the model's prediction does not appear at the end. It crystallizes. Early layers unembed to near-copies of the current token. Middle layers commit to a syntactic category. Somewhere around two-thirds depth the actual answer arrives, often in one or two layers. The rest of the network mostly sharpens a decision already made.

Key idea

The logit lens converts a static weight dump into a timeline. Instead of “the model knows Paris” you get “the model knows Paris by layer 8, and layers 9–12 only add confidence” — a claim precise enough to test, and a pointer to exactly where in the network to go looking.

The lens is not neutral

Applying

W_U

to an intermediate state assumes that state is already written in the output basis. Often it is not: later layers may expect a systematically rotated or rescaled input, so the raw lens can look like nonsense in models where it happens not to work (GPT-Neo is the standard example). The tuned lens of Belrose et al. fixes this by learning a small affine map per layer before unembedding — better calibrated, fewer artifacts, but now you are reading a probe you trained rather than the model's own head. Both readings are correlational. Neither shows the model uses what you see.

Learn

Probes, and the discipline that keeps you honest

The other standard observational tool: cache activations at some layer, label each one with a property you care about, and fit a linear probe — usually logistic regression — from activation to label.

\hat{y} = \sigma\big(w^\top a + b\big), \qquad a \in \mathbb{R}^{d_{\text{model}}}

If the probe gets high accuracy, the property is linearly decodable from the activation. That is a real, non-trivial finding: probes have found board state in a game-playing transformer, truthfulness-correlated directions, sentiment, syntactic role, and the geometry of numbers and time. It is also much less than it sounds like.

Key idea

A probe tells you the information is there. It says nothing about whether the model uses it. Those are different claims and they need different evidence.

Three ways a probe can mislead you, in increasing order of embarrassment:

The probe learned the task. A high-capacity probe on high-dimensional activations can fit labels that the model never represented. Random-vector baselines and control tasks exist for this reason.
The information is present but unread. The residual stream carries far more than any given layer consumes. Something can be decodable and causally inert.
The direction is right but the probe's is not. Two directions can both correlate with the property while only one feeds the computation. The probe optimizes for decoding accuracy, so it happily picks the wrong one. This is the case the widget below constructs, and it is the one people actually get wrong in practice.

correlational evidence

“When the model does X, this activation pattern is present.” Produced by looking: logit lens, probes, activation statistics, max-activating examples, attention-pattern eyeballing.

causal evidence

“When I change this activation, the behaviour changes in the predicted way.” Produced by intervening: ablation, activation patching, path patching, steering. Module 3.5 is entirely about doing this rigorously.

The move that converts one into the other is always the same: intervene. Add $\lambda w$ to the activation and see whether behaviour moves. Zero out the component along $w$ and see whether behaviour breaks. If the model's output does not care, then whatever you found is a readout, not a mechanism.

Why this discipline is a safety issue

The tempting application of probes is a lie detector: train a probe on “the model is being deceptive”, then monitor it in production. If the probe is correlational, an optimizer that pushes against it — RLHF, or the model itself if it can model the monitor — can move the activation off the probe's direction while leaving the mechanism untouched. You would see the probe go quiet and conclude you had fixed something. Interventional validation is not methodological fussiness here; it is the difference between a monitor and a placebo.

Explore

Look, then intervene

Two toys, in the order the field learned them. The first is pure observation: watch a prediction assemble itself across depth. The second is why observation is not enough — a probe that reads a property beautifully and points in a direction the model does not use.

Logit lens: the prediction crystallizing, layer by layer

Prompt: The·Eiffel·Tower·is·in·the·city·of. Click any read-out point to see what the model's own unembedding matrix says about the residual stream at that depth.

Before any block runs, the lens is unembedding the raw token embedding of “ of”. GPT-2 ties its embedding and unembedding matrices, so the read-out mostly echoes the input token back. No prediction is happening yet — this is the lens looking at itself.

At embed: ·Paris has probability 0.01% and rank 4210 out of 50,257.

Hand-authored illustration. These are not live GPT-2 numbers — they are written to match the shape of the published result (echo → grammar → category → answer → sharpening). Run the real thing yourself in the problem set; the qualitative story will hold, the digits will not.

Correlation vs causation: a probe that reads a direction the model ignores

240 cached activations from sentiment prompts, projected to 2-D. A linear probe is trained on the points below. Then push every activation along a direction and watch two things that do not have to move together: what the probe reads, and what the model does.

Push every activation by λ ·0.00

along

Probe says “positive”

50% of examples

Model outputs “ positive”

50% of examples

Behaviour changed on

0%

of examples

▬ probe direction & its boundary
Fill = what the model outputs now. Ring = the true label.

Probe accuracy on the unshifted data: 97.9%. The model's own behaviour matches the true label 93.8% of the time — the probe is more accurate than the model, which is already a warning sign: it is reading something the model does not fully act on. The learned probe direction sits 87° away from the direction the model reads.

Things to try: (1) In the lens, hit Play through depth and watch which tokens are competing at each stage — the category (“a famous city”) shows up several layers before the answer, which is a hint about how factual recall is organized. (2) In the probe demo, push λ to −2.5 along the probe direction: the probe goes from 50% positive to about 1%, while the model's output barely moves. You have fooled the monitor without touching the model. (3) Reveal the causal direction and push the same distance: now the probe reading does not move at all (its boundary is perpendicular to your push) while the model flips every positive example. A perfect double dissociation — and note the flip rate tops out near 50%, because the other half of the examples were already negative.

Practice

Problem set

The first two are quick and clarify the vocabulary. The code problems are your first real TransformerLens session — install it once here and you will reuse the setup for the rest of Part 3.

1.Is that a feature?pencil & paper

For each of the following, say whether it is best described as a feature, a circuit, or neither, and why:

Neuron 373 in layer 6 fires on French text.
The model attends from a pronoun back to the name it refers to, then copies that name into the output.
The model achieves 32% accuracy on GSM8K.
A direction in layer 8 whose presence predicts that the next token is a closing bracket.

2.Reading a lens plotpencil & paper

A logit lens on a 24-layer model shows the correct answer at rank 1 from layer 14 onward, with probability rising 0.09 → 0.11 → 0.14 → … → 0.72 by layer 24. A colleague concludes: “the fact is retrieved at layer 14.” Give two distinct reasons that conclusion might be wrong, and describe one intervention that would raise your confidence.

3.Logit lens in TransformerLenscode

In a Colab notebook, install transformer_lens and load gpt2-small. Run logits, cache = model.run_with_cache("The Eiffel Tower is in the city of"). For each layer L, take cache["resid_post", L] at the final position, apply model.ln_final, then model.unembed, and record the top-5 tokens and the probability of " Paris".

Success check: you produce a 13-row table like the widget above, and the probability of " Paris" rises monotonically over the last third of the network. Then repeat for two prompts of your own — one factual, one syntactic (e.g. a bracket-closing prompt) — and compare where the answer appears.

4.Train a probe, then try to break itcode

Build a dataset of ~400 short prompts labelled by a simple binary property (suggestion: the prompt is about a person vs about a place). Cache resid_post at layer 6 of gpt2-small at the final token. Fit sklearn.linear_model.LogisticRegression and report held-out accuracy.

Then run the control: shuffle the labels and refit. Then run the causal test: take the learned direction w, normalize it, and use a TransformerLens hook to add ±3·w to the layer-6 residual stream at every position. Measure how the model's output distribution changes.

Success check: real accuracy > 85%, shuffled accuracy ≈ 50%, and you can state — with numbers — whether steering along w changed behaviour.

5.Build the counterexample yourselfcode

Reproduce the widget from scratch in NumPy, without a transformer. Define a toy “model” whose output is sign(a · v) for a fixed direction v. Generate two classes of activations so that (i) both coordinates separate the classes, and (ii) the optimal linear probe direction is orthogonal to v.

Success check: your probe reaches >95% accuracy, and shifting every activation by 2·w (the probe direction) changes the model's output on <10% of points, while shifting by 2·v changes it on ~50%.

6.A first pass through Neuronpediaexplore

Open neuronpedia.org and pick any GPT-2 small SAE feature that looks interpretable from its top activating examples. Write down, in one sentence, what you think it detects.

Now look for evidence against your sentence: scroll to the weaker activations, check the negative logit effects, and see whether the feature fires on things your description does not cover. Then write down what kind of experiment would settle it.

0 of 6 problems marked done

Check

Check yourself

1.

A linear probe on layer 10 activations predicts “the model is about to refuse” with 96% accuracy. What have you established?

2.

Why can the logit lens be applied at every layer without retraining anything?

3.

In the probe widget, pushing every activation along the probe direction collapses the probe's readout but leaves behaviour nearly unchanged. What does that demonstrate?

4.

Which of these is the best statement of the universality hypothesis?

5.

A behavioural eval shows a model never produces harmful outputs across 10,000 red-team prompts. What is the strongest thing an interpretability microscope could add?

6.

The tuned lens learns a small affine map per layer before applying the unembedding. What is the cost of that improvement?

7.

You want to argue that a direction

w

in layer 8 is the model's representation of “the subject is plural”. Which single piece of evidence is worth the most?

Answer all 7 questions to submit.

Submit your answers to complete this check.

Go deeper

Zoom In is the field's founding document and worth your full attention. The logit lens post is short and you should read it before the problem set. Keep Nanda's glossary open for the whole of Part 3.

EssentialZoom In: An Introduction to Circuitspaper

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter (Distill) · 2020 · 1h

The three claims — features, circuits, universality — stated as speculative empirical claims with evidence for each. Read the whole thing; it is short and beautifully illustrated. Pay special attention to the curve-detector case study: it is the template for what 'we understand this circuit' should mean, and the standard it sets is higher than most later work meets.

Essentialinterpreting GPT: the logit lensblog

nostalgebraist · 2020 · 30 min

The original post. Read it for the plots, not the prose — the striking result is how early the prediction stabilizes and how much of the network is spent sharpening rather than deciding. Note the author's own hedging about what the lens does and does not show; it has aged better than most of the follow-up hype.

EssentialA Comprehensive Mechanistic Interpretability Explainer & Glossaryblog

Neel Nanda · 2022 · reference

Not a read-through — a lookup table for the rest of Part 3. When a paper uses 'OV circuit', 'ablation', 'privileged basis', or 'direct logit attribution' without defining it, come here. Skim the table of contents once now so you know what is in it.

Eliciting Latent Predictions from Transformers with the Tuned Lenspaper

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt · 2023 · 45 min

Read §1–3 and the figures. The important content for you is the diagnosis of why the raw lens fails on some models, and the honest accounting of what you give up by fitting a per-layer map. Skip the causal-basis-extraction section on a first pass.