Editing Weights & Learning on the Fly

“The Eiffel Tower is located in” → “Paris”. Somewhere in a hundred billion floating-point numbers, that association is stored. Not metaphorically — you can find it, and you can change it, and the second thing is the subject of this module.

The technique that located it is causal tracing, and it is activation patching (Module 3.5) pointed at a new question. Three runs:

Run that sweep and a specific shape appears — the one you will click around in the widget below. Two hot regions, not one. An early site at the last token of the subject in early-to-middle layers, and a late site at the final token in late layers. Restoring MLP outputs recovers the early site and not the late one; restoring attention outputs does the reverse.

Why MLPs? Because an MLP layer is shaped exactly like a lookup table. Write it as $\mathrm{MLP}(x) = W_{\text{out}}\,\sigma(W_{\text{in}} x)$: the rows of $W_{\text{in}}$ are keys that fire when the input matches a pattern, and the corresponding columns of $W_{\text{out}}$ are the values that get written to the residual stream when they do. Geva et al. (2020) argued this “key-value memory” reading directly; ROME took it seriously enough to edit with it.

Take the key-value picture literally. If the down-projection $W = W_{\text{out}}$ is a linear associative memory mapping keys to values, then inserting a new memory is a constrained least-squares problem: change $W$ as little as possible, subject to mapping one new key to one new value.

The solution has a closed form, and it is rank one:

Term by term. $k_*$ is the MLP's internal activation at the last subject token — the key for “Eiffel Tower”. $v_*$ is a value vector found by gradient descent, chosen so that writing it there makes the model say “Rome”. $C$ is the uncentered covariance $\mathbb{E}[kk^{\top}]$ of keys over a big text sample — it is what makes the update respect which directions in key space are actually used, so you disturb common memories least. The whole change is an outer product of two vectors: one new memory, written in place, in about a second.

It works. Editing papers report near-100% efficacy (the edited prompt gives the new answer) and strong generalization (paraphrases give it too), with specificity (unrelated facts untouched) and fluency as the guardrail metrics. MEMIT scales the same algebra to thousands of simultaneous edits by spreading the update over a range of critical layers instead of one.

And then you ask the model what country the Eiffel Tower is in, and it says France.

Cohen et al. (2023) made this measurable with a benchmark of ripple effects — logical implications, compositions, two-hop questions, subject aliasing — and found editing methods that score in the nineties on standard efficacy score far lower once the neighbourhood is probed. The simulator below lets you feel the shape of it: generalization and specificity trade off along one dial, and ripple consistency is not on that dial at all.

You came to this course partly wanting models that learn while you talk to them. Here is the honest state of it, arranged from cheapest to most invasive.

In-context learning is the one that already works. The residual stream is a scratchpad, induction heads (Module 3.2) are a copy-from-context mechanism, and the effect of a few-shot prompt is often well described as an implicit weight update — the “fast weights” framing. Nothing is stored, so nothing is corrupted; the trade is that it lasts exactly as long as the context window and costs tokens every single call.

Test-time training is the interesting middle. Hardt and Sun (2023) build a nearest-neighbour index over a large corpus, retrieve the neighbours of the current input, take a few gradient steps on them, answer, and throw the update away. Perplexity gains are large, and largest on exactly the data a model is worst at: rare, tail, domain-specific text. The cost is a training step inside your request path — which is a lot to ask of a serving stack, and the main reason you do not see it in products.

Fine-tuning and LoRA are the durable option, and the one that pays the classic tax:

Editing is the most surgical and the most brittle, for all the reasons above. And the brittleness compounds: sequential edits applied one after another degrade the model, and papers on large-scale sequential editing report accumulating damage to fluency and to unrelated knowledge well before you reach the number of edits a real deployment would need.

The open research direction is the one you would guess: architectures with a separate, addressable, revisable memory, so that learning a new fact is a write to a store rather than a perturbation of the function that computes everything. Nobody has made that work at frontier scale. It remains one of the more attractive open problems in the field, and it is unusually well suited to someone who thinks like an engineer about state management.

First locate the fact, then move it and watch the neighbourhood. These two widgets are the module in miniature: the first is why people believed editing would work, the second is why it mostly doesn't.

Things to try: (1) In the heatmap, switch between MLP-only and attention-only and watch the hot region jump from the subject to the last token — that single contrast is ROME's whole argument for editing MLPs rather than attention. (2) Click a cell at the last token in layer 5 and one at “Tower” in layer 5: same layer, wildly different effect. Position matters more than depth here. (3) In the simulator, set λ low, run all probes, and note which column is red; then set λ to 95 and run again. You should be able to state the tradeoff in one sentence. (4) Find the probe that is secretly useless and work out what makes it useless before reading its note — then go check whether your own evals at work contain one like it.