Superposition: Toy Models

Everything you have done so far — logit lens, probes, steering, reading off attention heads — quietly assumed one thing: that the model's concepts live in directions. It is worth making the assumption explicit, because it is the load-bearing hypothesis of the entire field, and because the rest of this module is about the complication that follows from it.

The evidence is circumstantial but broad. Word embeddings support vector arithmetic. Linear probes work. Adding a vector to the residual stream shifts behaviour in a semantically coherent way (Module 5.1). Sparse autoencoders — which assume exactly this structure — find dictionaries that reconstruct activations well and whose elements are frequently interpretable (Module 3.4). None of this proves the hypothesis; all of it would be surprising if it were badly wrong.

One immediate consequence, and it is the fork in the road: directions need not be axis-aligned. There is no law that says feature 7 lives along neuron 7.

Elhage et al. built the smallest system that can exhibit the phenomenon, so that everything about it could be understood. Here it is in full.

Take $n$ features. Each one is sparse — zero with probability $S$, and otherwise uniform on $[0,1]$ — and each has an importance $I_i$ saying how much the loss cares about reconstructing it. Squeeze them through $m < n$ dimensions and ask for them back:

Term by term: $W \in \mathbb{R}^{m \times n}$ has one column per feature, and that column is the direction the feature is stored in. Encoding and decoding share the same matrix, so the model cannot cheat by using a clever decoder. The bias $b$ and the ReLU are the only nonlinearity, and they turn out to matter enormously.

Notice what this model deliberately is not. There is no attention, no depth, no language. It is a pure question about representation: given a bottleneck, what does gradient descent choose to store, and how?

The model faces a trade with two sides. Representing more features reduces the loss on the features you would otherwise ignore entirely. Non-orthogonal directions cause interference: when feature $j$ fires, every feature whose direction is not perpendicular to it picks up a spurious signal $(f_i \cdot f_j) x_j$, and that costs loss too.

The ReLU and the bias are the model's defence. Train it at high sparsity and you will see $b_i$ go negative and $\|W_i\| > 1$. That combination is a threshold filter: small interference lands below zero and gets clipped away entirely, while a genuine activation is large enough to survive the negative bias — and the enlarged norm compensates for it. You can watch both numbers appear in the widget below.

Here is the result that made this paper the field's gateway drug. Sweep sparsity and relative importance, train a model at every point, and the outcome is not a smooth degradation. It is a phase diagram with sharp regions.

Read it as a decision. On the left — dense features — the model behaves like PCA: it keeps whichever feature is more important and throws the other away. On the right — sparse features — it stores both in the same one-dimensional space, antipodally, at 180°. The two regions meet at a boundary that moves with relative importance: the less the second feature matters, the more sparsity you need before it is worth representing at all.

The transition is sharp. Cross the boundary and the solution reorganizes rather than deforming. That is a phase change in the physics sense, and it is the first hint that these systems have discrete structure worth naming.

Go up to five features in two dimensions and the geometry gets genuinely beautiful. As you raise sparsity you pass through a sequence of arrangements:

• Two orthogonal features, the rest discarded — the dense regime. Dimensionality per feature: 1.
• Antipodal pairs (the paper calls a pair sharing an axis a digon): four features on two axes, 0.5 dimensions each.
• Pentagon: all five features at 72°, 2/5 = 0.4 dimensions each — the maximum-spread arrangement of five directions in a plane.

The paper measures this with a per-feature dimensionality

which reads 1 for a feature with its own orthogonal dimension, 1/2 for half of an antipodal pair, 2/5 for a member of a pentagon, and 0 for a feature that was not stored. Plot it across a sparsity sweep in a bigger model and the values do not spread out smoothly — they stick at particular fractions, with visible gaps between them. Those fractions correspond to uniform polytopes: digons, triangles, tetrahedra, square antiprisms. The model is solving something close to a Thomson problem — spread $n$ points on a sphere so they repel — and Thomson problems have discrete answers.

So far superposition is a storage story. The harder claim — and the one that decides whether interpretability is difficult or near-impossible — is that models also compute in superposition.

Elhage et al. demonstrate it on the smallest possible task: a network with fewer neurons than features learns to compute the absolute value of every input feature, with neurons that each participate in several of these computations at once. Nothing about the mechanism required one neuron per operation.

This is why so much of what you would naively try does not work:

• Neuron-level interpretability degrades. Look at a neuron's top activating examples and you get a story; look at the rest of its activation range and you get three more stories. The neuron was never one thing.
• Enumeration is impossible. If a layer with $d$ dimensions can hold far more than $d$ features — and it can, because you can pack exponentially many nearly-orthogonal directions into $d$ dimensions — then “list the features” has no obvious stopping point.
• Ablation results get harder to read. Zeroing a neuron removes a slice of several features at once, so a clean behavioural change does not localize to a clean concept.

One honest caveat about the whole framework: the toy model is a deliberately impoverished system, and the argument from it to real transformers is an argument from plausibility plus a growing pile of corroborating evidence — polysemantic neurons everywhere, sparse autoencoders finding far more features than dimensions, models with an architecture that discourages superposition (Softmax Linear Units) becoming more interpretable at a cost. It is a good theory. It is not a measurement of GPT-4.

This is the real model, not a simulation of one. Five features, two hidden dimensions, hand-written gradients, Adam, 512 samples per step, about 500 steps per second in your browser. The five arrows are the columns of $W$ — the direction each feature is stored in. The heatmap is $W^\top W$, so the off-diagonal cells are the interference the model has agreed to live with. The panel at the bottom fires one feature on its own and shows you what comes back out.

Things to try: (1) Start dense (S = 0) and let it settle: two arrows at 90°, dimensionality 1.00 each, three features with norm ~0 — the model has thrown them away. Now drag sparsity to about 0.7 and watch four arrows snap into two antipodal pairs at dimensionality 0.50. Keep going to 0.95 and the fifth arrow appears as the pentagon forms at 0.40 = 2/5. (2) With the pentagon formed, look at the bias column: every $b_i$ is negative and every $\|W_i\|$ is above 1. Fire a single feature in the bottom panel and you will see why — the neighbours receive real interference and the ReLU clips it to zero. (3) Set the importance decay to 0.5 at moderate sparsity: the model now allocates deliberately, giving $f_0$ a clean dedicated direction and forcing the cheap features to share. Superposition is not uniform; it is an allocation decision.