0.1 Linear Algebra as Geometry

Learn

Directions, not lists of numbers

Every sentence you will read in an interpretability paper is about directions in activation space. “The refusal direction.” “The feature points this way.” “This head writes into the residual stream.” None of that means anything until a vector stops being a list of numbers and becomes an arrow.

So: a vector is an arrow from the origin. Its length is how strongly something is present; its direction is what that something is. GPT-2 small carries a 768-dimensional arrow at every token position — impossible to picture, but every fact below is proved in 2-D and stays true in 768-D.

Two vectors and the angle between them. The dot product packs both lengths and that angle into a single number — which is why it is the only similarity measure a transformer ever needs.

The dot product of two vectors multiplies matching coordinates and adds them up. Geometrically it is something better:

a \cdot b = \sum_i a_i b_i = \lVert a \rVert \, \lVert b \rVert \cos\theta

Read the right-hand side term by term. $\lVert a \rVert$ and $\lVert b \rVert$ are the two lengths; $\cos\theta$ is pure direction agreement — 1 for parallel, 0 for perpendicular, −1 for opposite. Big dot product means long and aligned. Zero means perpendicular: as far as this measurement is concerned, the two vectors are about unrelated things.

unit vector

A vector scaled to length 1, written

\hat{u} = u/\lVert u \rVert

. Unit vectors are pure direction with the magnitude divided out, which is why feature directions are almost always reported normalised.

cosine similarity

\cos\theta = \frac{a \cdot b}{\lVert a \rVert \lVert b \rVert}

— the dot product with both lengths removed. It answers “same direction?” without caring “how much?”. Every time this course compares two features, it is this number.

Key idea

The dot product is the transformer's universal similarity operation. An attention score is a dot product of a query and a key. A logit is a dot product of the residual stream with an unembedding row. A linear probe is a dot product with a learned direction. Learn to read

a \cdot b

as “how much of

b

is in

a

” and half of mechanistic interpretability becomes readable.

Learn

Projection: the shadow and the leftover

Ask a sharper question than “are these similar?”: how much of this vector lies along that direction, and what is left over? That is a projection, and it is the single most-used operation in interpretability practice.

Shine a light straight down onto a line through the origin. The shadow the vector $v$ casts on that line is

\mathrm{proj}_{\hat{u}}(v) = (v \cdot \hat{u})\,\hat{u}

Two pieces: $v \cdot \hat{u}$ is a number — the signed length of the shadow — and multiplying it by $\hat{u}$ turns that number back into a vector pointing along the line. Whatever is left,

r = v - (v \cdot \hat{u})\,\hat{u}

is perpendicular to $\hat{u}$ — always, for any $v$ . You can check it in one line: $r \cdot \hat{u} = v \cdot \hat{u} - (v \cdot \hat{u})(\hat{u} \cdot \hat{u}) = 0$ , because $\hat{u} \cdot \hat{u} = 1$ . So every vector splits cleanly into the part along a direction and the part that knows nothing about it.

Key idea

Splitting a vector into “along

\hat{u}

” plus “perpendicular to

\hat{u}

” is what makes it sane to talk about one feature at a time inside a 768-dimensional activation. The residual stream carries hundreds of things at once; a projection is the instrument that reads out just one of them.

The same formula, written as a matrix, is $P = \hat{u}\hat{u}^{\mathsf{T}}$ . Applying $P$ keeps the component along $\hat{u}$ and deletes everything else; applying $I - P$ does the reverse — it erases that direction and keeps the rest. That second operation is exactly how directional ablation and concept erasure work in Part 5: you do not delete a neuron, you project the activation onto the subspace orthogonal to a direction and let the model continue.

Why the word “subspace” keeps appearing

A line through the origin is a 1-D subspace; a plane through the origin is 2-D. The formula generalises: with an orthonormal basis

\hat{u}_1, \dots, \hat{u}_k

of a subspace, the projection is the sum of the individual shadows,

\sum_j (v \cdot \hat{u}_j)\hat{u}_j

. When a paper says a behaviour lives in a “low-dimensional subspace,” this is the object it means.

Learn

Matrices are maps, and rank is how much they flatten

A matrix is not a grid of numbers. A matrix is a function that moves the whole space at once, in the most restrained way possible: grid lines stay straight and evenly spaced, and the origin stays put. That restraint is the entire content of the word linear.

Here is the trick that makes matrices readable at a glance: the columns of $M$ are where the basis vectors land. Column 1 is $M\hat{e}_1$ , column 2 is $M\hat{e}_2$ . Everything else follows, because every input is a combination of basis vectors and linear maps respect combinations:

M(x_1 \hat{e}_1 + x_2 \hat{e}_2) = x_1 (M\hat{e}_1) + x_2 (M\hat{e}_2)

rank

The dimension of the output space the map can actually reach. A 2×2 matrix of rank 2 spreads the plane over the plane; rank 1 squashes the entire plane onto a single line; rank 0 sends everything to the origin. Rank is information capacity: whatever the map flattens away is gone and cannot be recovered.

A rank-1 matrix is an outer product: $M = u v^{\mathsf{T}}$ , so that $Mx = u\,(v \cdot x)$ . Read that right-to-left and it is a two-stage machine — measure the input against $v$ (a dot product, one number), then write that number out along $u$ . Read-then-write is the shape of nearly every component you will meet later: attention heads, MLP neurons, SAE features, LoRA updates.

Key idea

Low rank is everywhere in transformers, and it is deliberate. In GPT-2 small each attention head uses

d_{\text{head}} = 64

inside a

d_{\text{model}} = 768

stream, so the head's effective 768×768 matrices have rank at most 64. A head cannot touch the other 704 directions. That constraint is why heads can be studied one at a time.

The singular value decomposition says that every matrix, no matter how ugly, is three simple steps:

M = U \Sigma V^{\mathsf{T}} \quad\text{(rotate, stretch along axes, rotate again)}

$V^{\mathsf{T}}$ rotates so that the interesting directions line up with the axes; $\Sigma$ is diagonal and just stretches axis by axis, by amounts $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ called singular values; $U$ rotates the result into its final pose. In pictures: the unit circle always becomes an ellipse, and the singular values are the ellipse's semi-axes. The number of non-zero singular values is the rank, and a small-but-nonzero $\sigma$ means “this direction survives, but faintly.”

Where you will meet this again

Module 2.2: LoRA fine-tunes with

\Delta W = BA

, a deliberately rank-

r

update — a few directions of change instead of millions. Module 3.2: the QK circuit is the low-rank matrix

W_Q^{\mathsf{T}} W_K

, one bilinear form scoring every token pair. Same geometry both times.

Learn

High-dimensional space is much roomier than it looks

Now the fact that makes modern interpretability necessary. In 2-D you can fit exactly 2 mutually perpendicular directions. In 768-D you can fit exactly 768. That sounds like a hard ceiling on how many separate things a model can represent — and it would be, if features had to be exactly perpendicular.

They do not. Drop the requirement to almost perpendicular and the ceiling explodes. Take two random unit vectors in $d$ dimensions; their cosine similarity has mean 0 and standard deviation

\mathrm{sd}(\cos\theta) = \frac{1}{\sqrt{d}}

At $d = 2$ that is 0.71 — random directions are all over the place. At $d = 768$ it is 0.036. At $d = 12{,}288$ (GPT-3 scale) it is 0.009. Two directions drawn at random in a big space are already nearly orthogonal, without anyone arranging it. And the tail is thin: the probability that a random pair exceeds $\varepsilon$ falls off like $e^{-d\varepsilon^2/2}$ , exponentially in the dimension.

Run the arithmetic the other way. If you tolerate interference up to $\varepsilon$ , the number of directions you can pack grows exponentially in $d$ rather than linearly. This is the geometric content of the Johnson–Lindenstrauss lemma: any $N$ points can be squeezed into about $O(\log N / \varepsilon^2)$ dimensions with all distances preserved to within $\varepsilon$ . The cost of an extra feature is not a whole dimension; it is a logarithm.

Same 8 directions, two dimensions. On the left (d = 2) they crowd each other: neighbours have cosine similarity 0.71. On the right, schematically, the same count in a higher-dimensional space — the pairwise angles all approach 90° and the vectors stop interfering. The explore widget below measures this for real.

Key idea

A

d

-dimensional space holds d exactly-orthogonal directions but exponentially many almost-orthogonal ones. So a model that has more things to represent than it has dimensions is not stuck: it can store them as nearly-orthogonal directions and pay a small interference tax. That is superposition, and Module 3.3 is where you will watch a model actually choose to do it.

Safety tie-in

This geometry is why interpretability is hard rather than merely tedious. If every feature owned a neuron, auditing a model would be reading a very long list. Because features are directions that do not line up with neurons — and there are more of them than there are neurons — you cannot audit a model by inspecting its coordinates. You have to find the directions first, which is what probes, sparse autoencoders, and circuit tracing all exist to do. Every safety technique later in this course is downstream of the fact you just measured.

A caveat worth keeping

“Features are directions” is the linear representation hypothesis. It is extremely productive and heavily evidenced, but it is a hypothesis, not a theorem — there is active work on circular, multi-dimensional, and non-linear feature structure. Hold it as a strong default that could be refined.

Explore

Play: shadows, maps, and crowded space

Three toys, in the order the lesson built them. The first makes the dot product physical. The second turns four numbers into a transformation you can watch flatten space. The third is the one that matters for the rest of the course: it measures how orthogonal random directions become as you add dimensions.

Projection playground

Drag anywhere in the panel to move the blue vector; the slider rotates the 1-D subspace (the dashed line). Orange is the shadow v falls on that line; aqua is what is left over.

subspace angle22°

v ₓ2.40

v ᵧ1.50

v: (2.40, 1.50), |v| = 2.83
û: (0.93, 0.37)
proj: (2.58, 1.04)
resid: (-0.18, 0.46)

v · û = 2.79 — the signed length of the shadow. Angle between them 10°, cosine similarity 0.98. Leftover length 0.49, and the leftover's own dot product with û is -0.00 — always zero, whatever you do.

A matrix is a map of the plane

The left panel is input space, the right is output space. Watch where the unit circle goes: it always becomes an ellipse, and the ellipse's two axis lengths are the singular values σ₁ ≥ σ₂.

a1.00

b1.20

c0.00

d1.00

M = [ 1.00 1.20 ; 0.00 1.00 ]

area preserved (det = 1) but directions are stretched unevenly

Full rank: the circle keeps its interior, so nothing is lost — the map is invertible.

input space

output space (after M)

det = 1.00 (area scale factor), σ₁ = 1.77, σ₂ = 0.57, rank = 2. Condition number σ₁/σ₂ = 3.1 — how much the map stretches its best direction relative to its worst.

How crowded can a space get?

Sample N random unit vectors in d dimensions and histogram every pairwise cosine similarity. Push d up and watch the whole distribution collapse onto zero — random directions become almost orthogonal for free.

dimension d8

vectors N120

tolerance ε0.10

Greedy packing: draw 400 random directions and keep each one only if it stays within ε of orthogonal to everything kept so far. Press Pack directions for d = 8, ε = 0.10.

7,140 pairs. Measured spread (RMS cosine) 0.355 vs the theoretical 1/√d = 0.354. Worst pair in this sample: |cos| = 0.972. Fraction of pairs inside ±ε: 22.7%.

Things to try: (1) In the projection playground, drag $v$ until the shadow disappears — note that the dot product hits 0 exactly when the arrow is perpendicular, and that $v$ can be enormous and still score zero. (2) In the matrix map, press rank 1 and then nudge any single slider by 0.05: watch the ellipse pop open from a segment, i.e. rank 1 is a measure-zero accident that real weight matrices never sit exactly on. (3) In the crowded-space lab, set $d = 2$ and look at the histogram, then walk $d$ up to 1024 and watch the whole distribution collapse into a spike at zero; check the measured spread against $1/\sqrt{d}$ in the footer at every step. (4) Set $\varepsilon = 0.2$ and press Pack directions at $d = 8$ , then $d = 64$ — the number of directions you can keep grows far faster than $d$ does.

Practice

Problem set

Do the pencil problems by hand, on paper, with actual arithmetic — the point is to make these operations physical before they show up buried inside an attention head. Budget 20 minutes for the code problem.

1.Shadow and leftoverpencil & paper

Let $v = (3, 4)$ and $u = (2, 1)$ .

Normalise $u$ to a unit vector $\hat{u}$ .
Compute $v \cdot \hat{u}$ , the projection $p = (v \cdot \hat{u})\hat{u}$ , and the residual $r = v - p$ .
Verify $r \cdot \hat{u} = 0$ and that $\lVert p \rVert^2 + \lVert r \rVert^2 = \lVert v \rVert^2$ .
What is the cosine similarity between $v$ and $u$ ? Does it change if you double $v$ ? Does the dot product?

2.An attention score, at 2×2 scalepencil & paper

Two token vectors (as columns): $x_1 = (1, 2)$ , $x_2 = (3, -1)$ . Weights $W_Q = \begin{bmatrix} 1 & 0 \\ 1 & 1\end{bmatrix}$ , $W_K = \begin{bmatrix} 0 & 1 \\ 1 & 0\end{bmatrix}$ .

Compute the queries $q_i = W_Q x_i$ and keys $k_j = W_K x_j$ , then the 2×2 score matrix $S_{ij} = q_i \cdot k_j / \sqrt{2}$ .
Apply a causal mask (position 1 may not look at position 2) and softmax each row. What does token 1 attend to? Token 2?
Now compute $A = W_Q^{\mathsf{T}} W_K$ and verify $x_1^{\mathsf{T}} A x_2$ equals the unscaled score you already found. What does that tell you about how many matrices an attention head really has?

3.Rank 1 means outer product (and LoRA is the payoff)pencil & paper

Let $M = \begin{bmatrix} 2 & 4 \\ 1 & 2\end{bmatrix}$ .

Show $\det M = 0$ and write $M$ as an outer product $u v^{\mathsf{T}}$ . Which line is the image? Which line is sent to zero?
Prove in one sentence that $\Delta W = BA$ with $B$ of shape $d \times r$ and $A$ of shape $r \times d$ has rank at most $r$ .
For $d = 4096$ and $r = 8$ , how many parameters does $\Delta W = BA$ have compared with a full $d \times d$ update?

4.Why the spread is 1/√d (guided derivation)pencil & paper

Let $u$ and $v$ be independent, uniformly random unit vectors in $\mathbb{R}^d$ . Show that $\mathbb{E}[u \cdot v] = 0$ and $\mathbb{E}[(u \cdot v)^2] = 1/d$ , so the standard deviation of the cosine similarity is $1/\sqrt{d}$ . Then state what this predicts for $d = 768$ and check it in the explore widget.

5.Measure near-orthogonality yourselfcode

In a notebook, for each $d \in \{2, 8, 64, 512, 4096\}$ : sample 2,000 random unit vectors (Gaussian entries, then divide by the norm — do not sample coordinates uniformly, that biases toward corners), compute the full pairwise cosine-similarity matrix, and report the standard deviation, the maximum absolute off-diagonal value, and a histogram.

Then answer with code: for $\varepsilon = 0.1$ , how many of the 2,000 vectors can you greedily keep such that all pairs satisfy $|\cos| < \varepsilon$ , at $d = 8$ versus $d = 64$ versus $d = 512$ ?

Success check: the measured standard deviation matches $1/\sqrt{d}$ to within a few percent at every $d$ , and the greedy count at $d = 512$ exhausts your 2,000 candidates while $d = 8$ saturates in the low tens.

6.Hunt for rank collapseexplore

Using the matrix-map widget above:

Find three different matrices with $\det M = 0$ but $M \neq 0$ . For each, read off the image line and describe what the map destroys.
Start from the squash preset ( $\sigma_2 = 0.15$ ) and drag the bottom-right entry toward zero until $\sigma_2$ nearly vanishes. Watch the condition number. Why do numerical people care about that ratio?
Now open the figures of Toy Models of Superposition and look at the plots of $W^{\mathsf{T}}W$ . Connect what you see there to what you just did: what would perfect orthogonality look like in those plots, and what does the model do instead?

0 of 6 problems marked done

Check

Check yourself

1.

Two activation vectors have dot product 0. The most useful reading of this is:

2.

You scale

v

by 3 and leave

\hat{u}

alone. What happens to

v \cdot \hat{u}

and to the cosine similarity?

3.

A 768×768 matrix has rank 64. Which statement is true?

4.

You want to remove a feature direction

\hat{u}

from an activation

v

without disturbing anything else. The right operation is:

5.

Random unit vectors are drawn in

d = 4096

dimensions. A typical pair has cosine similarity around:

6.

Why does near-orthogonality make superposition possible?

7.

The unit circle is mapped by

M

to an ellipse with semi-axes 3 and 0.02. What should you conclude?

Answer all 7 questions to submit.

Submit your answers to complete this check.

Go deeper

Watch the 3Blue1Brown series first — it is the visual backbone of everything above, and nothing else in this list replaces it. The rest are references you will return to rather than read straight through.

EssentialEssence of Linear Algebra (chapters 1–4, 9, 13–14)video

Grant Sanderson (3Blue1Brown) · 2016 · 2h

The core: ch. 1–4 for vectors, span, linear transformations and matrix multiplication as composition; ch. 9 for the dot product (watch the duality argument twice — it is why every measurement in a transformer is a dot product); ch. 13–14 for change of basis and eigenvectors. Skip determinants-by-cofactor and Cramer's rule; you will never need them here.

EssentialToy Models of Superposition — Introduction and “Superposition” sectionpaper

Elhage, Hume, Olsson, Schiefer et al. (Anthropic) · 2022 · 45 min (first pass)

Read only the introduction and the first section on superposition now — enough to see today's near-orthogonality fact used as the load-bearing argument for why features do not align with neurons. Module 3.3 does the full three-sitting read; resist the urge to go further today.

Deep Learning, Chapter 2: Linear Algebrabook

Goodfellow, Bengio & Courville · 2016 · reference

The notation dictionary for the rest of the course. Read §2.8 (SVD) and §2.7 (eigendecomposition) properly; treat the rest as a lookup table when a paper uses a symbol you cannot place.

Johnson–Lindenstrauss lemmablog

Wikipedia (after Johnson & Lindenstrauss 1984) · 1984 · 20 min

The formal statement behind the punchline: N points fit in O(log N / ε²) dimensions with distances almost preserved. Read the statement and the proof sketch, skip the constructions and the optimality literature. The takeaway is the shape of the bound — logarithmic in the number of things you want to store.

Privileged Bases in the Transformer Residual Streampaper

Elhage, Lasenby & Olah (Anthropic) · 2023 · 30 min

The honest complication to “only directions matter, never coordinates.” Anthropic finds that outlier coordinates do appear in real residual streams, and traces the cause provisionally to Adam's per-dimension normalisation. Read it for the epistemics as much as the result: a clean theoretical claim meeting messy empirical evidence.

MIT 18.06 Linear Algebra (lectures 1–6, 29–30)course

Gilbert Strang (MIT OpenCourseWare) · 2010 · optional, ~6h

Only if you want the full undergraduate treatment. Lectures 1–6 cover elimination and the four fundamental subspaces; 29–30 are the SVD, and Strang's derivation is the clearest on record. Everything else is optional for this course.