Training Dynamics & Scaling

You now know what a transformer block does. This module is about what happens when you run one for three months. The striking thing is how little the recipe changes: the objective that trains a 100M-parameter toy is the same objective that trains a frontier model. Only the budget moves.

That objective is next-token prediction. Take a document, feed it in, and at every position ask the model for a distribution over the next token. Because of the causal mask, position 5's prediction can't see token 6, so one forward pass over a 4,096-token document yields 4,096 independent training signals at once. Average their cross-entropy losses and you get the number everyone watches:

Term by term: $x_t$ is the token that actually came next, $p_\theta(\cdot \mid x_{<t})$ is the model's distribution given everything before it, and the log turns “probability I assigned to the truth” into an additive score. Units are nats per token. This is exactly the cross-entropy from Module 0.2, averaged over positions.

Two warnings about comparing loss numbers across models. First, loss depends on the tokenizer: a model with a bigger vocabulary packs more text into each token, so its per-token loss is higher even at equal quality. (Bits-per-byte fixes this and is what careful papers report.) Second, loss depends on the evaluation corpus. “Loss 1.9” means nothing without both.

Here is the result that turned deep learning into an industry. Plot loss against model size, or against data, or against compute, on log-log axes — and you get a straight line. Over seven orders of magnitude. Kaplan et al. found this in 2020, and it means you can train a set of small models, fit two numbers, and predict the loss of a model a thousand times bigger before you build it.

The modern form is Hoffmann et al.'s three-term fit, and it is worth reading slowly:

$N$ is parameters, $D$ is training tokens. $E$ is the entropy of natural language itself — the surprise no model can remove, because text is genuinely partly unpredictable. The other two terms are penalties: one for being too small to represent the structure, one for not having seen enough text to find it. Both decay as power laws, which is why they look linear on log-log axes and why progress feels smooth and expensive at the same time. Hoffmann's fitted values: $E = 1.69$, $A = 406.4$, $B = 410.7$, $\alpha = 0.34$, $\beta = 0.28$.

Training compute is well approximated by $C \approx 6ND$ FLOPs — roughly 2 FLOPs per parameter for the forward multiply-accumulate and 4 for the backward pass, at every token. So the real question a lab faces is not “how big?” but: given a fixed C, how should I split it between N and D?

Minimise $L$ subject to $C = 6ND$ and you get a clean answer:

Both exponents are near $\tfrac{1}{2}$: when your budget goes up 100×, you should make the model ~10× bigger and train it on ~10× more data. Kaplan's 2020 analysis had said to grow the model much faster than the data, and the field believed it — GPT-3 is 175B parameters trained on only 300B tokens. Hoffmann's team showed the earlier fit was distorted by a learning-rate schedule that wasn't re-tuned for each run length, then proved the point by training Chinchilla: 70B parameters, 1.4T tokens, same compute as the 280B-parameter Gopher, and better on essentially everything.

One more twist: compute-optimal is optimal for training. If you are going to serve a model to millions of users, inference cost scales with $N$ and not with $D$, so it pays to “overtrain” a small model far past the Chinchilla point. Llama 3 8B saw ~15T tokens — nearly 2,000 tokens per parameter, two orders of magnitude past compute-optimal. That is a deliberate, rational choice, not a mistake.

If loss falls this smoothly, why does everyone talk about abilities appearing “suddenly”? Three different phenomena get tangled together here, and pulling them apart is the point of this section.

1. Emergence (contested). Wei et al. (2022) showed benchmark after benchmark where accuracy sits at chance across several model sizes and then shoots up. Schaeffer et al. (2023) replied: look at your metric. Exact-match accuracy on a 5-digit arithmetic problem is a threshold applied to a smoothly improving per-token probability — get each of five digits right with probability $p$ and your score is $p^5$, which stays invisible until $p$ is already high. Swap in a continuous metric (token edit distance, log-probability of the answer) and many “emergent” curves straighten out. The honest position: some apparent emergence is a metric artifact; whether all of it is remains open.

2. Grokking (real, and mechanistically understood). Power et al. (2022) trained small transformers on modular arithmetic and found something odd: the model reaches 100% training accuracy quickly, sits at chance on held-out data for tens of thousands more steps, and then — long after the training loss stopped moving — generalises, abruptly.

Nanda et al. (2023) opened the model up and explained it. The network learns to do $(a + b) \bmod p$ by embedding numbers on circles at a handful of frequencies and applying trigonometric identities — a genuinely elegant algorithm. Crucially, that algorithm is not built abruptly. Three phases overlap:

3. Double descent (real, still surprising). Classical statistics says test error is U-shaped in model size: too small underfits, too big overfits. Nakkiran et al. (2019) showed that if you keep going past the interpolation threshold — the size at which the model can fit the training set exactly — test error falls again, often below the classical sweet spot. The same shape shows up in epochs (train longer, get worse, then better) and in sample count (more data can transiently hurt). Modern LLMs live far out on the second descent, which is one reason “overfitting” intuitions from a statistics course mislead here.

Two toys. The first is the Chinchilla loss surface with real fitted constants — every number it shows you is what the 2022 paper's parametric model actually predicts, including the historical models it was fit against. The second replays a grokking run so you can watch the two accuracy curves come apart and snap back together.

Things to try: (1) Load the GPT-3 preset, note the loss, then hit Snap to optimal — same compute, and you see the ~0.05 nats that the 2020 recipe left on the table; now do the same for Chinchilla and see how close it already is. (2) Load Llama 3 8B and look at the IsoFLOP panel: the orange dot sits way down the left wall, and the readout shows a large loss gap — that gap is the price paid, on purpose, for a model that is cheap to serve. (3) In the grokking widget, scrub to step 5,000 and read the phase blurb: train accuracy has been pinned at 100% for thousands of steps and test accuracy is still at chance. Ask yourself what an eval run at that moment would have concluded, and how you would have known better.