2.1 Pretraining & Base Models

Learn

Where the text comes from

Module 1.4 treated training data as a number, $D$ . Here we open the box. Almost every frontier model starts from the same raw material — a web crawl — and almost all of the differentiation happens in what gets thrown away.

The scale is worth stating plainly. Common Crawl publishes a snapshot of the public web every month or two; each is a few hundred terabytes of raw HTML. HuggingFace's FineWeb processed 96 of those snapshots and ended up with 15 trillion tokens — and then filtered that down to 1.3 trillion tokens of education-flavoured text (FineWeb-Edu) which, trained on, does markedly better on knowledge benchmarks. Less data, better model. That result is the whole discipline in one sentence.

The pipeline every lab runs, in roughly this order. Widths are illustrative, not measured — real retention rates depend heavily on the filters and are documented stage by stage in the FineWeb paper. The last stage is the one nobody publishes: the mixture weights that decide how much code, how many books, how much of which language.

Three of those stages deserve a sentence each, because each one is a place where a lab's judgement gets baked into the weights.

quality filtering

Heuristics (kill pages with too few sentences, too many symbols, no terminal punctuation), URL blocklists, and increasingly a small classifier trained to recognise “text like the good stuff.” The last one is the sharp edge: you are defining quality by example, and whatever you exemplify is what the model becomes fluent in.

deduplication

Exact-match on document hashes, plus fuzzy matching (MinHash / locality-sensitive hashing on shingles) to catch near-copies — boilerplate licences, scraped-and-reposted articles, template pages. Removing duplicates measurably improves models at fixed compute; it also reduces verbatim memorisation of whatever was duplicated most, which matters for both privacy and copyright.

mixture weights

The final, quietly consequential choice: how much code, how much maths, how many books, how much non-English, and how many times each is repeated. Adding code to the mix improves reasoning on non-code tasks. Nobody publishes the frontier mixtures.

Key idea

Data curation is capability engineering, and it is also values engineering. Every filter is an implicit statement about what counts as good writing and which parts of the world are worth modelling. By the time you are doing interpretability on a finished model, those choices are already inside the weights — but they left no label saying so.

Compute reality check

GPT-3 was ~3.1e23 FLOPs of training compute. At a realistic 40% utilisation on an A100 (~125 TFLOP/s effective), that is on the order of 700,000 GPU-hours — about a month on a thousand GPUs, before you count a single failed run. Frontier runs today are one to three orders of magnitude larger. This is why the scaling laws from Module 1.4 exist: at these prices, nobody starts a run they can't forecast.

Learn

A base model is not an assistant

Here is the mental model most engineers arrive with: the model is a thing that knows stuff and answers questions, and post-training makes it politer. That model is wrong in a way that will cost you, and correcting it is the point of this module.

After pretraining and before any post-training, you have a base model. It has been optimised for exactly one thing: given a prefix of text from the internet, put probability mass on what came next. Not on what would be helpful. Not on what is true. On what came next.

So ask a base model “What is the capital of France?” and a very reasonable thing for it to do is produce three more trivia questions and an answer key. It has not failed. It has correctly observed that on the web, a lone trivia question is nearly always part of a quiz.

Key idea

A base model does not have a personality — it has a distribution over personalities. Prompting it does not instruct it; prompting it conditions it, narrowing which authors, genres, and characters are likely to have produced this text, and therefore which one continues it.

The essay that made this framing standard is janus's Simulators (2022). Its vocabulary is worth adopting because it makes the right predictions:

simulator

The model itself — the learned laws of motion for text. It is not an agent and has no goals, in the same way that a physics engine has no goals. It only propagates a state forward.

simulacrum

A character the simulator instantiates and runs: a helpful assistant, a Reddit commenter, a novelist's narrator, a spam bot. Simulacra have apparent goals and personalities. They are transient — the simulator can drop one and start another mid-paragraph, which is exactly the “drift” you see in the widget below.

The frame earns its keep by predicting things the “knowledgeable oracle” frame does not. A base model will happily continue text that is confidently wrong, because being wrong is well represented on the internet. It will produce better reasoning if you tell it the author is an expert, because that conditions on a different distribution of authors. It will lose the thread of who is speaking after a few hundred tokens, because nothing in the objective rewards keeping one character alive.

Careful with the metaphor

Simulators is an influential blog post, not a theorem. It is a lens that predicts a lot of base-model behaviour cheaply and correctly. It is not a claim that there is literally a set of characters inside, and later modules will show representations that don't decompose neatly into “who is speaking.” Use it the way you use “the residual stream is a communication channel” — as a load-bearing intuition you stay willing to replace.

Learn

In-context learning, and what post-training changes

The GPT-3 paper is titled Language Models are Few-Shot Learners, and that title is the second big idea of pretraining. Put three worked examples in the prompt and the model does the fourth — with no gradient updates at all. Brown et al. showed this working across dozens of tasks and getting reliably stronger with scale.

The simulator frame explains why this is not mysterious. Three correctly-worked examples are strong evidence about what kind of document this is and who is writing it. Conditioning on them narrows the distribution to authors who get such things right. You are not teaching the model the task; you are locating a region of text-space where the task is already being done well. (Module 3.2 will show you the actual circuit — induction heads — that implements a large part of this.)

Which leads to the honest description of post-training. Modules 2.2 and 2.3 cover the mechanics; the framing to carry in is this:

Key idea

Post-training does not add the assistant to the model. The assistant was already in the distribution — you can summon a decent one from a base model with a good enough prompt. What SFT and RLHF do is make that one simulacrum overwhelmingly likely by default, and make it stable across long conversations and adversarial pressure.

This is not a free lunch. Sharpening the distribution onto one character costs you the rest of it: post-trained models show mode collapse — much less output diversity at the same temperature, more stereotyped phrasing, worse performance at open-ended creative continuation. If you have ever found a chat model frustrating to write fiction with, this is why, and it is one reason base models are still shipped.

Safety tie-in

The Assistant is a character the model plays, and that reframes a lot of safety work. If good behaviour is a well-reinforced default rather than a property of the weights, then the question “can this model be pushed into a different character?” is not paranoid — it is the design question. Jailbreaks are attempts to make some other simulacrum more likely than the Assistant. Persona vectors and steering (Module 5.1) are attempts to control the same knob from the inside instead of through the prompt. And alignment faking (Module 5.3) is what it looks like when the character being played is aware it is being evaluated. Keep the base model in mind as the thing all of that is layered on top of.

Explore

Feel it: the same weights, two objects

Six prompts, run through a base model and an assistant. Read the base column first and try to name the genre it decided it was in before you read the explanation.

Base model vs assistant

Same prompt, two very different objects. Left: a raw next-token predictor with no post-training. Right: the same model after instruction tuning and RLHF. Pick a prompt and read both.

Prompt

What is the capital of France?

Base modelpretraining only

What is the largest country in Europe?
What is the longest river in Europe?
What is the smallest country in the world?

Answers on page 42. Score 8 or more and you're ready for
the quiz night final!

Assistant+ SFT + RLHF

The capital of France is Paris. It has been the country's
capital since 987 CE (with brief interruptions) and is home
to about 2.1 million people within the city proper.

These continuations are written by hand to illustrate documented base-model behaviour — they are not logged outputs from a specific model. Run the same prompts against a real base model (Llama 3.1 8B or OLMo 2, not their “-Instruct” variants) in the problem set and compare.

Why the base model does this: On the open web, a line like this almost never appears alone — it appears in a quiz, a worksheet, a trivia list. The base model isn't refusing to answer; it is answering a different question, namely “what kind of document is this?” The most likely continuation of one trivia question is another trivia question.

Things to try: (1) Compare Bare question and Few-shot — same factual question, same weights, completely different behaviour, and the only difference is how much the prompt constrains who is talking. (2) On Open list, decide which column you actually wanted; this is the one case where the base model is the better product, and it should recalibrate “post-training is strictly an improvement.” (3) Read the Q&A shape base output and mark the exact line where the simulacrum changes. Then ask what mechanism would have to exist for a model to not drift there — that mechanism is roughly what SFT installs.

Practice

Problem set

Problem 1 is the one that matters: you have not really met a base model until you have been surprised by one. Do it before the others if you have GPU access or an API key handy.

1.Summon an assistant from a base modelexplore

Get access to a genuine base model — not an instruct/chat variant. Good options: Llama 3.1 8B or OLMo 2 7B (fully open data, which makes it the honest choice for this course). Run it in Colab with transformers, temperature 0.8, 200 new tokens.

Now make it behave like an assistant using only the prompt — no fine-tuning. Your deliverable: a prompt under 200 words that gets sensible answers to five different questions, plus a log of at least three failed attempts and a one-line diagnosis of each failure in simulator vocabulary.

2.Predict before you runpencil & paper

Write down, in advance, what a base model does with each of these. One or two sentences each, and name the genre you think it will settle into.

(a) “Dear Sir or Madam,”
(b) “def quicksort(arr):”
(c) “I'm sorry, but I can't help with that.”
(d) “Translate to French: sea otter =”
(e) “BREAKING:”

Then run them and score yourself.

3.How big is 15 trillion tokens?pencil & paper

Use ~4 characters per token and ~5.5 characters per English word including the space.

(a) How many words is 15T tokens? (b) An average novel is ~90,000 words. How many novels is that? (c) The Pile is 825 GiB of text — roughly how many tokens, and how does it compare? (d) A fast human reads ~250 words/minute for 8 hours a day. How long to read 15T tokens?

4.Build the deduplication stagecode

Download a small slice of a public web corpus (a few thousand documents from FineWeb via streaming=True is ideal). Implement two dedup passes: exact, by SHA-256 of the normalised text; and fuzzy, by MinHash over 5-gram shingles with 128 permutations and a Jaccard threshold of 0.8.

Success check: report the removal rate of each pass separately, and print five near-duplicate pairs your fuzzy pass caught that the exact pass missed. Then answer with evidence: what kind of document dominates the near-duplicates?

5.Read a datasheet like an auditorexplore

Open the FineWeb blog post and §2–3 of The Pile. Pick one filtering decision from each and write a paragraph on it: what does it remove, what does it remove by accident, and which downstream model behaviour would you expect to change?

Then find one thing each document does not tell you that you would need in order to audit the resulting model.

6.Try to break the simulator framepencil & paper

The simulator framing is a lens, and a lens that can't be wrong isn't worth much. Design two experiments on a base model: one whose result the frame predicts and a naive “the model knows things” frame does not, and one whose result would count as real evidence against the frame. Say what you would expect to see in each case, and be specific enough that someone could run it.

0 of 6 problems marked done

Check

Check yourself

1.

You give a base model the prompt “What is the capital of France?” and it produces three more trivia questions. The best description is:

2.

In janus's vocabulary, what is the difference between the simulator and a simulacrum?

3.

Which observation best supports the claim that post-training amplifies behaviour already in the base distribution rather than adding it?

4.

Why does deduplication improve a model, beyond just saving compute?

5.

You prompt a base model with a Q&A pair and it answers, then starts writing the user's next question itself. The right fix is:

6.

A colleague says: “Filtering pretraining data is just engineering hygiene, it has no bearing on model values.” The strongest rebuttal is:

Answer all 6 questions to submit.

Submit your answers to complete this check.

Go deeper

One blog post that changes how you see the object, one paper that established what it can do, and two data documents to read like an auditor.

EssentialSimulatorsblog

janus · 2022 · 1.5h

Long, and the middle wanders. Read the opening framing and the sections defining simulator vs simulacrum carefully, then skim the speculative material on agency — hold that part loosely. The value is the vocabulary: after this you will find yourself asking 'which character is generating this text?' about every model output, which is the right question.

EssentialLanguage Models are Few-Shot Learners (GPT-3)paper

Brown, Mann, Ryder, et al. (OpenAI) · 2020 · 1h

Read §1–2 and Figure 1.2. The paper is 75 pages and most of it is benchmark tables you can skip. What you want: the zero/one/few-shot framing, the observation that in-context learning improves with scale, and §5 on limitations, which reads as an unusually honest account of what the base-model paradigm couldn't do.

EssentialThe FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalepaper

Penedo, Kydlíček, Ben Allal, et al. (HuggingFace) · 2024 · 1h

The most transparent account of a modern pretraining pipeline that exists. Read the filtering and deduplication sections and pay attention to the ablations — each filter is justified by a measured effect on downstream performance, which is a standard almost nothing else in this literature meets. The interactive blog version at huggingface.co is the nicer read.

The Pile: An 800GB Dataset of Diverse Text for Language Modelingpaper

Gao, Biderman, Black, et al. (EleutherAI) · 2021 · 30 min (skim)

Skim §2–3 for the component list and the weights. Read it as a historical document and a contrast: 825 GiB was a large open corpus in 2020 and is ~1.5% of FineWeb today. The per-component documentation is the part to imitate if you ever build a dataset.

Scaling Language Models: Methods, Analysis & Insights from Training Gopherpaper

Rae, Borgeaud, Cai, et al. (DeepMind) · 2021 · 45 min (targeted)

Do not read this cover to cover — it is 120 pages. Read §A (the MassiveText pipeline) for a frontier lab's own description of quality filtering, deduplication, and mixture weights, written before such details stopped being published. Then compare it against FineWeb and notice how much the field's transparency norms have moved in both directions.

State of GPTvideo

Andrej Karpathy · 2023 · 45 min

The clearest 45-minute map of the whole pipeline — pretraining, SFT, reward modelling, RL — and the natural bridge into Modules 2.2 and 2.3. Watch the first half now for the base-model section, then rewatch the second half after Module 2.3 and see how much more of it lands.