Tokens & Embeddings

Before a transformer does anything, a completely separate program chops your string into pieces and looks each piece up in a table. That program is the tokenizer, the pieces are tokens, and the model only ever sees the results. It has no access to the original characters. Almost every surprising low-level failure in this course traces back to that sentence.

Why not feed the model characters? You could, and some models do, but an average English word is 4–5 characters, so every sentence becomes 4–5× longer. Attention cost grows with the square of sequence length, so character models spend their compute re-deriving spelling instead of meaning.

Why not whole words? Then the vocabulary is unbounded (every typo, every name, every URL is a new word), rare words get almost no training signal, and anything unseen becomes an <UNK> hole in the input.

Byte-pair encoding learns the vocabulary from data with an almost embarrassingly simple loop. Start with every character as its own symbol. Count every adjacent pair of symbols in the corpus. Merge the most frequent pair into one new symbol. Repeat until you have as many merges as you wanted.

The payoff shows up on words the corpus never contained. Feed lowest to the four merges above, in order, and you get low + est — two learned pieces, no <UNK>, and a segmentation that happens to be morphologically sensible. That is the whole trick: frequent things get short, rare things get decomposed.

Real implementations add two wrinkles. First, a pre-tokenization regex splits text into words, numbers, and punctuation runs before any merging, so merges can never straddle a space or glue a word to a comma. Second, byte-level fallback: GPT-2 works on the 256 raw bytes rather than Unicode characters, so any input at all — emoji, Klingon, binary — is encodable, just expensively. The widget below has the first wrinkle and not the second; characters missing from its small corpus get flagged instead.

A tokenizer is a lossy interface between the world and the model. Four failure families come straight out of that interface, and you can predict all four from the mechanism.

1. Arithmetic. A model can only learn a clean addition algorithm if digits arrive in consistent chunks. GPT-2 and GPT-3 gave hundreds of multi-digit numbers their own tokens with no consistent grouping, so 380 + 42 and 381 + 42 can have completely different token shapes. Beren Millidge's Integer tokenization is insane walks through how bad it was; his 2024 follow-up notes that newer tokenizers adopted consistent digit grouping, which is one reason arithmetic improved without anything changing in the architecture.

2. Spelling and counting letters. Asking how many rs are in strawberry asks the model to report on characters it never received. It can often answer anyway — spelling is recoverable from context and from the token string itself — but it is doing inference, not lookup, and it fails in the way that inference fails. Tokenization is not the whole story here, but it is why the task is hard at all.

3. Glitch tokens. In 2023 Jessica Rumbelow and Matthew Watkins found tokens like  SolidGoldMagikarp sitting near the centre of GPT-2's embedding cloud. These strings were frequent in the tokenizer's training data (Reddit usernames from counting threads) but essentially absent from the model's training data, so their embedding vectors were never trained away from their random initialisation. Prompting with them produced evasion, insults, hallucinated completions, and broke determinism at temperature 0. Land & Bartolo later automated detection of such under-trained tokens across many production models — they are not a historical curiosity.

4. Language inequity. Petrov et al. measured tokenized length for the same text across languages and found differences of up to 15×. Since APIs bill per token and context windows are counted in tokens, speakers of under-represented languages pay more money for less context and slower responses — a fairness problem baked in before inference starts.

A token id is just a row number. The embedding matrix $W_E \in \mathbb{R}^{n_{\text{vocab}} \times d_{\text{model}}}$ holds one learned vector per token, and “embedding a token” means taking that row:

Written as a matrix product it looks like computation; it is a table lookup. For GPT-2 small, $50{,}257 \times 768 \approx 38.6$ million parameters — roughly 31% of the model's 124M, spent before any thinking happens.

Because tokens with similar contexts get similar gradient pressure, embedding space acquires structure: numbers cluster with numbers, punctuation with punctuation, code keywords with code keywords. The famous stronger claim is analogy arithmetic — $\text{king} - \text{man} + \text{woman} \approx \text{queen}$ — which says related pairs are separated by a consistent offset vector, not just placed nearby.

At the other end of the model, the unembedding $W_U$ turns the final residual vector back into 50,257 logits — and each logit is a dot product between the residual stream and one row, exactly the similarity operation from Module 0.1:

That last observation is the whole idea behind the logit lens: if the unembedding can decode the final residual stream, point it at an intermediate layer and watch the prediction form. Module 3.1 does this properly. For now, notice that it is only possible because input and output live in the same vector space.

The first widget trains a genuine BPE tokenizer in your browser — 200 merges over a small corpus, learned when the page loads — and re-segments whatever you type on every keystroke. The second is a hand-built picture of embedding-space structure, which is honest about being a diagram rather than a projection of real weights.

Things to try: (1) Drag the merge slider from 0 to 200 with the prose sample loaded and watch tokens fuse — at 0 merges this is a character-level model, and the token count falls by more than half by the end. (2) Switch to the numbers sample and look at how 1024 and 2048 get carved up; the corpus contains both, so ask yourself what an addition algorithm would have to learn from these shapes. (3) Type the same word twice, once after a space and once at the start of a line, and confirm they produce different tokens. (4) In the map, hover actor and 42 and notice that nearest-neighbour structure is about role in text, not meaning in the world.