Inference, Performance & Reliability

Everything up to now produced one thing: a probability distribution over the next token. Turning that into text requires a decision rule, and the choice of rule changes the output more than most people expect — same weights, same prompt, wildly different behavior.

The obvious rule is to take the most likely token every time (greedy decoding), or to search for the highest-probability sequence (beam search). Both are wrong, and the way they are wrong is one of the more interesting empirical facts about language models.

The failure is self-reinforcing, which is what makes it dramatic. Once a phrase repeats, it becomes a strong contextual predictor of itself — the induction machinery you will meet in Part 3 — so its probability rises with every repetition. Greedy decoding on an open-ended prompt reliably falls into a loop it cannot leave.

So we sample. Three knobs, applied in this order:

• Temperature $T$ divides the logits before the softmax: $p_i \propto e^{z_i/T}$. Below 1 it sharpens, above 1 it flattens. It reshapes the whole distribution and removes nothing.
• Top-$k$ keeps the $k$ highest-probability tokens and zeroes the rest. Simple and blunt: $k$ is fixed whether the model is certain or not.
• Top-$p$ (nucleus sampling) keeps the smallest set of tokens whose probabilities sum to at least $p$. The cut is dynamic — a confident distribution keeps 2 tokens, an uncertain one keeps 200 — which is precisely Holtzman's contribution and why it became the default.

Whatever survives is renormalized to sum to 1 and sampled from. The widget below shows every step, including which rule killed which token.

Generating a token requires a forward pass over the whole context. Generating the next one requires a forward pass over the whole context plus one. Done naively, producing $n$ tokens costs $O(n^2)$ work that is almost entirely redundant — because attention at position $t$ needs the keys and values of every earlier position, and those never change.

The cache is not a footnote. For a 70B-class model at a 128K context it runs to tens of gigabytes for a single request, which is why serving systems are designed around it: paged allocation so requests do not need contiguous blocks (vLLM's PagedAttention), and prefix sharing so a common system prompt is cached once for thousands of users. The calculator below makes the arithmetic concrete.

The architectural fix is grouped-query attention (GQA): let several query heads share one key/value head. Llama-3 70B has 64 query heads and 8 KV heads, cutting the cache by 8× for a small quality cost. Multi-query attention (MQA) is the extreme, with one KV head. Toggle the “70B without GQA” preset in the widget to see what this bought.

A second inefficiency is subtler. Decoding is memory-bandwidth bound, not compute bound. Producing one token requires reading every weight of the model from memory in order to do a handful of arithmetic operations on each. The GPU sits mostly idle waiting on memory. Two techniques exploit this.

Quantization attacks the same bottleneck by making the weights smaller: store them in 8-bit or 4-bit instead of 16-bit, and you read a quarter as many bytes per token. What degrades is specific rather than uniform. Dettmers et al. found that transformers past a few billion parameters develop outlier features — a small number of dimensions with enormous magnitudes that dominate the layer's behavior and are destroyed by naive rounding. LLM.int8() keeps exactly those in 16-bit and quantizes the rest; later methods (GPTQ, AWQ) refine which weights are worth protecting.

“Hallucination” gets used for two failures with different causes and different fixes. Separating them is most of the work.

Kadavath et al. (2022) is the load-bearing result here, and it is more optimistic than the discourse suggests. Large models are well calibrated on multiple-choice and true/false questions when asked in the right format. They can be asked to propose an answer and then evaluate the probability $P(\text{True})$ that it is correct, with decent calibration and improvement as models scale. They can even be trained to predict $P(\text{IK})$ — “do I know this?” — before answering, though calibration of $P(\text{IK})$ transfers poorly to new task distributions.

RLHF makes this worse in a specific and well-documented way. OpenAI's GPT-4 system card reported that the pre-RLHF base model was well calibrated on MMLU and the post-RLHF model was noticeably less so. That should be unsurprising after Module 2.3: humans prefer confident answers, so preference optimization pushes stated confidence up regardless of what the underlying distribution says. Alignment training and calibration are in direct tension.

Kalai et al. (2025) push the argument one step further and blame the scoreboard. Almost every benchmark grades binary — right or wrong — with no credit for “I don't know.” Under that rule, guessing strictly dominates abstaining whenever you have any information at all, exactly as it does for a student on a multiple-choice exam with no penalty for wrong answers. Models are optimized to be good test-takers, and we built a test that rewards bluffing. Their proposed fix is socio-technical rather than algorithmic: change the scoring of the benchmarks that dominate leaderboards.

An eval is a measurement instrument, and the failure modes are the ones every measurement instrument has: measuring something adjacent to what you meant, measuring something you already trained on, or measuring with less precision than your decisions require.

• Multiple-choice benchmarks (MMLU, GPQA) are cheap, reproducible, and heavily contaminated by now. They also measure recognition rather than generation, and are graded binary — the scoring problem above.
• Verifiable-answer benchmarks (GSM8K, HumanEval, SWE-bench) have real ground truth and are the best instruments we have, restricted to the domains where a checker exists — the Module 2.4 story again.
• Human preference arenas (Chatbot Arena) measure what people like, which is genuinely useful and is also, as Module 2.3 established, precisely the signal that rewards confident formatting.
• LLM-as-judge scales, and inherits every bias of the judge model, including a well-documented preference for its own outputs and for longer answers.
• Red-teaming and behavioral evals target a specific hazard rather than average quality. This is where safety-relevant evaluation actually happens.

The deeper problem is Goodhart, one last time. Every eval that becomes important becomes a training target, whether deliberately (training on the benchmark) or structurally (the field selects for methods that score well on it). The half-life of a useful benchmark is short, and holding out an eval you never train on is worth more than any single number it produces.

The sampling playground uses one fixed illustrative distribution. Temperature is applied first, then top-$k$, then top-$p$, then what survives is renormalized — the same order as a real sampler. Every removed token is labeled with the rule that removed it.

The calculator is the arithmetic every serving engineer runs before promising a context length.

Things to try: (1) Set $T$ = 0.2 and draw fifty tokens — one token takes essentially all the draws. That is degeneration in miniature, and it is why greedy decoding loops. (2) Set top-$p$ to 0.90 and sweep temperature from 0.5 to 2.0, watching the nucleus size change without touching $p$ — a fixed top-$p$ is not a fixed vocabulary. (3) In the calculator, load Llama-3 70B at 128K, then hit “70B without GQA” and watch the line jump above the H100 reference. Then set batch to 32 with GQA back on: this is why context length is priced the way it is.