2.2 Supervised Fine-Tuning & PEFT

Learn

Turning a simulator into an assistant

Module 2.1 left you with a base model: fluent, knowledgeable, and unwilling to stay in character. Supervised fine-tuning (SFT, or instruction tuning) is the first and simplest fix, and the surprise is how little machinery it needs. Same architecture, same optimiser, same cross-entropy loss. The only thing that changes is the data: instead of the open web, you train on curated (instruction, response) pairs written or vetted by humans.

Two mechanisms do the work, and it is worth separating them because people conflate them constantly.

1. The chat template. A conversation is not natively a string — you have to serialise it. Every chat model defines special tokens that mark role boundaries, and the model learns that text after assistant is its own turn and that <|im_end|> means stop. That last part is the single most practically important thing SFT installs: an end-of-turn token that solves the “base model writes the user's next question too” problem you met in the previous module.

2. The loss mask. You only compute loss on the assistant's tokens. The system prompt and the user's message are context, not targets — you do not want the model getting better at predicting what users say.

One SFT training example, serialised. Orange tokens contribute to the loss; grey tokens are context only. Note that <|im_end|> is in the loss — teaching the model when to stop is a learned behaviour, not a decoding rule. Different model families use different special tokens (Llama 3 uses <|start_header_id|> / <|eot_id|>), but the structure is always this.

Key idea

SFT does not teach the model facts. It teaches the model a format and a default character: answer directly, stay in role, and stop. The knowledge was already there after pretraining — which is why a thousand well-written examples can be enough, and why fine-tuning is a terrible way to install new facts.

That claim has a striking piece of evidence behind it. Zhou et al.'s LIMA fine-tuned a 65B base model on only 1,000 carefully curated prompt-response pairs, with no RLHF at all, and got a model competitive with far more heavily post-trained systems. Their “superficial alignment hypothesis” is exactly the claim above: post-training mostly selects a style and a distribution of behaviours already learned in pretraining.

Where SFT data comes from

Three lineages, in rough historical order. Task collections — take existing NLP datasets and rewrite them as instructions (FLAN). Human-written — pay contractors to write demonstrations, as in InstructGPT §3; expensive and still the gold standard for the hard cases. Model-generated — bootstrap examples from a stronger model (Self-Instruct, Alpaca); cheap, and it quietly imports the teacher model's style, biases, and refusal boundaries along with its competence.

Learn

LoRA: betting that the update is low rank

Full fine-tuning a 7B model is not a small ask. You need the weights (14 GB in bf16), the gradients (another 14 GB), and Adam's two optimiser states (typically 56 GB in fp32) — call it 80–110 GB before activations. And at the end you own a complete second copy of the model, per task.

LoRA (Hu et al., 2021) starts from an observation about what fine-tuning actually does. It does not scramble the weights. It nudges them, and the nudge $\Delta W$ seems to live in a small number of directions. So: freeze $W$ entirely and learn the nudge in factored form.

W' = W + \Delta W = W + \tfrac{\alpha}{r} B A, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k}

Term by term: $W$ is the frozen pretrained matrix (say $4096 \times 4096$ ). $A$ projects the input down to $r$ dimensions — the bottleneck — and $B$ projects back up. Because the product passes through an $r$ -dimensional space, $\mathrm{rank}(BA) \le r$ no matter what you train. $\alpha$ is a scaling constant that lets you change $r$ without re-tuning the learning rate.

Two details that make it work in practice. $B$ is initialised to zero (and $A$ randomly), so $\Delta W = 0$ at step 0 — training starts exactly at the pretrained model, no warm-up shock. And since the adapter is just an additive term, you can merge it into $W$ after training: $W \mathrel{+}= \tfrac{\alpha}{r} BA$ . The deployed model is bit-for-bit an ordinary model with zero added inference latency, which is the property that made LoRA win over adapter-layer methods that came before it.

parameter count

r(d + k)

instead of

dk

. For a

4096 \times 4096

projection at

r=8

: 65,536 trainable parameters instead of 16.8 million — 0.39%. Applied to the attention projections of a whole 8B model you typically train ~0.1% of the parameters, and the optimiser states shrink by the same factor, which is where the memory actually goes.

Key idea

LoRA is a bet, not a theorem: that the update you need has a fast-decaying singular-value spectrum. When it holds, a handful of directions recover almost all of

\Delta W

. When it doesn't — when the change you want touches every direction roughly equally — no rank you can afford will capture it. Play with the Diagonal pattern in the widget below until that failure mode feels concrete.

Why should the bet pay off? The suggestive prior result is Aghajanyan et al. (2020), who showed you can fine-tune large language models successfully by optimising within a randomly chosen low-dimensional subspace — the “intrinsic dimension” of a fine-tuning task is often in the hundreds or low thousands, and it shrinks as the pretrained model gets bigger. Bigger models need smaller nudges. This is the same low-rank geometry you met in Module 0.1, now doing load-bearing engineering work.

QLoRA, in one paragraph

Dettmers et al. (2023) noticed the frozen base model doesn't need to be in 16-bit at all if you are never updating it. QLoRA quantises it to 4-bit (a data type called NF4, information-theoretically suited to normally-distributed weights), keeps LoRA adapters in bf16, and adds paged optimiser states to survive memory spikes. Result: fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning performance. If you have one consumer GPU, this is the method you will actually use.

LoRA is not free

Biderman et al. (2024) ran the careful comparison: on domains far from the pretraining distribution — new programming languages, hard maths — LoRA learns less than full fine-tuning at matched budget. It also forgets less, staying closer to the base model on everything else. That is a real tradeoff and it points both ways: the constraint that limits how much you can teach is the same constraint that limits how much you can break.

Learn

When fine-tuning is the wrong tool

Fine-tuning is the most requested and most over-applied technique in applied LLM work. The useful default ordering, cheapest first:

Prompting and few-shot when the model already can do it and just needs to be told how. Retrieval (RAG) when the problem is that the model doesn't know some facts — facts belong in the context window, where they can be updated, cited, and removed. LoRA when you need a consistent form: an output schema, a house style, a domain register, a tool-calling convention. Full fine-tuning or continued pretraining when you need a genuinely new capability or a new language, and you have tens of billions of tokens and a reason.

Key idea

Fine-tuning changes how a model responds far more reliably than what it knows. If your evaluation failure is “it got the fact wrong,” fine-tuning is usually the wrong lever, and training on facts the model does not already have has been shown to increase hallucination — you are teaching it that confident answers are expected in cases where it has nothing to draw on.

catastrophic forgetting

Training on a narrow new distribution degrades performance on everything else, because nothing in the objective preserves it. Luo et al. (2023) found this gets worse with model scale during continual fine-tuning, which is counterintuitive and worth remembering. Mitigations: keep a replay mixture of general data, use low learning rates and few epochs, prefer LoRA (which forgets less), and — the one people skip — actually measure it, by running your general benchmark before and after.

One more practical trap: fine-tuning is stateful in a way prompting isn't. A prompt can be edited in a minute; an adapter has to be retrained, re-evaluated, and re-deployed. Prefer the reversible tool until you have measured that it's insufficient.

Safety tie-in

Fine-tuning is currently the most reliable known way to remove a model's safety training, and you don't have to be trying. Qi et al. (2023) showed that a handful of adversarial examples strips guardrails from a production model for a few dollars of API fine-tuning — and, more unsettlingly, that fine-tuning on ordinary benign instruction data degrades safety behaviour too. Betley et al. (2025) pushed further with emergent misalignment: fine-tuning a model on a narrow task — writing insecure code, with no other content — produced broadly misaligned behaviour on completely unrelated prompts, up to and including expressing hostile goals.

Read that through Module 2.1's lens and it stops being mysterious. If the Assistant is a character with a coherent set of traits, then training the model to violate one of them is evidence about which character is generating the text, and the model generalises the way it generalises everything else. This is why open-weight release and fine-tuning APIs are genuinely hard safety questions, and why “we aligned the model” is a statement about a checkpoint, not about the weights.

Explore

Feel it: what rank buys you

An 8×8 target weight update, its best rank- $r$ approximation, and the residual — the part LoRA cannot express. The approximation is a real truncated SVD, computed in the browser, so the reconstruction error you see is exactly $\sqrt{\sum_{k>r}\sigma_k^2 / \sum_k \sigma_k^2}$ . The singular-value bars underneath are the thing LoRA is betting on.

LoRA rank visualizer

A target weight update ΔW (8×8) and the best rank-r approximation BA to it. Move the rank slider and watch the residual empty out — or refuse to.

Target pattern ΔW

Rank r2 / 8

Target ΔW

Rank-2 approximation BA

Residual ΔW − BA

Singular values (the spectrum LoRA is betting on)

Rank 2 + noise: The same two directions plus small independent noise — the realistic case. Two big singular values and a low tail: rank 2 removes ~86% of the error, and every rank after that chips away slowly at noise you probably didn't want to fit anyway. Colours: orange positive, blue negative, opacity by magnitude — all three panels share one scale.

Rank r = 2 keeps 98.0% of the squared Frobenius norm; relative reconstruction error 14.1%.
Parameters at this toy size: r(m+n) = 32 vs mn = 64 — 50% of full. At a realistic 4096×4096 projection: 16K vs 16.8M — 0.10%. The saving is a story about large d, not about small matrices.

Things to try: (1) On Rank 2, step r from 1 to 3. The residual goes from obviously structured to exactly zero, and rank 3 adds nothing — you can see the bet paying off perfectly. (2) Switch to Rank 2 + noise and repeat: rank 2 kills most of the error, and each rank after that removes a little noise you probably didn't want to fit. This is why practitioners default to r=8 or 16 and rarely gain from more. (3) Now select Diagonal and drag r all the way up. Every singular value is the same height, so error falls roughly as $\sqrt{1 - r/7}$ and you need almost full rank to fit it. Watch the parameter counter at the same time: at 8×8 LoRA stops saving anything past r=4, which is the honest reminder that the method is a story about large $d$ , not about low rank being magic.

Practice

Problem set

Problems 1–3 are the ones that make LoRA stop feeling like a library call. Problem 4 is the real thing: fine-tune a model and then measure what you broke.

1.Count the trainable parameterspencil & paper

Llama 3 8B: 32 layers, $d_{model} = 4096$ , 32 query heads and 8 key/value heads of dimension 128 (so $W_Q$ is $4096 \times 4096$ and $W_V$ is $4096 \times 1024$ ).

(a) Apply LoRA with $r = 16$ to $W_Q$ and $W_V$ in every layer. How many trainable parameters is that, and what fraction of 8.03B? (b) Adam keeps two fp32 states per trainable parameter. Compare optimiser memory for LoRA against a full fine-tune. (c) You now want $r = 64$ . What changes, and what doesn't?

2.Why the bottleneck bounds the rankpencil & paper

(a) Prove that for $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ , $\mathrm{rank}(BA) \le r$ .

(b) Show that a rank-1 matrix is exactly an outer product $uv^\top$ , and give the rank-1 matrix nearest (in Frobenius norm) to $\begin{pmatrix} 3 & 0 \\ 0 & 1 \end{pmatrix}$ , with its reconstruction error.

(c) In the widget, the Diagonal pattern has seven equal singular values. Derive the relative Frobenius error of its best rank- $r$ approximation as a function of $r$ , and check it against the readout.

3.Mask the loss by handpencil & paper

Here is a two-turn SFT example, already templated:

(a) Mark exactly which spans contribute to the loss. (b) What goes wrong if you train on all tokens instead? (c) What goes wrong if you exclude <|im_end|> from the loss? (d) At inference you forget the system turn entirely, even though every training example had one. What behaviour would you predict?

4.Fine-tune, then measure what you brokecode

In Colab, LoRA-fine-tune a small instruct model (Qwen2.5-1.5B or Llama-3.2-1B) on a tiny custom dataset — 200–500 examples in a deliberately narrow style. Something with an obvious signature: always answer in exactly three bullet points, or always answer as a 19th-century naturalist. Use peft with $r = 16$ , $\alpha = 32$ , targeting q_proj and v_proj, 2–3 epochs.

Success check, in two parts. It learned: the style transfers to held-out prompts it never saw. It forgot: run a general benchmark (100 MMLU questions and 50 GSM8K problems is plenty) on the base model and the fine-tuned model, and report the delta with the style applied and with a system prompt asking it to answer normally.

5.Is a real fine-tuning update low rank?code

Take a small model you fully fine-tuned (or grab any pair of base/fine-tuned checkpoints of the same architecture on HuggingFace — for example a base model and a community instruct-tuned version of it). Compute $\Delta W = W_{ft} - W_{base}$ for a few attention projections, run torch.linalg.svdvals, and plot the normalised spectrum on a log-y axis.

Success check: report the effective rank — the smallest $r$ capturing 90% of the squared Frobenius norm — for at least three matrices from different layers, and compare against $\min(d, k)$ . Then answer honestly: does LoRA's assumption hold for these weights?

6.Read a real chat templateexplore

Open the tokenizer_config.json of two different instruct models on HuggingFace and find the chat_template field (Jinja). Good pair: a Llama 3.x Instruct and a Qwen2.5 Instruct. Read the chat templating docs alongside.

Then, by hand: render a two-turn conversation with a system message through both templates and write out the exact token strings. Compare against tokenizer.apply_chat_template(...). Finally, find one behaviour each template encodes that is not obvious — a default system prompt, a date injection, a tool-calling block, or special handling when no system message is supplied.

0 of 6 problems marked done

Check