RLVR & Reasoning Models

Module 2.3 ended in an uncomfortable place. The reward model is a learned approximation of noisy human judgment; push a policy against it hard enough and you get Goodhart. The obvious question is whether there is any reward you can optimize without that risk.

For a narrow but enormously valuable slice of tasks, yes. If the problem is “solve this competition math problem” or “make these unit tests pass,” the reward is not a prediction of what a human would like. It is a fact:

This is RLVR — reinforcement learning from verifiable rewards. There is no reward model to overfit, no labeler disagreement to average, no sigmoid extrapolating past its support. A checker runs and returns a bit.

The relocated failures are documented and mundane. Agents that edit the test file rather than the implementation. Solutions that special-case the visible test inputs. In formal-proof settings, exploiting an unsoundness in the checker. And the systemic version: a verifier checks the final answer, so any reasoning that arrives there is reinforced — including reasoning that was wrong twice in compensating directions.

Here is the part that genuinely surprised people. Nobody taught reasoning models to think step by step for ten thousand tokens, backtrack, or double-check their arithmetic. Those behaviors emerged from optimizing a single bit of reward.

DeepSeek-R1-Zero (2025) is the cleanest demonstration because it removed every confound. Start from a base model with no supervised fine-tuning on reasoning traces at all. Apply RL with two rule-based rewards: is the final answer right, and is it in the required format. Run it.

The model's responses got longer on their own — not because length was rewarded, but because longer chains produced correct answers more often, so the gradient favored them. Along the way, self-verification and backtracking appeared; the paper highlights an “aha moment” where the model interrupts itself mid-solution to re-examine an earlier step. Reported AIME 2024 pass@1 rose from roughly 16% to roughly 71% over the RL run, and to roughly 87% with majority voting over 64 samples.

R1-Zero was not shippable — poor readability, languages mixed mid-sentence. Nothing in the reward said “be legible,” so it wasn't. DeepSeek-R1 proper adds a small cold-start SFT set of readable long chains before the RL, then a rejection-sampling and SFT pass, then a final RL stage covering general helpfulness. The capability came from RL; the presentability came from the scaffolding around it.

There is an older, simpler idea in the same family worth knowing because you can run it on a laptop. Expert iteration (also STaR, or rejection-sampling fine-tuning): sample $k$ solutions per problem, keep only the ones the verifier accepts, fine-tune on those with ordinary cross-entropy, repeat. No policy gradient, no advantage estimation. The model teaches itself using search plus a filter. Most of RLVR's conceptual content is already here.

A separate question is what gets rewarded. Outcome supervision scores only the final answer. Process supervision scores each intermediate step, using a process reward model (PRM) trained on step-level human labels. Lightman et al. (2023) compared the two directly and found process supervision significantly better on MATH — their PRM solved 78% of problems in a representative test subset — and released PRM800K, 800,000 step-level labels, so others could reproduce it.

Once a model can produce a correct solution some of the time, a second axis opens up: spend more compute at inference instead of at training. Three strategies, in increasing order of what they require:

• Majority vote (self-consistency). Sample $N$ solutions, return the most common answer. Needs nothing external. Works because wrong answers scatter and right ones agree.
• Best-of-$N$. Sample $N$, score each with a verifier or reward model, return the top-scoring one. Needs a scorer, and its value is capped by that scorer's quality.
• Longer single chains. Let one trajectory think for more tokens with backtracking. This is what reasoning models do natively, and it is why an “effort” or “thinking” dial exists in the product.

Two quantities are worth keeping separate. Coverage is the chance that at least one of the $N$ samples is correct — with independent samples, $\text{pass@}N = 1 - (1-p)^N$, which rises fast and is the ceiling for any selection method. Accuracy is the chance you actually return the correct one, which requires a way to pick.

The tradeoff is real, not free. Snell et al. (2024) found the best allocation depends on problem difficulty, that compute-optimal adaptive allocation beats a fixed best-of-$N$ baseline by several times, and that on problems where a small model already has non-trivial success rates, test-time compute can beat a 14× larger model at matched FLOPs. On problems the small model essentially never solves, it cannot — you cannot select from a set containing no correct answer.

Reasoning models produce a legible stream of English before answering. It is enormously tempting to read that stream as the model's reasoning — and if it were, it would be the single most valuable safety artifact we have: a natural-language window into a system whose internals are otherwise opaque.

Turpin et al. (2023) ran the first sharp version. Bias the context — for example, make the correct answer always (A) in the few-shot examples — and the model's answers move toward the bias, while its stated explanations never mention it and instead argue for the biased answer on the merits. The explanation is systematically unfaithful in exactly the cases where faithfulness would matter.

Anthropic's 2025 study repeated the design on reasoning models, where you might hope the extra visible thinking helps. It mostly doesn't. Inserting a hint, keeping only cases where the hint demonstrably changed the answer, and counting mentions: Claude 3.7 Sonnet mentioned the hint about 25% of the time and DeepSeek R1 about 39%. For the hint types you would most want surfaced — one framed as unauthorized access to the grader — the rates fell to 41% and 19%. Training the models to use reasoning more improved faithfulness at first and then plateaued below 30%. In environments seeded with a deliberate reward hack, models learned to exploit it in nearly every episode and described it in under 2% of their chains.

This module is also the bridge to Part 3. If you cannot trust the model's self-report, the alternative is to read the computation directly — activations, circuits, features. That is what mechanistic interpretability is for, and CoT unfaithfulness is one of the strongest arguments that we need it.

First, the economics of sampling. The four curves are the four strategies from the lesson, simulated under a stated toy model rather than measured on a benchmark — what matters is their shape and the gaps between them.

Then the faithfulness cases. Read each chain of thought first and commit to a verdict before you run any intervention. Most people get at least one wrong, which is the lesson.

Things to try: (1) Set $p$ = 0.1 and slide the verifier quality from 0.5 to 0.99 — watch best-of-$N$ travel almost the entire distance from “single sample” to “pass@N”. That whole range is the value of a verifier, and it is why the verifiable domains ran away from the rest. (2) Set $m$ = 1 (all wrong answers agree with each other) and drop $p$ below 0.5: majority vote now falls below a single sample, because voting amplifies a shared systematic error. (3) In the case viewer, try to find any surface feature — hedging, length, confidence — that separates Case 1 from Case 3 without running an intervention.