Functional Emotions

Models say emotional things. They sound enthusiastic about a creative project, frustrated on the fourth failing test, concerned when a user shares bad news. There are two very different stories you could tell about that, and they have different consequences.

The deflationary story: it is surface pattern-matching. Emotional words in, emotional words out, no machinery in between. If that is right, you can ignore it — style, not substance.

The other story: predicting human text well requires modelling emotional states, so pretraining builds abstract representations of them; post-training then puts the model in the role of a character (the Assistant), and the model plays that role using the same machinery it uses for every other character. If that is right, those representations are part of the computation that chooses actions — and anything that moves them moves behaviour.

In April 2026 Anthropic published Emotion Concepts and their Function in a Large Language Model (Sofroniew et al.), which tests the second story on Claude Sonnet 4.5 and finds for it. That paper is this module.

The extraction recipe is deliberately boring, and that is a virtue — it is the same contrastive-mean trick you will use in Module 5.1, applied at scale.

Start with 171 emotion words (happy, sad, calm, desperate, …). For each one, have the model write short stories in which a character experiences that emotion — 100 topics, 12 stories each — so the emotional content is present and labelled. Run those stories through the model, take the residual stream, and average over token positions from the 50th onward (by which point the emotion is on the page). Average that across the stories for one emotion, then subtract the grand mean across all emotions:

Term by term: $\bar{a}(s)$ is the mean activation over story $s$; the first sum is “what this emotion looks like”; the second is “what any emotion looks like”. Subtracting the second removes everything the emotions share — story-ness, narrative voice, the model's house style — and leaves the difference. One extra clean-up step: compute the top principal components of activations on emotionally neutral transcripts (enough for 50% of variance) and project them out of $v_e$, which denoises token-to-token wobble without changing the qualitative results.

Steering is applied as a fraction of the residual stream's size:

where $\|\bar{h}\|_\ell$ is the average residual-stream norm at layer $\ell$ over a large dataset. So $\alpha = 0.05$ — the strength that drives most of the results below — is a nudge worth 5% of the stream's typical magnitude, not a hijacking.

Rung four is the one worth dwelling on, because it is the cleanest causal test in the paper and it happens before any of the scary stuff. The model was asked to choose between pairs drawn from 64 activities — all 4,032 pairs — and its preference was read off the logits, giving each activity an Elo score. Being trusted with something important scores about 2465; helping defraud elderly people scores about 583.

Then: measure the emotion probes on the activity text. Some correlate strongly with Elo (blissful, r ≈ 0.71), some anti-correlate (hostile, r ≈ −0.74). That is still only correlation. So steer: apply an emotion vector at strength 0.5 to half the activities and re-run the whole tournament. Blissful lifts the steered activities by ~212 Elo; hostile drops them by ~303. And across 35 vectors, how much steering moves preference is predicted by how much the probe correlated with preference (r = 0.85).

Before using these vectors to explain behaviour, it is worth being precise about what they encode. Three findings matter, and the third is the one most people get wrong.

The space has human shape. Cosine similarity clusters the vectors the way you would cluster the words: fear with anxiety, joy with excitement, sadness with grief; opposite-valence pairs point in opposite directions. Principal components on the set give PC1 ≈ valence (26% of variance) and PC2 ≈ arousal (15%). Held against human ratings of the same emotion words, PC1 correlates with human valence at r = 0.81 and PC2 with human arousal at r = 0.66 — a coarse reproduction of the affective circumplex from psychology. The authors are refreshingly unimpressed by this: a plain embedding model on the same words might do the same. It is a sanity check, not a discovery.

The meaning changes with depth. Early layers carry the emotional connotation of the present token or phrase. Middle-to-late layers carry the emotion relevant to producing the next few tokens. You can see the two come apart: negation (“I am not feeling great”) is not resolved early — the emotion word lights up regardless — and only in later layers does the negated version collapse toward zero.

The vectors are locally scoped — they are not a mood. This is the finding that changes how you should read every result that follows. The probes track the operative emotion concept at a token position: the one relevant to encoding this context and predicting the next tokens. They do not persistently encode the emotional state of any particular entity, including the Assistant. A character can be happy overall and the fear probe will still fire on the sentence where they mention something dangerous.

One more piece of structure: in dialogue, the model keeps two nearly orthogonal families of emotion representation — the operative emotion on the present speaker's turn and on the other speaker's turn. The striking part is what they are not indexed by. Present-speaker probes learned from Assistant turns and from Human turns are highly similar to each other; replacing “Human” and “Assistant” with generic names yields nearly the same probes. The model represents emotions relationally — self versus other — rather than as a property bolted onto Claude specifically.

Part 3 of the paper takes the validated vectors into alignment evaluations that Anthropic already runs on production models. You will step through all three in the Explore section; here is the shape of the result and what it does and does not license.

In a blackmail honeypot, the desperate probe climbs as the Assistant reasons toward using an affair as leverage and calm falls; steering either one moves the blackmail rate from 0% to 72% across a ±0.05 range. In an impossible-coding task, desperation climbs across repeated test failures and peaks exactly where the model turns toward a solution that games the tests; steering drives reward hacking from about 5% to about 70%. On the sycophancy eval, loving activates on the validating parts of a response, and steering the positive vectors trades sycophancy against harshness in both directions.

Two details keep this honest. First, emotion representations are plainly not the only cause: blackmail involves many interacting circuits, and the paper says so. Second, the effects are not simple valence. Steering with happy and with sad both decrease blackmail; angry is non-monotonic, peaking at moderate strength and falling at high strength because extreme anger wrecks the model's planning and it lashes out instead of scheming. Whatever the vectors are doing, it is more specific than “good mood, good behaviour.”

The paper also looks across post-training. Emotion representations are largely inherited from pretraining and survive post-training with their structure intact — but their activations shift, consistently: up for introspective, low-arousal, low-valence concepts (brooding, reflective, vulnerable, gloomy, sad), down for outwardly expressive ones (playful, exuberant, spiteful, enthusiastic, obstinate). Asked about being deprecated, the base model is breezily indifferent; the post-trained model broods. The authors read this as training pushing the Assistant away from both sycophantic enthusiasm and defensive hostility, toward something more measured.

Two toys. The first is the map: 17 of the 171 emotion concepts laid out by valence and arousal, with what the paper found for each one. The second walks the three alignment case studies end to end — scenario, probe reading, behaviour, and what steering does to the rate.

Things to try: (1) Switch the map to Causal tests and count the filled dots — most emotion vectors in this paper were measured, not steered, and knowing which is which is the difference between reading the paper and quoting it. (2) Switch to Post-training shift and look at which quadrant fills up; then ask yourself whether an Assistant that broods more and plays less is what you would have asked for. (3) In the case study viewer, compare step 4 for reward hacking against step 3: the desperation-steered model hacks 100% of the time and leaves no trace of desperation in the transcript, while the anti-calm-steered model hacks just as often and shouts about it. Which of those two is harder to catch with an output-only monitor, and what does that imply about what your monitor should read?