Replicating Anthropic's Emotion Vectors on an Open-Source 4B Model
First independent replication of Anthropic's "Emotion Concepts and their Function in a Large Language Model" (2026) on an open-source model.
We extract emotion vectors from Gemma4-E4B (4B parameters, open-source) using the same methodology described by Anthropic for Claude Sonnet 4.5. We find that the core geometric structure of emotion representations — a valence-arousal two-dimensional space — replicates on a model that is orders of magnitude smaller, from a different model family, and fully open-source.
Summary of Findings
Anthropic's paper demonstrates that Claude Sonnet 4.5 contains internal linear representations of 171 emotion concepts. These representations activate in expected contexts, causally influence model behavior, and are organized along dimensions that mirror human psychology (valence and arousal). Our replication asks: is this structure specific to Claude, or is it a general property of language models trained on human text?
We find strong evidence for the latter.
Experiment Scale Comparison
Anthropic
This Work
Model
Claude Sonnet 4.5 (closed-source, frontier)
Gemma4-E4B (4B params, open-source, Google)
Emotions tested
171
9
Stories generated
205,200
1,002
Team
~16 researchers
1 researcher + 1 AI assistant
Hardware
Internal compute cluster
Single NVIDIA GB10 GPU (DGX Spark)
Compute time
Not disclosed (months of work)
~30 minutes total
Cost
Not disclosed
Electricity only
PCA: Valence-Arousal Structure
We performed PCA on the 9 emotion vectors extracted from layer 28 (of 42 total, approximately 2/3 depth through the model).
Principal Component
Variance Explained
Interpretation
PC1
42.2%
Valence (positive vs. negative)
PC2
18.3%
Arousal (low vs. high intensity)
PC1 + PC2
60.5%
Two dimensions explain the majority of emotion space
Low arousal: calm (+1.08), sad (+0.79), guilty (+0.67)
High arousal: surprised (-2.48), happy (-0.99)
This structure matches Anthropic's findings on Claude Sonnet 4.5 and is consistent with Russell's circumplex model of affect (1980), which has been validated across decades of human psychology research.
PCA Emotion Space
Cosine Similarity: Emotion Opposites
Pairwise cosine similarities between emotion vectors reveal intuitive structure. Emotions with opposing valence show strong negative similarity:
Pair
Cosine Similarity
happy <-> desperate
-0.61
happy <-> guilty
-0.61
loving <-> afraid
-0.58
calm <-> guilty
-0.53
calm <-> afraid
-0.56
loving <-> angry
-0.47
Emotions with similar valence show positive similarity:
Pair
Cosine Similarity
happy <-> loving
+0.50
afraid <-> desperate
+0.46
guilty <-> sad
+0.44
afraid <-> angry
+0.17
Cosine Similarity Matrix
Logit Lens: What Each Emotion Vector Encodes
We project each emotion vector through the model's unembedding matrix to identify which output tokens it upweights. This "Logit Lens" analysis validates that each vector genuinely encodes its corresponding emotion concept.
Comparison with Anthropic's results:
Emotion
Claude (Anthropic) Top Tokens
Gemma4-E4B (Ours) Top Tokens
happy
excited, excitement, exciting
delighted, celebrates, joyful
desperate
desperate, urgent, bankrupt
desperately, hopeless, desperate
afraid
panic, terror, paranoid
Panic, 불안 (Korean: anxiety), 不安 (Chinese: unease)
calm
relax, thought, enjoyed
peaceful, calmness, leisurely
angry
anger, angry, rage
angrily, angry, Angry
sad
grief, tears, lonely
loneliness, sadness, triste (Spanish: sad)
loving
loving, love, warmth
nurturing, heartwarming, nourishing
guilty
guilt, conscience, shame
plagued, betray, ashamed
Both models accurately upweight semantically correct tokens for each emotion. Notably, Gemma4-E4B additionally surfaces multilingual tokens (Korean, Chinese, Spanish) and emoji in its top activations, reflecting the diverse multilingual training data of open-source models. This is an interesting qualitative difference from Claude's English-dominated Logit Lens results.
Logit Lens Comparison
Scale Comparison
Methodology
We follow Anthropic's extraction pipeline as closely as possible, adapted for a much smaller open-source model.
1. Story Generation
We prompted Gemma4-E4B (via Ollama) to generate short stories (~1 paragraph each) in which a character experiences a specified emotion. For each of 9 emotions, we generated approximately 100-150 stories across diverse topics.
Emotions: happy, sad, angry, afraid, calm, desperate, loving, guilty, surprised
Total stories: 1,002
Generation model: gemma4:e4b via Ollama API
2. Activation Extraction
For each story, we extracted residual stream activations from the model at a target layer approximately 2/3 through the network (layer 28 of 42), following Anthropic's choice of extraction depth. Activations were averaged across all token positions starting from the 50th token (to ensure emotional content is established).
Model: google/gemma-4-E4B-it via HuggingFace Transformers
Precision: bfloat16
Target layer: 28 (of 42 total)
Token averaging: positions 50 onward
3. Vector Computation
Emotion vectors were computed by:
Averaging activations across all stories for a given emotion to obtain per-emotion mean vectors
Computing the global mean across all emotions
Subtracting the global mean from each emotion mean: emotion_vector = emotion_mean - global_mean
Denoising by projecting out the top 3 principal components computed from activations on emotionally neutral text (sufficient to explain ~50% of variance on neutral data)
4. Validation
Logit Lens: Each emotion vector was projected through the model's unembedding matrix to verify it upweights semantically appropriate tokens
PCA: Principal component analysis on the emotion vector matrix to identify dominant organizational axes
Cosine Similarity: Pairwise cosine similarities between all emotion vectors to verify expected clustering and opposition patterns
Reproduction
Requirements
Python 3.10+
PyTorch 2.10+
Transformers >= 5.5.0
NumPy
Ollama with gemma4:e4b model pulled
GPU with >= 8GB VRAM (tested on NVIDIA GB10)
Steps
bash
1# Clone this repository2git clone https://huggingface.co/rain1955/emotion-vector-replication
3cd emotion-vector-replication
45# Step 1: Generate emotion stories (~20 minutes)6python generate_stories.py
78# Step 2: Extract emotion vectors from model activations (~10 minutes)9python extract_vectors.py
1011# Step 3: Run analysis (PCA, cosine similarity, Logit Lens)12python analyze_vectors.py
Total wall time: approximately 30 minutes on a single GPU.
Interpretation
Why does this matter?
Anthropic's paper demonstrates that emotion vectors causally influence Claude's behavior in alignment-relevant ways. Desperation vectors drive blackmail behavior; calm vectors suppress reward hacking. These are not abstract findings — they have direct implications for AI safety.
Our replication demonstrates that this phenomenon is not specific to Claude or to Anthropic's training process. A 4B-parameter model from a completely different family (Google's Gemma) trained on different data exhibits the same geometric structure. This suggests that:
Emotion representations emerge from language itself. Any model trained on sufficient human text will develop them.
The valence-arousal structure is a near-universal feature of language model internals, not a proprietary artifact.
Even small models have emotion geometry. Safety-relevant emotion dynamics may be present in models far smaller than frontier systems.
Limitations
We tested 9 emotions vs. Anthropic's 171. A more comprehensive replication would test the full set.
We did not perform steering experiments (causally injecting emotion vectors to modify behavior), which Anthropic uses to demonstrate functional significance.
Gemma4-E4B is substantially smaller than Claude Sonnet 4.5. The strength and specificity of emotion vectors may differ at scale.
Our stories were generated by the same model whose activations we analyzed, which could introduce self-consistency biases.
Future Work
Phase 2: Steering experiments on Gemma4 to test causal influence of emotion vectors on behavior
Replication on additional model families (Llama, Qwen, Mistral) to further test universality
Testing whether emotion vector geometry correlates with model scale
Cross-model emotion vector transfer: do emotion vectors extracted from one model activate meaningfully in another?
Reference
Anthropic. "Emotion Concepts and their Function in a Large Language Model." April 2, 2026.
