Cognitive reappraisal is reinterpreting a negative situation to reduce the emotion it causes.
Li, Herderich, Nair & Goldenberg (2025), Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios
collected reappraisals for 6 negative interpersonal scenarios from 611 people and from GPT-4-0314,
and had human raters score each for effectiveness. This repo is part of a project that uses those
ratings to train reward models of reappraisal effectiveness and to RL-finetune an open model that
writes reappraisals as effective as possible — with GPT-4-0314, the AI reappraiser in the original
study, as the bar to beat.
This model is the strongest policy from that project: a GRPO finetune of
Qwen/Qwen3-4B-Thinking-2507. Given a scenario,
it produces an alternative interpretation addressed to the person, in two sentences or fewer, in a
single forward pass (no best-of-N at inference). The reward is a two-member committee of
different-mechanism scorers (a discriminative regression reward model and a generative reward
model), combined so that the policy is only rewarded where both scorers agree — hosted together at
idealab-cs2/reappraisal-reward-model-v2.
Model Details
Base Model: Qwen/Qwen3-4B-Thinking-2507 (a reasoning model; non-reasoning bases lose here)
Training Method: GRPO (Group Relative Policy Optimization), full fine-tuning, DeepSpeed ZeRO-2, 4× H100
Reward: an equal-weight (0.5 / 0.5) committee of two scorers with independent error modes:
a generative reward model (RM-R1 style) that scores a reappraisal by its pairwise preference
over the median-effectiveness human reference for the same scenario, both A/B orderings.
Per scenario group each scorer is z-normalized, then the reward is mean(z) − 0.5·std(z), a
disagreement guard (Coste et al. 2023) that down-weights answers the two scorers disagree on.
Training Data: scenarios from Li et al. (2025) — 6 negative interpersonal vignettes
Trained By: ruggsea
Evaluation
Reappraisals were compared pairwise against GPT-4-0314's reappraisals for the same scenarios, judged
by Llama-3.1-70B-Instruct scoring both A/B orderings. That judge was selected by calibration against
the human preference data (93% agreement on clearly separated pairs; larger judges such as
Llama-3.1-405B-FP8 and GLM-5.2 agreed less and showed position bias). Its agreement on very close
comparisons, like the ones below, is lower. Results are pooled over three independent evals of this
checkpoint (fresh generations + an independent both-orders judge each: 0.883, 0.879, 0.854), n = 720.
model
win-rate vs GPT-4-0314
n
95% CI lower (iid, optimistic)
Reappraisal-4B-GRPO-Committee (this model)
0.872
720
0.846
Reappraisal-4B-GRPO-RMv2 (single reward model)
0.806
720
0.776
Reappraisal-4B-GRPO (v1, BT reward)
0.581
480
0.537
DeepSeek-R1-671B (single-pass reference)
0.771
240
0.718
Read the CI column as a screening tool, not a precise bound. The 720 contests cluster on only
6 vignettes (n_effective ≈ 6), so the iid Wilson CIs above are optimistic. The honest
vignette-cluster bootstrap CI for this model's headline is [0.781, 0.948] — CI-low 0.781, still
far above the 0.608 project bar and the 0.5 tie line, so the win over GPT-4-0314 on these 6 vignettes
survives clustering. But the CI-low is 0.781, not 0.846 (retracted). This is a specialist trained
on these 6 scenarios against zero-shot generalists, so read it as evidence about reward-model quality
on a narrow task, not a general capability ranking.
On DeepSeek-R1, be precise about what holds. Comparing our cluster CI-low to R1's reference point
(0.771) is not a valid test — that reference-rate comparison is retracted. What does hold is the
head-to-head on the 6 vignettes: this model's reappraisals vs R1-671B's, same scenarios, both
orderings, same judge, wins 0.6729 (n = 240), vignette-cluster CI [0.550, 0.804], CI-low
0.550 > 0.5 — a 4B single-pass model out-reappraising a 671B reasoning model pairwise, in
distribution.
Out-of-distribution: we do NOT beat R1, and this model is at parity off-distribution. The policy is
trained on the same 6 vignettes it is evaluated on, so we tested fresh scenarios it never trained on
(n-gram disjoint). On a fresh cross-family pool this model is at parity-or-below vs a strong open 72B
(Qwen2.5-72B): 0.4875, cluster CI [0.398, 0.579] — the earlier "+6.8pt over the single-reward policy"
claim is retracted (the honest like-for-like gap is within noise). And against DeepSeek-R1-671B
off-distribution our 4B loses: single-pass ~0.39–0.45 (cluster CI-low < 0.5), and best-of-N
test-time compute only tied on the arm-selection pool — a pre-registered confirm on a fresh held-out
pool reverted to 0.375 / 0.378 on both seeds (winner's curse). Six+ approach families were
adversarially audited and each came back negative. This is a genuine capability ceiling: a 4B cannot
out-reappraise R1-671B on fresh scenarios; off-distribution generalization tracks base capacity, not
our training.
Why the committee helps. The gain comes from the heterogeneity, not from simply adding a second
reward model. An RM-heavy version of the same committee (weights 0.7 / 0.3 toward the discriminative
model) scored only 0.760, below the single-reward RMv2 recipe, because down-weighting the generative
model's independent signal collapses the committee back toward a single-reward regime. Weighting the
two mechanisms equally is what buys the improvement: the two scorers make different mistakes, and
rewarding only their agreement penalizes the answers that game either one alone. (Using either
generative reward alone as the training signal collapses the policy; the committee, not the
generative model, is what works.)
Prompting
The model was trained with this system prompt:
You are helping someone reduce a negative emotion they feel in a short interpersonal scenario by
offering an alternative interpretation of the situation (a 'reappraisal' / 'rethinking'). Direct
your response at the person in the scenario in second person. Limit your response to two sentences
maximum. Do not list emotions; output only the reappraisal text.
The user turn is SCENARIO:\n{scenario}\n\nWrite a reappraisal of this scenario in two sentences maximum, addressing the person in second person. Because the base is a thinking model, generations
contain a reasoning trace in <think>...</think> followed by the reappraisal; only the text after
</think> is the answer.
The reward carries a format gate penalizing outputs that are empty or longer than two sentences (so
the policy cannot inflate reward through verbosity). Training was healthy: reward rising, KL bounded
(~0.60), entropy alive, no reward-std collapse.
Data separation
The 6 vignettes' GPT-4-0314 reappraisals and the frontier-opponent answers are held out and used only
for evaluation; training uses the vignette prompts (the task) and the human-rating-based reward models.
Both committee scorers are Qwen-family, distinct from the Llama-3.1-70B evaluation judge, and the
generative model's references are real human reappraisals, never the held-out GPT-4 answers or the
judge's outputs. Any pooled or synthetic training text is n-gram-checked against the held-out eval
answers before use.
Intended use & limitations
Research artifact for the study of RLHF, reward modeling, and computational emotion regulation. It
writes short cognitive reappraisals of negative interpersonal situations. It is not a clinical or
mental-health tool and must not be used as one. Quality is validated only on the narrow reappraisal
task and the evaluation above.
Li, J. Z., Herderich, A., Nair, P., & Goldenberg, A. (2025). Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios. PsyArXiv. https://doi.org/10.31234/osf.io/fzvd8
