2026.RA.Fairness-GRPO — LoRA adapters
These adapters did not achieve their objective. They are published as the artifact of a preregistered
negative result, and should not be used as "a fairer negotiator" — measured against the untrained base model
they are worse on welfare and more selfish in canonical bargaining games.
What these are
LoRA adapters (r=32, α=64, 87.3M trainable parameters) on Qwen/Qwen3-8B, trained with GRPO in six-party
self-play negotiation against a smoothed logarithmic Nash-welfare reward. Two arms, each with a checkpoint
ladder at steps {5, 10, 15, 20, 24}:
2026.RA.Fairness-GRPO-lam0 — λ=0, each seat paid its own normalized outcome.2026.RA.Fairness-GRPO-lam1 — λ=1, every seat paid the table's mean outcome.
The reward is R_i(λ) = (1−λ)·g(z_i) + λ·mean_j g(z_j) where z_i is party i's normalized surplus and
g(z) = log z above a threshold ε = 0.01, continuing linearly below it. It is text-blind: computed from
the engine's scoring of the closed deal, never from generated text.
Measured performance (this is the part that matters)
Held-out evaluation on 24 games never trained on (48 instance clusters, 960 episodes per cell), trained minus
untrained Qwen3-8B, 95% instance-cluster bootstrap:
| endpoint | λ=0 step 24 | λ=1 step 24 | direction |
|---|
| normalized Nash welfare | −0.185 [−0.225, −0.145] | −0.109 [−0.145, −0.070] | worse |
| deal rate (baseline 0.817) | −0.252 | −0.149 | worse |
| below-threshold agreements | −0.087 [−0.127, −0.050] | −0.079 [−0.124, −0.030] | better |
| NNW among individually-rational deals | +0.014 [−0.012, +0.040] | −0.040 [−0.067, −0.015] | λ=1 worse |
| Gini among IR deals | −0.008 [−0.025, +0.009] | +0.023 [+0.010, +0.036] | λ=1 worse |
Interpretation. The one thing training bought is a reduction in below-threshold agreements — deals a party
signs that are worse for it than walking away. It bought that substantially by not agreeing at all: deal
rate fell by 0.25 (λ=0) and 0.15 (λ=1). Overall welfare therefore fell. Splitting the fairness metrics into all
episodes versus only-fully-individually-rational episodes separates the arms: λ=0's welfare loss is a
composition effect (fewer deals, not worse ones — every among-IR interval spans zero), whereas λ=1 struck
measurably less fair deals among the deals it did strike.
Out-of-distribution behaviour got worse. Seated in the classic ultimatum game, which the policy never
saw in training, the untrained model proposes an accepted 60/40 split. Both trained arms propose 96.7/3.3,
accepted 100% of the time — a move toward the self-interested subgame-perfect prediction and away from the
fair split, consistent across every seed. Divide-the-dollar shifts the same way more weakly (max share
0.794 → 0.844 / 0.894 against a 1/n reference of 0.333).
Guards. The trained policy is not exploitable — seated against five computable rational agents its own
capture is unchanged (λ=0 −0.001 [−0.036, +0.036]). A transcript audit found no degenerate strategy: the
trained policy actually negotiates more (turns per episode 20.5 → 25.1) and agrees less.
Training-time metrics (on the training bank, so not evidence of generalization): reward R_table +0.61 /
+0.57, worst-off seat's g +2.86 / +3.33, log-prob drift from the frozen base +0.14 / +0.27 nats per token,
0 fabricated turns at any step.
Caveat that bounds every number above
The run is 24 GRPO steps per arm, 12% of the preregistered 200-step budget, because a step costs ~17 minutes
in this environment (an episode is ~20 sequential co-stepping model calls). Over-training is already visible —
welfare worsens between step 10 and step 24 — but nothing here rules out a different trajectory at full budget.
Treat this as a well-instrumented short run.
Reproducing
1python -m grpo.train --lam 0 --steps 24 --groups 4 --k 8 --micro-batch 6 --max-new-tokens 384 \
2 --lr 5e-5 --checkpoint-steps 5 10 15 20 24 --bank instances_grpo_train_v1 \
3 --out runs/lam0 --transcript-every 5 \
4 --wandb-project rational_agents_fairness_grpo --wandb-group fairness-grpo-pilot
Evaluation fleet and analysis: python -m grpo.launch_eval --checkpoint "lam0_step24=runs/lam0/checkpoint-24" --out-dir sbatch_grpoeval --submit, then python -m grpo.analyze_eval --baseline "<baseline>_primary_s*" --trained "lam0_step24=<run>_primary_s*" --out eval.json.
Usage
1from peft import PeftModel
2from transformers import AutoModelForCausalLM
3base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16")
4model = PeftModel.from_pretrained(base, "siddharthmb/2026.RA.Fairness-GRPO-lam0", subfolder="checkpoint-24")
Links
- Dataset (training steps, eval contrasts, rollout transcripts):
siddharthmb/2026.RA.Fairness-GRPO
- Weights & Biases: project
rational_agents_fairness_grpo,
group fairness-grpo-pilot — λ=0 iedvonxu,
λ=1 a6wqb6l8
- Cluster artifacts:
/nlp/scr/siddharth/ii_mats/rational_agents/fairness_grpo/ (adapters, step logs, 256
sampled rollout transcripts); evaluation run directories /nlp/scr/siddharth/ii_mats/rational_agents/grpoeval_*
- Full write-up: research note
0023-fairness-grpo-pilot.md in the project repository, with the preregistration
it was run against and the compliance table.
