Hanabi Qwen3-8B — RL step 100 (20k data, deduction-weighted)
Qwen3-8B trained with RL on Hanabi belief-state tracking + move rating. This is the
best-validating checkpoint produced across every run in this project (held-out
val reward 1.3429).
Lineage — read this before using the adapter
This model is the result of two RL stages, applied in sequence:
| stage | platform | data | steps | notes |
|---|
| 1 | Tinker (hosted) | 3.5k (Hanabi-init-30turns) | 110 | LoRA r=32 a=32, batch 512 |
| 2 | prime-rl (local, 8xA100) | 20k (Hanabi-merged-without_init-30turns) | 110 | deduction weight 2.0, 2 trainer + 6 inference |
merged/ is the standalone model (stage 1 + stage 2 both merged in) — use this.
adapter/ is the stage-2 LoRA only. Its base is NOT vanilla Qwen/Qwen3-8B; it is the
stage-1 merged model. Applying it to stock Qwen3-8B produces a broken model. If you only want
the adapter, you must reconstruct the stage-1 base first.
Results
Held-out val reward (32 problems, 1 rollout each; reward = deduction_score + move_reward, max 2.0):
| step | 50 | 60 | 70 | 80 | 90 | 100 |
|---|
| val | 1.084 | 0.918 | 1.113 | 1.077 | 1.107 | 1.343 |
Train reward: 1.086 (steps 0-9) -> 1.142 (steps 100-109).
For context, stage 1 alone peaked at val 1.008. The stage-2 changes (5.7x more data +
deduction weighted 2x vs move) are what moved it to 1.343.
Honest caveats
- Game score is not yet measured for this checkpoint. The deduction-task reward is a
single-turn proxy. In a previous experiment, a checkpoint that improved this reward by +0.13
scored worse at actual multi-turn Hanabi (6.67 -> 6.32 mean final_score). Improved val here
does not imply better play until a game eval confirms it.
- The training task always supplies the previous belief state in the prompt, so the model is
never trained to consume its own belief output. That exposure mismatch is the leading
explanation for why stage-1 gains did not transfer to multi-turn play.
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2m = AutoModelForCausalLM.from_pretrained("Mahesh111000/hanabi-qwen3-8b-rl-step100",
3 subfolder="merged", dtype="bfloat16", device_map="auto")
4t = AutoTokenizer.from_pretrained("Mahesh111000/hanabi-qwen3-8b-rl-step100", subfolder="merged")
Or serve directly:
1huggingface-cli download Mahesh111000/hanabi-qwen3-8b-rl-step100 --local-dir ./ckpt
2vllm serve ./ckpt/merged --tensor-parallel-size 4
Reward definition
reward = deduction_score + move_reward, each in [0, 1].
- deduction_score — per-card agreement with the programmatic belief state. Cards whose
belief changed since the player's last turn are weighted 1.0; carried-over cards 0.1. This
asymmetry matters: without it, echoing the previous belief state scores 0.58; with it, 0.17.
- move_reward — the dataset's rating for the chosen action, clipped to [0, 1].
Reference points: random legal move 0.33, oracle (argmax) 0.99.
