Hanabi Qwen3-32B LoRA — step 10 (RL / GRPO via prime-rl)
RL training state for resuming on another machine. Base model: Qwen/Qwen3-32B (bf16).
Note: 97.5% of checkpoint_step_10/ is the frozen base model stored as fp32 master
weights (32.76B x 4 bytes = 131 GB). Only ~3.2 GB is unique. If you don't need the exact
optimizer state and step counter, just use adapter/ (1.07 GB) with a fresh optimizer — it
holds all the learning, and it lets you change the learning rate (a resume does not).
Contents
| path | size | purpose |
|---|
adapter/ | 1.1 GB | LoRA adapter (r=32, alpha=64). The entire trained delta. PEFT/vLLM loadable. |
checkpoint_step_10/ | 126 GB | prime-rl DCP checkpoint — model + optimizer + scheduler, exact resume. |
code/ | small | the custom reward env, config, and launcher (not on PyPI — required). |
Training state at step 10
- Checkpoint saved after step 9 completed; resuming runs step 10 next.
- LoRA r=32, alpha=64, lr 1e-5, all 7 linear projections.
- batch 256 = 16 problems x 16 rollouts,
max_tokens 26624, seq_len 32768.
- GPUs: 4 inference (TP=4) + 4 trainer (cp=4), 8x A100-SXM4-80GB.
- Reward =
deduction_score + move_reward (range [0,2]).
Observed through step 101
Train reward (256 rollouts/step, so SEM ~0.02 — this metric is resolvable):
| window | mean train reward |
|---|
| steps 0–50 | ~1.28 |
| steps 67–81 | 1.499 |
| steps 82–101 | 1.557 |
Recent steps: 95 1.6600 · 96 1.5811 · 97 1.5674 · 98 1.4306 · 99 1.6806 ·
100 1.5396 · 101 1.6122. Grad norm ~0.0006, entropy ~0.578, mismatch KL ~0.005.
Train reward has genuinely risen (~1.28 → ~1.56, roughly 15 SEM). Mean completion length
grew from ~6,145 tokens (base model) to ~9,865, i.e. the policy reasons ~1.6x longer.
Held-out val, however, is flat at baseline:
| step | 0 | 70 | 80 | 90 | 100 |
|---|
| val reward | 1.2401 | 1.3656 | 1.1854 | 1.2400 | 1.2669 |
Read this metric with care — and do not over-read it in either direction. [orchestrator.val]
is num_examples = 32, rollouts_per_example = 1, temperature unset (server default, i.e.
stochastic). So each point is 32 one-shot samples, giving a noise band of roughly ±0.12 at
2σ — every reading above overlaps every other one and the baseline. A 32-sample stochastic
eval cannot resolve the size of change a LoRA run produces. It is not evidence of learning, and
it is not evidence of failure; it is under-powered.
To actually settle generalization, run an offline eval instead: all 64 held-out examples x 8
rollouts (~512 samples, SEM ~0.02) on the base model vs this checkpoint, using the same rubric.
is_truncated = 0.0 throughout, so nothing is hitting the 26,624-token cap. copy_rate ~0.55
and flat, i.e. the degenerate "copy the previous belief state" policy is not being learned.
Entropy drifts down (0.60 → 0.578) and mismatch KL up (0.0029 → 0.005), both consistent with a
policy slowly concentrating — worth watching, not yet a problem.
The small grad norm is expected, not a fault: the loss is token-mean-reduced over ~3.9M tokens
per step and advantages are raw within-group deviations (no std normalisation), while Adam is
scale-invariant so updates remain ~lr-sized.
Resume on a new server
1git clone https://github.com/PrimeIntellect-ai/prime-rl.git
2cd prime-rl && git checkout 16e747c25e81c5c90bd860c68a588f7b151353a6
3uv sync
4
5huggingface-cli download Mahesh111000/hanabi-qwen3-32b-lora-step10 --local-dir ./hanabi-state
6
7# custom reward env (required — not on PyPI)
8uv pip install --python .venv/bin/python --no-deps -e ./hanabi-state/code/hanabi-deduction
9
10cp ./hanabi-state/code/rl_32b.toml configs/hanabi_deduction/
11mkdir -p outputs/checkpoints
12cp -r ./hanabi-state/checkpoint_step_10 outputs/checkpoints/step_10
13
14VLLM_ATTENTION_BACKEND=FLASHINFER \
15 uv run rl @ configs/hanabi_deduction/rl_32b.toml --ckpt.resume-step 10
Dataset: Mahesh111000/Hanabi-init-30turns (or point dataset_path at a local copy).
Expect: the first step after any (re)start is a cold start while the async pipeline refills
(Async Level climbs from -2 to 0) — measured 148 min, then 85, then settling to ~60–92 min
per step. The trainer needs ~40 min and hides entirely inside the rollout phase, so step time is
inference-bound. Needs 8x80 GB GPUs.
VLLM_ATTENTION_BACKEND=FLASHINFER in the command above is not cosmetic — it is the single
biggest win found, measured over full training steps on 8x A100-SXM4-80GB:
| backend | steps | mean min/step |
|---|
| FlashAttention | 52–66 | 124.5 |
| FlashInfer | 67–81 | 81.6 |
| FlashInfer | 67–101 (35 steps) | 91.1 |
~1.37x end-to-end, measured over 35 steps. Note the earlier step-65 README quoted 1.56x from
the first 15 FlashInfer steps (81.6 min); with 20 more steps the mean regressed to 91.1 min, so
1.37x is the honest figure and the 15-step window was optimistic. A standalone decode
benchmark predicted 1.26x, so a modest overlap benefit is real, just smaller than first measured.
Since decode here is KV-bandwidth-bound (256 KiB/token, ~5,400-token must-attend prompt = ~47%
of KV), attention-kernel efficiency dominates step time.
Gotchas baked into the config (learned the hard way)
dist_timeout_seconds = 18000 — the default 600s kills the trainer on the cold-start step.[orchestrator.client] timeout = 7200 — the default 1200s silently drops long rollouts.ckpt.keep_last = 2 — without it, checkpoints fill the disk (~126 GB per save).[trainer.tokenizer] name pinned — prime-rl otherwise resolves it from a stale default.- Resume always inherits the checkpoint's LR; the config's
lr is ignored on resume. To
change the learning rate you must start fresh from merged weights, not resume.
- prime-rl's
outputs/weights/ are NOT merged (bitwise identical to base) — don't rely on
them. clean_lora_state_dict() (trainer/lora.py:220) discards the adapter. Use merge_lora.py.
- A checkpoint named
step_N is the state to resume at N, i.e. saved after step N-1 finished.
- vLLM prefix caching is inert here:
verifiers sets prompt_logprobs=True, which makes vLLM
set skip_reading_prefix_cache and bypass the cache entirely (0 queries, not 0 hits).
