You can deploy this model and start earning money today!

Mahesh111000/
hanabi-qwen3-32b-lora-step75

Name: hanabi-qwen3-32b-lora-step75
Availability: InStock

Views

No views yet

Hanabi Qwen3-32B LoRA — step 75 (RL / GRPO via prime-rl)

RL training state for resuming on another machine. Base model: Qwen/Qwen3-32B (bf16).

Note: 97.5% of checkpoint_step_75/ is the frozen base model stored as fp32 master weights (32.76B x 4 bytes = 131 GB). Only ~3.2 GB is unique. If you don't need the exact optimizer state and step counter, just use adapter/ (1.07 GB) with a fresh optimizer — it holds all the learning, and it lets you change the learning rate (a resume does not).

path	size	purpose
`adapter/`	1.1 GB	LoRA adapter (r=32, alpha=64). The entire trained delta. PEFT/vLLM loadable.
`checkpoint_step_75/`	126 GB	prime-rl DCP checkpoint — model + optimizer + scheduler, exact resume.
`code/`	small	the custom reward env, config, and launcher (not on PyPI — required).

Training state at step 75

Checkpoint saved after step 74 completed; resuming runs step 75 next.
LoRA r=32, alpha=64, lr 1e-5, all 7 linear projections.
batch 256 = 16 problems x 16 rollouts, max_tokens 26624, seq_len 32768.
GPUs: 4 inference (TP=4) + 4 trainer (cp=4), 8x A100-SXM4-80GB.
Reward = deduction_score + move_reward (range [0,2]).

Observed at steps 65–75

step	reward	grad norm	entropy	mismatch KL
65	1.4244	0.0008	0.5994	0.0029
66	1.3072	0.0007	0.5936	0.0029
67	1.4862	0.0007	0.6049	0.0030
68	1.4189	0.0009	0.6000	0.0030
69	1.3700	0.0008	0.6041	0.0032
70	1.4810	0.0007	0.5889	0.0033
71	1.5295	0.0007	0.5794	0.0037
72	1.6249	0.0007	0.5823	0.0040
73	1.6333	0.0006	0.5819	0.0038
74	1.4401	0.0007	0.5918	0.0038
75	1.4628	0.0007	0.5847	0.0039

Mean reward over 67–75 is 1.494, against 1.424 at step 65. Treat that as flat, not as progress: the step-to-step spread (1.31–1.63) is far wider than the gap between the endpoints, and a 9-point window at this noise level cannot resolve a trend. Entropy drifts down slightly (0.60 → 0.58) and mismatch KL up (0.0029 → 0.0039), both consistent with a policy that is slowly concentrating — worth watching, not yet a problem.

is_truncated = 0.0 throughout, so nothing is hitting the 26,624-token cap. copy_rate ~0.55 and flat, i.e. the degenerate "copy the previous belief state" policy is not being learned.

The small grad norm is expected, not a fault: the loss is token-mean-reduced over ~3.9M tokens per step and advantages are raw within-group deviations (no std normalisation), while Adam is scale-invariant so updates remain ~lr-sized.

Resume on a new server

bash

git clone https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl && git checkout 16e747c25e81c5c90bd860c68a588f7b151353a6
uv sync

huggingface-cli download Mahesh111000/hanabi-qwen3-32b-lora-step75 --local-dir ./hanabi-state

# custom reward env (required — not on PyPI)
uv pip install --python .venv/bin/python --no-deps -e ./hanabi-state/code/hanabi-deduction

cp ./hanabi-state/code/rl_32b.toml configs/hanabi_deduction/
mkdir -p outputs/checkpoints
cp -r ./hanabi-state/checkpoint_step_75 outputs/checkpoints/step_75

VLLM_ATTENTION_BACKEND=FLASHINFER \
  uv run rl @ configs/hanabi_deduction/rl_32b.toml --ckpt.resume-step 75

Dataset: Mahesh111000/Hanabi-init-30turns (or point dataset_path at a local copy).

Expect: the first step after any (re)start is a cold start while the async pipeline refills (Async Level climbs from -2 to 0) — measured 148 min, then 85, then settling to ~60–92 min per step. The trainer needs ~40 min and hides entirely inside the rollout phase, so step time is inference-bound. Needs 8x80 GB GPUs.

VLLM_ATTENTION_BACKEND=FLASHINFER in the command above is not cosmetic — it is the single biggest win found, measured over full training steps on 8x A100-SXM4-80GB:

backend	steps	mean min/step
FlashAttention	52–66	124.5
FlashInfer	67–75	79.7

1.56x end-to-end. A standalone decode benchmark predicted only 1.26x; the extra gain comes from the trainer overlapping better once rollouts finish sooner, which a generation-only benchmark cannot see. Since decode here is KV-bandwidth-bound (256 KiB/token, ~5,400-token must-attend prompt = ~47% of KV), attention-kernel efficiency dominates step time.

Full code, configs and profiling tooling: https://github.com/kaousheik-26/hanabi_latest_train_32b

Gotchas baked into the config (learned the hard way)

dist_timeout_seconds = 18000 — the default 600s kills the trainer on the cold-start step.
[orchestrator.client] timeout = 7200 — the default 1200s silently drops long rollouts.
ckpt.keep_last = 2 — without it, checkpoints fill the disk (~126 GB per save).
[trainer.tokenizer] name pinned — prime-rl otherwise resolves it from a stale default.
Resume always inherits the checkpoint's LR; the config's lr is ignored on resume. To change the learning rate you must start fresh from merged weights, not resume.
prime-rl's outputs/weights/ are NOT merged (bitwise identical to base) — don't rely on them. clean_lora_state_dict() (trainer/lora.py:220) discards the adapter. Use merge_lora.py.
A checkpoint named step_N is the state to resume at N, i.e. saved after step N-1 finished.
vLLM prefix caching is inert here: verifiers sets prompt_logprobs=True, which makes vLLM set skip_reading_prefix_cache and bypass the cache entirely (0 queries, not 0 hits).

Mahesh111000/hanabi-qwen3-32b-lora-step75