Qwen3-4B — Vanilla PPO on Symbolic Tool-Calling (cmp-vanilla-ppo-4gpu-v1)
Qwen/Qwen3-4B-Instruct-2507 trained with vanilla PPO (Schulman et al. 2017 —
clipped surrogate, learned value head, trainer-side GAE) on the deterministic
symbolic tool-calling benchmark from the GRPO-vs-PPO-under-rollout-compaction
study (prime-rl fork: ChinmayK0607/blog-rl, branch feat/vanilla-ppo).
This is the uncompacted real-PPO baseline that study was missing: its
compacted PPO (rollout-reward critic targets, no GAE) collapsed to
0.25/0.14 pass@4; this run shows actual PPO nearly matches GRPO.
Results (val = 72 held-out mixed tasks, greedy)
| step | 10 | 60 | 80-100 | 140 | 150 |
|---|
| val reward | 0.556 | 0.764 | 0.000 | 0.903 | 0.972 |
Reference (same env, Phase-A regime, 8xH100): full GRPO 1.000, compacted
GRPO 0.986, segment-normalized GRPO 0.986.
Notable dynamics: entropy collapse around step 80 made a corrupted tool-call
token the greedy argmax (val 0 for ~20 steps, 2-turn episodes) — the policy
self-recovered by step 110 and finished at 0.972 with more efficient
solutions (turns 14.3 -> 11.8). Critic explained variance rose from a cold
start (0.01 -> 0.07+) — no warm-start needed, unlike the compacted-PPO arm.
Training setup
- Algorithm: prime-rl
ppo — per-token terminal-reward stream, trainer-side
GAE (gamma=1.0, lambda=0.95), clipped surrogate (eps=0.2), clipped value
loss (coef 0.5), entropy_coef=0 (a 0.01 arm is a follow-up), LR 1e-6
- Data: symbolic-curriculum-v2 — 588 train / 72 val pass@4-mixed tasks
(frozen-model pass@4 sweep over deterministic pools, keep tasks solved
1-3 of 4 times), depth 3-5, max 24 turns, 32k-token episodes
- Batch 64 rollouts x group 8, 150 steps, temp 0.7, 4096 completion tokens
- Hardware: 4x RTX PRO 6000 Blackwell 96GB (2 inference / 2 trainer), ~35 s/step
- W&B: https://wandb.ai/krishnapg2315/blog-rl/runs/5c4f977de99343c7b44edfe8d05fb5c0
Files
model.safetensors — step-150 policy, value head stripped (vLLM-loadable)value_head.safetensors — the trained PPO value head (value_head.weight,
[1, hidden]) for critic warm-start experiments
(trainer.model.ppo_value_head_init)
Caveats
- Step-150 weights. The greedy policy is functional (0.972 val) but the run
showed a mid-training degeneracy window — prefer temp>0 sampling or verify
greedy behavior on your tasks.
- Tool-call format: Qwen3/hermes (
<tool_call> JSON blocks).
