Qwen3.5-4B-Soyuz-GRPO-v3 (merged bf16)
GRPO (Group Relative Policy Optimization) fine-tune of Soyuz-4B, where the reward is a
separate Claude Code instance grading each answer against an explicit per-task criteria checklist.
This card documents how the GRPO actually affected the model — including a negative transfer
result, reported honestly.
Setup
- Policy: Soyuz-4B (
Qwen3_5ForCausalLM) + LoRA r=32, on-policy REINFORCE with a group baseline
(G=6 rollouts/task) + KL to the frozen base (LoRA-disabled reference). Custom bare-torch loop.
- Reward:
claude -p (headless Claude Code) scores each rollout passed/total against the
task's explicit criteria → reward in [0,1]. Criteria-checklist grading has much lower variance
than free-form quality scoring (this was the single biggest fix for a stable signal).
- Per-turn recovery shaping: each task is a 2-turn rollout (answer → self-revise). Generation and
judging are interleaved so the turn-2 critique is grounded in the judge's actual reason and is
harsh when the answer is bad ("scored N/10, problems: …, scrap it and rewrite"). Turn-2 reward
=
score + 0.5·(score − score_prev) → rewards climbing out of a bad answer, penalises regression.
- Data: 44 tasks × 2 epochs — algorithms/data-structures, library use (pandas, numpy,
requests, asyncio, SQLAlchemy, pytest, click), real-code fixes (SQL-injection, async race,
FastAPI pagination, vectorize, retry bug, refactor), and creative (micro-fiction, ELI5,
limerick, haiku, dialogue, tone-rewrite).
How GRPO affected the model (all numbers file-backed)
On the trained distribution — clear gain:
| metric | epoch 1 | epoch 2 | Δ |
|---|
| reward (criteria) | 0.700 | 0.757 | +0.057 (27/44 tasks up) |
| turn-0 score | 0.652 | 0.723 | +0.071 |
| recovery (climb from a bad turn-0) | 0.301 | 0.273 | −0.028 |
The recovery skill internalised into the first turn. Recovery fell not because the model got
worse at fixing — but because turn-0 got better, so there were fewer bad starts left to recover from.
The model learned to do it right the first time (e.g. product-description turn-0 0.23→0.60,
cover-letter 0.3→0.93, ELI5 0.0→0.73). Biggest gains were on the hard tail (semver-compare +0.35,
weighted-choice +0.27, streaming-median +0.22, SQL-injection-fix +0.18).
Negative transfer to agentic terminal tasks. Tested on the terminal-bench-2 tasks the base model
partially solved, GRPO-v3 regressed: tasks-with-any-pass dropped 11/11 → 4/11 (e.g.
hf-model-inference 4/5→0/5, openssl 3/5→0/5, sqlite 3/5→0/5). General single-response coding/creative
GRPO does not transfer to — and hurts — multi-step shell-agent behaviour. To improve agentic
terminal performance you must run GRPO on the agentic rollouts themselves (verifier reward), not on
single-shot coding questions.
App generation. Works well as a single-shot generator (one prompt → full index.html), which
is the model's strength; it fails inside a heavy agent loop (drowns in planning). Suppress its
<think> ("output only the HTML, do not plan") or it spends its token budget reasoning and never
emits the code.
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3m = AutoModelForCausalLM.from_pretrained("AlexWortega/qwen35-4b-soyuz-grpo-v3", dtype=torch.bfloat16, device_map="cuda")
4tok = AutoTokenizer.from_pretrained("AlexWortega/qwen35-4b-soyuz-grpo-v3")
Best for single-response coding / writing / rewriting. For multi-step shell agents, prefer the
base Soyuz-4B.
Related
