The 4B SFT critic from Steer, Don't Solve: Training Small Critic Models for Large Code Agents. It is Qwen3-4B-Instruct-2507 fine-tuned on the same corpus as the main 8B critic, and it is the starting point for the DPO critic Qwen3-4B-Critic-SFT-DPO.

A critic sits next to a frozen coding agent. Every k agent steps it reads the trajectory so far and returns a short structured critique: which error categories it detects, the evidence, a recovery action, the task status, and one line of overall guidance. It steers the agent; it does not write the patch.

All released models and datasets are listed on the organization page. Code and configs are in the critic-training repository.

Where it appears in the paper

Paper location	Row label
Table 1, every agent block	`Qwen3-4B + SFT`
Table 3, corpus ablation	`4B, Qwen+CWM`
Section 3.4 and Figure 3	the SFT checkpoint that DPO starts from

Original run name: qwen3-4b-instruct-2507-full-sft-prm-r2egym-swebench-instructions-k5-cwm-plus-qwen-opus-distill-32k-multiturn. The repository code-critic-model/qwen3-4b-sft-prm holds a byte-identical copy of these weights under the name the DPO runs referenced.

Training data

code-critic-model/critic-sft-cwm-qwen, 6,447 examples.

Tasks: 500 R2E-Gym instances from matplotlib, moto, and sympy, disjoint from SWE-bench Verified.
Agents that produced the trajectories: CWM-32B (500 trajectories, 4,532 examples) and Qwen3-Next-80B-A3B-Instruct (483 trajectories, 1,915 examples).
Teacher: Claude Opus 4.6, queried every 5 agent steps with the high-level prompt.

Training setup

Full-parameter SFT with LLaMA-Factory. The config is finetuning/qwen3_4b_critic_full_sft_l40s_train_multiturn_resumable.yaml in the repository.

Setting	Value
Base model	Qwen/Qwen3-4B-Instruct-2507
Chat template	`qwen3_nothink`
Sequence length	32,768 tokens
Loss	final critique turn only (`mask_history: true`)
Hardware	8 x L40S, per-device batch 1, effective batch 8
Optimizer	AdamW, lr 5e-6, cosine schedule, warmup ratio 0.1
Epochs	3
Precision	bf16

Results

Resolve rate on SWE-bench Verified (500 instances), from Table 1 of the paper, best of k=5 and k=10 per configuration. The untrained base model is included so the effect of SFT is visible.

Coding agent	No critic	Untrained Qwen3-4B-Instruct-2507	+ Qwen3-4B-Critic-SFT	+ Qwen3-4B-Critic-SFT-DPO
Qwen3-32B	8.8	10.2	11.4	14.4
Qwen3-Next-80B-A3B	20.0	20.2	24.2	26.2
GPT-OSS-20B	3.0	6.8	9.8	14.8
GLM-4.7-Flash-30B-A3B	21.6	29.8	35.2	35.8
GPT-OSS-120B (medium reasoning)	20.4	16.8	31.2	34.8
o3-mini	19.0	20.4	27.6	28.2

How to use

Serve with vLLM in bf16 and run an agent through the repository's mini-swe-agent fork, which inserts a critique every k steps.

bash

vllm serve code-critic-model/Qwen3-4B-Critic-SFT \
    --served-model-name Qwen3-4B-Critic-SFT \
    --dtype bfloat16 --max-model-len 65536 --port 8071

bash scripts/run_critic_max150.sh prm_issue_res_instructions_step_aware 5 0 qwen3-80b \
    --prm Qwen3-4B-Critic-SFT --prm-node <vllm-host>:8071 --slice :500 \
    --prefix-dir <path to the matching no-critic run>

The --prm name goes to LiteLLM, which needs a matching entry in mini-swe-agent/configs/litellm_model_registry.json to price the calls. Copy the block for the original run name to a new key Qwen3-4B-Critic-SFT, or serve under the original run name. Without an entry the critic call fails and the agent runs without critiques.

To call the critic directly, take any training record, drop its final teacher critique, and generate. The system message and trajectory encoding in the records are exactly what the model saw in training. A complete snippet is on the Qwen3-8B-Critic-SFT card; only the repo name changes.

Citation

bibtex

@misc{gandhi2026steerdontsolvetraining,
  title={Steer, Don't Solve: Training Small Critic Models for Large Code Agents},
  author={Shubham Gandhi and Yiqing Xie and Atharva Naik and Ruichen Zhu and Carolyn Rose},
  year={2026},
  eprint={2606.21811},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2606.21811}
}

shubhamrgandhi/qwen3-4b-instruct-2507-full-sft-prm-r2egym-swebench-instructions-k5-cwm-plus-qwen

Qwen3-4B-Critic-SFT