Checkpoints from RDAN-GRPO (Rubric Decoupled Advantage Normalization GRPO), a rubric-based RL method for instruction following, trained on Qwen3-4B-Instruct-2507.

Each folder is a complete, independently loadable checkpoint.

What it is for

Instructions carry two kinds of requirement: objectively checkable ones ("start with 'My answer'", "under 500 words") and matters of judgment (is it well written). Reward judged quality directly and the policy learns that good prose earns credit while the hard constraints become optional. RDAN-GRPO prevents that.

Method

Sample G = 8 responses per prompt, score each twice, and normalize the two scores over different populations:

A_out  = (r_out  - mu)   / sigma      over all G responses
A_proc = (r_proc - mu_C) / sigma_C    over constraint-satisfying responses only
A      = A_out + A_proc

The second denominator is the whole idea. A response failing any hard constraint is excluded from the process statistics entirely, so quality is a tiebreaker among compliant responses and never a substitute for compliance. It also keeps learning alive once the outcome channel saturates and carries no gradient.

Results

Full official datasets, greedy decoding, each benchmark scored by its own evaluator.

Checkpoint	IFEval	IFBench	MulDimIF	MATH-500	GPQA
base	82.99	30.95	57.17	87.60	55.05
`step-000100`	85.21	34.35	68.92	88.40	59.09
`step-000200`	87.06	35.71	72.58	88.80	58.08
`step-000300`	86.69	34.69	73.25	89.60	57.58
`step-000400`	86.69	35.37	74.08	86.40	58.59
`step-000420`	86.88	33.67	74.67	87.20	55.56

Gain over base at step 200: +4.07 IFEval, +4.76 IFBench, +15.42 MulDimIF.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "beingamanforever/Qwen3-4B-RDAN-GRPO"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="step-000200", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="step-000200")

messages = [{"role": "user", "content": "Write exactly two friendly sentences about rain."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=512)[0], skip_special_tokens=True))

Trained with thinking disabled; evaluate the same way.

Training


Hardware	2x A100-80GB
Framework	ROLL, FSDP2 actor with vLLM rollout
Throughput	6.1 min/step
Wall clock	~41 h to step 400
Prompts	18,096
Rubrics	125,278 (77,615 deterministic, 47,663 judged)
Judge	`qwen/qwen3.7-flash`, three-point scale
Prompts per step	64
Samples per prompt (G)	8
Global batch	256
Prompt / response length	1,280 / 2,048 tokens
Learning rate	1e-6
PPO clip (low / high)	0.2 / 0.27
KL penalty	none