You can deploy this model and start earning money today!

npatel121/
qwen2-0.5b-grpo-gsm8k

Name: qwen2-0.5b-grpo-gsm8k
Availability: InStock

Qwen/Qwen2-0.5B-Instruct

finetune

apache-2.0

Views

No views yet

Qwen2-0.5B-Instruct, GRPO-fine-tuned on GSM8K

Qwen/Qwen2-0.5B-Instruct fine-tuned with a from-scratch GRPO (Group Relative Policy Optimization) implementation — no TRL/RLHF library, just a custom training loop — on the GSM8K math reasoning dataset. Produced as part of a take-home engineering exercise; not a production model.

Training setup

1000 GRPO iterations, 16 sampled completions ("chains") per question, group-relative-mean advantage baseline.
Reward = correctness (+2 / -1) + integer-format + strict-XML-format + soft-XML-format + XML-tag-count shaping rewards.
KL penalty (Schulman approximation, beta=0.04) against a frozen reference copy of the same base model.
LR 5e-6, linear warmup over the first 18% of steps, AdamW, gradient clipped to norm 0.1.
1x H100 SXM, ~1h50m wall-clock.

Results

GSM8K held-out eval accuracy (74-example split): 1.7% → 38.5% over training, monotonically increasing (no late-training degradation observed for this model size — contrast with the 1.5B variant trained alongside it).

Output format

The model is trained to respond in:

<reasoning>
...step-by-step reasoning...
</reasoning>
<answer>
INTEGER
</answer>

Caveats

Single run, no repeated seeds.
Eval split is small (74 examples) — treat the accuracy trend as indicative, not a precise benchmark number.
This checkpoint is the final training step; not separately checkpoint-selected for best eval accuracy (though for this model size, final and peak accuracy coincide).

npatel121/qwen2-0.5b-grpo-gsm8k