You can deploy this model and start earning money today!

npatel121/
qwen2-7b-grpo-gsm8k

Name: qwen2-7b-grpo-gsm8k
Availability: InStock

safetensors

qwen2

grpo

reinforcement-learning

gsm8k

math

openai/gsm8k

Qwen/Qwen2-7B-Instruct

finetune

apache-2.0

Views

No views yet

Qwen2-7B-Instruct — GRPO fine-tuned on GSM8K

Qwen/Qwen2-7B-Instruct fine-tuned with GRPO (DeepSeek-style, no external RL library) on GSM8K for 1000 iterations, as part of a scaling-laws comparison across 0.5B/1.5B/7B model sizes. See npatel121/qwen2-0.5b-grpo-gsm8k and npatel121/qwen2-1.5b-grpo-gsm8k for the smaller models in the same comparison.

Training setup

Trained via 2-way data-parallel GRPO (torchrun --nproc_per_node=2 DDP) on a rented 4x H100 NVL instance — each rank held a policy+frozen-reference GPU pair. Getting a 7.6B-parameter model to fit required reducing two hyperparameters from the 0.5B/1.5B baseline, due to real memory constraints at this scale (see the training repo's scaling_analysis.md for the full debugging story):

Hyperparameter	0.5B / 1.5B baseline	This 7B run
`num_chains` (GRPO group size)	16	4
`max_completion_length`	786	400
`gradient_accumulation_steps`	4	8
`learning_rate`	5e-6	5e-6 (unchanged)
`kl_weight_beta`	0.04	0.04 (unchanged)

This is a real deviation from the other two models in the comparison, not just a different model size — treat cross-model comparisons as directional rather than a fully controlled experiment.

Results

Eval accuracy at step 0 (before any GRPO update): 75.3% — a strong base model already.
Eval accuracy at step 900 (last available eval point — the training loop's eval trigger never fires on the literal final step): 92.9%, which is also the highest value seen across the whole run.
Unlike the 1.5B run in this same comparison (which peaked at step 500, 72.5%, then declined to 55.4% by step 900), this 7B run shows no overtraining decline — accuracy oscillates in a fairly tight 87-93% band from step 100 onward and ends at its peak. This published checkpoint (step 1000, the only one saved) is very likely close to the best accuracy achieved during the run, unlike the 1.5B checkpoint published alongside it.
Whether the lack of a decline is a genuine effect of model scale or a byproduct of the smaller num_chains/max_completion_length forced by memory constraints is not resolved by this single run — see scaling_analysis.md in the training repo for the full discussion.

Usage

Standard GSM8K-style prompting: the model was trained to respond in a strict <reasoning>...</reasoning><answer>INTEGER</answer> XML format (see the training repo's system prompt in rl_datasets.py).

What's included

Model weights only (bf16 safetensors) + tokenizer. Optimizer state (training_state.pt, needed only to resume training via --continue_train, not for inference) is not included.

npatel121/qwen2-7b-grpo-gsm8k