MathReasoner-Mini-1.5b

🚨 We recommend using this model for math problems. It works better to ask the question in English. We do not advise using it for other tasks.

Introduction

This is a reasoning model trained on top of Qwen2.5-Math-1.5B-base and has been trained in Three stages (SFT, DPO and GRPO), to progressively improve mathematical reasoning with structured outputs on GSM8K dataset, a benchmark targeting math problems.

Evaluation (GSM8K Pass@1 Zero shot)

ModelPass@1	Math Accuracy %
Base Qwen2.5-Math-1.5B	54%
After SFT	67.5%
After SFT + DPO	70%
After SFT + DPO + GRPO (MathReasoner-Mini-1.5b)	~83.7%

Evaluation was run on GSM8K test with: temperature=0.3, top_p=1.0,

XML structured output accuracy improved from 71% (Qwen2.5-1.5B-base) to 99% (MathReasoner-Mini-1.5b)

MathReasoner's pass@8 math accuracy is 94.1% showing that there still more improvement possible on scaling RL Accuracies shown above take structured output format in consideration, requiring reasoning to be enclosed within think tags and numerical answer between answer tags

Training Stages

Stage 1 — Supervised Fine-Tuning (SFT)

• Dataset: curated GSM8K subset with self-verified generations
• Epochs: 10
• LR: 3e-6
• Batch size: 4
• Gradient accumulation: 4
• Only correct & well‑formatted CoT samples used to minimize model entropy

Stage 2 — Direct Preference Optimization (DPO)

• Dataset: ~1,000 preference pairs
• Mostly hard pairs (correct vs incorrect)
• Some soft preferences (shorter correct CoT)
• For each GSM8K problem, 4 samples were generated → chosen = correct, rejected = incorrect
• Epochs: 3
• β = 0.1, LR = 3e‑6

Stage 3 — GRPO Reinforcement Learning

• Steps: 400
• Loss type: DAPO
• Rollouts per prompt: 4
• Gradient accumulation : 8
• Custom reward: format strictness + correctness
• vLLM enabled rollout, TRL trainer

Prompt Template

Code for inference

Scope for Improvement

• Train with more diverse math data leading to better generalization. Currently all the three stages were trained using the entire or subsets of GSM8K trainset. Including diverse datasets will lead to further improvement and the model will be able to solve harder problems. This path can be pursued in the future if more compute resources are available
• Optimizing current training by using curriculum learning with temperature and rollout scaling especially in RL phase, as done by PhysicsWallahAI/Aryabhata-1.0
• Exploring methods like Model merging and on-policy Distillation from larger models

Contact