The Qwen 2.5 3B model has been enhanced to interact with a recursive calculator environment that supports four basic arithmetic operations.
The agent generates structured tool calls in both XML and YAML format, enabling precise execution of complex expressions.
After the calculation is performed by the environment, the model formulates a final human-readable answer.
✅ Key Achievements
Training Method: GRPO, using a hybrid reward signal combining LLM-as-a-judge feedback (Claude-3.5-Haiku) and programmatic verification.
Evaluation Accuracy:
Before RL: 27%
After RL: 89%
Absolute Gain: +62 pts
Training Cost: $23.50 (£17.55) on 4x A100 (80GB) GPUs
Total Training Time: ~3 hours
🧪 Evaluation Dataset
The evaluation dataset consists of synthetically generated arithmetic problems designed to be difficult for humans to solve without a calculator. Questions include nested operations and real-world phrasing diversity.
This model was trained using parts of the Verifiers framework for structured reinforcement learning. If you use this model or build upon this work, please consider citing:
@article
{brown2025verifiers,
title={Verifiers: Reinforcement Learning with LLMs in Verifiable Environments},
author={Brown, William},
year={2025}
}
