The Qwen 2.5 0.5B model was adapted to interface with a recursive calculator environment that supports addition, subtraction, multiplication, and division.
The agent generates structured tool calls in XML and YAML format, which are then executed by the calculator.
After receiving the computed result from the tool, it formulates a final human-readable response.
✅ Key Achievements
Training Method: GRPO, using a hybrid reward signal combining LLM-as-a-judge feedback and programmatic verification.
Evaluation Accuracy:
Before RL: 0.6%
After RL: 34%
Absolute Gain: +33.4 pts
Training Cost: $18 (£13.47) on 8x RTX6000 Ada GPUs
Total Training Time: ~3 hours
🧪 Evaluation Dataset
The evaluation dataset consists of synthetically generated arithmetic problems designed to be difficult for humans to solve without a calculator. Questions include nested operations and real-world phrasing diversity.
This model was trained using parts of the Verifiers framework for structured reinforcement learning. If you use this model or build upon this work, please consider citing:
@article{brown2025verifiers,
title={Verifiers: Reinforcement Learning with LLMs in Verifiable Environments},
author={Brown, William},
year={2025}
}
