When making the Andy series of models, I have been using PPO techniques to train models.
But as the bleeding edge of small models is becoming clear, reasoning models are the winners.
So, in order to learn the nuances of training models, I decided to train a small 3B model using GRPO techniques instead of PPO.
The base model was Qwen2.5 3B, it is very smart as is, and even smarter with reasoning.
<think>
--reasoning content here--
</think>
<answer
--answer content here--
</answer>
Similar to the XML reasoning format but changed to use DeepSeek-R1 / QwQ thinking blocks.
