ReMix (Reincarnating Mix-policy Proximal Policy Gradient) is a general approach to enable on-policy RFT methods like PPO and GRPO to leverage off-policy data.

[2025/08/30] We release Remix-R1-Distilled-Qwen-1.5B and Remix-R1-Distilled-Qwen-7B, the checkpoints of the ReMix-PPO reported in our paper. We also release a ready-to-use [evaluation scripts] that reproduces all benchmark results.

[2025/07/11] We release paper Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model presenting ReMix, a simple yet effective approach that equips on-policy proximal policy gradient methods (eg. PPO and GRPO) with off-policy replay to slash LLM reasoning-finetuning costs while setting new SOTA math performance and, for the first time, exposing how off-policy RL shapes the emergence of reasoning behaviors.

Below are the pass@1 results of our models versus all baselines on five widely-used benchmarks. For each entry we report two figures in the format “greedy / temperature-0.1”. On the two small, high-variance datasets (AIME and AMC) we average 32 runs at temperature 0.1

In the table, “cost” refers to the total rollout data volume generated during training. Rollout data volume defined as the total number of rollouts generated by the model during the training process. It reflects the total amount of model inference, which is usually \textbf{the dominant source} of computational cost during training.

If you find ReMix helpful, please cite us.