On-policy distillation (Agarwal et al., Thinking Machines writeup): the student generates rollouts on math prompts, the teacher scores every student token, and the per-token reward is the negative student–teacher log-probability gap (reverse KL). Implemented with SkyRL's examples/train/on_policy_distillation recipe (GRPO trainer with a pass-through advantage and the teacher in the ref-model slot).

Student: Qwen/Qwen3-1.7B with enable_thinking=false
Teacher: Qwen/Qwen3-4B-Instruct-2507
Prompts: DAPO-Math-17k (no ground-truth rewards used — the teacher signal is the only supervision)
Training: 10 steps × (128 prompts × 8 samples), max 8192 generated tokens, lr 1e-5, importance-sampling policy loss
Hardware: 4× NVIDIA L40 (48GB), FSDP + vLLM via SkyRL

Results (AIME 2024, pass@8, temperature 1.0 / top-p 0.7)

step	0 (this base)	5	10 (this model)	15
pass@8	6.7%	43.3%	50.0%	56.7%

A note for practitioners: with reverse-KL distillation, teacher and student must have matched termination behavior. A thinking-mode teacher, or a base-model student, each cause length collapse (the student learns to never emit EOS). Non-thinking instruct models on both sides trained stably.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("lecunyin/Qwen3-1.7B-OPD-Math", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("lecunyin/Qwen3-1.7B-OPD-Math")

messages = [{"role": "user", "content": "Find the sum of all positive integers n such that n^2 + 12n - 2007 is a perfect square. Put your final answer in \\boxed{}."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tokenizer.decode(model.generate(inputs, max_new_tokens=4096)[0]))