Qwen3-0.6B quantized to 2.79-bit weights, then recovered with three stages of
on-policy distillation (K1 reverse-KL as a per-token reward + GRPO) from a
Qwen3-1.7B teacher.
trust_remote_code=True is required, and edgerazor must be installed — weight
quantization is baked in, but A8 activation and KV-cache quantization act on runtime
tensors and need the code path. Without it you get an ImportError (a loud failure,
unlike the latent checkpoints).
GSM8K exceeds the FP baseline. AMC23 recovery vs. the QAT start is significant
(p = 3.9e-4).
⚠️ AMC23 must be measured at avg@16 or higher. The benchmark has 40 problems, so a
single greedy pass has 2.5%/problem resolution and is unusable in this score range — we
once read 12.5% at n=1 where avg@8 gave 4.06%.
⚠️ Code benchmarks are contaminated — do not quote them
The pipeline includes a deliberately contaminated stage that trains directly on
MBPP-test-448 and HumanEval. Its scores (MBPP-448 169/448, HumanEval 58/164) measure
fit to those evaluation sets, not generalization, and must not be reported as
code-generation ability. They are published only for reproducibility of the recipe.
The math benchmarks above are clean: the training data was decontaminated against
AMC23 (18 overlapping items removed) and MATH-500 (12).
How it was made
Phase
Data
Steps
Effect
① unified math OPD
GSM8K + MATH + DAPO-17k, 2500 each
80
GSM8K 45.72 / MATH 25.20 / AMC 4.22
② contaminated code OPD
MBPP-test + HumanEval
120
code up, MATH → 19.60
③ math refresh
same as ①
60
restores MATH, lifts AMC to 4.69
Teacher: Qwen3-1.7B (bf16)
Objective: K1 sampled-token reverse KL as per-token reward, combined with GRPO
policy gradient and a strict \boxed{} task reward. LR 3e-6.
Non-thinking mode. Quantization collapses this model's thinking pathway and six
recovery attempts failed — see
docs/LESSONS.md.
An independent rerun of all three phases from the shipped QAT checkpoint reproduced
four of five benchmarks inside their acceptance intervals (GSM8K 44.28, MATH-500 22.80,
MBPP-448 160, HumanEval 53). AMC23 came out 2.50 against 4.69 — below the interval,
cause not yet established. Recorded openly in docs/RECIPE_3PHASE.md; treat the AMC
figure as the least settled number here.
Quantization scheme
weights
mixed int1.58/int4, block 256, 50% high-precision → 2.79 bit
