Qwen3-4B distilled from Qwen3-30B-A3B — On-policy 5k (continual² , prompts 35k–40k)
Second continual on-policy KD round for a Qwen3-4B student toward the Qwen3-30B-A3B (MoE) teacher, on 5,000 math prompts from openthoughts3_math_50k (positions 35k–40k).
The student for this run is
RockToken/qwen3_30b_a3b_to_4b_onpolicy_5k_src30k-35k_cont, so the effective training exposure of this checkpoint is:
- Off-policy KD on 20k teacher responses (positions 0–20k of
openthoughts3_math_50k)
- On-policy KD on 10k prompts (positions 20k–30k) → produced the 10k model
- On-policy KD on 5k prompts (positions 30k–35k) → produced the
_cont model
- On-policy KD (this run) on 5k prompts (positions 35k–40k) → this checkpoint
Stages 2–4 share the same on-policy pipeline; only the seed weights and prompt slice differ.
Models
| Role | Model |
|---|
| Student | Qwen3-4B-Instruct-2507 (after stages 1–3 above) |
| Teacher | Qwen/Qwen3-30B-A3B-Instruct-2507 (MoE, ~3B active) |
enable_thinking=False throughout.
Training data
- Source:
openthoughts3_math_50k (math split of OpenThoughts-3)
- Slice used here: prompts 35k–40k → RockToken/openthoughts_prompt_math_5k_src35k-40k
- 5,000 single-user-turn prompts, mean length ~261 chars
Note on data overlap. The parent openthoughts3_math_50k[30k:40k] window contains 418 internal duplicate prompts across its two halves, so ~8.4% of the 5k prompts here also appeared in the prior stage-3 slice (30k–35k). This is an upstream data artifact of the OpenThoughts-3 math split, not from the slicing; the file itself is still exactly parent[5000:10000].
Training setup
Framework:
KDFlow — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.
Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).
Key hyperparameters (identical to stages 2 and 3)
| Group | Value |
|---|
| Backend | fsdp2, bf16, gradient ckpt on |
| Epochs | 1 |
| Train batch | 4 (micro 1) |
| Learning rate | 2e-6, cosine, warmup 5% |
| KD ratio | 1.0 |
| KD loss | reverse KL (rkl) |
| KD algorithm | vanilla_kd |
| Temperature (KD) | 1.0 |
| Rollout engine | SGLang, TP=2, 1 engine |
| Rollout batch | 2 prompts × 4 samples/prompt |
generate_max_len | 8000 |
prompt_max_len | 800 (total max_len 8800) |
| Sampling | temperature 1.0, top-p 1.0 |
| Teacher | TP=4, sleep/wakeup enabled |
Intended use
Research on distillation dynamics and the effect of successive continual on-policy rounds on the same student. Domain: math (OpenThoughts-3 math split).
Limitations
- Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
enable_thinking=False — this student does not emit <think> traces.- Chain-of-continual training may drift capabilities relative to earlier checkpoints in the chain; see the wandb curves for loss dynamics.
