qwen3-4b-gdn-hybrid-gkd50-baseline
GKD 50/50 baseline — stage-3 trainer with half of each step's loss tokens off-policy (teacher traces), reverse KL on both halves. Concave dose-response: MATH think 0.782, AIME 0.167/0.150, vt@32k 0.984 — ~85-95% of OPD's reasoning at 50% on-policy — while inheriting the full reverse-KL drift (PPL 16.44): drift is objective-driven, not data-driven.
Part of a study converting full-attention Qwen3-4B into a GDN (gated
DeltaNet) hybrid (27 of 36 layers converted, uniform 1:4 retention) and
recovering capability via staged distillation, culminating in on-policy
distillation (OPD): student rollouts scored per-token by the frozen teacher
under reverse KL. Checkpoints are stock Qwen3NextForCausalLM — load with
AutoModelForCausalLM (transformers >= 4.57). vLLM serving needs a small
registry shim (all-dense MoE + tied-embeddings fixes) from the project repo.
Pipeline: surgery/init -> stage-1 hidden-state alignment (100M tok) ->
stage-2a forward-KL KD @4k (600M) -> stage-2b long-context KD @32k (294M) ->
stage-3 OPD (200M gen tokens, horizon 512->16k, token-keyed LR).
Teacher/reference: Qwen/Qwen3-4B (thinking mode).