A 3.43B dense research checkpoint made by selecting 8 of the parent MoE's 256 experts per layer, concatenating them into a single FFN (d_dense = 8 × 512 = 4096), and distilling from Qwen/Qwen3.6-35B-A3B for 550M tokens of FineWeb-Edu.

⚠️ Experimental — heavily undertrained, do not expect a usable model

550M distillation tokens is roughly 1/25th of what a comparable pretrained model sees. It is beaten by Qwen3.5-0.8B, a model a quarter its size, on average downstream accuracy:

	params	WikiText PPL	avg downstream¹
parent Qwen3.6-35B-A3B	35B	6.25	—
Qwen3.5-4B	4B	8.65	—
Qwen3.5-0.8B	0.8B	15.71	0.545
this model	3.43B	12.12	0.536

¹ arc_easy / arc_challenge / hellaswag / lambada / piqa, acc_norm where available.

It does beat the 0.8B on arc_easy (0.621 vs 0.617) and on perplexity, and the scaling law was still clean when the hardware was reclaimed (PPL ∝ T^−0.206, extrapolating to 4B-level perplexity at ~3B tokens). Nobody should treat that extrapolation as a result.

Loading

Loads in stock transformers 5.x with no custom code — it is emitted as a Qwen3_5MoeForCausalLM with num_experts=1, which is a plain dense FFN as a function.

python

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("senorperez/qwen3.6-35b-a3b-dense-3.4b", dtype="bfloat16")

For speed, dense_fastpath.py in the repo below skips the now-vestigial MoE dispatch (bit-identical, 1.18× on forward).

Details

Replication and extension of "Pruning and Distilling Mixture-of-Experts into Dense Language Models". Two findings from the work, both in the repo:

Selecting experts by marginal routing frequency weighted into a log-det diversity objective beats the paper's activation-weighted conditional probability by 27.8% PPL (72/72 windows, p=4e-22) on this 256-expert parent.
Never reuse data when distilling on cached logits. A second epoch cost 12.8pp of arc_easy accuracy (z=8.9) while perplexity moved 0.08% (p=0.41).

Code, artifacts and open questions: https://github.com/sootaugur/moe2dense (private).