MLX (Apple Silicon) conversion of deepseek-ai/DeepSeek-V4-Flash-0731,
built from mixed-4_8bit (4-bit routed experts, 8-bit elsewhere) with 25% of routed experts pruned by REAP saliency.
DeepSeek-V4-Flash is a 304B-parameter sparse-MoE model. Its deepseek_v4
architecture is not implemented in any released runtime — not transformers,
and not mlx-lm (which carries deepseek, _v2, _v3, _v32). This repo bundles
a from-scratch MLX port under deepseek_v4_mlx/, validated against DeepSeek's own
reference implementation.
Perplexity is teacher-forced over wikitext-2-raw-v1 test, 199,485 tokens in 195
windows of 1024 — identical windows for every build. Differences are established by
a paired bootstrap over shared windows, which removes window-difficulty variance:
independent intervals overlap almost entirely and settle nothing, while the paired
interval is ~10x tighter. Every gap quoted here is significant at 95%.
Why mixed-4_8bit is the recommended build
The routed experts are 98.1% of all quantized parameters, and they ship natively
as FP4. Keeping them at 4 bits while raising everything else — attention,
embeddings, shared experts, about 5B parameters — to 8 bits costs 3 GB and
recovers 56% of the gap between 4-bit and 8-bit. Per parameter, the non-expert
weights are roughly 50x more quantization-sensitive.
Two variants were built, measured, and not published:
group-size 32 (matching FP4's native 32-value scale blocks) is 0.5% worse
than group 64 and 18 GB larger. Finer scaling improves raw tensor reconstruction
but not prediction — the mismatch is that FP4's levels are non-uniform while
affine's are evenly spaced, which no group size fixes.
3-bit experts collapse the model entirely (perplexity 1.4e7, degenerate token
loops). 4-bit sits just above a cliff for this checkpoint, not in the middle of a
gentle curve.
REAP-pruned builds
REAP (Cerebras, arXiv:2510.13999) drops the
lowest-saliency routed experts, where saliency is the mean of
router_weight x ||expert_output|| over the tokens routed to that expert — its
actual contribution, not how often it fires. The router renormalizes over survivors.
Only 40 of the 43 layers are pruned. The first three route through a fixed
tid2eid token-id table with no router to renormalize, so dropping an expert there
would leave table entries pointing at nothing. They keep all 256, and the config
records two expert counts which the loader resolves per layer.
Saliency was profiled over 150k tokens of English prose, ten other languages and
real code — deliberately disjoint from the evaluation set, verified by 32-gram
overlap (0 of 4687), so pruning decisions are not fit to the text they are judged on.
Build
Experts kept
Saliency retained
ppl
vs unpruned
mixed-4_8bit
256
100%
6.1262
—
REAP25
192
96.2%
6.4025
+4.5%
REAP37
161
91.5%
6.9634
+13.7%
REAP50
128
83.9%
7.7520
+26.5%
REAP25 is the one to take — its perplexity is statistically indistinguishable
from plain 4-bit (paired ratio 1.016, CI [0.998, 1.038], crossing 1.0) at 33 GB less.
Note how poorly saliency retention predicts damage: 96.2% retained still costs 4.5%,
and the last 12 points of retention cost 22 points of perplexity. Treat it as a
screen for whether pruning is viable, not as an estimate of what it will cost.
⚠️ Loading requires the bundled loader
pip install mlx mlx-lm transformers
python
1from deepseek_v4_mlx.load import load
2from deepseek_v4_mlx.generate import greedy_generate, load_tokenizer
34model, args = load("/path/to/this/repo")5tok = load_tokenizer("/path/to/this/repo")# bundled — no download6ids = tok("The capital of France is")["input_ids"]7print(tok.decode(greedy_generate(model, args, ids, max_new_tokens=32)))
The tokenizer is bundled in this repo, so nothing is fetched at runtime. (The base repo
shipped no tokenizer at first — only encoding/encoding_dsv4.py, a prompt-string renderer;
DeepSeek has since published one. DeepSeek-V3's tokenizer, used here in the interim, produces
identical ids for text and identical special-token ids, so builds made either way agree.)
What this port implements
Mechanism
Notes
Hyper-Connections
the residual stream carries 4 parallel copies, mixed by a Sinkhorn-normalized combination matrix
Hash-routed experts
the first 3 MoE layers pick experts from a tid2eid[vocab, 6] table keyed on token id — routing independent of context
Learned KV compression
gated pooling over spans, alternating ratio 4/128, overlapping windows at ratio 4
Sparse-attention indexer
scores compressed spans, keeps the top 512 for attention to read
MLA + grouped output LoRA
single shared KV vector; o_groups: 8 block-diagonal output projection
The source checkpoint is doubly pre-quantized — FP8 e4m3 with e8m0 128x128
block scales for most matmuls, FP4 e2m1 packed two-per-byte for routed experts — so
conversion dequantizes both formats before requantizing. Both decode paths are
bit-exact against an independent implementation.
Dropped: the multi-token-prediction head and DSpark (speculative decode over
5-token blocks in the last three layers). Both hang off the reference's forward_spec
path and are unused for ordinary generation.
wo_a is kept unquantized: it is a block-diagonal projection the model reshapes by
group, and a packed quantized weight cannot be reshaped that way. Costs ~2% of build size.
