AWQ-calibrated mixed-precision quantization of DeepSeek-V4-Flash-0731
(304B MoE, 43 layers, 256 routed experts/layer, Hyper-Connections, HISA
attention) for Apple Silicon via oMLX.
Routed experts carry the aggressive quantization; everything else stays at
8-bit.
component
precision
switch_mlp.gate_proj / up_proj
2-bit, group size 128
switch_mlp.down_proj
3-bit, group size 64
attention, shared experts, embeddings, head
8-bit, group size 64
MTP / DSpark drafter (-mtpq)
2-bit gs128 / 3-bit gs64 — same recipe
Backbone: 278.6B quantized params · 101.4 GB · BPW 2.91oMLX build with quantized MTP drafter: 108.2 GB total (drafter 7.04 GB,
down from 10.86 GB unquantized)
The MTP/DSpark speculative drafter is quantized to the same 2/3-bit recipe,
not left at native precision. BPW above is quoted for the backbone weights;
the drafter is additional and is not part of the 278.6B count.
Recommended sampling
temperature = 1.0
top_p = 0.95
min_p = 0.05
For long-context work, enable KV cache quantization at 4-bit or 6-bit —
this model's context cost is dominated by KV, and 6-bit is essentially free
quality-wise while 4-bit buys substantially more usable context.
reasoning_effort supports low, high, max (not medium/xhigh — the
chat template rejects those). Pass via chat_template_kwargs.
Performance (M5 Max, 128 GB, oMLX, MTP enabled)
Single request
test
TTFT (ms)
TPOT (ms)
prefill tok/s
gen tok/s
E2E (s)
peak mem
pp 1024 / tg 128
1,650.8
19.9
620.3
50.8
4.2
102.4 GB
pp 4096 / tg 128
5,927.1
20.3
691.1
49.7
8.5
103.4 GB
pp 8192 / tg 128
12,823.2
21.7
638.8
46.4
15.6
104.5 GB
pp 16384 / tg 128
30,535.1
21.4
536.6
47.0
33.3
106.7 GB
Generation speed is essentially flat with context — TPOT stays at ~20–22 ms
from 1k to 16k, i.e. ~47–51 tok/s throughout. The MTP/DSpark drafter is doing
its job.
Batched
batch
gen tok/s
prefill tok/s
avg TTFT (ms)
E2E (s)
speedup
1×
50.8
620.3
1,650.8
4.2
1.00×
2×
36.9
466.5
4,389.7
11.3
0.73×
4×
54.6
467.4
8,620.2
18.1
1.07×
8×
75.2
470.5
16,935.1
31.0
1.48×
Two things worth knowing
Build the custom kernels for long context. If oMLX reports
dsa_indexer_scores/dsa_topk_indices unavailable (glm_moe_dsa extension not built), the sparse-attention indexer falls back to a slow MLX path and
long-context prefill is several times slower than it needs to be. Rebuild
with OMLX_WITH_CUSTOM_KERNEL=1. Generation speed is unaffected — this is
purely a TTFT issue, and it is the largest single speedup available.
Memory and context. KV growth is modest (~4 GB from 1k to 16k), but the
weights alone occupy ~102 GB, so headroom runs out well before very long
contexts on a 128 GB machine. Use 4-bit or 6-bit KV cache quantization for
long-context work — 6-bit is effectively free, 4-bit buys substantially more
usable context.
Measured results
Benchmarks via oMLX's own omlx.eval harness (n=200 MMLU/GSM8K, n=164
HumanEval), MTP enabled:
Benchmark
v1 (single-turn calib)
this build (v2)
MMLU
34.5%
61.5%
GSM8K
77.0%
93.5%
HumanEval
72.6%
87.2%
The v2 recalibration used a broader, multi-turn agentic calibration set plus
down_proj at group size 64. Both changes were necessary — the v1 calibration
was single-turn only and left the model badly mismatched on real conversation
structure.
How much does quantization cost?
Measured against the unquantized teacher on identical questions
(likelihood-scored MMLU, n=200, paired):
MMLU
teacher (native)
81.5%
this build
64.5%
gap
−17.0pp (95% CI [10.1, 23.9])
Paired: 42 questions the teacher answers and this build misses, versus 8 the
other way. McNemar p = 3.06e-06.
This is an honest statement of what 2/3-bit costs on knowledge recall. It is
not a reason to avoid the model — it is a 304B model running on a single
Apple Silicon machine at 2.91 BPW, and GSM8K (93.5%) and HumanEval (87.2%)
remain close to ceiling. Knowledge-recall breadth is where the compression is
paid for.
How this recipe compares (all measured, same 200 questions)
Every number below is likelihood-scored MMLU on the identical stratified
sample, scored by the same code, paired question-by-question. Absolute values
are not interchangeable with the generation-scored table above; the
comparisons are what matter.
build
MMLU
size
vs this build (paired)
teacher (native mxfp4/mxfp8)
81.5%
155.0 GB
+42/−8, p=3.06e-06
this build (AWQ 2/3-bit)
64.5%
108.2 GB
—
+ DWQ distillation
66.5%
108.2 GB
+24/−20, p=0.65
same recipe, no calibration
28.5%
92 GB
emits gibberish
1. Quantization costs ~17 MMLU points, and that gap is real
Against the native teacher on identical questions: 42 questions the teacher
answers that this build misses, versus 8 the other way. McNemar p=3.06e-06.
This is an honest statement of what 2/3-bit costs on knowledge recall — and it
is unclaimed headroom, not something any method below recovered.
2. AWQ calibration is load-bearing, not a refinement
A plain round-to-nearest build at the same bits and group sizes scores 28.5%
— barely above MMLU's 25% chance floor — and generates literal gibberish
('}<?iger}<?codeline...'). Weight reconstruction error was verified at 44.5%
RMS, exactly what honest 2-bit RTN produces, so this is not a broken build: at
2-bit, without activation-aware scaling, the model simply collapses.
Calibration is the difference between a working model and noise.
3. DWQ distillation added nothing
Distilling the quantized student toward the native teacher (100 steps,
expert scales only, agentic-weighted calibration) cut held-out KL by ~89–91%
and moved MMLU not at all (p=0.65; near-symmetric 24 gained / 20 lost — churn,
not learning). Two loss variants came out statistically identical (p=0.75).
A cautionary result worth repeating: KL-to-teacher over-credits badly. A 91%
improvement in the distillation objective bought zero measurable capability.
Judge distillation on exact-answer or behavioural evals, never on the loss.
Method note
Teacher and low-bit builds were measured with a streaming likelihood scorer
(reap_stream/mmlu_streamed.py) that reads one block at a time — ~8 GB
resident — so a 155 GB model can be benchmarked on a 128 GB machine. Prompting
is oMLX's own 5-shot MMLU format and stratified sample.
Notes
oMLX-specific. Uses oMLX's custom deepseek_v4 loader; not portable to
generic MLX/GGUF runtimes as-is.
MTP/DSpark drafter is quantized to the same 2-bit gs128 / 3-bit gs64
recipe (7.04 GB, saving 3.82 GB over native). Safe to quantize aggressively:
DSpark uses exact rejection sampling, so every drafted token is verified by
the main model — drafter error costs accept-rate (speed) only, never output
correctness.
Native tool calling is DSML (｜DSML｜ markup), not the generic
<tool_call> convention.
Sibling model
DeepSeek-V4-Flash-0731-DWQ
— distillation-trained variant of this build. See its card before using it:
it shows no measured capability gain over this model and has a suspected
early-turn-termination issue.
