DeepSeek-V4-Flash-0731 DSpark drafters (GGUF, for mainline llama.cpp)
A DSpark speculative-decoding drafter extracted from the official
DeepSeek-V4-Flash-0731 weights,
in the
dflash architecture that mainline llama.cpp uses.
Which file to use
| file | size | routed experts | decode @12K [*] |
|---|
DeepSeek-V4-Flash-0731-DSpark-Q2K-Q4K.gguf | 7.83 GiB | Q2_K gate/up, Q4_K down | 16.5 t/s |
DeepSeek-V4-Flash-0731-DSpark-BF16.gguf | 10.15 GiB | MXFP4 (converter default) | 15.2 t/s |
Use the Q2K-Q4K file. It is 2.3 GiB smaller and ~8% faster. Decode on this hardware is
bandwidth-bound, so a smaller drafter streams fewer bytes per draft step; the accuracy lost by
dropping the experts to 2-3 bpw costs less than the bandwidth saved. ffn_down is kept at Q4_K
because it is the error-sensitive projection - a Q3_K variant measured 15.9 t/s, and an
IQ2_XXS/Q4_K variant 15.8 t/s.
[*] 11,929-token prompt, --spec-draft-n-max 2, same target and method as the table below.
Confirmed over a 768-token generation: 16.0 t/s vs 14.9 t/s at n-max 3.
It pairs with any 0731 target GGUF. The drafter is a separate model, so a target quantized
before MTP support existed - which is most of them, including files with no mtp.* tensors at
all - still gets speculation with no rebuild.
Target model used for testing
All measurements below use
antirez/deepseek-v4-gguf,
specifically
DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.gguf
(90.9 GiB). It is a good reference target for two reasons:
- Its importance matrix was computed on the 0731 weights themselves - 202,100 chunks of
chat-template-rendered prompts, covering the routed expert tensors.
- It embeds a chat template. The 0731 release ships no Jinja template at all (only the
encoding_dsv4 Python scripts), so quants that rely on a converter's bundled template may
behave differently for tool calls and reasoning.
That repo also has a plain IQ2XXS-...-0731 variant (80.8 GiB) and a larger
Q4KExperts-F16HC-F16Compressor-F16Indexer-...-0731 (153 GiB) if you have the memory.
Note this target contains no mtp.* tensors - it was built before MTP export existed. That
is precisely the case this drafter is for: speculation without rebuilding the target.
Requirements
Mainline llama.cpp with commit 596a579 ("DeepseekV4 MTP + DSpark", #25784, merged 2026-08-02)
or newer. Nothing else is needed to use this file; the extra PR below was only needed to
build it.
Usage
1llama-server \
2 -m DeepSeek-V4-Flash-...-fixed-0731.gguf \ # e.g. from antirez/deepseek-v4-gguf
3 -md DeepSeek-V4-Flash-0731-DSpark-Q2K-Q4K.gguf \
4 --spec-type draft-dspark --spec-draft-n-max 2 \
5 -ngl 99 -ngld 99 -fa on
Draft depth
--spec-draft-n-max 2 is the optimum here; the llama.cpp default of 3 is worse. Measured on the
Q2_K/Q3_K drafter, 11,929-token prompt:
n-max | 2 | 3 | 4 | 6 |
|---|
| decode t/s | 15.9 | 14.1 | 13.8 | 13.4 |
Deeper drafting loses: DeepSeek-V4's MTP head does not stay accurate past ~2 tokens, so the extra
drafts are rejected and their verification cost is wasted.
How the Q2K-Q4K file was built
Mainline's converter repacks DeepSeek-V4's routed experts to MXFP4 and llama-quantize
refuses MXFP4 as input, so there is no supported path to a smaller drafter. Producing this file
needed a converter patch adding --dspark-expert-type, which dequantises the packed E2M1 experts
to f32 (or Q8_0) instead of repacking them:
1python convert_hf_to_gguf.py DeepSeek-V4-Flash-0731 --dspark \
2 --target-model-dir DeepSeek-V4-Flash-0731 \
3 --dspark-expert-type q8_0 --outfile dspark-q8-intermediate.gguf # 19.7 GiB
4
5llama-quantize --allow-requantize \
6 --tensor-type "ffn_gate_exps=q2_k" --tensor-type "ffn_up_exps=q2_k" \
7 --tensor-type "ffn_down_exps=q4_k" \
8 dspark-q8-intermediate.gguf DeepSeek-V4-Flash-0731-DSpark-Q2K-Q4K.gguf Q8_0
The patch is not upstream; you do not need it to use this file, only to rebuild it.
Measured results
AMD Radeon 8060S (gfx1151, Strix Halo), 128 GB unified memory, ROCm 7.2.4, HIP backend,
mainline llama.cpp 0b14b87. Target as described above. Temperature 0, f16 KV cache.
Swap disabled, one discarded warmup request per server, distinct filler text per depth so no
prompt-cache prefix is shared between measurements.
Speedup vs context depth (--spec-draft-n-max 2)
Decode, tokens/s. Every cell measured on the same machine and target, swap disabled, one
discarded warmup request per server, and distinct filler text per depth so no prompt-cache
prefix is shared between measurements. -c 49152 for the first three rows, -c 131072 for the
last two.
| prompt tokens | mainline | mainline + drafter | gain | hipCUB [1] | hipCUB + drafter |
|---|
| cached (4) | 13.61 | 19.47 | +43% | 13.50 | 19.31 |
| 11,929 | 9.84 | 14.18 | +44% | 12.14 | 16.30 |
| 24,530 | 9.38 | 13.65 | +45% | 11.34 | 15.04 |
| 39,628 | 9.48 | 13.35 | +41% | 10.67 | 13.12 |
| 84,829 | 7.45 | 10.95 | +47% | 9.05 | 10.95 |
The drafter gives +41-47% at every depth tested, and the gain does not fade as context grows
- at 85K it is the largest measured. Draft acceptance was 0.67-0.76 across the range.
Note the cached row: 13.61 vs 13.50 without the drafter, 19.47 vs 19.31 with it. With a prompt
that short the hipCUB code path is never reached, so those columns are a control - they should
be identical, and they are.
About the hipCUB columns
They are a separate, unofficial AMD patch [1], included because the interaction turns out to be
depth-dependent:
- Without the drafter it helps everywhere: +23% at 12K, +21% at 24K, +13% at 40K, +22% at 85K.
- With the drafter it helps only up to ~32K: +15% at 12K and +10% at 24K, but -2% at 40K and
0% at 85K (10.95 either way).
The likely reason is that speculation cuts the number of target forward passes, so the indexer's
CPU fallback stops being the limiting factor at depth and memory bandwidth over the 90 GiB of
weights dominates instead. If you run this drafter, the patch is worth having for mid-length
contexts and is neutral beyond that; if you run the target without a drafter, it helps throughout.
[1]
argsort/
top_k above 1024 columns have no GPU path on HIP, so they fall back to the CPU
backend - for DeepSeek-Sparse-Attention models that puts V4's per-token lightning-indexer
top_k
on the CPU, and the cost grows with context depth (visible as ~416% CPU on mainline at depth
versus ~100% with the patch, the latter being the backend's normal busy-wait). Enabling hipCUB
(rocPRIM's CUB-compatible API) restores the GPU path. Branch:
https://github.com/singulared/llama.cpp/tree/hipcub-port - not upstream, feature-detected so
builds without
hipcub-dev are unchanged, and
-DGGML_HIP_NO_HIPCUB opts out.
Note on -ffast-math (2026-08-09). Enabling hipCUB is safe on mainline, which compiles
ggml-hip with -funsafe-math-optimizations and deliberately not -ffast-math (see the comment
in ggml/src/ggml-hip/CMakeLists.txt). If you build a fork that adds -ffast-math — ROCmFPX
does — hipCUB's radix sort is miscompiled on gfx1151: DeviceRadixSort/DeviceSegmentedSort
return out-of-range values for most ncols > 1024, which corrupts the indexer's token selection
or faults in GET_ROWS. Reproduced standalone with no ggml involved: identical hipCUB call,
-O2 correct, -O2 -ffast-math gives 3072 bad indices at ncols=1536; -O3 alone is fine.
In that case compile argsort.cu and top-k.cu with -fno-fast-math. Verified on this branch
rebased onto master 69bf643: all sizes correct.
Choosing --spec-draft-n-max (short prompt, -c 32768)
| n-max | decode | draft acceptance |
|---|
| none (baseline) | 14.03 t/s | - |
| 1 | 17.73 t/s | 0.801 |
| 2 | 20.43 t/s | 0.759 |
| 3 | 16.04 t/s | 0.591 |
| 4 | 16.10 t/s | 0.540 |
| 6 | 13.75 t/s | 0.439 |
n-max 2 is the knee. Acceptance falls monotonically as drafting goes deeper, and past 2 the
extra verification costs more than the accepted tokens save; by n-max 6 the gain is gone.
KV cache: f16 measured fastest here (20.61 vs 17.06 q8_0 vs 17.76 q4_0 t/s on one prompt), and
V4's MLA already keeps the cache small, so there is little to save by quantizing it. Those three
figures are from a single prompt at one context length - treat them as provisional.
Memory, measured at -c 131072 on a 128 GB machine: the target alone occupies ~92 GiB of GTT
(~21 GiB free), and target + drafter ~104 GiB (~11 GiB free). It fits, but with little room -
run with swap disabled if you are benchmarking, otherwise memory pressure degrades throughput
silently rather than failing.
Contents
81 tensors, 3 blocks, 10.15 GiB. The converter fixes the layout - MXFP4 experts, Q8_0 attention
and shared experts, BF16/F32 router, norms, sinks and hyper-connections - so it is not
re-quantizable (llama-quantize rejects mxfp4 input). Precision outside the experts is
deliberately high: drafter precision drives acceptance.
How it was built
1python convert_hf_to_gguf.py <fp8-checkpoint-dir> \
2 --dspark --target-model-dir <same-dir> --outtype bf16 \
3 --outfile DeepSeek-V4-Flash-0731-DSpark-BF16.gguf
--target-model-dir is mandatory: the drafter needs the target's tokenizer.
--dspark was added by PR
#26452, merged as
3581ba0, so current master has it. It is needed because the 0731 checkpoint ships
three
mtp.* blocks (matching
dspark_target_layer_ids: [40, 41, 42]) while
config.json declares
num_nextn_predict_layers: 1 - so plain
--mtp fails with
ValueError: Unexpected DeepSeek-V4 MTP layer 1.
Credits
DeepSeek for the weights (MIT),
am17an for the llama.cpp DSpark implementation and the
--dspark export path,
antirez for the target quant and its
imatrix, used for every measurement here.
