You can deploy this model and start earning money today!

bullerwins/
DeepSeek-V4-Flash-GGUF

Name: DeepSeek-V4-Flash-GGUF
Availability: InStock

gguf

deepseek-ai/DeepSeek-V4-Flash

Views

No views yet

DeepSeek-V4-Flash GGUF quantizations

High-quality GGUF quantizations of DeepSeek-V4-Flash, produced with llama.cpp.

Important model detail

The source HF model is already natively quantized:

Dense weights: fp8, converted to Q8_0/BF16 in the reference GGUF
Routed MoE experts: fp4 MXFP4

Most parameters are in the routed experts, so normal Q4_K_M is not a useful upload target here. It is only marginally smaller than the reference, and it was dominated by better measured options. The final set below keeps only the quants that make practical sense.

Final files

File	Size	BPW	KLD	Notes
`DeepSeek-V4-Flash.IQ2_S.gguf`	82.0 GiB	2.48	0.151700	Smallest option. Experts use the IQ2_S recipe; dense tensors are protected at Q8_0.
`DeepSeek-V4-Flash.IQ3_XXS-AS.gguf`	95.4 GiB	2.88	0.111648	Asymmetric small-mid option. Down/up experts use iq3_xxs; gate experts use iq2_xxs.
`DeepSeek-V4-Flash.IQ3_XXS.gguf`	106.1 GiB	3.21	0.077500	Standard 3-bit option. All experts use iq3_xxs; dense tensors are protected at Q8_0.
`DeepSeek-V4-Flash.IQ3_XXS-L21.gguf`	124.8 GiB	3.77	0.053911	High-quality compressed option. Final 21 expert layers stay native MXFP4; earlier experts use iq3_xxs.
`DeepSeek-V4-Flash.IQ3_XXS-L39.gguf`	140.9 GiB	4.26	0.043924	Near-reference compact option. Final 39 expert layers stay native MXFP4; first four expert layers use iq3_xxs.
`DeepSeek-V4-Flash.IQ4_XS.gguf`	144.4 GiB	4.36	0.038715	Best non-reference standard quant found. Experts use IQ4_XS; dense tensors are protected at Q8_0.
`DeepSeek-V4-Flash.Q8_0.gguf`	145.6 GiB	4.40	0.000000	Highest fidelity reference. Dense Q8_0/BF16 plus native MXFP4 experts.

KLD is the mean KL divergence of each quantized model against the Q8_0/reference logits on a held-out evaluation split. Lower is better. The reference-vs-self KLD is effectively zero.

Recommendations

Best quality: DeepSeek-V4-Flash.Q8_0.gguf. This is the exact GGUF reference.
Best near-reference non-reference quant: DeepSeek-V4-Flash.IQ4_XS.gguf.
Compact near-reference option: DeepSeek-V4-Flash.IQ3_XXS-L39.gguf.
High-quality compressed option: DeepSeek-V4-Flash.IQ3_XXS-L21.gguf.
Standard 3-bit option: DeepSeek-V4-Flash.IQ3_XXS.gguf.
Smaller mid option: DeepSeek-V4-Flash.IQ3_XXS-AS.gguf.
Smallest option: DeepSeek-V4-Flash.IQ2_S.gguf.

Measurement methodology

Reference: converted GGUF with Q8_0/BF16 dense tensors and native MXFP4 experts.
Calibration: imatrix computed on a training split disjoint from evaluation data.
Evaluation: KLD measured only on the held-out evaluation split against the reference logits.
Metric used for selection: KLD only.

Key findings

Q4_K_M is not included because it was almost the same size as the reference and was dominated by IQ4_XS and IQ3_XXS-L39.
iq3_xxs worked unusually well for these MXFP4-source expert tensors, beating larger nominal 3-bit variants in the tested recipes.
Keeping later expert layers at native MXFP4 gives a strong size-quality ladder; L21 and L39 were kept as the most useful points from that ladder.
IQ4_XS was the best standard near-reference quant found, with lower KLD than the removed Q4_K_M file.

Running

Example:

llama-cli -m DeepSeek-V4-Flash.IQ3_XXS.gguf -ngl 999 -fa on -c 8192

Flash attention (-fa on) is recommended when supported.

Reproducibility notes

The quantization recipes used llama.cpp llama-quantize --allow-requantize with imatrix guidance and tensor-type override files. The hash-routing tensor ffn_gate_tid2eid must stay i32. Small F32 tensors such as ape, hc_*, attn_sinks, and output_hc_* must stay native for low-bit imatrix-requiring ftypes.

External Antirez comparison

The Antirez DeepSeek-V4-Flash GGUFs were also evaluated against the same Q8_0/reference KL base and held-out split. The MTP side GGUF was skipped because it is not a standalone full model.

File	Size	KLD	Result
`DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf`	80.8 GiB	0.184799	Slightly smaller than our IQ2_S, but higher KLD.
`DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf`	80.8 GiB	0.183665	Slightly smaller than our IQ2_S, but higher KLD.
`DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf`	90.9 GiB	0.158371	Dominated by our IQ2_S: larger and higher KLD.
`DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf`	153.3 GiB	0.035836	Low KLD, but larger than the Q8_0 reference, so not a practical upload target.
`DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf`	153.3 GiB	0.035170	Low KLD, but larger than the Q8_0 reference, so not a practical upload target.

Combined graphics: kld_size_comparison_with_antirez.png and kld_size_comparison_with_antirez.svg.

bullerwins/DeepSeek-V4-Flash-GGUF