A custom mixed quant of DeepSeek V4-Flash (284B MoE, 13B active) sized so the entire model + 256K-512K of KV cache fits on one 96 GB card (RTX PRO 6000 Blackwell class) with measurably better quality than the 2.39 bpw baseline it derives from.

Recipe

Base recipe: antirez's IQ2_XXS chat-v2 imatrix quant (81 GB, 2.39 bpw), including his 90K-chunk imatrix.
Change: ffn_down_exps upgraded Q2_K → IQ3_XXS, re-quantized from a lossless MXFP4 staging GGUF (not from the q2, so no double-quantization loss).
Result: 85.3 GiB weights, 2.58 bpw.
Quality: wikitext perplexity 4.505 vs 4.717 for the 2.39 bpw baseline (−4.5%).
HumanEval+ pass@1 93.1% (official localmaxxing eval shards, temp 0).
Aider polyglot (Python subset, diff format): pass@2 64.7% (22/34).

Measured performance (RTX PRO 6000 Blackwell, 96 GB, single card)

Config	decode	prefill	context	VRAM
llama.cpp dsv4 fork, `-np 4`, `-ub 1024`	42-47 t/s	~2200-2600 t/s	512K (4x128K slots)	93.3 GB
+ in-graph K=1 MTP (see below), `-ub 512`	59-81 t/s	~1700 t/s	256K (1 slot)	94.5 GB

The 81 t/s figure is steady-state, held across a continuous 47K-token generation at temp 0 during a 10-hour soak.

Prefill numbers include two CUDA patches in the mtp-k1 branch (2026-07): a batched lightning-indexer top-k (the per-row CUB loop was a >1M-kernel launch storm) and a compact MoE stream-k work-list (DSV4_MOE_COMPACT=1, fixes last-wave quantization). Together: ~4.5x prefill over the base fork (548 → ~2500 t/s at pp4096).

Running it

DeepSeek V4 needs a llama.cpp build with dsv4 support. This quant is served daily by the mtp-k1 branch of sigkaiser/llama.cpp (fairydreaming's dsv4 graph/kernels + an in-graph K=1 MTP port, lineage below).

bash

DSV4_MOE_COMPACT=1 \
llama-server -m DeepSeek-V4-Flash-custom-iq3down-00001-of-00003.gguf \
  -c 524288 -ngl 99 -np 4 -b 2048 -ub 1024 \
  -n 16384 --dry-multiplier 0.8 --dry-allowed-length 4 --dry-penalty-last-n 8192 \
  --cache-reuse 256 --no-mmap --flash-attn on

Optional: +40% decode via in-graph MTP speculative decoding

The mtp-k1 branch can side-load antirez's 3.8 GB MTP module GGUF and let the DS4 graph draft its own next token every decode (verified in a constant nt=2 batch; sampling distribution is unchanged). Draft acceptance 75-83% on code/prose:

bash

DSV4_MTP_GGUF=DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf DSV4_MTP_DEV=CUDA0 DSV4_MTP_SPEC=1 \
DSV4_MOE_COMPACT=1 \
llama-server -m DeepSeek-V4-Flash-custom-iq3down-00001-of-00003.gguf \
  -c 262144 -ngl 99 -np 1 -b 2048 -ub 512 --spec-type dsv4-mtp \
  -n 16384 --dry-multiplier 0.8 --dry-allowed-length 4 --dry-penalty-last-n 8192 \
  --cache-reuse 256 --no-mmap --flash-attn on

Notes: MTP mode is single-slot (-np 1) — the ring/seed state is per-server. The -n / --dry-* flags are doom-loop guards for unattended agentic use (reasoning models can ruminate unboundedly at temp 0); drop them if you want raw sampling.

Attribution

Quant recipe base + imatrix + MTP module: antirez
dsv4 llama.cpp graph and CUDA kernels: fairydreaming
In-graph K=1 MTP design: Alexey Fateev (alesha-pro), re-expressed on the fairydreaming memory architecture in the mtp-k1 branch
Base model: DeepSeek

starvingcheetah/DeepSeek-V4-Flash-custom-iq3down-GGUF

DeepSeek V4-Flash — custom 2.58 bpw GGUF for single 96 GB GPUs

Recipe

Measured performance (RTX PRO 6000 Blackwell, 96 GB, single card)

Running it

Optional: +40% decode via in-graph MTP speculative decoding

Attribution

starvingcheetah/
DeepSeek-V4-Flash-custom-iq3down-GGUF