DeepSeek V4 Flash — REAP K128 Mixed NVFP4 (NVFP4 + Q2_K + Q8)
REAP-pruned DeepSeek V4 Flash at K128 (128 routed experts per MoE layer, 50%
pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark
(GB10 Blackwell, sm_121).
At a Glance
| |
|---|
| Base model | DeepSeek V4 Flash |
| Source checkpoint | deepseek-ai/DeepSeek-V4-Flash (MXFP4/MXFP8 mixed precision) |
| Pruning method | REAP (Router-weighted Expert Activation Pruning) — Cerebras Research |
| Routed experts | 128 per layer (down from 256) |
| Hash-preserved | Layers 0-2 (256 experts each — hash-routed layers must remain full-width) |
| Pruned | Layers 3-42 (128 experts each) |
| Format | ds4-compact-v2 GGUF with NVFP4 multi-tensor convention |
| File size | ~76 GiB |
Mixed Quantization Structure
This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth
profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):
| Component | Quant | bpw | BW (GB10) |
|---|
| Routed experts — gate (w1) | NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2) | 4.50 | ~140 GB/s |
| Routed experts — up (w3) | NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2) | 4.50 | ~140 GB/s |
| Routed experts — down (w2) | Q2_K | 2.625 | ~160 GB/s |
| Attention (q, k, v, o) | Q8_0 | 8.5 | ~228 GB/s |
| Shared experts (w1, w2, w3) | Q8_0 | 8.5 | ~228 GB/s |
| Output head | Q8_0 | 8.5 | ~228 GB/s |
| Token embeddings | F16 | 16 | ~208 GB/s |
| Norms, HC base/scale | F32 | 32 | ~247 GB/s |
| HC fn weights | F16 | 16 | ~208 GB/s |
Why this mix?
The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes).
Standard GGUF quants use IQ2_XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw)
but offers substantially better precision at the same bit budget, with dequant
bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than
IQ2_XXS due to the higher byte volume, but the precision gain is well worth the
small throughput cost. Down experts stay Q2_K (already saturating at ~160 GB/s).
Attention and shared experts stay Q8_0 for quality.
NVFP4: lossless MXFP4→NVFP4 conversion
The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32
block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1
nibbles are identical — the conversion is a scale-only transform. No weight
requantization, no quality loss from the quantization step. The NVFP4 weights are
bit-identical to the MXFP4 originals; only the scale format changes.
Domain Split (Calibration)
8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096
token context. REAP activation_energy_sum2 score metric.
| Domain | Share |
|---|
| Coding & development | 35–40% |
| Agentic tool-calling | 16% |
| Research & knowledge | 15–20% |
| Math & science | 10–15% |
| Design & planning | 5–10% |
| Trivia & general QA | 3–5% |
How to Run
⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a
custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2)
plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.
Required Runtime
Clone and build the custom ds4 engine:
1git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
2cd ds4-nvfp4-spark
3make
Inference (DGX Spark / NVIDIA GB10)
1# Basic usage — K128 fits comfortably without managed memory
2./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072
3
4# With managed memory — saves ~10 GiB RAM, costs 1-3 t/s (optional for K128)
5DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072
6
7# With FP8-packed KV cache (save ~7 GiB at 1M ctx)
8DS4_KV_TURBO=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 1048576
API Server
1./ds4-server --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf \
2 --host 0.0.0.0 --port 17777 --ctx 131072
Memory Budget (DGX Spark, 128 GB unified memory)
| Context | K128 resident |
|---|
| Short (8K) | ~80 GiB ✅ |
| 256K | ~86 GiB ✅ |
| 1M (FP8 KV) | ~100 GiB ✅ |
| 1M (FP32 KV) | ~107 GiB ✅ |
K128 is the most comfortable fit on 128 GB — ample headroom for KV cache growth.
How It Was Built
- Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
- REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark →
activation_energy_sum2 per-expert scores → top-128 per layer (layers 3–42), layers 0–2 preserved at 256
- NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale_2; down experts kept as Q2_K
- Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
- Emission: single-pass GGUF generation via
deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)
No fine-tuning. Purely structural expert removal + lossless NVFP4 repack.
Weights are unmodified — a subset of the original MXFP4 experts.
Why REAP + Hybrid NVFP4?
Standard GGUF quants (IQ2_XXS, Q2_K, Q4_K) are compact but sacrifice precision at
low bit widths. NVFP4 trades some speed for substantially better quality: it reads
more bytes per weight (4.50 vs 2.06 bpw) but preserves the lossless MXFP4→NVFP4
mapping, meaning no requantization error on expert weights. The result is a model
that prioritizes precision over raw throughput — slightly slower decode than an
all-IQ2_XXS quant, but with dramatically better weight fidelity on the expert path.
Combined with REAP (50% expert pruning), the result is a model that:
- Fits comfortably on a single DGX Spark (128 GB)
- Runs 1M context with FP8-KV
- Preserves attention quality (Q8_0)
- Has no quantization quality loss on experts (lossless MXFP4→NVFP4)
Variants
| Variant | Experts | Size | Best for |
|---|
| K128 (this) | 128 (50% pruned) | 76 GiB | Comfortable fit, 1M ctx |
| K150 | 150 (41% pruned) | 86 GiB | Better quality, ~256K ctx |
| K180 | 180 (30% pruned) | 99 GiB | Best quality, managed memory required |
Acknowledgments
- DeepSeek — DeepSeek V4 Flash base model
- antirez — ds4 engine and GGUF quants
- Cerebras Research — REAP expert pruning
- NVIDIA — DGX Spark (GB10) hardware
- eouya2 — reap-for-ds4 tooling
