These files REQUIRE a patched llama.cpp - stock llama.cpp crashes after 2-3 prompts due to non-unique expert IDs in REAP's tid2eid routing tensors (issues #24591, #25598, labeled wontfix).

TLDR: Download the GGUF + patched files from patches/, build llama.cpp with CUDA, run with --tensor-split 1,1 --split-mode layer. See instructions below.

Files

File	Size	BPW	Max Context (96GB VRAM)	Speed (2x RTX 4090 48GB)
`DSV4-Flash-162B-REAP-Q2_K.gguf`	56 GB	~2.96	200K	28-29 tok/s
`DSV4-Flash-162B-REAP-Q3_K_M.gguf`	73 GB	~3.74	128K (Q4 KV)	~13 tok/s

License

This model inherits the MIT License from the original DeepSeek-V4-Flash-162B by 0xSero. The llama.cpp patches are also MIT-licensed (llama.cpp is MIT).

Source

Original model: 0xSero/DeepSeek-V4-Flash-162B (BF16 safetensors)
Inference engine: llama.cpp (MIT License)
Architecture: DeepSeek-V4 with REAP pruning (284B -> 162B)
MoE: 144 experts, MXFP4 expert weights, MHC (Manifold Hyper-Connections)

CRITICAL: You NEED the patched llama.cpp

The REAP model has non-unique expert IDs in tid2eid routing tensors. Stock llama.cpp crashes after 2-3 prompts with:

CUDA error: illegal memory access

This is a known issue labeled wontfix by llama.cpp maintainers because REAP's routing is non-standard.

Patch 1: Fix duplicate expert ID crash in `mmid.cu`

File: ggml/src/ggml-cuda/mmid.cu

Problem: The mm_ids_helper kernel uses warp_reduce_any which only checks if ANY thread in the warp matched. When the same expert appears twice for one token (as REAP does), multiple threads try to write to the same memory slot, causing a race condition and illegal memory access.

Fix: Replace warp_reduce_any with a warp-level prefix sum (inclusive scan). This:

Counts the total number of matching threads per token (not just 0/1)
Gives each matching thread a unique write offset in the store array
The kernel correctly handles N copies of the same expert for one token

The patch modifies two code paths:

Generic path (n_expert_used_template == 0): Uses full-warp prefix sum
Optimized path (templated expert count): Uses prefix sum within each neu_padded group

Full patched file is in patches/mmid.cu in this repo. The unified diff is in patches/mmid.cu.patch.

Patch 2: Exclude i32 routing tensors from quantization

File: src/llama-quant.cpp

Problem: When requantizing from one GGUF format to another, llama.cpp tries to dequantize all tensors. The tid2eid tensors are i32 type and cannot be dequantized, causing:

llama_model_quantize: failed to quantize: cannot dequantize/convert tensor type i32

Fix: Add one line to tensor_allows_quantization():

cpp

// do not quantize i32 routing tensors (DeepSeek V4 tid2eid)
quantize &= name.find("tid2eid") == std::string::npos;

Full diff in patches/llama-quant.cpp.patch.

Build Instructions (Multi-GPU)

bash

# 1. Clone llama.cpp (latest version)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# 2. Copy patched files over the originals
# Download mmid.cu and llama-quant.cpp from the patches/ folder in this repo
cp mmid.cu ggml/src/ggml-cuda/mmid.cu
cp llama-quant.cpp src/llama-quant.cpp

# 3. Build with CUDA
# Adjust CUDA_ARCHITECTURES for your GPU:
#   89 = RTX 4090 (Ada)
#   86 = RTX 3090 (Ampere)
#   90 = H100 (Hopper)
mkdir build && cd build
cmake .. -DGGML_CUDA=ON \
         -DCMAKE_CUDA_ARCHITECTURES=89 \
         -DCMAKE_BUILD_TYPE=Release \
         -DLLAMA_BUILD_TESTS=OFF \
         -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build . -j$(nproc) --target llama-server llama-quantize

Running (Multi-GPU)

Q2_K with 200K context (RECOMMENDED - best speed + max context)

bash

./build/bin/llama-server \
  --model DSV4-Flash-162B-REAP-Q2_K.gguf \
  --ctx-size 204800 \
  --n-gpu-layers 99 \
  --tensor-split 1,1 \
  --split-mode layer \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --port 8032 --host 0.0.0.0 \
  -t 8 --parallel 1 -fa on

Q3_K_M with 128K context

bash

./build/bin/llama-server \
  --model DSV4-Flash-162B-REAP-Q3_K_M.gguf \
  --ctx-size 131072 \
  --n-gpu-layers 99 \
  --tensor-split 1,1 \
  --split-mode layer \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --port 8032 --host 0.0.0.0 \
  -t 8 --parallel 1 -fa on

Benchmark Results (2x RTX 4090 48GB, 96GB VRAM, Multi-GPU)

Config	Context	Speed	Stability	VRAM
Q2_K + Q4 KV	200K	28-29 tok/s	8/8 tests passed	88 GB
Q3_K_M + Q4 KV	128K	~13 tok/s	Crashed after 1 prompt	95.9 GB

Hardware Requirements

Minimum: 2x GPU with 48GB VRAM each (96GB total) - e.g. 2x RTX 4090 modificated to 48GB VRAM each
Architecture: sm_89 (Ada Lovelace) or newer
System RAM: 112GB+ recommended
Disk: 56-73GB per quantization file
Multi-GPU: Required - uses --tensor-split 1,1 --split-mode layer to distribute across 2 GPUs

Technical Details

Why is DSv4 slower than other MoE models?

DSv4 uses a novel architecture with multiple complex attention mechanisms per layer:

CSA (Compressed Sparse Attention) with Lightning Indexer
HCA (Heavily Compressed Attention)
MHC (Manifold Hyper-Connections) with Sinkhorn normalization
MLA (Multi-head Latent Attention) for KV cache compression

This results in ~5x more operations per layer compared to standard transformer MoE models like MiniMax-M2.7 (which runs at 95 tok/s on the same hardware).

Acknowledgments

Original model: 0xSero/DeepSeek-V4-Flash-162B
Inference engine: llama.cpp
REAP prunes the full 284B DeepSeek-V4 to 162B while retaining quality

xik94/DeepSeek-V4-Flash-162B-REAP-GGUF

DeepSeek-V4-Flash-162B REAP - GGUF Quantizations (Multi-GPU)

Files

License

Source

CRITICAL: You NEED the patched llama.cpp

Patch 1: Fix duplicate expert ID crash in mmid.cu

Patch 2: Exclude i32 routing tensors from quantization

Build Instructions (Multi-GPU)

Running (Multi-GPU)

Q2_K with 200K context (RECOMMENDED - best speed + max context)

Q3_K_M with 128K context

Benchmark Results (2x RTX 4090 48GB, 96GB VRAM, Multi-GPU)

Hardware Requirements

Technical Details

Why is DSv4 slower than other MoE models?

Acknowledgments

xik94/
DeepSeek-V4-Flash-162B-REAP-GGUF

Patch 1: Fix duplicate expert ID crash in `mmid.cu`