Views
No views yet
UD-Q4_K_XL, 144 GiB) and verify shard sha256s against the LFS manifest.| metric | value |
|---|---|
| generation | 12.9 t/s (-b 2048 -ub 1024) |
| prefill @ 8K | 205–209 t/s |
| scaling | 8 GPUs optimal — 10 ≈ par, 14 halves throughput (PCIe island hops) |
| quality | legal-drafting screen 6/6 + verbatim needle retrieval at ~7K tokens |
--cache-type-k q8_0, this model emits confident gibberish behind a
healthy /health — on CPU and CUDA alike. Cause: llama.cpp's quantized-KV
Hadamard rotation diverts DeepSeek-V4 off its sparse attention paths into a
fallback with broken rotation math. Either run --cache-type-k f16
(zero-cost, MLA's cache is tiny) or apply the patches here (also on the
ds4-volta-fix
branch), which disable the rotation for this architecture — verified with a
full K/V-type matrix on CPU and GPU.1git clone -b ds4-volta-fix https://github.com/Mermiges/llama.cpp
2cmake -S llama.cpp -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_BUILD_TYPE=Release
3cmake --build build --target llama-server -j
4build/bin/llama-server -m DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf \
5 -ngl 999 -sm layer -fa 1 -b 2048 -ub 1024 -c 8192 \
6 --cache-type-k f16 --cache-type-v f16patches-rotation-aware/
(branch ds4-rotation-aware)
makes DeepSeek-V4's attention paths rotation-aware and re-enables the quantized-KV
incoherence rotation — verified coherent across the full K/V-type matrix, ~6.8%
decode cost, and in a ~6000-token needle screen the rotation-ON build retrieved a
needle that plain quantized-K missed. patches-tensor-parallel/ carries the
experimental de-blacklisting of DeepSeek-V4 for llama.cpp's tensor-parallel mode
(loads and reaches decode on 8× V100; one scheduler assert remains).