ExLlamaV3 (exl3) quantizations of DeepSeek-V4-Flash.
[!WARNING]
This quant does NOT run on stock exllamav3 or stock tabbyAPI.
DeepSeek-V4-Flash uses an architecture that upstream exllamav3 does not implement, so
loading these files with an official release will fail with an unknown-architecture error.
You must install the exllamav3 fork branch described in Installation.
This is experimental software — expect rough edges and breaking changes.
Available quants
Weights live on a branch named after the bit-rate. The model card (this page) is on main.
Branch / revision
Bits per weight
Approx. size
Notes
4.0bpw
4.0 bpw
~145 GB
tested up to 256k context
More bit-rates may be published as additional branches.
Why it isn't supported natively in exllamav3
DeepSeek-V4-Flash is not a standard transformer, and none of the machinery it needs exists in
any official exllamav3 release:
No DeepseekV4ForCausalLM architecture. Upstream has no model definition for it, so the
loader rejects the checkpoint outright.
Custom compressed-attention CUDA kernels. The sliding-window + compressor + indexer
attention is implemented in bespoke kernels (dsv4.cu) that ship only on the fork.
QAT fake-quant path. The model was trained with quantization-aware training; reproducing
its fp8/fp4 fake-quants at inference (for output fidelity) requires extra code paths upstream
does not have.
Non-standard cache model. The compressor and sliding-window state is stateful and kept
inside the attention module, not laid out like exllamav3's normal paged KV cache. The fork
adds two bespoke paths to make caching work anyway — append-only prefix reuse (default) and an
optional native-paging mode for multiple concurrent sessions (see
Cache reuse & multi-session) — neither of which upstream carries.
Custom MoE router and loader/generator hooks that aren't part of the stock modules.
All of this lives on the fork branch below.
Installation
Three steps: install the tabbyAPI fork, install the exllamav3 fork into it, download the model.
You'll need an NVIDIA CUDA toolkit (the exllamav3 fork builds a custom extension) and enough VRAM
(4.0bpw weights are ~145 GB).
1. Install the tabbyAPI fork
The fork branch adds the DeepSeek-V4 tool-call parser and an example config on top of current
tabbyAPI. Clone it and set up its venv:
This compiles the custom CUDA extension (needs the CUDA toolkit) and cleans up after itself. The
fork is a drop-in superset of exllamav3 v1.0.0, so the rest of tabby keeps working.
3. Download the model
The weights are on the 4.0bpw branch, not main — select that revision:
[!TIP]
If autosplit OOMs while loading across multiple GPUs, give each GPU a few GB of headroom with
a per-GPUautosplit_reserve list — one entry per GPU, e.g. [3072, 3072, 3072, 3072]. A
single-element list only reserves GPU0. In paged mode the cache is larger (see below), so you may
need to lower the reserve instead so it fits.
Cache reuse & multi-session
Because the compressed attention state lives inside the module rather than in a standard paged KV
pool, the fork provides two ways to avoid re-prefilling a prompt from scratch. Both decode at the
same speed — reuse only affects prefill / time-to-first-token.
Default — append-only prefix reuse. The cache stays in its compact non-paged layout (~3.6 GB at
256k) and a single, evolving conversation continues from the previous turn's cache instead of
replaying the whole prompt each turn. This is the common agent / tool-loop case: a long context
that grows by a few tokens per turn keeps TTFT low. Only the one resident sequence's prefix is
reusable — switching to a different conversation re-prefills.
DSV4_PAGED=1 — native paging + multi-session. Materializes the per-page cache state so
exllamav3's own page table can share and reuse pages by content hash. Several independent
conversations can stay resident at once, and switching between them reuses each one's cache without
re-prefilling — even after other sessions were served in between. The trade-off is memory: paging
stores per-page state, so the cache is roughly 3× larger (~11 GB at 256k) and must fit alongside
the weights (lower autosplit_reserve if the split doesn't fit).
Mode
Env
Cache @ 256k
Reuse
Default
(none)
~3.6 GB
append-only, single session
Paged
DSV4_PAGED=1
~11 GB
native page table, multiple sessions
Both are set via environment variables when launching tabby: DSV4_PAGED (default 0) toggles
paging; DSV4_REUSE (default 1) toggles the append-only reuse in the non-paged mode.
For the curious, this is what the fork adds on top of exllamav3:
Compressed sparse attention. Each layer uses one of three KV regimes: a 128-token sliding
window; CSA — a 4:1 compressor plus a top-512 indexer that selects the most relevant
compressed blocks; or HCA — a 128:1 compressor only. Only a small, selected slice of the past
is ever attended to, which is why the KV cache stays around 3.6 GB even at 256k tokens.
Latent attention (MQA, not MLA). Keys and values are a single shared 512-dim latent
(num_key_value_heads == 1), used directly — there is no per-head KV up-projection, so this is
MQA rather than MLA. Queries and the attention output use low-rank projections, and RoPE is
decoupled and scaled with YaRN for long context.
Mixture-of-Experts. 256 fine-grained routed experts with sigmoid gating and routed-scaling,
6 active per token, plus 1 always-on shared expert.
Fidelity-preserving fake-quant. The model was trained with quantization-aware training. The
fork reproduces the reference fake-quants at inference — fp8 on the attention KV "nope" dims,
fp4 for the indexer/compressor, and the SwiGLU activation clamp — so the quant matches how the
model was trained. Weights use exl3's trellis quantization at the target bit-rate; activations
stay in fp16 (higher precision than the reference's fp8 activation path, which is harmless).
Custom kernels + engine hooks. New CUDA kernels implement the compressed attention, with
hooks into the loader, cache, generator (chunked page-aligned prefill, append-only prefix reuse,
and an optional native-paging cache layer for multi-session reuse) and the MoE path.
Limitations
Experimental / not upstream. Branches and behavior may change.
Cache reuse has two modes. By default a single evolving conversation reuses its own growing
prefix (append-only), so a multi-turn chat / tool loop continues from the previous turn instead of
re-prefilling. Cross-session reuse — keeping several conversations resident and switching between
them — requires paged mode (DSV4_PAGED=1), which costs more cache memory. See
Cache reuse & multi-session.
One sequence per forward pass. Tested with max_batch_size: 1. Paged mode lets multiple
sessions share the cache and be served in turn, but concurrent batched decoding of several
sequences at once is untested.
Build required. A CUDA toolkit is needed to compile the custom extension.
Large. 4.0bpw is ~145 GB of weights — you need substantial multi-GPU VRAM.
