An exl3 quantization of DeepSeek-V4-Flash-0731 at 2.32 bpw, with one bit-width per layer
chosen by measured quantization hardness. It is the maximum-reach rung of a three-pack ladder
sized for a single 96 GB card, and the only one that runs MTP speculative decoding and a
1M-token context at the same time.
The three packs, and why these bitrates
All three target one machine: a single NVIDIA RTX PRO 6000 Blackwell, 96 GB. The bitrates
are not round numbers — each is the largest that still clears a specific capability on that
card, with an 8-bit KV cache.
2.72, per-expert — every one of the 11,008 routed experts carries its own bit-width,
allocated from measured routing traffic. Best quality of the three, and the reason the
ANEMONE exllamav3 fork exists: stock exllamav3 asserts one bit-width per layer and will not
load it. On this card it serves 256k with the drafter off; MTP does not fit alongside it.
2.54, per-layer uniform — one bit-width per layer, chosen by measured usage-weighted
quantization hardness rather than by position. Buys back the templated kernels and enough
VRAM for either the MTP drafter at 256k, or a 1M context without it.
2.32, per-layer uniform — the only one that runs the drafter and a 1M context at the
same time.
What the measurements actually say
All figures: one RTX PRO 6000 Blackwell, batch 1, 8-bit KV cache, --no-rlp, same engine
build and same harness for every pack, measured in the same sessions.
Perplexity and NLL are over 364,544 held-out tokens spanning 24 engineering and software
domains, with each domain's calibrated extent computed from the calibration manifest and
excluded — text none of the packs were calibrated on. The corpus is not published, so the
absolute values are not comparable with anyone else's perplexity number; the differences
between the packs are the transferable part, and the token counts match exactly.
Three things are worth reading off the table:
Quality degrades gently. The full 14.5% bitrate cut from 2.72 to 2.32 costs 1.00%
perplexity. The 2.54 → 2.32 step costs only 0.34%.
The decode gain comes from the allocation style, not the bitrate. Going per-expert →
per-layer is worth ~6% (86.4 → 91.6 tok/s) because it restores the compile-time-specialised
kernels; going 2.54 → 2.32 on top of that is worth another 0.6% (91.6 → 92.2). Decode here is
not strongly bandwidth-bound, so lower bitrate buys context and headroom, not speed.
Drafting is the largest single speed lever, where it fits. MTP drafting adds ~21% on 2.54
and ~24% on 2.32. Acceptance falls as the draft model gets coarser — 137 of 256 tokens
accepted on 2.54 against 104 on 2.32 — but the net is still faster. Every drafted token is
verified by the main model, so drafting changes speed and never output quality.
Not measured, so not claimed: prefill timings for the two uniform packs.
hybrid (routed experts K=2, attention/shared K=3), built in place
size
85 GB, 12 shards
Which layers get the extra bit
Not the ends. The conventional heuristic promotes the first and last layers; this pack ranks
all 43 by usage-weighted quantization hardness measured from a prior conversion, and promotes
the hardest 14:
Measured reconstruction error came out as the allocation intends — K=3 layers average about
half the error of K=2 layers — but the ranking is an imperfect predictor of which individual
layers reconstruct worst, and the spread within each class is roughly as large as the gap
between them. It is a better criterion than position, not a solved problem.
Quality
Held-out perplexity 1.9126, NLL 0.6485, over 364,544 tokens across 24 engineering and
software domains — text excluded from calibration by computing each domain's calibrated
extent from the calibration manifest.
Against the 2.72 per-expert pack measured identically that is +1.00%, and against the
2.54 pack +0.34% — for a 14.5% and 8.4% cut in expert bitrate respectively. The
degradation is gentle, which is what makes this rung worth having.
Build for plain 12.0 on Blackwell, never12.0a — a single-arch build is unloadable on
anything else.
Configuration these numbers were measured with:
max_seq_len / cache_size 1048576 (1M, with or without the drafter)
cache_mode 8,8 8-bit K and V
gpu_split [95.5]
max_batch_size 1
env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
EXL3_NO_RLP=1
expandable_segments is not optional at these context lengths: without it the allocator
fragments and the load fails tens of MiB short. A complete config, launcher and systemd unit
are in the recipe repository.
On the fork requirement: the 2.72 pack genuinely needs the fork, because stock exllamav3
asserts a single bit-width per layer and refuses a per-expert pack. This pack is uniform
per layer, so that assertion is satisfied — but it has not been tested against a stock build,
and every number here was taken with the fork. Treat stock compatibility as plausible and
unverified.
Serve it the way DeepSeek specify
Two settings outside this pack dominate how the model behaves, and both are easy to get wrong.
There is no official Jinja chat template. DeepSeek's model card says so and ships a Python
encoder (encoding/encoding_dsv4.py) instead, so any chat_template.jinja beside an exl3
pack is a community reconstruction. The reasoning-effort prompt is the mechanism that makes
effort levels work, and a paraphrase is not equivalent — DeepSeek's max prompt instructs the
model not to stop reasoning until it has independently verified the solution from multiple
angles, where the version that circulated with these packs merely said "think exhaustively
before answering". The chat_template.jinja here carries their verbatim strings. Valid levels
are low, high, max — there is no medium, and low injects no prompt at all.
Sampling should be temperature 1.0, top_p 1.0 (0.95 for agentic use), per both their
card and the model's own generation_config.json — and nothing else. min_p is not one of
their recommendations. Low temperature or aggressive truncation drives this model into
repetition loops during long reasoning: at temperature 0.2 it will emit tens of thousands of
characters of self-checking and never answer. serve/sampler_overrides/deepseek_v4.yml
encodes the recommended values as TabbyAPI fallbacks (force: false, so a client that sends
its own values still wins).
Give the reasoning room — and this pack can. DeepSeek recommend a maximum output length of
384K tokens at high and max effort, and the model genuinely uses it: a hard
competitive-programming problem measured on the 2.72 pack spent ~45,000 tokens reasoning
before answering. The 2.72 pack's 262,144-token context is less than that recommended budget
on its own; this pack reaches 1M, including with the MTP drafter, so it can hold DeepSeek's recommended reasoning
budget with room to spare. That is a practical argument for the lower-bitrate packs beyond
raw speed.
If a reply ever comes back with a large reasoning_content and an empty content, it ran out
of context mid-deliberation — raise max_tokens or lower the effort level.
The MTP drafter
The MTP module is built into the shards, so nothing extra needs downloading. Enable it with
--mtp. It drafts a block of tokens per verification round and the main model verifies every
one, so it changes speed and never output quality.
Measured at 96,801 MiB with a 1,048,576-token cache — the drafter and a 1M context fit
together on one 96 GB card, which neither the 2.72 nor the 2.54 pack manages.
Acceptance is lower than on the higher-bitrate packs (104 of 256 generated tokens against 137
on 2.54), because the draft model is quantized alongside everything else. The net is still
faster: ~114 tok/s against ~92 without drafting.
Reproducing
The full recipe — routing census, hardness ranking, calibration, conversion and gates — is at
anoane/anemone-recipe, including the measured
usage counts this allocation derives from:
The conversion took 12 h 31 m on one 96 GB card, all-or-nothing with no checkpoint resume,
and needs ~215 GB of free disk.
Limitations
Lowest quality of the three packs. If you do not need 1M context or the drafter, the
2.54 or 2.72 pack is a better choice — see the table above.
Single-GPU fit assumes one process, batch 1, and an 8-bit KV cache. With one sequence
slot a second concurrent request waits rather than running.
Prefill timings are unmeasured for this pack. Decode is characterised above; prefill is
not, and should not be assumed identical to the other packs'.
Calibration domain. Bit allocation derives from engineering and software prompts.
Routing tracks the prompt distribution — including style and source, not only subject — so
a very different workload would ideally re-profile.
License
Inherits the license of the base model, DeepSeek-V4-Flash-0731. Quantization metadata and
scripts are MIT.
