The main decoder was quantized from the original BF16 model. The MTP component
was sourced from SpectreSystems/KAT-Coder-V2.5-Dev-MTP,
which contains the corresponding BF16 MTP tensors. Existing MTP tensors are
preserved and quantized at approximately 8 bpw. This conversion does not
synthesize a new MTP head.
This is not an official Kwaipilot release. It is an independently converted
community quantization.
Default sampling
The published generation_config.json now defaults to deterministic coding
behavior:
Clients may override these values per request. For debugging path handling,
tool calls, or malformed output, start with temperature: 0 and
enable_thinking: false.
Parameter count note
The upstream KAT-Coder-V2.5-Dev model is a 35B total parameter MoE with
approximately 3B activated parameters per token. The Hugging Face file viewer
may display a lower number such as 12B for this repository because its generic
parameter counter does not understand EXL3 packed tensors and counts their
storage representation incorrectly. That generated UI value is not the model's
logical parameter count and does not indicate missing weights.
Q8 KV cache at the full tested context: approximately 10 GiB
(10.737 GB) of KV memory, before other runtime workspaces
Recommended total GPU memory: two 24 GiB cards
Recommended cache: Q8
Recommended batch size: 1
The context value above is the validated operating limit for the documented
TabbyAPI setup. Lower context limits are recommended on smaller or more
imbalanced GPU configurations.
Quick setup with TabbyAPI and ExLlamaV3
The following is a minimal agent-friendly setup for a fresh Linux machine.
Adapt the CUDA device IDs if the two 24 GiB inference GPUs use different IDs.
Install the serving stack according to the TabbyAPI and ExLlamaV3 instructions
for the target CUDA/PyTorch combination. Do not mix an unrelated system Python
with the environment containing TabbyAPI dependencies.
3. Create model-local TabbyAPI overrides
Save this as tabby_config.yml in the model directory, or pass equivalent
settings through the TabbyAPI configuration:
The model uses its native chat_template. Thinking is selected per request.
Thinking disabled:
{"enable_thinking": false}
Thinking enabled:
{"enable_thinking": true}
With thinking disabled, the response is clean assistant content without visible
thinking markers. With thinking enabled, the raw response contains reasoning
followed by </think> and the final answer. Clients should either render the
reasoning separately or remove <think>...</think> before displaying only the
final answer.
For tool-calling clients, preserve the Qwen3 coder tool format and the native
chat template:
These measurements were made with two RTX 3090 cards, Q8 KV cache, MTP draft
mode, one active sequence, and TabbyAPI/ExLlamaV3. Actual speed depends on
prompt length, generated length, cache state, GPU split, sampling settings,
and whether thinking is enabled.
Long-context performance
Content tokens
End-to-end
Prefill
Generation
Draft acceptance
32,256
0.15 s
2,848 t/s
135.91 t/s
2 / 2
65,024
1.40 s
1,623.62 t/s
111.95 t/s
2 / 2
130,560
48.86 s
1,347.66 t/s
82.29 t/s
2 / 2
196,096
71.30 s
923.37 t/s
64.91 t/s
2 / 2
258,048
88.25 s
706.0 t/s
26.86 t/s
1 / 5
For ordinary short requests, expect roughly 80-136 generated tokens/second in
this setup when thinking is disabled. Long-context generation becomes slower as
KV memory grows; near the full tested context, approximately 27 tokens/second
was measured.
Thinking-mode microbenchmark:
Thinking disabled: 3/3 checks passed, average 0.115 seconds
Thinking enabled: 3/3 checks passed, average 0.172 seconds
Tool calls: valid qwen3_coder calls in both modes
Replacement characters: 0 in tested responses
These are local independent measurements, not upstream benchmark scores. No
perplexity or public benchmark score is claimed for this quantization.
Attribution and license
This is a quantized derivative and is not an official Kwaipilot release.
Preserve the upstream model attribution and Apache-2.0 license declaration.
