INT4 (W4A16) quantization of Kwaipilot/KAT-Coder-V2.5-Dev — a 35B-A3B coding Mixture-of-Experts model from Kwaipilot (qwen3_5_moe: hybrid GatedDeltaNet linear-attention + full-attention over 40 layers, 256 routed experts + a shared expert with 8 active per token; base Qwen3.6-35B-A3B, SFT+RL post-training). Supports a <think> reasoning/thinking mode (on by default) and tool use. (The arch config declares a vision tower, but this checkpoint ships language-model weights only.)

Variant: W4A16 — 4-bit symmetric integer weights, group size 128, activations BF16. Round-to-nearest (data-free, no activation-aware scaling). Quantized by: sahilchachra Tooling: llm-compressor model_free_ptq (data-free, RTN) -> compressed-tensors

This is a quantized derivative. Weights, behavior, and license follow the base model — see the original card for full details, benchmarks, and citation.

What is quantized

Quantized to 4-bit:

routed experts mlp.experts.*.{gate,up,down}_proj (all layers)
shared expert {gate,up,down}_proj
full-attention self_attn.{q,k,v,o}_proj

Kept in BF16: GatedDeltaNet linear_attn (mamba) layers, MoE router mlp.gate + shared_expert_gate, token embeddings, lm_head, all norms (incl. q_norm / k_norm).

Calibration

Data-free — weight-only (model_free_ptq, round-to-nearest); no calibration data. Weights are quantized by streaming the safetensors from disk.

Prompt template & sampling

Uses the Qwen3.5 chat template — ChatML (<|im_start|>role … <|im_end|>) with a <think>…</think> reasoning trace (thinking mode on by default; disable with enable_thinking=False). Apply it via tokenizer.apply_chat_template(messages, add_generation_prompt=True); supports tool-calling.

Recommended sampling: temperature=1.0, top_p=0.95, top_k=20 for general/coding tasks (the card also suggests temperature=0.7, top_p=0.8 for instruction mode and presence_penalty=1.5).

Usage (vLLM)

python

from vllm import LLM, SamplingParams

# NOTE: --language-model-only equivalent. This arch carries a vision config
# but the checkpoint has no vision weights, so vLLM skips the vision tower.
llm = LLM(
    model="sahilchachra/KAT-Coder-V2.5-Dev-W4A16",
    trust_remote_code=True,
    hf_overrides={"language_model_only": True},
)
out = llm.chat(
    [{"role": "user", "content": "Hello!"}],
    SamplingParams(temperature=0.6, top_p=0.95, max_tokens=512),
)
print(out[0].outputs[0].text)

Serving via the CLI, pass the flag directly:

bash

vllm serve sahilchachra/KAT-Coder-V2.5-Dev-W4A16 \
    --language-model-only --trust-remote-code \
    --max-model-len 262144 --reasoning-parser qwen3