KAT-Coder-V2.5-Dev — 11.7 GB Asymmetric GGUF for QuarkStar
These GGUF weights are built for QuarkStar, a small native inference engine specialized for Qwen3.6-35B-A3B.
This repository contains an imatrix-calibrated GGUF quantization of Kwaipilot/KAT-Coder-V2.5-Dev, a 35B-A3B Mixture-of-Experts model focused on agentic coding and repository-level software engineering.
KAT-Coder-V2.5-Dev is post-trained from Qwen/Qwen3.6-35B-A3B. It is designed for tasks such as:
understanding unfamiliar repositories;
locating and fixing bugs;
editing multiple files;
using shell, search and file tools;
running and interpreting tests;
implementing features from issue descriptions;
long-horizon coding-agent workflows.
The quantization recipe follows the asymmetric MoE strategy explored by Salvatore Sanfilippo in DS4: aggressively compress the routed experts, which contain most of the model weights, while retaining substantially higher precision for the attention path, shared experts and output tensors.
The resulting model is approximately 11.7 GB, making a 35B-class coding model practical on machines with 16 GB of unified memory.
This is a quantized derivative. Refer to the upstream KAT-Coder model card and technical report for training details, intended use, benchmark methodology and model limitations.
Model details
Property
Value
Upstream model
Kwaipilot/KAT-Coder-V2.5-Dev
Foundation model
Qwen/Qwen3.6-35B-A3B
Architecture
Qwen3.6 MoE / HF qwen3_5_moe
Total parameters
Approximately 35B
Active parameters
Approximately 3B per token
Transformer layers
40
Routed experts
256
Experts selected per token
8
Maximum upstream context
262,144 tokens
Format
GGUF
Calibration
Importance matrix / imatrix
Primary use
Coding and agentic software engineering
Quantized size
Approximately 11.7 GB
The upstream context limit is an architectural maximum. Practical context size depends on available memory, KV-cache precision, runtime buffers and the inference engine.
The routed experts account for most of the model's storage footprint. Compressing them aggressively provides most of the size reduction while preserving higher precision in components that directly control attention, routing and token prediction.
The imatrix calibration data is used to make the low-bit expert quantization more sensitive to tensor importance than a purely weight-only conversion.
Why this quantization?
A conventional Q8 GGUF of this model is roughly 37 GB, while the original BF16 weights are roughly 69 GB.
This asymmetric release targets a different operating point:
a fully local 35B-class coding model;
approximately 11.7 GB of model weights;
enough remaining memory on a 16 GB unified-memory system for runtime buffers, the operating system and a moderate KV cache;
higher precision for attention, shared experts and output generation;
aggressive compression concentrated in the routed experts.
This is especially useful for Mixture-of-Experts models because only a subset of the routed experts is activated for each token.
QuarkStar / q36
This GGUF targets the Qwen3.6-35B-A3B-compatible tensor layout used by QuarkStar.
QuarkStar is a small native inference engine with Vulkan and Metal backends. It is deliberately specialized rather than being a general-purpose GGUF runner.
Vulkan
bash
1git clone https://github.com/Ninnix/q36
2cd q36
34make5mkdir -p gguf
67./download_model.sh kat-coder
89./q36 \10 -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \11 --ctx 32768\12 -p "Inspect this repository and explain its architecture."
Apple Silicon / Metal
bash
1make metal
23./q36 \4 --metal \5 -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \6 --ctx 32768\7 -p "Write a minimal Redis-compatible server in C."
KAT-Coder-V2.5 is trained for autonomous work inside executable repositories rather than only single-turn code generation. The upstream project combines supervised training, reinforcement learning, executable environments, verifiable rewards and multi-teacher on-policy distillation.
Acknowledgements
The asymmetric expert-quantization strategy and runtime direction were inspired by Salvatore Sanfilippo's work on:
