Qwen3.6-35B-A3B — GGUF for QuarkStar
These GGUF weights are built for
QuarkStar , a small native inference engine specialized for Qwen3.6-35B-A3B.
The quantization recipe is inspired by Salvatore Sanfilippo's asymmetric DeepSeek-V4 quantization work in DS4: aggressively quantize the routed experts, which contain most of the parameters, while keeping routing, shared experts, attention, recurrent blocks, embeddings, and output tensors at higher precision.
The result is a roughly 11 GB model capable of fitting fully resident on a 16 GB unified-memory machine, with enough memory left for the runtime, KV cache, and operating system.
Files
File Size Purpose Qwen3.6-35B-A3B-AntirezExperts-IQ2XXS-gateup-Q2K-down-Q8rest.gguf11.7 GB Default Q2 model for fully resident use on a 16 GB BC-250 Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.gguf13.1 GB Higher-quality mixed Q2/Q4 model; SSD streaming recommended on 16 GB Qwen3.6-35B-A3B-MTP-Q4K-Q8_0-F32.gguf1.59 GB Optional MTP model for experimental speculative decoding
The MTP file is not a standalone language model. Pair it with either main model and enable it explicitly with --mtp.
Which model should I use?
16 GB unified memory
Use the default Q2 model:
./download_model.sh q2-imatrixIt is designed to fit fully resident on 16 GB while leaving memory for the operating system, scratch buffers and KV cache.
Higher quality
Use the mixed Q2/Q4 model:
./download_model.sh q2-q4-imatrixThe last six transformer layers, layers 34 through 39, use Q4_K routed experts.
On a 16 GB, run this model with SSD streaming:
1 ./q36 \
2 -m gguf/Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.gguf \
3 --ssd-streaming \
4 -p "Hello"
Optional MTP
Download the MTP component separately:
Enable it explicitly:
1 ./q36 \
2 --mtp gguf/Qwen3.6-35B-A3B-MTP-Q4K-Q8_0-F32.gguf \
3 --mtp-draft 2 \
4 -p "Hello"
The current MTP path is experimental and generally provides only a slight speedup.
Quantization recipe
Default Q2 quantization recipe
File:
Qwen3.6-35B-A3B-AntirezExperts-IQ2XXS-gateup-Q2K-down-Q8rest.ggufTensor class Quantization blk.*.ffn_gate_exps.weightIQ2_XXSblk.*.ffn_up_exps.weightIQ2_XXSblk.*.ffn_down_exps.weightQ2_KShared experts Mostly Q8_0 Attention projections Mostly Q8_0 Recurrent and SSM matrices Mostly Q8_0 Token embeddings Q8_0Output head Q8_0Norms, gates, biases and small tensors F32
Only the routed MoE expert tensors are changed from the source Q8 GGUF.
Tensor counts
Type Count F32301 Q8_0312 IQ2_XXS80 Q2_K40 Total 733
Conversion summary
1 80 routed gate/up tensors -> IQ2_XXS
2 40 routed down tensors -> Q2_K
3 all other tensors -> unchanged
Property Value Source size 36,903,140,320 bytes Final size 11,737,316,384 bytes Changed tensors 120 Imatrix entries 120 Imatrix chunks 6,249
Mixed Q2/Q4 quantization recipe
File:
Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.ggufLayers 0 through 33 use the default asymmetric Q2 expert recipe:
Tensor class Quantization Gate experts IQ2_XXSUp experts IQ2_XXSDown experts Q2_K
Layers 34 through 39 use Q4_K for all three routed-expert matrices:
Tensor class Quantization Gate experts Q4_KUp experts Q4_KDown experts Q4_K
Everything outside the routed experts retains the same higher-precision source type.
Tensor counts
Type Count F32301 Q8_0312 IQ2_XXS68 Q2_K34 Q4_K18 Total 733
Usage
1 git clone https://github.com/Ninnix/q36
2 cd q36
3 ./download_model.sh q2-imatrix # 16 GB RAM machines
4 ./download_model.sh q2-q4-imatrix # >= 24 GB or enable ssd-streaming on 16 GB
5 ./download_model.sh mtp # optional MTP / speculative decoding
6 make
7
8 ./q36 -p "Explain Redis streams in one paragraph."
9 ./q36-server --ctx 32000 --kv-disk-dir /tmp/q36-kv --kv-disk-space-mb 8192
The download_model.sh script fetches from this repository, resumes partial downloads, and points ./q36moe.gguf at the selected variant.
Acknowledgements
The asymmetric expert-quantization strategy and much of the runtime direction are inspired by Salvatore Sanfilippo's work on:
QuarkStar is a Vulkan-oriented redesign of ideas explored by DwarfStar, retargeted at Qwen3.6-35B-A3B and the AMD BC-250.
This work also depends heavily on the formats, quantization methods, kernels, tests and engineering knowledge developed by:
llama.cpp GGML
The Qwen team
Salvatore Sanfilippo
Georgi Gerganov
llama.cpp and GGML contributors
License
The Qwen3.6-35B-A3B base model and these derived GGUF weights are distributed under the Apache License 2.0 .
The q36 / QuarkStar inference-engine source code is distributed separately under the MIT License .
See the original model repository for the complete base-model license and usage terms:
