Self-quantized IK_K GGUFs of Qwen/Qwen3.6-35B-A3B — a 35B Mixture-of-Experts (~3B active), so it decodes at small-model speed. Quantized with an importance matrix (qwen35moe.imatrix, included) using ik_llama.cpp's IK_K quant types, which give noticeably better quality-per-bit than standard K-quants on this MoE.

⚠️ These require ik_llama.cpp. The IQ*_K types are an ik_llama extension and will NOT load in mainline llama.cpp, LM Studio, Ollama, or koboldcpp. If you need a portable quant, use a standard Q4_K_M/Q5_K_M GGUF instead.

Files

File	Quant	Size
`qwen35moe-IQ4_K.gguf`	IQ4_K	19.7 GB
`qwen35moe-IQ5_K.gguf`	IQ5_K	24.0 GB
`qwen35moe-IQ6_K.gguf`	IQ6_K	28.8 GB
`qwen35moe.imatrix`	imatrix used for quantization	0.11 GB

IQ4_K fits a single 24 GB GPU with room for context; IQ5_K / IQ6_K are for more VRAM or dual-GPU.

Usage (ik_llama.cpp)

Build ik_llama.cpp, then:

bash

./build/bin/llama-server \
  -m qwen35moe-IQ4_K.gguf \
  -ngl 99 -fa 1 -c 32768 \
  -ctk q8_0 -ctv q8_0 \
  -fmoe -rtr \
  --jinja --host 127.0.0.1 --port 8080

Flag notes (ik_llama dialect differs from mainline):

-fa 1 — flash attention (not --flash-attn on).
-fmoe — ik_llama's fused-MoE kernel; meaningfully faster MoE decode. Recommended for this model.
-rtr — run-time tensor repack for your GPU; can speed up decode on some setups (optional).
Do NOT use -mtp — speculative decoding regresses on MoE (measured ~−28% on this model). These quants also ship no MTP head. Leave it off.
-ub 512 (default) is fastest here; -ub 2048 is ~8% slower on this MoE.

Single-stream decode is very fast for a 35B thanks to the ~3B active params — on the order of 175–200+ t/s at IQ4_K on a single modern 24 GB GPU.

Credits & license

Base model: Qwen/Qwen3.6-35B-A3B, Apache-2.0. Quantized locally with ik_llama.cpp + imatrix. Quant types courtesy of ik_llama.cpp.

hyperhuzaifa/Qwen3.6-35B-A3B-IK-K-GGUF

Qwen3.6-35B-A3B — ik_llama IK_K quants (GGUF)

Files

Usage (ik_llama.cpp)

Credits & license

hyperhuzaifa/
Qwen3.6-35B-A3B-IK-K-GGUF