A GGUF quantization of Qwen3.6-35B-A3B built for larger-than-RAM /
SSD-streaming inference. Rather than quantizing every tensor to the same width,
it quantizes asymmetrically: the routed-expert weights (the bulk of the bytes
moved per token in a Mixture-of-Experts model) are pushed to ~2 bits, while
everything quality-sensitive (attention, router, shared experts, embeddings,
norms) stays at Q8_0.
It matches the 4-bit IQ4_XS baseline on quality (identical 9/10 coding) while
being ~33% smaller and streaming fewer cold bytes per token.
Why this exists - intelligence that fits on basic hardware
This quant is built for Fono, whose aim is to
run the most capable model that will actually fit on ordinary, memory-limited
hardware. The guiding principle is intelligence per gigabyte: spend precision only
where the model is genuinely sensitive, and compress everything else hard. That lets
a larger, smarter model drop into the same memory budget, so a modest machine gets
to punch well above its weight instead of settling for a smaller, weaker model.
What "asymmetric quant" means here
In an MoE, decode speed under memory pressure is dominated by how many
routed-expert bytes must be pulled from SSD per token. So this recipe spends
bits where they matter and saves them where they don't:
Tensor group
Precision
Why
Routed experts ffn_gate_exps / ffn_up_exps
IQ2_XXS (~2.06 bpw, imatrix)
dominant streamed mass — the big lever
Routed expert ffn_down_exps
Q2_K (imatrix)
smart 2-bit; dims are natively ÷256, so no padding needed
Note the architectural contrast with the gemma-4 sibling: Qwen uses separate
gate/up/down expert tensors plus always-on shared experts, and its routed
reduction dims are already 256-aligned (down_exps = 512, gate/up = 2048).
So the smart imatrix i-quants tile directly — no zero-padding required. This
is what makes the technique general: the only arch-specific wrinkle (alignment)
is a no-op here.
Only imatrix-aware i-quants (IQ2_XXS, Q2_K) are used for the 2-bit tensors;
crude legacy 2-bit formats at the same bit budget collapse model capability. The
importance matrix was computed with llama-imatrix over Bartowski's
calibration_datav3 (generic web/wiki text, deliberately disjoint from our
benchmark tasks so scores aren't inflated), 100 chunks × 512 tokens.
Why you might use it
There are two distinct wins:
It fits in RAM where the full-size model won't. At 11.74 GB the asymmetric
quant can sit entirely in memory on a machine where the 17.7 GB 4-bit baseline
would have to spill to SSD (or fail to load). Running fully in RAM is far faster
than streaming — so on that hardware the smaller-but-equally-smart model simply
is the faster model, and you get 35B-class capability where you otherwise
couldn't.
When it must stream, it streams less. Past the RAM budget it moves fewer cold
bytes per token than the uniform 4-bit build, so decode stays quick.
Other reasons: a Qwen3.6 GGUF that keeps full coding capability at ~⅔ the disk of the
4-bit build, and low bytes-moved-per-token rather than just a small file.
(If the full-size model already fits comfortably in your RAM, a uniform 4-bit build
is marginally faster there — fewer dequant ops. The asymmetric quant's advantage is
exactly the common case where it fits and the bigger one doesn't.)
The single lost coding point is the same hard Rust task the baseline's asym-class
peers also miss; general prose, polish, and the other 9 tasks are unaffected.
Files
qwen3.6-35b-a3b-asym.gguf (11.74 GB)
Usage (llama.cpp)
llama-cli -m qwen3.6-35b-a3b-asym.gguf -p "Write a prime sieve in Rust."
Limitations
~2-bit routed experts trade a little robustness on the hardest reasoning/parsing
tasks vs the 4-bit baseline.
