Sensitivity-guided mixed-precision quantization of Qwen/Qwen3.5-122B-A10B. 122 billion parameters, 10 billion active per token. Hybrid SSM/attention MoE architecture with 256 experts per layer, 8 active.
Compressed from 228 GB (BF16) to 32.7 GB at 2.30 bits per weight. Runs on a single RTX 3090 (24 GB VRAM + 34 GB system RAM) at 25 tokens/second.
Quantization Strategy
Single-pass from BF16 safetensors using llama.cpp llama-quantize:
Base: IQ2_XXS across all tensors
Promotion: ffn_down_exps elevated to Q2_K (these carry the actual expert computation)
No demotions — IQ1_S on expert weights destroys MoE routing quality
This targets the projection weights that matter most for output quality while keeping the model small enough for consumer hardware.
Benchmarks
Measured directly on this GGUF with llama.cpp on RTX 3090. The model-index metadata in this card's frontmatter mirrors these numbers.
Benchmark
Score
Perplexity (wikitext-2, 2048 ctx)
6.73
HellaSwag
92.04%
ARC-Challenge
95.56%
EvalPlus HumanEval (base)
59.8%
EvalPlus HumanEval+
54.9%
Unquantized Qwen 3.5 122B-A10B baseline for comparison: HellaSwag ~95%, ARC ~97% (estimated from Qwen's published results on similar benchmarks). Reasoning capability is well-preserved; code generation takes the expected hit at 2-bit precision.
Recommended Settings
Temperature: 0.6 for general use, 0.0 for benchmarks and deterministic tasks.
Thinking mode: This model supports Qwen's thinking/reasoning mode. For general chat, leave it enabled. For structured tasks (benchmarks, classification, code completion), disable it:
--chat-template-kwargs '{"enable_thinking":false}' --reasoning off
This keeps KV cache and attention layers on GPU while offloading the large expert gate/up matrices to system RAM. The 10B active parameters per token means only a fraction of the 122B total moves through the GPU each forward pass.
Memory usage: ~18 GB VRAM, ~34 GB system RAM.
Hardware requirements:
Minimum: 24 GB VRAM GPU + 40 GB RAM
Recommended: RTX 3090/4090 + 64 GB RAM for comfortable headroom
Build History
This is the result of iterative experimentation on this architecture:
v1 — Uniform IQ2_XXS baseline (34 GB). Functional but no targeted optimization.
v3 — Attempted IQ1_S demotions on gate/up expert weights. PPL exploded to 8.02. Proved that MoE expert weights cannot survive sub-2-bit quantization.
v4 (this release) — Promoted ffn_down_exps to Q2_K, no demotions. 32.7 GB, strong reasoning scores, best balance of size and quality.
Key insight: MoE models are fragile in the opposite way from dense models. In dense architectures, FFN gate weights can be crushed freely. In MoE, the expert weights carry everything — crush those and routing falls apart.
About Cerebellum
Cerebellum is a sensitivity-guided quantization methodology. Instead of uniform compression, it measures which weight groups survive extreme quantization and which need protection, then allocates precision accordingly. The goal is making large models accessible on consumer hardware without destroying the capabilities that make them worth running.
