Qwen3.5-9B — BitClass2 Mixed-Precision GGUF
Mixed-precision GGUF quantizations of
Qwen3.5-9B
using
Hessian-informed per-tensor bit allocation. Each tensor group receives
the precision level that minimizes quality loss for its measured sensitivity.
Available Quantizations
Recommended: Q4_K_M — nearly matches Q6_K quality (PPL 1.875 vs 1.863) at 17% less size.
How It Compares
| Model | BPW | Size | PPL ↓ | Source |
|---|
| ByteShape IQ3_S 3.00bpw | 3.0 | 3.37 GB | 2.069 | byteshape |
| ★ Ours Q3_K_S | 3.3 | 3.66 GB | 2.001 | This repo |
| ★ Ours Q4_K_M | 4.7 | 5.21 GB | 1.875 | This repo |
| ★ Ours Q5_K_M | 5.1 | 5.70 GB | 1.865 | This repo |
Our Q3_K_S beats ByteShape's 3.00bpw 9B on perplexity (2.001 vs 2.069 — 3.3% better), at a larger file (3.66 vs 3.37 GB). ByteShape's higher-BPW rows reach lower PPL.
The PPL curve is remarkably flat from Q6_K to Q4_K_M: going from 6.30 GB down to
5.21 GB (saving ~1.1 GB) only costs 0.012 PPL. This is the mixed-precision allocation
working — gate_proj/up_proj drop to Q4_K while down_proj and attention stay at Q6_K.
Key Sensitivity Findings (Qwen3.5-9B)
The Hessian sensitivity pattern for 9B is fundamentally different from 4B:
- blk.3 (early layer) is most sensitive — score 1.0 for k/v. On 4B it was blk.34 (late layer).
- Sensitivity peaks at both ends AND middle: blk.3 (1.0), blk.7 (0.78), blk.23 (0.78), blk.27 (0.86), blk.31 (0.87)
- ffn_down at blk.4-5 is near-zero sensitivity (0.0003) — safe for aggressive quantization
- This confirms: model-specific Hessian data matters. You cannot assume late layers are always most sensitive.
How It Works
- Hessian sensitivity — compute H_diag = mean(X²) per layer on calibration data
- LP-optimal allocation — solve knapsack: minimize Σ(sensitivity × quant_error) subject to size ≤ target
- Per-layer variation — within each suffix group, vary types by layer using imatrix + Hessian blend
- GGUF export — llama.cpp
--tensor-type-file for per-tensor overrides
Usage
1huggingface-cli download sh111111111111111/Qwen3.5-9B-BitClass2-GGUF \
2 Qwen3.5-9B-Q4_K_M.gguf --local-dir .
3
4llama-cli -m Qwen3.5-9B-Q4_K_M.gguf -cnv
5llama-server -m Qwen3.5-9B-Q4_K_M.gguf --port 8080
6
Benchmark Details
NVIDIA GB10 ATOM (128GB unified memory, aarch64). llama.cpp commit 406f4e3.
PPL via llama-perplexity (2 chunks, 851 context). tok/s via llama-bench (tg128, ngl=999).
Disclaimer
Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.
License
Apache 2.0, inherited from
Qwen3.5-9B.
