Qwen3.5-9B — BitClass Mixed-Precision GGUF
Mixed-precision GGUF quantizations of
Qwen3.5-9B using
learned per-tensor quantization profiles. Each tensor group receives the precision level that minimizes quality loss for its importance — more bits where they matter, fewer where they don't.
Qwen3.5-9B uses a hybrid DeltaNet + Attention architecture (24 DeltaNet layers + 8 standard attention layers). Our pipeline includes full support for DeltaNet's unique tensor groups (in_proj_qkv, in_proj_z, out_proj) alongside standard attention and MLP tensors — 10 suffix groups, 250 weight tensors total.
Seven precision levels from compact (3.0 bpw) to high quality (5.0 bpw).
Models
Target BPW is the planner's per-tensor bit budget (and the filename label). The actual
whole-file BPW runs ~0.3–0.4 higher, because output/embedding tensors are kept at higher
precision and GGUF carries metadata overhead — see the Size column for the real footprint.
Recommended: MX-3.5bpw for the best quality-to-size ratio. MX-3.0bpw for maximum compression. MX-4.5bpw for the highest quality in this ladder.
How It Compares
| Model | BPW | Size | PPL ↓ | Source |
|---|
| ByteShape IQ3_S 2.81 | 2.81 | 3.15 GB | 2.218 | byteshape |
| ByteShape IQ3_S 3.00 | 3.00 | 3.37 GB | 2.069 | byteshape |
| ★ Ours MX-3.0 | 3.0 | 3.79 GB | 2.099 | This repo |
| ByteShape IQ3_S 3.15 | 3.15 | 3.53 GB | 2.033 | byteshape |
| ★ Ours MX-3.2 | 3.2 | 3.82 GB | 2.072 | This repo |
| ★ Ours MX-3.5 | 3.5 | 4.38 GB | 1.957 | This repo |
| ByteShape IQ4_XS 3.60 | 3.60 | 4.04 GB | 1.947 | byteshape |
| ★ Ours MX-4.0 | 4.0 | 4.59 GB | 1.922 | This repo |
| ByteShape IQ4_XS 4.20 | 4.20 | 4.71 GB | 1.866 | byteshape |
| Bartowski Q3_K_S | 4.40 | 4.93 GB | 1.898 | bartowski |
| ★ Ours MX-4.5 | 4.5 | 5.24 GB | 1.879 | This repo |
| ★ Ours MX-5.0 | 5.0 | 5.46 GB | ~1.87 | This repo |
| Bartowski Q4_K_M | 5.50 | 6.17 GB | 1.856 | bartowski |
All models benchmarked in the same session on identical hardware (NVIDIA GB10 ATOM, GPU) for fair comparison.
Key Results
- Near-parity with ByteShape at low BPW: MX-3.0 (PPL 2.099) vs ByteShape 3.00 (PPL 2.069) — 1.4% behind, at a larger file (3.79 vs 3.37 GB)
- Competitive at mid-range: MX-3.5 (PPL 1.957) vs ByteShape 3.60 (PPL 1.947) — 0.5% behind, at 4.38 vs 4.04 GB
- Beats Bartowski Q3_K_S: MX-4.5 (PPL 1.879) vs Bartowski Q3_K_S (PPL 1.898) — 1.0% better, at a larger file (5.24 vs 4.93 GB)
DeltaNet Architecture
Qwen3.5-9B is not a standard transformer. It uses a hybrid architecture:
- 24 DeltaNet layers with linear attention (in_proj_qkv, in_proj_z, out_proj tensors)
- 8 standard attention layers (q_proj, k_proj, v_proj, o_proj)
- 32 MLP layers (gate_proj, up_proj, down_proj)
Our pipeline handles all 10 tensor suffix groups with appropriate quantization profiles for each.
Running with llama.cpp
1# Chat
2llama-cli -m Qwen3.5-9B-MX-3.5bpw.gguf -cnv
3
4# Server (OpenAI-compatible API)
5llama-server -m Qwen3.5-9B-MX-3.5bpw.gguf --port 8080
6
7# Benchmark
8llama-perplexity -m Qwen3.5-9B-MX-3.5bpw.gguf -f your_eval_data.txt
Benchmarking Details
All benchmarks run with llama.cpp (commit 406f4e3) on NVIDIA GB10 ATOM GPU with full offload (-ngl 999). Perplexity measured via llama-perplexity on a held-out evaluation set (20 chunks, 512 context). Throughput via llama-bench (512 prompt / 128 generation tokens). All models benchmarked in the same session.
Disclaimer
Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.
License
Apache 2.0, inherited from
Qwen3.5-9B.
Acknowledgments
- Base model by Qwen Team
- Importance matrix data from Bartowski
- Quantization infrastructure built on llama.cpp
