Mixed-precision GGUF quantizations of Qwen3-4B-Instruct-2507 using learned per-tensor quantization profiles. Each tensor group receives the precision level that minimizes quality loss for its importance — more bits where they matter, fewer where they don't.

Seven precision levels from ultra-compact (3.0 bpw) to high quality (4.5 bpw). Low-BPW models use IQ types for better quality-per-bit; higher levels use KQ types for faster inference.

Models

File	Target BPW	Size	PPL ↓	Family
Qwen3-4B-Instruct-2507-MX-3.0bpw.gguf	3.0	1.72 GB	3.073	IQ
Qwen3-4B-Instruct-2507-MX-3.2bpw.gguf	3.2	1.79 GB	3.034	IQ
Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf	3.4	1.88 GB	2.971	IQ
Qwen3-4B-Instruct-2507-MX-3.6bpw.gguf	3.6	2.02 GB	2.936	KQ
Qwen3-4B-Instruct-2507-MX-3.8bpw.gguf	3.8	2.03 GB	2.951	KQ
Qwen3-4B-Instruct-2507-MX-4.0bpw.gguf	4.0	2.01 GB	2.928	KQ
Qwen3-4B-Instruct-2507-MX-4.5bpw.gguf	4.5	2.47 GB	2.932	KQ

Target BPW is the planner's per-tensor bit budget (and the filename label). The actual whole-file BPW runs ~0.3–0.4 higher, because output/embedding tensors are kept at higher precision and GGUF carries metadata overhead — see the Size column for the real footprint.

Recommended: MX-3.4bpw for the best quality-to-size ratio. MX-3.0bpw for maximum compression. MX-4.0bpw for the highest quality in this ladder.

How It Compares

Model	BPW	Size	PPL ↓	Source
ByteShape KQ 3.34	3.34	1.69 GB	3.121	byteshape
★ Ours MX-3.0	3.0	1.72 GB	3.073	This repo
★ Ours MX-3.2	3.2	1.79 GB	3.034	This repo
Unsloth Q3_K_S	3.75	1.75 GB	3.007	unsloth
★ Ours MX-3.4	3.4	1.88 GB	2.971	This repo
★ Ours MX-3.6	3.6	2.02 GB	2.936	This repo
★ Ours MX-4.0	4.0	2.01 GB	2.928	This repo
Unsloth Q4_K_M	4.97	2.32 GB	2.956	unsloth
★ Ours MX-4.5	4.5	2.47 GB	2.932	This repo

All models benchmarked in the same session on identical hardware (NVIDIA GB10 ATOM, GPU) for fair comparison.

Key Results

Lower perplexity than ByteShape KQ 3.34: MX-3.0 (PPL 3.073) vs 3.121 — 1.5% better, at comparable size (1.72 vs 1.69 GB)
Lower perplexity than Unsloth Q3_K_S: MX-3.4 (PPL 2.971) vs 3.007 — 1.2% better, at 1.88 vs 1.75 GB
Lower perplexity than Unsloth Q4_K_M: MX-4.5 (PPL 2.932) vs 2.956 — 0.8% better, at 2.47 vs 2.32 GB

Running with Ollama

bash

# Pick any precision level
ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-BitClass-GGUF:Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf

Running with llama.cpp

bash

# Chat
llama-cli -m Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf -cnv

# Server (OpenAI-compatible API)
llama-server -m Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf --port 8080

# Benchmark
llama-perplexity -m Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf -f your_eval_data.txt

Benchmarking Details

All benchmarks run with llama.cpp (commit 406f4e3) on NVIDIA GB10 ATOM GPU with full offload (-ngl 999). Perplexity measured via llama-perplexity on a held-out evaluation set (20 chunks, 512 context). Throughput via llama-bench (512 prompt / 128 generation tokens). All models — ours, Unsloth, ByteShape — benchmarked in the same session.

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.

License

Apache 2.0, inherited from Qwen3-4B-Instruct-2507.

Acknowledgments

Base model by Qwen Team
Importance matrix data from Unsloth and Bartowski
Quantization infrastructure built on llama.cpp