Qwen3-8B GSQ Q3_K_M GGUF
Quantized with GSQ (Gumbel-Softmax Quantization) using the Q3_K_M mixed-precision recipe.
Benchmark: WikiText2 Perplexity (llama-perplexity, 512 context)
| Model | Size | PPL | Delta vs F16 |
|---|
| F16 baseline | 16 GB | 10.36 | — |
| GSQ (this model) | 4.74 GB | 10.59 | +0.23 (+2.2%) |
| Unsloth Dynamic Q3_K_M | ~4 GB | 10.90 | +0.54 (+5.2%) |
| Standard Q3_K_M | ~4 GB | 11.39 | +1.03 (+9.9%) |
Quantization Recipe
Per-projection assignment (uniform across all 36 layers):
- Q3_K: gate_proj, up_proj, q_proj, k_proj
- Q4_K: down_proj, v_proj, o_proj
- Q6_K: embed_tokens
Effective average: ~3.4 bits per weight.
Method
GSQ applies GPTQ initialization followed by 2000 steps of Gumbel-Softmax refinement per linear layer, optimizing quantized weights against the Hessian-weighted reconstruction loss.
