Qwen3.6-27B-INT8-Autoround-V2
This is an unofficial mixed-precision INT8 quantized version of
Qwen3.6-27B.
It was created using
AutoRound.
Quantization details
Some layers are not quantized as quantizing them causes a heavy downgrade, or AutoRound simply doesn't support quantizing them.
| Precision | Layer Count | Target Layers |
|---|
| INT8 (group_size=-1, symmetric) | 410 | gate/up/down_proj, q/k/out_proj, in_proj_qkv, in_proj_z, mtp |
| FP16 (unquantized) | 98 | in_proj_a / in_proj_b of linear_attn, embed_tokens, visual |
- AutoRound doesn't support quantizing
in_proj_a and in_proj_b.
- Calibration configuration:
iters=250, nsamples=1024, batch_size=4, with gradient_accumulate_steps=2.
You need to change a line in utils.py of the AutoRound library
from:
layer_config[match] = val
to:
layer_config[match] = copy.deepcopy(val)
in order to quantize linear_attn.
To whoever wants to open a pull request upstream: if you believe this is actually a bug, I encourage you to do so.
Evaluation Results
1. KL Divergence (KLD)
The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.
| Metric @ctx 512 | Value |
|---|
| Mean KLD | 0.00182 ± 0.00014 |
| Median KLD | 0.000672 |
| P90 / P95 / P99 KLD | 0.00232 / 0.00349 / 0.00931 |
| P99.9 / Max KLD | 0.0668 / 9.156 |
| Top-1 Agreement | 98.32% |
| PPL (Base → Quantized) | 7.7415 → 7.7442 |
| ln(PPL Ratio) | 0.000343 |
Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.
- I heard quantizing linear_attn impacts the performance quite badly, but that's not the case here. Maybe not quantizing
in_proj_a / in_proj_b is the reason.
2. Benchmarks
Measured via lm-evaluation-harness (0.4.12) using the vLLM OpenAI-compatible API.
- Settings: 0-shot,
temperature=0.6, top_p=0.95, max generation tokens 65,536.
- Comparison baseline: Official metrics from the Qwen3.6-27B model card.
| Benchmark | This Model | Base Model (Official) | Difference | n |
|---|
| MMLU-Redux | 91.9% ± 1.6 | 93.5 | -1.6 | 285 |
| MMLU-Pro | 83.8% ± 1.8 | 86.2 | -2.4 | 420 |
- Performance drops in knowledge-focused benchmarks (MMLU-Redux, MMLU-Pro) are minor and lie mostly within the standard error. I wanted to run more benchmarks, but it's quite hot here and AC doesn't seem to work well. You have to do it yourself.
How to Use
This model has been tested using the latest vllm-openai:nightly.
On a 2x RTX 3090 setup, you can get 204,800 context without kv-cache quantization.
You can append --kv-cache-dtype fp8 to optimize memory further.
Acknowledgements
- Alibaba / Qwen team for the base Qwen3.6-27B model.
- Intel AutoRound team for the quantization framework.
- vLLM project for the inference engine and Qwen3.x MTP support.
