Muse-Glimmer-30B-W8A16-AutoRound
This is an unofficial mixed-precision INT8 quantized version of
Muse-Glimmer-30B. It was created using
AutoRound.
Quantization details
Some layers are not quantized as quantizing them causes a heavy downgrade, or there is no calibration signal available to quantize them with. The model shrinks from 59.5 GB to 33.5 GiB.
| Precision | Layer Count | Target Layers |
|---|
| INT8 (group_size=-1, symmetric) | 260 | o_proj, gate_proj of self_attn, gate/up/down_proj of mlp |
| FP16 (unquantized) | 461 | q_proj / k_proj / v_proj of self_attn, embed_tokens, lm_head, vision_tower, vision_adapter, vision_projection |
k_proj and v_proj are kept unquantized because this model uses GQA with a 16:1 ratio (32 query heads, 2 KV heads), so an error in one KV element is amplified across 16 query heads. They are also only [256, 6656] each, so keeping all 104 of them in FP16 costs about 354 MB.q_proj is unquantized for a mechanical reason rather than an accuracy one. vLLM's native implementation fuses q/k/v into a single QKVParallelLinear, and one packed tensor cannot hold two different quantization schemes, so the loader rejects any checkpoint where the three disagree. Raising q_proj to FP16 costs 1.32 GiB; dropping k_proj and v_proj to INT8 instead would have been cheaper but would have discarded the reason above. Revisions of this repo before 2026-08-11 had q_proj at INT8 and fail to load on that path.- The vision tower is kept unquantized.
- Calibration configuration:
iters=250, nsamples=512, batch_size=4, with gradient_accumulate_steps=2.
A note for anyone reproducing this with a different layer configuration: the shared-dict aliasing bug in AutoRound is still present in 0.15, it has only moved from compressors/utils.py to compressors/layer_config/resolver.py. The regex expansion still assigns the same dict object to every matched layer, and the later shape check that forces bits=16 on layers whose weights are not divisible by 32 mutates that shared dict, silently dragging every sibling layer to 16 bit with it. Change
layer_config[match] = val
to
layer_config[match] = copy.deepcopy(val)
if any of your keywords can also match a shape-incompatible layer. It did not affect this model, since every layer selected for INT8 here has both dimensions divisible by 32, and the only shape-incompatible layer in the model is the vision patch embedding, which is excluded anyway.
Evaluation Results
1. KL Divergence (KLD)
The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.
| Metric @ctx 512 | Value |
|---|
| Mean KLD | 0.00130 ± 0.000015 |
| Median KLD | 0.000644 |
| P90 / P95 / P99 KLD | 0.00275 / 0.00403 / 0.00928 |
| P99.9 / Max KLD | 0.0338 / 0.8118 |
| Top-1 Agreement | 98.53% |
| PPL (Base → Quantized) | 7.8194 → 7.8241 |
| ln(PPL Ratio) | 0.000608 |
Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.
- The tail is light for an INT8 quant, with a maximum per-token KLD of about 0.81 nats. Whether the untouched KV projections deserve the credit is a guess on my part, not something I isolated with an ablation.
- Raising
q_proj from INT8 to FP16 is the only difference from the previous revision, and both runs were scored against the same cached reference distribution, so this one is a clean single-factor comparison. It moved mean KLD from 0.00136 to 0.00130, max KLD from 1.049 to 0.8118, and ln(PPL ratio) from 0.000750 to 0.000608. It says nothing about k_proj and v_proj, which were FP16 in both runs.
2. Benchmarks
Not run. you have to do it yourself.
How to Use
There are two ways to serve this model as of 2026-08-11 and they behave differently.
Native support lives in
vLLM PR #51655, which is still open and is in neither a release nor
main. The preview image
vllm/vllm-openai:muse-glimmer carries it, resolves the model to
MuseGlimmerForCausalLM, and ships the
muse_glimmer tool-call and reasoning parsers. This is the faster path and the one this quant is now aligned with.
Released and nightly vLLM have no native muse_glimmer implementation and fall back to TransformersMultiModalForCausalLM. That path still works, with these caveats, none of which are caused by the quantization.
--tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer do not exist there and will abort startup if you pass them. Without them the model still generates fine, but the reasoning channel is not stripped from the response.- With tensor parallelism the vision tower fails during multimodal memory profiling with
mat1 and mat2 shapes cannot be multiplied (16384x768 and 1536x1536), where 768 is 1536 split across 2 ranks. This happens inside default_unquantized_gemm, on weights this quant never touched. Pass --language-model-only. Setting mm_encoder_tp_mode=data does not help, because the Transformers backend reports supports_multimodal_encoder_tp_data=False and falls back to weight splitting.
- The Transformers backend does not fuse q/k/v, which is why revisions of this repo before 2026-08-11 loaded there despite mixing INT8
q_proj with FP16 k_proj and v_proj. The native implementation does fuse them, and that is what forced the layer table above to change.
Measured on 2x RTX 3090 with --tensor-parallel-size 2 --language-model-only --gpu-memory-utilization 0.95 --max-model-len auto --max-num-seqs 1 -O3 --async-scheduling, without kv-cache quantization.
| Path | Weights + non-torch per rank | KV cache | Concurrency @ 131,072 ctx |
|---|
Native, vllm/vllm-openai:muse-glimmer | 15.54 GiB | 822,190 tokens | 6.27x |
Transformers fallback, vllm/vllm-openai:nightly | 18.92 GiB | 460,638 tokens | 3.51x |
You can append --kv-cache-dtype fp8 to optimize memory further.
Speculative decoding with the
DFlash drafter that the vLLM recipe advertises does not currently work, for reasons unrelated to this quant. Four separate defects are present at the PR head: the draft architecture name that
EAGLEConfig derives,
DFlashMuseGlimmerAssistantModel, is absent from the model registry, which only has the unprefixed
MuseGlimmerAssistantModel; the default
SupportsEagle3 hooks assume a two-level
language_model.model layout that
MuseGlimmerForCausalLM does not have; vLLM parses the drafter config as
Qwen3Config, which discards
sliding_window unless
use_sliding_window is also set; and the Qwen3 DFlash loader does not recognise the drafter's
encoder. weight prefix. The first of these fires before any weight is read, so the feature cannot have been exercised on any hardware.
Acknowledgements
- Meta Superintelligence Lab for the base Muse-Glimmer-30B model.
- Intel AutoRound team for the quantization framework.
- vLLM project for the inference engine.
