This is an INT4 W4A16 quantization of meta-models/Muse-Glimmer-30B produced with AutoRound (auto_round 0.14.2). It quantizes the 52-layer text decoder to 4-bit weights (group size 128, symmetric) while keeping the vision tower, embeddings, norm layers, adapter and the LM head in BF16 — so the full multimodal (image/text) capability of the original model is preserved.

The quantization is text-coherent at essentially BF16 quality: on a held-out general-prose set it matches (and even slightly improves on) the BF16 base in perplexity, with a mean KL divergence of only ~0.001 nats/token.

Scope note on metrics: PPL and KL are measured on a small held-out set (4 general-prose paragraphs in history / biology / literature / astronomy — not from the calibration domains, 431 tokens total) via a disk-offload streaming prefill of the text decoder. These are relative precision indicators against the same BF16 base, not broad general benchmarks.

Key results (text decoder)

Quant	PPL (lower better)	Mean NLL/token (nats)	Mean KL vs BF16 (nats/token)
BF16 base (reference)	1.00218	0.00218	— (0.0)
AutoRound INT4 w4g128 (this repo)	1.00163	0.00163	0.001114

Per-example KL(BF16 ‖ AutoRound): 0.00127, 0.00032, 0.00068, 0.00217.

The AutoRound INT4 output is three orders of magnitude tighter to the BF16 base than the earlier AWQ INT4 attempts on this model (AWQ-v2 measured ~0.46 nats/token), and it reproduces coherent reasoning, tool calling, and vision understanding when served.

Quantization details

Tool: AutoRound (auto_round==0.14.2)
Scheme: INT4 W4A16 — bits=4, group_size=128, sym=True, data_type="int"
Calibration: nsamples=72 (28 valid after the seqlen≤1024 filter), iters=300, seqlen=1024, batch_size=1, lr=5e-3, gradient_accumulate_steps=2, seed=42
Quantized blocks: model.language_model.layers (all 52 text-decoder layers, 8 linears each — all g128-divisible)
Kept BF16: vision tower (model.vision_tower.*, ViT-G, 50 layers), model.vision_adapter, model.vision_projection, embeddings, norms, lm_head
Exported format: auto_round:auto_gptq packing (transposed qweight/qzeros/scales)
Size: ~21 GB across 8 safetensors shards; 416/2268 tensors quantized INT4
block_name_to_quantize in quantization_config.json is set to model.layers — this is the serving-compatible naming required by SGLang/vLLM to actually dequantize the text decoder on load (the internal AutoRound naming model.language_model.layers would be treated as unquantized and OOM on small GPUs).

Full calibration dataset blend

Calibration used a text blend of NVIDIA Hugging Face instruction/reasoning/agentic datasets (the same blend pipeline used for the sibling Qwen3.8-27B AutoRound work), chat-formatted via the model's own chat template and capped to the calibration sequence length. The on-disk blend is composed of the following 25 categories:

Category	# samples	Source dataset
Terminal / agentic	20	`nvidia/Nemotron-RL-Agentic-Terminal-Pivot-v1`
SWE / code	16	`nvidia/Nemotron-SFT-SWE-v3`
Reasoning / thinking	18	`nvidia/AceReason-1.1-SFT`
Chat / instruction-following	14	`nvidia/Nemotron-SFT-Instruction-Following-Chat-v3`
Function calling	14	`nvidia/Nemotron-RL-Agentic-Function-Calling-Pivot-v1`
OpenCode (input/output)	14	`nvidia/OpenCodeInstruct`
HelpSteer3 preference	12	`nvidia/HelpSteer3` (`preference`)
RAG / long-context	10	`nvidia/ChatRAG-Bench` (`coqa`)
Knowledge / web search	8	`nvidia/Nemotron-RL-knowledge-web_search-mcqa`
Reasoning gym	8	`nvidia/Nemotron-RL-ReasoningGym-v1`
Safety	8	`nvidia/Nemotron-RL-Safety-v1`
Math (AceReason)	8	`nvidia/AceMath-Instruct-Training-Data` (`math_sft`)
CUDA	6	`nvidia/Nemotron-SFT-CUDA-v1`
Structured outputs	6	`nvidia/Nemotron-RL-Instruction-Following-Structured-Outputs-v2`
Math (human reasoning)	6	`nvidia/Nemotron-Math-HumanReasoning`
Math (Nemotron)	6	`nvidia/Nemotron-Math-v2` (`medium`)
Workplace agent	6	`nvidia/Nemotron-RL-agent-calendar_scheduling`
Multilingual code (hi)	5	`nvidia/Nemotron-SFT-Multilingual-v2` (`code_hi`)
Multilingual math (ja)	4	`nvidia/Nemotron-SFT-Multilingual-v2` (`math_ja`)
HelpSteer3 edit	4	`nvidia/HelpSteer3` (`edit`)
Prompt injection	4	`nvidia/Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1`
Abstention	4	`nvidia/Nemotron-RL-QA-Abstention-v1`
Identity following	4	`nvidia/Nemotron-RL-Identity-Following-v1`
Multilingual STEM (pt)	3	`nvidia/Nemotron-SFT-Multilingual-v2` (`stem_pt`)
Multilingual code (pt)	2	`nvidia/Nemotron-SFT-Multilingual-v2` (`code_pt`)

Total: 210 samples across 25 categories. For this AutoRound run, 72 samples were drawn from the blend (with 28 surviving the seqlen≤1024 filter used for the Muse text-decoder calibration).

The vision tower is not quantized — it remains the original BF16 weights — so no vision calibration data is needed for the quantized text path.

Usage

The checkpoint is a standard AutoRound export and is intended to be served with SGLang or vLLM using the native auto-round quantization path.

vLLM / SGLang (validated on 2× RTX 3090)

bash

# SGLang
python -m sglang.launch_server \
  --model-path woooflet/Muse-Glimmer-30B-AutoRound-INT4 \
  --served-model-name "Muse Glimmer 30B" \
  --quantization auto-round \
  --tp-size 2 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-multimodal \
  --reasoning-parser muse --tool-call-parser muse

Transformers (AutoRound)

python

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "woooflet/Muse-Glimmer-30B-AutoRound-INT4",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("woooflet/Muse-Glimmer-30B-AutoRound-INT4")

Note: for inference, load with the auto-round / AutoRound quantized path in SGLang or vLLM (both verified). If you load raw weights with Transformers without the AutoRound runtime, tensors are stored in the packed qweight/qzeros/scales layout and require AutoRound's dequantization.

Validated behavior

When served with SGLang (--quantization auto-round, TP2, fp8_e4m3 KV, muse parsers), the quantized model was verified to be coherent:

Reasoning: correct reasoning↔content separation on math and multi-step problems
Tool calling: multi-step tool-use chains (search → read → contacts → calc → email) handled correctly; tool-eval-bench 87/100
Vision: reads charts, dashboards and photos coherently (chart values, UI elements, scene descriptions)
Performance: prefill ~1846 tok/s, generation ~54 tok/s, TTFT ~1.03s (2× RTX 3090)

License & attribution

Base model: meta-models/Muse-Glimmer-30B by Meta (License: Apache 2.0). This quantization inherits the base model's Apache 2.0 license.
Quantization performed with AutoRound (Apache 2.0).
Calibration data sampled from the NVIDIA Hugging Face datasets listed above; see each dataset for its individual terms.

This model is a community quantization of a Meta model and is not an official Meta release. Use and redistribute in accordance with the base model's license.

Author

Ford Morris