This repository contains the W2A16 quantized versions of Meta's Muse Glimmer 30B, optimized for extreme memory efficiency during local inference. The model was quantized using Intel AutoRound to 2-bit precision. To preserve the model's multimodal and agentic capabilities, the Vision Encoder was explicitly kept in its original precision (BF16).
The quantization was performed with high-accuracy calibration settings tailored for 2-bit compression:
Scheme: W2A16 (2-bit weights, 16-bit activations)
Group Size: 32 (Lowered to preserve accuracy in 2-bit)
Symmetric: True
Iterations: 1000
Vision Module: Kept unquantized (quant_nontext_module=False) to ensure maximum vision-language alignment.
Calibration: 512 samples, sequence length of 2048.
By compressing the language model weights to 2-bit precision, this version vastly reduces VRAM requirements, leaving ample room for the KV cache and the unquantized perception encoder to run smoothly on lower-VRAM consumer GPUs.
Usage with vLLM
For optimal performance, including agentic function calling and reasoning parsing, use the vllm/vllm-openai:muse-glimmer image or the latest vLLM build supporting Muse Glimmer.
Run the following command to serve the model (adjust the model path to the AutoRound or GPTQ repo as needed):
--tool-call-parser / --reasoning-parser: Configures the endpoint to natively handle Muse Glimmer's multi-step thinking and function schema outputs.
About the Base Model (Muse Glimmer 30B)
Muse Glimmer is a 30-billion-parameter causal language model with a dedicated perception encoder (~1.8B param ViT-G/14), distilled from Muse Spark by Meta Superintelligence Lab. It is purpose-built for autonomous agentic tasks on consumer hardware.
Key Capabilities:
End-to-end Agentic Task Completion: High success rates on DeepSearch QA, MCP-Atlas, 𝛕3-Bench, and SWE-Bench.
Multimodal Reasoning: Interprets interleaved text and images (charts, documents, screenshots).
Reliable Tool Use & Failure Recovery: Handles complex tool schemas and automatically diagnoses/retries failed calls.
Controllable Effort: Supports customizable reasoning strengths (low/medium/high/xhigh) via system prompts.
Multilingual: Trained on data from more than 100 languages.
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
