AWQ (W4A16) quantization of microsoft/Mage-VL — a ~5B streaming vision-language model from Microsoft (mage_vl, custom architecture): a Mage-ViT visual encoder + a Qwen3-4B-Instruct language backbone, with native support for proactive streaming and neural-codec video. Only the Qwen3 LM backbone is quantized here; the vision encoder is kept in BF16.

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16. Quantized by: sahilchachra Tooling: llm-compressor (AWQModifier + QuantizationModifier) -> compressed-tensors pack-quantized

This is a quantized derivative. Weights, behavior, and license follow the base model — see the original card for full details, benchmarks, and citation.

What is quantized

Quantized to 4-bit:

LM backbone model.language_model.*.self_attn.{q,k,v,o}_proj
LM backbone model.language_model.*.mlp.{gate,up,down}_proj (all 36 layers)

Kept in BF16: Mage-ViT vision encoder (model.visual.*), token embeddings, lm_head, all norms (incl. q_norm / k_norm).

Decode speed (measured)

Benchmarked on an NVIDIA Thor (Blackwell, aarch64) via transformers generate(), single 448px image + text prompt, greedy (do_sample=False), 96 new tokens, GPU otherwise idle:

Variant	Decode speed	On-disk size
BF16 (base)	~16.8 tok/s	~9.5 GB
NVFP4A16	~14.0 tok/s	~6.0 GB
AWQ W4A16	~11.7 tok/s	~3.9 GB

Quantization here reduces size, not decode latency. On a 5B VLM at batch=1 on the eager transformers path, decode is compute / vision-encoder bound rather than weight-bandwidth bound, so 4-bit weights bring no bandwidth win and the on-the-fly dequant (FP4→BF16 for NVFP4, int4-unpack for AWQ) adds a little overhead — so the quantized variants are actually slightly slower than BF16 in this setup. The benefit is memory / disk footprint (AWQ ≈ 40% of BF16). Turning that size reduction into a throughput win would require a serving stack with native 4-bit kernels (vLLM / TensorRT-LLM), which do not yet implement the mage_vl architecture.

Runtime

Load with transformers + trust_remote_code=True and the repo's AutoProcessor (custom mage_vl code); vLLM has no mage_vl implementation yet.

Calibration

AWQ: 64 sequences x 512 tokens of HuggingFaceH4/ultrachat_200k rendered through the tokenizer's chat template (text-only calibration of the LM backbone). NVFP4 is data-free.

Prompt template & sampling

Custom mage_vl architecture — load with trust_remote_code=True and the repo's AutoProcessor. OpenAI-style messages with image content, e.g. messages=[{"role":"user","content":[{"type":"image"},{"type":"text","text":"Describe this image."}]}], rendered via processor.apply_chat_template(...), images passed as PIL objects via images=[...]. The card recommends do_sample=False for deterministic outputs. Note: vLLM does not yet implement mage_vl, so run it via transformers.

Recommended sampling: do_sample=False (deterministic) per the model card; max_new_tokens per use case.

Usage (vLLM)

python

from vllm import LLM, SamplingParams

# Weight-only quantized (custom architecture -> requires trust_remote_code).
# Load like the original model in any runtime that implements this arch.
llm = LLM(
    model="sahilchachra/Mage-VL-AWQ",
    trust_remote_code=True,
)
out = llm.chat(
    [{"role": "user", "content": "Hello!"}],
    SamplingParams(temperature=0.6, top_p=0.95, max_tokens=512),
)
print(out[0].outputs[0].text)

Serving via the CLI, pass the flag directly:

bash

vllm serve sahilchachra/Mage-VL-AWQ \
    --trust-remote-code \
    --max-model-len 262144