NVFP4A16 (4-bit weights, 16-bit activations) quantization of meta-models/Muse-Glimmer-30B, produced with llm-compressor and verified end-to-end on an RTX PRO 6000 Blackwell (sm_120).

22.2 GiB of weights (down from 55.8 GiB BF16), vision intact, and it works with the model's DFlash speculative decoding drafter.

The repo id keeps the NVFP4 name for URL stability, but this checkpoint is the A16 variant: weights are NVFP4, activations stay 16-bit. It replaced a W4A4 build that was both slower and less accurate — see Why A16 and not W4A4.

Requires a patched vLLM. Muse Glimmer support is not in any vLLM release yet — it is upstream PR #51655, which needs five further fixes before DFlash will start. See Running it.

What was quantized


Format	`compressed-tensors`, `nvfp4-pack-quantized`
Scheme	NVFP4A16: 4-bit float weights, group size 16, activations left at 16-bit
Rounding	GPTQ error-compensating (Hessian-weighted), not round-to-nearest
Calibration	512 samples × 2048 tokens from `HuggingFaceH4/ultrachat_200k`
Quantized	language-model `q/k/v/o_proj`, `mlp.gate/up/down_proj`, `self_attn.gate_proj`
Left in BF16	the entire ViT-G/14 vision tower, vision adapter + projection, `lm_head`, embeddings, all norms

The vision stack stays BF16 deliberately: it is a small share of the weights, it runs once per image rather than once per token, and FP4 activations through a ViT cost more accuracy than they save time. Image understanding is unaffected — the quantized model still reads the 中華門 characters off a test photo of a Chinatown gate.

Note that self_attn.gate_proj (Muse Glimmer's per-head sigmoid attention output gate) is a different tensor from mlp.gate_proj. The recipe's regexes are scoped so they cannot collide.

Why A16 and not W4A4

This repo originally held a W4A4 build. A16 replaced it because W4A4 was worse on both axes at single-stream:

	VRAM	Decode	+ DFlash	PPL
NVFP4 W4A4	22.2 GiB	54.3 tok/s	97.9	5.7660
NVFP4A16 + GPTQ (this)	22.2 GiB	74.7 tok/s	139.3	5.5631

Identical weight format and identical bytes streamed per token — only the activation side differs. W4A4 is slower because at batch size 1 a quantized matmul is a GEMV: FP4 tensor cores never engage, but dequant and block-scale work still costs. vLLM picks CutlassNvFp4LinearKernel for W4A4 and MarlinNvFp4LinearKernel for W4A16, and Marlin is built for low-batch decode. Measured against this card's 1629 GB/s read-only bandwidth, W4A4 reaches 56% of the roof; A16 reaches 77%.

W4A4 only earns its keep at high concurrency, where the GEMM is compute-bound. If you serve many concurrent requests, build the W4A4 variant (--scheme NVFP4) — but pick its kernel explicitly, see below.

Pick the FP4 kernel explicitly; vLLM's default is not the fastest

On W4A4, vLLM's automatic choice measured 54.3 tok/s. Forcing FlashInfer's CUTLASS path measured 75.7 tok/s — 1.39x faster for free:

W4A4 kernel	tok/s
`CutlassNvFp4LinearKernel` (auto)	54.3
`FlashInferCutlassNvFp4LinearKernel`	75.7

vllm serve ... --kernel-config '{"linear_backend":"flashinfer_cutlass"}'

Two traps. First, FlashInfer silently disables itself if nvcc is not on PATH — vLLM JIT-compiles its kernels and falls back without a clear error, so export PATH=/usr/local/cuda-<ver>/bin:$PATH before serving. Second, the CuteDSL backend is gated to sm_10x (B200-class); on RTX Blackwell (sm_120) it is unavailable no matter what you pass. W4A16 always uses Marlin, since FlashInfer's FP4 kernels need quantized activations.

Quality

Perplexity on wikitext-2, 24 x 1024-token chunks, scoring only the second half of each chunk with BOS prepended — matching llama-perplexity's const int first = n_ctx/2, so these are directly comparable to llama.cpp numbers.

Variant	read/token	PPL	vs BF16
BF16 (original)	55.4 GB	5.4555	—
llama.cpp Q4_K_XL (GGUF)	15.11 GB	5.5421	+1.59%
This model (NVFP4A16 + GPTQ)	16.85 GB	5.5631	+1.97%
NVFP4A16, plain round-to-nearest	16.85 GB	5.5965	+2.58%
NVFP4 W4A4	16.85 GB	5.7660	+5.69%

Be aware of the honest comparison here: Unsloth's Q4_K_XL GGUF is better quality and smaller. K-quants allocate bits non-uniformly, spending more on layers that need them, whereas this recipe applies uniform 4-bit to every targeted linear. If your priority is quality per byte on a single stream, llama.cpp with Q4_K_XL is the better artifact. This checkpoint's advantages are throughput with DFlash (136.5 vs 116.7 tok/s), concurrent serving, and the vLLM stack.

GPTQ is what got this checkpoint from 5.5965 to 5.5631, closing 61% of the gap between plain round-to-nearest and Q4_K_XL. It is the lever that matters for weight-only schemes: with activations unquantized there are no activation scales to fit, so simply adding calibration data does comparatively little. Build it with --scheme NVFP4A16 --gptq.

Performance

RTX PRO 6000 Blackwell (95 GiB, sm_120), vLLM 0.26.1rc1.dev564, single request, greedy, 256 output tokens:

Config	Decode	+ DFlash
BF16	27.8 tok/s	60.0 tok/s
FP8	49.5 tok/s	94.3 tok/s
NVFP4 W4A4	54.3 tok/s	97.9 tok/s
NVFP4A16 + GPTQ (this model)	74.7 tok/s	139.3 tok/s

139.3 tok/s is 5.0x the BF16 baseline, and faster than llama.cpp's best configuration on the same GPU (116.7 tok/s with Q4_K_XL + DFlash).

On bytes per token

Muse Glimmer is a dense 29.6B model — every parameter is read on every decoded token. This checkpoint streams 16.85 GB per token (14.16 GB of layers plus a 2.69 GB BF16 lm_head). Measured read-only bandwidth on this card is ~1633 GB/s, giving a hard ceiling near 97 tok/s before speculation, independent of kernel quality.

If you have seen substantially higher tok/s from a similarly-sized dense NVFP4 model, the likely explanation is speculative decoding, not a different architecture. The bandwidth ceiling above limits forward passes per second, not tokens per second — speculation emits several tokens per target forward and legitimately exceeds it. This very checkpoint does: its no-speculation ceiling is ~97 tok/s, yet it measures 136.5 tok/s with DFlash.

Engine support

engine	this checkpoint	speed (single stream, +DFlash)
vLLM (PR #51655 + fixes)	yes	74.7 -> 139.3 tok/s
SGLang (fork below)	yes	70.7 -> 111-207 tok/s
SGLang upstream	not yet	no dense NVFP4 weight-only scheme (sgl-project/sglang#33711)
llama.cpp	n/a	GGUF only

Running on SGLang

Upstream SGLang cannot serve weight-only NVFP4 yet; support lives on cloudnathan5/sglang@nvfp4-w4a16-compressed-tensors (Muse Glimmer branch from sgl-project/sglang#34262 plus a CompressedTensorsW4A16Fp4 scheme over SGLang's existing FP4 Marlin kernels — 201 lines, no new kernels).

bash

pip install "sglang[all]"        # released wheels supply the compiled kernels
git clone -b nvfp4-w4a16-compressed-tensors https://github.com/cloudnathan5/sglang
pip install -e sglang/python             # overlays the branch and upgrades
                                         # transformers/sglang-kernel to what it needs
export FLASHINFER_DISABLE_VERSION_CHECK=1  # branch/wheel cubin skew; benign

Serve with DFlash (note the reasoning parser is registered as muse in SGLang, not muse_glimmer as in vLLM):

python -m sglang.launch_server --model-path cloudnathan5/Muse-Glimmer-30B-NVFP4 --speculative-algorithm DFLASH --speculative-draft-model-path meta-models/Muse-Glimmer-30B-assistant --speculative-dflash-block-size 16 --reasoning-parser muse

Measured on an RTX PRO 6000 Blackwell (suite: 3 prompt classes x 2 lengths x repeats; baseline 70.7 tok/s in every cell):

workload	256 tok	1024 tok
open-ended	110.8	137.0
coding	160.7	195.5
agentic	166.0	206.6

That makes SGLang + this checkpoint the fastest quality-preserving configuration measured for this model: ~200 tok/s on the coding/agentic workloads it is built for, at the best quantized perplexity (5.5631). A W4A4 build is faster only on short open-ended generations, and costs quality (5.7209).

Two notes if you go off this path: SGLang's fp4 backend auto-selection breaks on SM120/SM121 for W4A4 checkpoints (pass --fp4-gemm-runner-backend flashinfer_cutlass; not needed for this W4A16 checkpoint, which uses Marlin directly), and upstream W4A16 support is being worked in #33711 — prefer upstream once it lands.

Related checkpoints

Preyazz/Muse-Glimmer-30B-NVFP4 is the same scheme (compressed-tensors nvfp4-pack-quantized, W4A16) and is tagged for DGX Spark / GB10. Its card states it is weight-only with no calibration data — i.e. plain round-to-nearest. This checkpoint is GPTQ-calibrated, which measured 5.5631 vs 5.5965 PPL for RTN under the same methodology. If you are on a Spark, that card also documents the aarch64 install path, which is the same VLLM_USE_PRECOMPILED=1 route used here.

Running it

Muse Glimmer is not in any vLLM release (0.26.0 is current; the official recipe targets an unreleased 0.27.0+). Support is upstream PR vllm-project/vllm#51655.

For DFlash you need more than that PR. As published it cannot start an engine with speculative decoding — five separate blockers, each surfacing only after the previous one is fixed. The PR author has them fixed in xianbaoqian/vllm#1 (registry name rewriting, the Eagle3 .model unwrap, a real MuseGlimmerAssistantConfig preserving sliding_window and vocab_size, and the encoder.* draft weight names). Use that; it is the canonical fix and is more complete than the branch these benchmarks were run on.

bash

git clone https://github.com/vllm-project/vllm.git && cd vllm
git fetch origin pull/51655/head:muse-glimmer && git checkout muse-glimmer
# until xianbaoqian/vllm#1 is merged into that branch, also apply it:
git fetch https://github.com/xianbaoqian/vllm.git fix-spec-decode && git merge FETCH_HEAD

uv venv --python 3.12 && source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install -e .

Python 3.12 is required — vLLM does not support 3.13+ yet. Every change is pure Python, so VLLM_USE_PRECOMPILED=1 avoids a CUDA build. If it 404s looking for a wheel matching the branch's base commit, pin a nearby nightly explicitly:

export VLLM_PRECOMPILED_WHEEL_LOCATION="https://wheels.vllm.ai/3a79957b62ade336010cc052e322d0005eb091a2/vllm-0.26.1rc1.dev554%2Bg3a79957b6-cp38-abi3-manylinux_2_28_x86_64.whl"

This checkpoint was verified against that upstream route directly: loaded on PR vllm-project/vllm#51655 merged with fix-spec-decode, it serves at 141.9 tok/s with DFlash, matching the 139.3 measured on the branch below (run-to-run variance). No quantization-specific changes are needed on either.

The numbers on this card were measured on cloudnathan5/vllm@muse-glimmer-fork, an independently-derived branch carrying equivalent fixes. It is kept only as provenance for those measurements — prefer the upstream PR above.

Then serve. The reasoning and tool-call parsers must be used together — Muse Glimmer emits neither <think> tags nor JSON tool calls, only channel-scoped messages:

vllm serve cloudnathan5/Muse-Glimmer-30B-NVFP4 --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --max-model-len 32768

With DFlash speculative decoding (drafter is a separate ~5 GB download):

vllm serve cloudnathan5/Muse-Glimmer-30B-NVFP4 --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --max-model-len 32768 --speculative-config '{"method":"dflash","model":"meta-models/Muse-Glimmer-30B-assistant","num_speculative_tokens":16}'

num_speculative_tokens should stay at 16 — that is the drafter's native block_size. Lowering it to 4 measured slower (89.6 vs 97.9 tok/s).

Sampling defaults recommended by Meta: temperature=1.0, top_p=0.95, top_k=64.

Notes and caveats

Greedy output under DFlash differs from non-speculative decode. This is expected: DFlash is deterministic run-to-run, and every measured divergence sits at a near-tie (top1−top2 logprob gaps of 0.000, 0.063, 0.125) — bf16 tie-breaking under different GEMM shapes, not a verify-path defect.
Raw LLM.generate() returns the channel framing verbatim (assistant to=self / assistant to=user); use the server with --reasoning-parser muse_glimmer to split reasoning from content.
vLLM logs Multi-modal warmup failed at startup. Non-fatal; images work.
Unsloth's Q4_K_XL GGUF beats this on quality per byte; see Quality above.
The perplexity comparison is one corpus. It ranks the quants consistently but is not a substitute for a task benchmark on agentic workloads, which is what this model is built for.
PR #51655 is unmerged. Once it lands and is rebased, some of the five fixes may become unnecessary.

License and attribution

Apache 2.0, inherited from meta-models/Muse-Glimmer-30B. This is a quantized derivative; all model capabilities and limitations are those of the original. Meta's USAGE_POLICY.md from the base repository applies.

Quantized with llm-compressor 0.12.1a (--scheme NVFP4A16). Upstream vLLM model support is the work of PR #51655 by @xianbaoqian.