LLM Compressor converted the BF16 weights of the language model's linear layers to MXFP8. No calibration data or fine-tuning was used. The vision tower, vision adapter, vision projection, and lm_head remain unquantized.

Quantization details

Property	Value
Format	OCP MXFP8 E4M3
Group size	32 elements
Scale data type	Unsigned 8-bit E8M0
Weight quantization	Static, per 32-element group
Activation quantization	Dynamic, per 32-element group
Layers quantized	416 language-model linear weights in 52 decoder layers
Layers skipped	`lm_head`, vision tower, vision adapter, vision projection
Checkpoint format	`compressed-tensors`
Conversion software	LLM Compressor 0.13.0, Compressed Tensors 0.18.0

The quantized layers cover the attention and feed-forward projections in each decoder layer. The checkpoint contains eight safetensor shards.

Hardware requirements

MXFP8 acceleration requires OCP MXFP8 hardware and a compatible inference backend.

This checkpoint passed a deterministic tensor-parallel-1 language-only vLLM smoke test on AMD Instinct MI350X (CDNA4, gfx950) with HIP runtime 7.2.53211. vLLM selected RocmDotScaledMxfp8LinearKernel for the quantized linear layers.

Note: Multimodal generation, model-quality parity, and other hardware configurations have not been validated with this checkpoint.

Usage with vLLM

The validated path loads only the language model:

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="rkayaith/Muse-Glimmer-30B-mxfp8",
    language_model_only=True,
)

outputs = llm.generate(
    ["Tell me about AMD Instinct MI350X."],
    SamplingParams(max_tokens=200, temperature=0.7),
)
print(outputs[0].outputs[0].text)

Use this command to start a language-only server:

bash

vllm serve rkayaith/Muse-Glimmer-30B-mxfp8 \
  --language-model-only

Comparison variants

Variant	Repository	Format
BF16	`meta-models/Muse-Glimmer-30B`	Source BF16 checkpoint
Dynamic block FP8	`RedHatAI/Muse-Glimmer-30B-FP8-block`	FP8 E4M3, 128x128 weight blocks, dynamic 128-element activation groups
Block MXFP8	`rkayaith/Muse-Glimmer-30B-mxfp8`	OCP MXFP8, 32-element groups

License

This derivative is distributed under the source model's Apache License 2.0. Use of the model must also comply with the source model's Usage Policy.

Modifications: The source BF16 language-model linear weights were quantized to MXFP8. The vision components and lm_head were retained without quantization. No fine-tuning occurred.