MLX (Apple Silicon) MXFP8 quantization of microsoft/Mage-VL, a ~5B vision-language model (Mage-ViT codec-native visual encoder + Qwen3-4B-Instruct text backbone).

This is a full vision-language quant — not text-only. The Qwen3 language backbone is quantized to MXFP8 (OCP microscaling FP8, group size 32); the Mage-ViT vision tower is kept in bf16 (the mlx-vlm default for multimodal towers) to preserve image understanding.

	value
Precision (LLM)	MXFP8, group 32
Vision tower	bf16 (not quantized)
Bits/weight (overall)	8.79 bpw
On-disk size	~4.9 GB
Decode speed¹	48.4 tok/s
Peak memory¹	~6.6 GB

¹ Measured on an Apple M5 Pro (24 GB), single image prompt, 100-token greedy decode. Decode speed is LLM-bound (the vision tower runs once at prefill).

Speed comparison (same machine, full model):

build	decode	peak RAM
bf16	30.1 tok/s	10.9 GB
MXFP8 (this)	48.4 tok/s	6.6 GB
MXFP4	62.7 tok/s	4.5 GB

Verification

Quantization was verified against the original torch model (bf16, CPU) on the same image + prompt. The MLX vision tower matches the reference vision features (cosine 0.99992, fp32-vs-fp32). End-to-end, this MXFP8 build reproduces the reference caption exactly:

"A wizard reading a book with the words Mage-VL above him."

and on a second image (the Mage-VL framework diagram):

"A diagram showing the process of generating event-triggered commentary for a live match stream."

Usage

⚠️ These are MLX weights (Apple Silicon), for mlx-vlm — not loadable with 🤗 Transformers. mage_vl support is added in mlx-vlm PR #1751. Once it merges, plain pip install -U mlx-vlm is enough. Until then, either install from the PR branch, or copy the bundled model definition into your mlx-vlm (one-time), then use mlx_vlm.generate normally:

bash

# option A — install mlx-vlm with mage_vl support from the PR
pip install "git+https://github.com/SahilChachra/mlx-vlm.git@add-mage-vl"

bash

# option B — stock mlx-vlm + copy the bundled definition (one-time)
pip install mlx-vlm
python - <<'PY'
import mlx_vlm, os, shutil
dst = os.path.join(os.path.dirname(mlx_vlm.__file__), "models", "mage_vl")
shutil.rmtree(dst, ignore_errors=True); shutil.copytree("mlx_vlm_mage_vl", dst)
print("installed mage_vl ->", dst)
PY

# native mlx-vlm CLI (verified)
python -m mlx_vlm generate --model . --image your_image.jpg \
  --prompt "Describe this image." --max-tokens 128 --temperature 0.0

Or in Python:

python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load(".")
config = model.config.__dict__
prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
print(generate(model, processor, prompt, image=["your_image.jpg"], max_tokens=128))

A standalone run_mage_vl_mlx.py (uses the model's HF processor for preprocessing) is also bundled as an alternative.

Notes / limitations

Image path verified. Video inference (the DCVC neural codec) is not included here; copy neural_codec/ from the base repo if you need it.
The streammind_gate (streaming EPFE) component is omitted — not used on the standard image/video path.
MXFP8 tracks the bf16 model closely; for the smallest/fastest build see the MXFP4 sibling (slightly more verbose on short prompts).

Attribution

Base model: microsoft/Mage-VL (Apache-2.0)
Quantized to MLX MXFP8 by @sahilchachra using mlx-lm / mlx-vlm.

sahilchachra/mage-vl-mxfp8-mlx

Mage-VL — MLX MXFP8

Verification

Usage

Notes / limitations

Attribution

sahilchachra/
mage-vl-mxfp8-mlx