Muse-Glimmer-30B-MLX-4bit
MLX (Apple Silicon) build of
Muse-Glimmer-30B, quantized to
4-bit (group size 64).
These files are modified — the weights have been converted to MLX and quantized. The
architecture is unchanged.
You need the runtime
muse_glimmer is carried by neither mlx-lm nor mlx-vlm, so mlx_vlm.load() cannot read this
repository (or any other MLX build of this model — there are several, and none of them load with
stock tooling):
1>>> importlib.import_module("mlx_vlm.models.muse_glimmer")
2ModuleNotFoundError
Use the port:
1git clone https://github.com/PipeNetwork/muse-glimmer-mlx
2cd muse-glimmer-mlx && pip install -r requirements.txt
1from muse_glimmer_mlx.load import load
2model = load("/path/to/Muse-Glimmer-30B-MLX-4bit")
That port is validated against transformers 5.15 — text stack to 2.0e-05, vision tower to
5.5e-06, vision geometry exactly — and it also loads every other published MLX conversion of this
model unmodified.
Size
| |
|---|
| this build | 21.4 GB |
| bfloat16 | 59.6 GB |
Quantization covers the 52 text layers, lm_head, and the vision adapter and projection — 420
layers. The 50-layer vision tower (3.7 GB) and the embedding table (2.7 GB) stay bfloat16, as in
every published build, so 6.4 GB of this file is unquantized and the bit width understates the
download.
Quality
Measured against bfloat16 on ~98k tokens of wikitext-2, scored over identical windows with
differences taken per window and a bootstrap over one shared index set. Perplexity varies far more
between windows than between quants, so pairing is what makes the comparison readable at all.
| build | size | perplexity | ΔNLL vs bf16 [95% CI] | windows worse |
|---|
| bfloat16 | 59.6 GB | 6.3224 | — | — |
| 8-bit | 34.6 GB | 6.3224 | −0.0000 [−0.0003, +0.0003] | 25/48 |
| 4-bit | 21.3 GB | 6.5206 | +0.0309 [+0.0285, +0.0334] | 48/48 |
| 3-bit | 18.1 GB | 7.4400 | +0.1628 [+0.1547, +0.1718] | 48/48 |
| 2-bit (withheld) | 14.7 GB | 78.1019 | +2.5139 [+2.4730, +2.5552] | 48/48 |
Read the last column with the interval. 8-bit is worse on only 25 of 48 windows — a coin flip —
and its interval straddles zero: it is statistically indistinguishable from bfloat16 here. Every
other width is worse on every window, because quantization damage is systematic, so once it exists
the paired test finds it everywhere. That also means 4-bit's +3.1% is unambiguous and small:
significant is not the same as large.
The knee is 4 → 3 bits (5.3x the delta) and the cliff is 3 → 2 (15.4x).
What that looks like
Greedy decoding, same prompts, against bfloat16:
| prompt | bfloat16 | 3-bit |
|---|
| The capital of France is | "Paris. It is the most populous city in France…" | fluent and on-topic, but never directly completes the statement — "a city of contrasts…" |
| Explain why the sky appears blue, in two sentences | Rayleigh scattering, two distinct sentences | correct physics, but repeats the same sentence as (1)(2)(3) |
| Write a Python function that reverses a linked list | def reverse_linked_list(head): then explains | restates the problem first, reaches class ListNode: |
So: grammatical, factually largely intact, but weaker at following an instruction directly and
prone to repetition loops that bfloat16 does not exhibit on the same prompts. Choose it to fit the
model in ~18 GB, not for fidelity.
2-bit is not published. It was built and measured: perplexity 78.1, and it emits "Paris, not
Paris. The rest of the world, the rest of the world…". It would also have saved only 3.3 GB over
this build, because the unquantized tower and embeddings dominate either way.
License
