NVIDIA-Nemotron-3-Ultra-550B-A55B-MLX-8bit
Other quantizations: 4-bit ·
5-bit ·
6-bit ·
8-bit (this)
8-bit (MLX) quantization of NVIDIA's
Nemotron-3 Ultra 550B-A55B. Converted with
mlx-lm from the official BF16 release.
Distributed-only. At ~584 GB this does
not fit a single 512 GB Mac — it targets a
multi-machine MLX shard (e.g. 512 GB + 256 GB M3 Ultras over Thunderbolt 5, 768 GB
combined). For a single 512 GB machine use the
4-bit.
| |
|---|
| Precision | MLX affine 8-bit, group 64 (~8.5 bpw) |
| Disk size | ~584 GB |
| Runs on | multi-Mac MLX shard (>=~640 GB combined unified memory) |
Architecture
Hybrid Mamba-2 + Attention + Latent-MoE (nemotron_h): 108 layers (48 Mamba-2, 48 MoE,
12 Attention), 512 routed experts top-22 + 1 shared, moe_latent_size=2048.
Usage (distributed)
Run sharded across machines with mlx.distributed pipeline/tensor parallelism. See mlx-lm's
distributed docs; the model path must exist on every node. mlx-lm >= 0.31.2.
Quantization
- MLX affine 8-bit, group 64. Source:
nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.
- Command:
mlx_lm.convert --hf-path <bf16> --mlx-path <out> -q --q-bits 8 --q-group-size 64
- Note: the BF16 config omits
num_hidden_layers (required by mlx-lm's nemotron_h
loader; = 108, from layers_block_type). Inject it into config.json before converting.
Quality note
On the 550B, 4/5/6/8-bit are quality-indistinguishable in our testing (all within noise).
8-bit is provided for max-precision / distributed use, not as a quality upgrade over 4-bit.
Known limitations
- No MTP / speculative decoding (mlx-lm drops the
mtp.* layers).
- Context: config sets 256K positions; long contexts grow KV-cache memory.
License
Derived from NVIDIA Nemotron-3 Ultra under OpenMDW-1.1 (included as LICENSE).
"Nemotron" is a trademark of NVIDIA. Converted with mlx-lm 0.31.2. Not affiliated with NVIDIA.
Measured eval results
Internal relative-quality harness (n=60, zero-shot length-normalized continuation scoring —
not leaderboard-comparable; this method underrates reasoning models vs. their official
scores). BF16 reference not measurable (won't fit in memory).
| Build | corpus ppl | ARC-Challenge acc | ARC-Challenge acc_norm |
|---|
| 6-bit | 2.586 | 0.617 | 0.633 |
| 4-bit | 2.620 | 0.617 | 0.617 |
4-bit and 6-bit are within noise on both perplexity and ARC-Challenge — confirming on a real
task that 4-bit is quality-equivalent at the 550B scale. (5-bit and 8-bit were not
separately ARC-tested; quant-vs-quant perplexity placed all four within ~2%.)
