4-bit (MLX) quantization of NVIDIA's Nemotron-3 Ultra 550B-A55B, runnable on a single Apple Silicon machine with enough unified memory. Converted with mlx-lm from the official BF16 release (not from the NVFP4 checkpoint, which MLX cannot read).

This quant: Recommended. Smallest and fastest; quality indistinguishable from 5/6-bit.

At a glance


Precision	MLX affine 4-bit, group size 64 (~4.5 bits/weight)
Disk size	288 GB
Peak RAM	310 GB (measured)
Generation speed	18.9 tok/s (measured, M3 Ultra)
Min unified memory	~330 GB

Why this exists

There was no MLX build of Nemotron-3 Ultra. NVIDIA ships BF16 (~1.1 TB), FP8, and NVFP4 checkpoints; NVFP4 is NVIDIA-GPU specific and isn't loadable by MLX. This repo is a straight post-training quantization of the BF16 weights into MLX's affine format so the model runs natively on Apple Silicon.

Architecture

Hybrid Mamba-2 + Attention + Latent-MoE (nemotron_h):


Total / active params	550B / 55B (A55B)
Layers	108 (48 Mamba-2, 48 MoE, 12 Attention)
Experts	512 routed, top-22, + 1 shared
MoE latent size	2048
Hidden size	8192
Vocab	131072
Max position (config)	262144 (256K)

Requirements

Apple Silicon with ~330 GB free unified memory (peak observed/expected: 310 GB). In practice a Mac Studio M3 Ultra (512 GB).
mlx-lm >= 0.31.2 (supports nemotron_h + LatentMoE + shared experts).

Usage

bash

pip install -U mlx-lm
mlx_lm.generate --model pipenetwork/NVIDIA-Nemotron-3-Ultra-550B-A55B-MLX-4bit \
  --prompt "Explain MoE routing." --max-tokens 512
# OpenAI-compatible server:
mlx_lm.server --model pipenetwork/NVIDIA-Nemotron-3-Ultra-550B-A55B-MLX-4bit --port 8080

Quantization details

MLX affine, 4-bit, group size 64 (~4.5 bits/weight).
Source: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.
Command: mlx_lm.convert --hf-path <bf16> --mlx-path <out> -q --q-bits 4 --q-group-size 64
Conversion note: the BF16 config omits num_hidden_layers (required by mlx-lm's nemotron_h loader; derivable from layers_block_type, length 108). Inject num_hidden_layers: 108 into config.json before converting if reproducing.

Known limitations

No Multi-Token Prediction / speculative decoding — mlx-lm drops the MTP layers (mtp.*), so the original's native speculative-decoding speedup is unavailable here.
Context: config sets 256K positions; long contexts grow KV-cache memory.
Post-training quantization, not NVIDIA's QAT NVFP4.

Evaluation

Internal relative-quality harness (not leaderboard-comparable): perplexity on a small mixed corpus, and ARC-Challenge via zero-shot length-normalized continuation scoring (underrates reasoning models vs. official scores). BF16 cannot be measured (won't fit).

Quant-vs-quant perplexity (short mixed passages, 6-bit baseline):

Quant	mean ppl	Δ vs 6-bit	size
6-bit	2.580	baseline	416 GB
5-bit	2.527	−2.1%	352 GB
4-bit	2.564	−0.6%	288 GB

All within noise; generations near-identical. The 30B Nano sibling showed a large 4-bit penalty (+22% ppl), so this robustness is a property of the 550B scale (NVFP4-QAT origin), not of 4-bit in general.

License & attribution

Derived from NVIDIA Nemotron-3 Ultra under the OpenMDW-1.1 license (included as LICENSE); this derivative is provided under the same terms. "Nemotron" is a trademark of NVIDIA Corporation. Converted on a Mac Studio M3 Ultra with mlx-lm 0.31.2 / mlx 0.31.1. Not affiliated with or endorsed by NVIDIA.

Measured eval results

Internal relative-quality harness (n=60, zero-shot length-normalized continuation scoring — not leaderboard-comparable; this method underrates reasoning models vs. their official scores). BF16 reference not measurable (won't fit in memory).

Build	corpus ppl	ARC-Challenge acc	ARC-Challenge acc_norm
6-bit	2.586	0.617	0.633
4-bit	2.620	0.617	0.617

4-bit and 6-bit are within noise on both perplexity and ARC-Challenge — confirming on a real task that 4-bit is quality-equivalent at the 550B scale. (5-bit and 8-bit were not separately ARC-tested; quant-vs-quant perplexity placed all four within ~2%.)