Nemotron-Labs-3-Puzzle-75B-A9B — oQNVFP4 + MTP sidecar (MLX)
Experimental, AI-built — not production software. This artifact and its serving
stack were built largely by AI agents under human direction and validated by
measurement, not review. Enjoy the speed, hack on it, review it before you depend
on it — and if you'd like to adopt and polish it properly, please do; the author
would be delighted and simply doesn't have the time.
NVIDIA's Nemotron-3 Puzzle 75B (A9B active, hybrid Mamba/attention/MoE) quantized to
oQNVFP4 (NVFP4 MoE weights with per-layer time-scale sidecars, 6-bit affine elsewhere,
~48 GB) for Apple Silicon, with the checkpoint's multi-token-prediction head included
as an mtp.safetensors sidecar (5.5 GB, bf16, bit-exact extraction of NVIDIA's
"training-only" MTP tensors).
Most community conversions drop the MTP tensors. Kept and served, they roughly halve
decode latency: the head drafts multi-token chains that the trunk verifies in a single
fused pass, and acceptance on natural text is high enough (~80–86%) that speculation pays
at every context depth.
Measured throughput
Apple M3 Ultra 512 GB,
omlx fork,
streaming API, salted natural-document prompts (no repeated-filler inflation), temperature 0,
1000-token generations at depth:
| context | decode tok/s (MTP on) | decode tok/s (MTP off) | prefill tok/s |
|---|
| 0.6k | 85.0 | 55.0 | — |
| 2k | 80.8 | 54.8 | 977 |
| 26k | 63.0 | — | 977 |
| 53k | 57.3 | — | 897 |
| 108k | 47.5 | — | 771 |
MTP census over full runs: 2.99 tok/cycle at short context, ~1.86 at 26k+ (the adaptive
depth controller shallows as verify forwards grow), acceptance stable at 80–86% throughout.
Prefill is near-flat with depth — the Mamba backbone's signature.
Requirements
This artifact is
not loadable by stock mlx-lm. The NVFP4 time-scale sidecars
(
fc1_ts/
fc2_ts) and the MTP sidecar head require the
omlx fork:
1git clone https://github.com/josefprusa/omlx
2cd omlx && uv sync --python 3.12
3OMLX_WITH_CUSTOM_KERNEL=1 uv pip install -e . # builds the native kernels (needs full Xcode)
4omlx serve
Place this repository's contents in your omlx models directory, enable mtp_enabled for
the model in model_settings.json, and the server logs will show
[Nemotron-H MTP] sidecar head attached (pattern=*E, depth=3) plus a per-request
acceptance census.
Works without MTP too (55 tok/s class): leave mtp_enabled off and the model serves as a
plain checkpoint — the sidecar is simply ignored.
Provenance
- Base: nvidia/Nemotron-Labs-3-Puzzle-75B-A9B (OpenMDW v1.1)
- MTP tensors: bit-exact extraction from the base checkpoint's
mtp.* block (bf16,
unquantized) — provenance-verifiable by hash against the source shards.
- Quantization: omlx oQNVFP4 pipeline (NVFP4 MoE + ts sidecars); conversion script in the
fork under
experimental/.
License: OpenMDW v1.1 (NVIDIA's open model license), carried unchanged in LICENSE.
