29.1 GB · ~110.0 tok/s decode on M5 Max · 31B total / ~3B active · hybrid Mamba-2 + Attention + MoE · native MLX
Full multimodal (text + image + audio + video) port of NVIDIA's
Nemotron-3-Nano-Omni-30B-A3B-Reasoning
to Apple MLX — calibrated with a measured Hessian, an imatrix fit and AWQ,
with the RADIO vision tower and the Parakeet speech encoder kept at fp16 so
every modality actually works.
Modality
Status
Text
✅ reasoning on by default
Image (RADIO ViT, 512px tiles + thumbnail)
✅
Video (frame sampling + EVS pruning)
✅
Audio / ASR (Parakeet 24-layer Conformer)
✅
Tool calling (agentic)
✅ XML <tool_call><function=…>
Quantization recipe
JANG_6M is a mixed-precision allocation (6-bit, 8-bit), not a single width.
The bit map is chosen per tensor role, then calibrated:
Component
Precision
Routed experts switch_mlp.fc1 / fc2 (93% of params)
6-bit / 6-bit
Shared expert, attention q/k/v/o, lm_head
8-bit
Mamba-2 in_proj / out_proj, embeddings
8-bit
conv1d, A_log, D, dt_bias, router gate, all norms
fp16
Vision tower, Parakeet encoder, both projectors
fp16 (never quantized)
Group size 64. 164 quantized modules; 1106 tower tensors passed
through untouched.
Why there is no 2-bit variant
This model's expert MLP is unusually sensitive to low-bit quantization, so the
routed experts carry a hard minimum width regardless of profile. A nominal
"2-bit" build lands within ~7% of the 4-bit one and is measurably worse — not a
2-bit bundle in any useful sense, so we do not ship one.
Calibration
Quantization is calibrated against measured activations from a
domain-weighted corpus (coding, agentic/tool, reasoning, general, multilingual,
long-context), not fitted to the weights alone:
imatrix — the affine fit is weighted toward the channels that actually
carry signal.
AWQ — per-channel scales, folded so the result is numerically equivalent
and costs nothing at inference.
Hessian — captured per MoE layer and used to steer the bit allocation.
QAT is not available for this model: NVIDIA's own quantized releases are
post-training (modelopt, MIXED_PRECISION), and NVFP4 codes carry no meaning
in an affine (scale, bias) layout.
Sampling — use the right preset
Reasoning is ON by default (enable_thinking); turning it off prefills an
empty <think></think> block rather than omitting it.
Mode
temperature
top_p
top_k
max_tokens
reasoning_budget
grace
Thinking (default)
0.6
0.95
—
20480
16384
1024
Instruct (non-thinking)
0.2
—
1
1024
—
—
ASR / transcription
1.0
—
1
—
—
—
Stamped into bothgeneration_config.json and
jang_config.chat.sampling_defaults. EOS is [2, 11] — dropping 11 means the
model never stops.
Runs in Osaurus / vMLX, which carries the
native MLX NemotronHOmni stack (RADIO vision, Parakeet audio, EVS video). The
LLM half also loads directly with mlx_lm:
