[!TIP]
KV-cache quantization without any fork (recommended, 2026): upstream
llama.cpp/Ollama now cover this natively — use
-ctk q8_0 -ctv q8_0
(~half KV memory, negligible quality loss: perplexity +0.002–0.05) or
-ctk q4_0 -ctv q4_0 (~quarter memory, ≈7.6% perplexity increase). In
Ollama:
OLLAMA_KV_CACHE_TYPE=q8_0 with
OLLAMA_FLASH_ATTENTION=1. Keep
K and V types symmetric to stay on the fast fused Flash-Attention path.
Since April 2026, mainline llama.cpp also applies Hadamard rotation to
KV activations (
PR #21038),
which greatly improves low-bit KV quality (opt-out:
LLAMA_ATTN_ROT_DISABLE=1).
The RotorQuant/TurboQuant fork flow below is experimental/legacy: the
TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork
is unmaintained relative to mainline. It is NOT required to use this model.
Nemotron-3-Nano-Omni-30B-A3B-Reasoning - TurboQuant MLX 5-bit
MLX 5-bit quantization of the text tower of Nemotron-3-Nano-Omni-30B-A3B-Reasoning (nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16)
with TurboQuant weight method. Apple Silicon native via mlx-lm.
This variant covers the LLM backbone only. Vision (CRADIO v4-H) + audio (Parakeet-TDT-0.6B-v2)
encoders are NOT included — the MLX-VLM Nemotron-Omni model class is supported upstream
as of mlx-vlm v0.4.4 (PR #1518, merged 2026-07-07; audio weight fixes in PR #1743, merged
2026-07-29). For multimodal inference, use mlx-vlm (v0.4.4+) or the GGUF variants with
llama-mtmd-cli.
For the matched-KV stack — TurboQuant weights + TurboQuant KV-cache modifier —
For the runtime KV-cache modifier itself, see
majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant.
Quickstart
1# Today (mlx-lm 0.31.x): the NemotronH_Nano_Omni_Reasoning_V3 model class
2# is not yet registered in mlx-lm. The cell below is the API shape that WILL
3# work once upstream lands the class (track ml-explore/mlx-lm#386).
4
5from mlx_lm import load, generate
6
7model, tokenizer = load("majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-5bit")
8
9prompt = tokenizer.apply_chat_template(
10 [{"role": "user", "content": "Solve: 17 * 23"}],
11 add_generation_prompt=True,
12 enable_thinking=False, # set True to enable extended reasoning (default)
13)
14
15response = generate(
16 model, tokenizer,
17 prompt=prompt,
18 max_tokens=512,
19 sampler=lambda x: x.argmax(axis=-1), # or use mlx_lm.sample_utils.make_sampler(temp=0.6, top_p=0.95)
20)
21print(response)
⚠️ This variant covers the text tower only. For multimodal inference (vision + audio + video), use the GGUF variants with llama-mtmd-cli — see the GGUF cards in this family.
Modality matrix
| Modality | Encoder | Quantization in this variant |
|---|
| Text | LLM backbone (Mamba-2 + Transformer hybrid Sparse MoE) | per the variant suffix |
| Image | CRADIO v4-H | BF16 (kept full-precision in every non-GGUF variant; GGUF uses mmproj-F16 split file) |
| Audio | Parakeet-TDT-0.6B-v2 | BF16 (same rationale) |
| Video | Parakeet-TDT-0.6B-v2 + frame sampler | BF16 (≤ 2 min, 256 frames @ 2 FPS) |
NVIDIA's official FP8 / NVFP4 recipe keeps both encoders + the cross-modal
MLP projectors in BF16 to preserve multimodal accuracy. We follow that
convention in every quantized variant we ship.
Runtime quirks
MLX-LM (text-only)
This variant covers the LLM backbone only. Vision + audio encoders
are NOT included — the MLX-VLM Nemotron-Omni model class is
supported upstream as of mlx-vlm v0.4.4 (PR #1518, merged
2026-07-07; audio weight fixes in PR #1743, merged 2026-07-29).
Use the mlx_lm.generate API; enable_thinking is a runtime flag
(see below).
Reasoning mode
enable_thinking defaults to True. To disable extended reasoning
(e.g., for latency-sensitive cases), pass enable_thinking=False
to the chat template / generate call. No separate "no-think"
variant card exists — this is a runtime flag, not a model variant.
Quant trade-off (MLX lane)
| Bits | Approx size | Use case | Recommendation |
|---|
| 2-bit | ~8.1 GB | Aggressive quantization | Very low-RAM Macs |
| 3-bit | ~11 GB | Lossy but small | Low-RAM Macs |
| 4-bit | ~13 GB | Balanced default | Recommended for most Macs |
| 5-bit | ~16 GB | Higher fidelity | Quality-sensitive |
| 6-bit | ~19 GB | Approaching FP16 quality | High-fidelity |
| 8-bit | ~24 GB | Near-lossless reference | Fidelity-critical work |
(Current variant — 5bit — is bolded.)
Variants in this family
(Showing 56 sibling variants under majentik/nemotron3-nano-omni-30b-*. The current variant — TurboQuant-MLX-5bit — is bolded.)
| Variant | Runtime | Approx size | Use case |
|---|
| TurboQuant-MLX-5bit | mlx-lm | ~23 GB | Apple Silicon, higher fidelity |
About the RotorQuant / TurboQuant labels
RotorQuant and TurboQuant are this project's release labels, not distinct
quantization algorithms — for any given tier, both brand repos carry
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
No brand-specific speedup is claimed or measured.
