AWQ 4-bit (group_size=64) quantization of nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning, targeting 2× RTX 3090 (48 GB) and similar 40–48 GB inference fleets. Verified serving + 6/6 capabilities at full 262144 context on SGLang v0.5.13.post1.


Architecture	`NemotronH_Nano_Omni_Reasoning_V3` (multimodal wrapper) → `NemotronHForCausalLM` (Mamba2 + Attention + MoE hybrid)
Total params	31B (23× Mamba2 / 6× Attention / 23× MoE language layers + CRADIO vision tower + Parakeet audio tower)
Active params	~3B per token (top-6 of 128 routed experts + 1 shared expert)
Modalities	Text, Image, Video, Audio → Text
Quantization	INT4 W4A16 via GPTQ (compressed-tensors recipe) → repacked to native AWQ
Group size	64 (Nemotron-H's `moe_intermediate_size=1856` is not divisible by 128, so the standard AWQ default doesn't apply — group_size=64 is required)
AWQ size on disk	~21 GB (34 shards)
Source size on disk	~62 GB BF16
Reasoning	On by default; toggle via `enable_thinking` (preserved from upstream)
256K decode	~98 tok/s flat 1K→255K (Mamba2 O(1) recurrent state); beats R9700 FP8 reference

What's quantized vs passthrough

INT4 W4A16 (GPTQ-calibrated, 5934 tensors): all language_model.backbone.layers.*.mixer.experts.*.{up_proj,down_proj}.weight MoE expert linears, plus shared_experts.{up_proj,down_proj}.weight. NemotronH's MoE is non-gated (squared-ReLU activation, up_proj only — no separate gate_proj), so the per-expert weight count is 2× per layer, not 3× as in a standard SwiGLU MoE.
BF16 passthrough: 23 Mamba2 SSM layers (in_proj, out_proj, conv1d, dt_bias, A_log, D, RMSNorms), 6 attention layers (q_proj, k_proj, v_proj, o_proj), all router gates, all mlp projector heads (mlp1), embeddings + LM head, the CRADIO v4-H vision tower (vision_model) in full, and the Parakeet audio tower (sound_encoder + sound_projection) in full.

The quantization_config.modules_to_not_convert and .ignore fields in config.json are populated with the exact 14-entry exclusion list — the AWQ loader must see this to keep the BF16 passthrough modules unquantized. (Earlier ships had this empty, causing SGLang to attempt AWQ-wrapping the Mamba in_proj and crashing on shape constraints — see Changelog v3.)

Calibration recipe

omni_thinking_tools recipe, 1024 samples × 2048 tokens (~2.1 M cal tokens). Eight-slice mix that covers the model's live token routes:

Slice	%	Source	Purpose
`am_thinking`	18%	`a-m-team/AM-Thinking-v1-Distilled`	Reasoning traces with `<think>…</think>` tags
`llava_instruct`	15%	`liuhaotian/LLaVA-Instruct-150K`	Image-conversation rows
`llava_video_178k`	15%	`lmms-lab/LLaVA-Video-178K`	Video-caption rows
`hermes_tools`	12%	`NousResearch/hermes-function-calling-v1`	XML function-call exchanges — exercises `<tool_call>`
`common_voice_audio`	10%	`openslr/librispeech_asr` (config `clean`)	ASR transcripts
`covost2_audio`	10%	`facebook/voxpopuli` (config `en`)	Speech instruction
`numina_math`	10%	`AI-MO/NuminaMath-CoT`	Math + chain-of-thought
`ultrachat`	10%	`HuggingFaceH4/ultrachat_200k`	Plain multi-turn dialogue

drop_images=True is used in rows_to_text — image/audio/video bytes are dropped before tokenization. The language-model submodule sees text + the model's own modality placeholder tokens, mirroring inference where the CRADIO / Parakeet towers produce their own embeddings into the LM's input stream at those positions. This is the same text-with-placeholders approximation used for the rest of our multimodal AWQ ships (Qwen3-VL, gemma-4, Devstral-vision).

Pipeline: llmcompressor.oneshot(model=model.language_model, …) targeting the inner NemotronHForCausalLM only (the multimodal wrapper's image_flags.squeeze(-1) AttributeErrors on text-only rows; the encoders + Omni wrapper remain attached so the saved checkpoint still ships them). GPTQModifier(scheme="W4A16", targets="Linear", offload_hessians=True, bypass_divisibility_checks=True), group_size=64, moe_calibrate_all_experts=True (default in llmcompressor ≥0.11 — forces every token through every expert during cal so rare-routed experts don't get degenerate scales).

Wall clock: Phase 2 calibration ran 9 days 6 h 7 min end-to-end on a single CPU host (no GPU; Hessian build is GPU-light; expert quantization dominates).

Audit (`check_awq_scales.py --base`)

Scanned 5934 *.scales and 5934 *.qweight tensors against the BF16 base (dead-channel comparator enabled). 0 flagged. No all-zero scales, no majority-zero qweights, no NaN/Inf, no live-base / zero-scale mismatches.

This is the structural gate that catches the v2 "scales=0 NaN logits" and v3 "qweight=0 rare-expert-under-cal" failure modes we've seen on other MoE quantizations. Both clean here.

Per-layer GPTQ error pattern (audit-only, not a quality gate)

Elevated down_proj and up_proj errors were observed during calibration on later MoE layers — most prominent on layer 47 (peak expert.99.down_proj error 2311), layer 49 (sustained 400–900 across mid-rank experts), layer 34 (range 384–689), and layer 27 (range 65–347). The structural audit above is clean, so these are quality-signal-only flags, not defects. Live capability validation (6/6 PASS — see below) and 256K decode benchmarking both succeed, indicating the elevated errors did not translate into observable inference defects.

Inference

SGLang (verified working — 2×RTX 3090, full 262144 ctx)

This artifact serves end-to-end on SGLang v0.5.13.post1 + two open-source-friendly patches plus the standard nemotron3-omni launcher preset:

bash

# scripts/launch.sh nemotron3-omni on the v0.5.13.post1 tree
# QUANT=moe_wna16, TP=2, --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder
# --trust-remote-code, MambaRadixCache hybrid_ssm=True, fp8 KV
# Boots at full 262144 with max_total_num_tokens=5250973 (5.25 M-tok KV pool,
# ~20× the context — only 6 of 52 layers are attention; Mamba layers use
# the SSM-state cache, not the regular KV pool).

Required patches (both backend-independent, both upstream-PR-worthy):

Patch	Touches	What it does
`052-moe-wna16-nongated-moe.patch`	`python/sglang/srt/layers/quantization/moe_wna16.py` (6 hunks)	Adds non-gated MoE support to `moe_wna16`: when `is_gated=False`, builds the fused `gate_up` param at 1× intermediate (not the hardcoded 2×) and loads `up_proj` at offset 0. Required for NemotronH's squared-ReLU experts (`moe_intermediate_size=1856`).
`053-evs-video-combined-path-routing.patch`	`python/sglang/srt/managers/mm_utils.py` (1 hunk)	Routes EVS video items through the combined chunked-prefill path (which unwraps the `EVSEmbeddingResult`) instead of the per-image path that would `.reshape()` the result object and crash the scheduler.

Build receipts in mattbucci/2x-3090-GA102-300-A1-sglang-inference — see patches/052-*.patch, patches/053-*.patch, and benchmarks/nemotron3-omni-awq-serve-2026-06-16.md for the full serving recipe.

Also required in the env: librosa==0.11.0 (the Parakeet audio extractor imports it).

6-modality capability validation (`validate_capabilities.py`)

All six capabilities pass on the served model (receipt: benchmarks/nemotron3-omni-v0513-caps.json):

Check	Result	Note
basic	✅ PASS	`finish=stop answer='paris'`
thinking	✅ PASS	`reasoning_content` via `nemotron_3` parser, 191 tok, terminated cleanly
tool_call	✅ PASS	`get_weather` with `{"location":"Paris"}`, structured tool_calls
vision	✅ PASS	Red circle on white — `saw=['red','circle','round']`
video	✅ PASS	Red circle moving right — `saw=['move','right','circle','red']`
audio	✅ PASS	Parakeet on synthetic tone — `saw=['sound']`

Server stayed alive through every modality. Zero reshape / EVS errors.

256K decode performance (single-user M=1, cuda-graph ON)

Context	tok/s
1024	102.7
32 K	99.9
131 K	97.9
255 K	97.8

~5% decay over 250× depth. NemotronH's hybrid is 23 Mamba layers (O(1) recurrent state) + 6 attention layers — so deep-decode bandwidth is dominated by the int4 expert weights, not by KV reads. Beats the R9700 FP8 reference (74.79 short / 49.22 @ 230 K) on both ends. Prefill is also cheap at depth: TTFT ~1.1 s @ 200 K.

Open lever for further wins: the int4 MoE currently runs an untuned default Triton config (E=128,N=464,…int4_w4a16.json not yet supplied). A tuned fused-MoE config would lift decode further.

transformers (CPU/MPS or single-GPU fp16)

python

from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained(
    "mattbucci/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ",
    trust_remote_code=True,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "mattbucci/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ",
    trust_remote_code=True,
)

The custom modeling files (modeling.py, modeling_nemotron_h.py, configuration*.py, audio_model.py, etc.) are bundled — trust_remote_code=True resolves the architecture via auto_map.

Recommended sampler

Mode	temperature	top_p	top_k	max_tokens	reasoning_budget	grace_period
Thinking	0.6	0.95	—	20480	16384	1024
Instruct	0.2	—	1	1024	—	—

(Identical to upstream — preserved through quantization.)

License & attribution

Inherits the NVIDIA Open Model Agreement from the upstream base — the model is available for commercial use. See the agreement for full terms.

This quantization derivative is published non-commercially by mattbucci for research/community use; downstream users are bound by the same upstream license.

Upstream attribution:

NVIDIA Nemotron 3 Nano Omni is a multimodal LLM unifying video, audio, image, and text understanding for enterprise Q&A, summarization, transcription, and document intelligence. Developed by NVIDIA as part of the Nemotron model family. Improved using Qwen3-VL-30B-A3B-Instruct, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, Qwen2.5-VL-72B-Instruct, and gpt-oss-120b — see the upstream training-data section.

Provenance

Quantization rig: 2× RTX 3090 (GA102), 92 GB RAM + 64 GB swap, Arch Linux, miniforge3 quant env (torch 2.12.0+cu130, safetensors 0.7.0, llmcompressor 0.11.0).
Source repo: letsrtfm/2x-3090-GA102-300-A1-sglang-inference — see scripts/quantize/quantize_nemotron3_nano_omni.py, scripts/quantize/convert_moe_ct_to_awq.py, patches/052-*, patches/053-*, benchmarks/nemotron3-omni-awq-serve-2026-06-16.md, benchmarks/nemotron3-omni-v0513-caps.json.

Changelog

v3 (2026-06-16, late evening) — config + docs only; weights byte-identical to v2. Two things landed:
1. config.json: populated quantization_config.modules_to_not_convert and .ignore with the 14-entry exclusion list verified by serving (["sound_encoder","sound_projection","mlp1","vision_model","in_proj","out_proj","conv1d","q_proj","k_proj","v_proj","o_proj","gate","lm_head","embeddings"]). v1 and v2 both shipped these fields empty, which caused SGLang's AWQ loader to attempt to wrap the BF16 Mamba2 in_proj and crash on the Marlin shape constraint. The calibration script never wrote its ignore list to the output config; fixed in the upstream script too so future ships emit it automatically.
2. README: corrected the calibration recipe section (was missing 4 of 8 slices and incorrectly implied multimodal bytes were processed during cal — they're dropped via drop_images=True); replaced the "SGLang preset pending" + "6-modality validate deferred" placeholders with the actual verified serving recipe, 6/6 caps result, and 256K decode benchmark.
v2 (2026-06-16, evening) — stripped 5934 stale .weight_zero_point int8 tensors that leaked through from compressed-tensors → AWQ conversion (one per quantized Linear). The AWQ format already encodes the symmetric zero-point in .qzeros; the leftover .weight_zero_point is dead weight at best and broke SGLang's MoE loader at worst — it tried to map the residue onto an experts.w2_weight_zero_point parameter that doesn't exist on the AWQ Linear, raising KeyError on model load (caught after 5 SGLang load attempts across the kernel/TP matrix). convert_moe_ct_to_awq.py upstream is patched so future ships are clean by construction.
v1 (2026-06-16, afternoon) — initial AWQ ship from Phase 2 calibration. Inadvertently included the .weight_zero_point residue (see v2) and shipped empty modules_to_not_convert (see v3).

mattbucci/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ