4-bit weight, 8-bit activation (W4A8) quantization using NVIDIA ModelOpt NVFP4
Pre-quantized checkpoint — weights are packed as FP4 (uint8), not online quantization
~3.6x compression on MoE expert weights (54 GB → 15 GB)
Same capabilities as the original: text, image, audio, video input → text + speech output

Quantization Details

Component	Precision	Quantized?
Thinker MoE experts (gate_up_proj, down_proj)	NVFP4 packed uint8	Yes — pre-quantized
Thinker attention (q/k/v/o_proj)	NVFP4 (calibrated)	Yes
Thinker lm_head	BF16	No
Thinker MoE router gates	BF16	No
Audio Encoder	BF16	No
Vision Encoder	BF16	No
Talker (MoE)	BF16	No
Code2Wav	BF16	No

Weight Format

MoE expert weights are stored as:

gate_up_proj: packed uint8 (2× FP4 values per byte)
gate_up_proj_scale: float8_e4m3fn (per-block-of-16 FP8 scale)
gate_up_proj_scale_2: bfloat16 (global per-tensor scale)
Same pattern for down_proj

Quantization Config

json

{
  "quant_algo": "NVFP4",
  "quant_method": "modelopt",
  "group_size": 16,
  "weights": {"num_bits": 4, "type": "float"},
  "input_activations": {"num_bits": 8, "type": "float"},
  "kv_cache": {"num_bits": 8, "type": "float"}
}

Memory Comparison

Config	Checkpoint Size	Notes
BF16 (original)	~60 GB	Full precision
FP8 (ModelOpt)	~40 GB	Online quantization
NVFP4 (this model)	~27 GB	Pre-quantized FP4 experts

Hardware Requirements

Purpose	Hardware	Notes
Inference	RTX 5090, B100, B200 (Blackwell SM 100+)	Native FP4 hardware support required
Inference (not supported)	RTX 4090, A100, H100	No FP4 hardware — use FP8 instead

Quantization Methodology

Base model: Qwen/Qwen3-Omni-30B-A3B-Instruct (BF16)
Tool: NVIDIA ModelOpt v0.42.0
Method: NVFP4 (W4A8) with static calibration (256 diverse prompts)
Calibration: 256 English + Chinese prompts covering factual, creative, code, math, and reasoning tasks
Attention layers: Quantized via mtq.quantize with calibration
Expert layers: Manually packed via NVFP4QTensor.quantize() (fused expert tensors not supported by mtq.quantize)
Excluded: lm_head, MoE router gates, all non-thinker stages

Original Model

This is a quantized version of Qwen/Qwen3-Omni-30B-A3B-Instruct. Please refer to the original model card for full capabilities, benchmarks, and usage documentation.

License

Apache License 2.0 (same as the original model)

shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4

Qwen3-Omni-30B-A3B-Instruct NVFP4 (W4A8)

Key Features