INT4 post-training quantization of Qwen/Qwen3-Omni-30B-A3B-Instruct — the 30B omni model with audio, vision, and speech generation. ~22 GB on disk. Runs on a single RTX 4090 or A6000.

Attention-only quantization via AutoRound W4A16 G32. MoE expert weights stay BF16 — no quality sacrifice on the sparse expert path.

At a Glance

Property	Value
Base model	`Qwen/Qwen3-Omni-30B-A3B-Instruct`
Release tier	Attention-gold (AutoRound attention-only)
Quant method	AutoRound W4A16-G32 attention-only iters=200 (MoE experts BF16)
FLAC status	Not measured (T+7d milestone)
Architecture	Sparse MoE + Whisper audio + ViT vision + speech decoder
Quant format	compressed-tensors (native vLLM)
Quantized	48 attention layers (q/k/v/o_proj) — 192 tensors total
Kept BF16	MoE experts, audio_tower, visual, talker, code2wav
Disk size	~22 GB
Min GPU	1× RTX 4090 24GB or A6000 48GB

Why attention-only?

Qwen3-Omni's MoE expert weights are sparse by construction — aggressively compressing them trades quality for minimal size gain. The 192 attention projections (48 layers × q/k/v/o_proj) are the quality-critical path: they participate in every token of every forward pass. Quantizing only those at W4 achieves the large memory reduction while leaving the routed and shared experts untouched at BF16.

This is also why this model has no full-W4 compressed-tensors release yet — this is the first.

Memory Requirements

Configuration	BF16	W4A16 (this model)
Weights on disk	~60 GB	~22 GB
VRAM at batch=1, 32k ctx	~66 GB	~23 GB
Min GPU	2× A100 40GB	1× RTX 4090 24GB

Quick Start

Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.

vLLM — text output only

bash

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \
  88plug/Qwen3-Omni-30B-W4A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

Weights are in compressed-tensors format — no --quantization flag needed. Mainline vLLM returns text only. Audio input works; speech output does not.

vLLM-Omni — full audio output

vLLM-Omni enables real-time speech output. Required if you need the model to speak.

bash

docker run --gpus device=0 -p 8080:8080 \
  vllm-omni-image vllm serve \
  88plug/Qwen3-Omni-30B-W4A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

Recommended Sampling Parameters

Mode	Temperature	Top-P	Top-K	Min-P	Use When
Thinking (default)	0.6	0.95	20	0.0	Reasoning, math, code
Non-thinking	0.7	0.8	20	0.0	Chat, creative, fast response

Enable/disable thinking via chat_template_kwargs={"enable_thinking": True/False}. Default is thinking-enabled.

Quantization Recipe

Parameter	Value
Method	AutoRound
Scheme	W4A16
Group size	32
Targets	`re:.*self_attn\.(q\|k\|v\|o)_proj$` (192 modules)
Ignored	`lm_head`, `embed_tokens`, `norm`, `audio_tower`, `visual`, `talker`, `code2wav`
Calibration data	75% UltraChat-200k + 25% WikiText-103
Calibration samples	1024 × 2048 tokens
Iterations	200

What's Quantized, What's Not

Component	Precision	Reason
Attention q/k/v/o_proj (all 48 layers)	W4A16 INT4	Quantized — quality-critical path
MoE experts (routed + shared)	BF16	Sparse weights — kept intact for quality
`model.thinker.audio_tower.*`	BF16	Whisper encoder — excluded
`model.thinker.visual.*`	BF16	ViT — excluded
`model.talker.*`	BF16	Speech decoder — excluded
`model.code2wav.*`	BF16	Waveform codec — excluded
Embeddings, LM head, norms	BF16	Standard practice

Comparison to Other W4 Releases

Model	Method	Coverage	Speech output	Format
88plug/Qwen3-Omni-30B-W4A16 (this)	AutoRound W4A16 G32	Attention-only (MoE BF16)	Yes (vLLM-Omni)	compressed-tensors
Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound	AutoRound W4	Full model — all Linear	Yes	compressed-tensors
cyankiwi/Qwen3-Omni-30B-A3B-Instruct-AWQ-4bit	AWQ	Full model	Yes	AWQ
ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF	GGUF	Full model	No (mainline llama.cpp)	GGUF

Key differentiator vs Intel's AutoRound release: Intel quantizes all Linear layers uniformly to W4, including MoE experts. This release leaves MoE experts at BF16, preserving expert routing quality. The tradeoff is slightly larger size in exchange for higher output fidelity on complex reasoning and long-context tasks.

This is the only W4A16 compressed-tensors release for this model.

SGLang

SGLang compressed-tensors support is under active development. For baseline throughput comparisons, run the BF16 base model:

bash

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --tp 2 \
  --port 30000

SGLang compressed-tensors support — verify with your SGLang version before production use. Speech output (talker/code2wav) requires vLLM-Omni regardless.

llama.cpp / GGUF

For CPU inference or Apple Silicon, use the GGUF variant from ggml-org or unsloth:

bash

# llama.cpp (text-only; speech output not supported in mainline llama.cpp)
llama-cli -hf ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF \
  -m Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf \
  --chat-format chatml

Note: GGUF runs on CPU and Apple Silicon without vLLM. Speech synthesis (talker + code2wav pipeline) is not available in mainline llama.cpp — use vLLM-Omni for full audio output. This compressed-tensors W4A16 checkpoint is optimized for GPU serving with vLLM.

Limitations

Mainline vLLM (text-only): Audio input is supported; speech output requires vLLM-Omni. The talker and code2wav components are BF16 and inactive in standard vLLM serve.
MoE expert quality: Expert weights remain at BF16 — this quant targets attention projections only. Full-W4 releases (e.g., Intel's) quantize experts too, trading quality for additional size reduction.
Audio/vision tower untouched: audio_tower, visual, talker, and code2wav are fully BF16. Quantization affects the LLM thinker backbone only — audio/vision pipelines are identical to the BF16 base model.
Group size 32: Finer-grained than standard G128, improving quality on attention heads at modest overhead.
vLLM ≥ v0.21.0 required: Older vLLM versions do not support compressed-tensors natively.
Context length: Tested up to 32k. Qwen3-Omni supports up to 128k — longer contexts require proportionally more KV cache VRAM.

Quality Targets

Metric	Target
KL divergence vs BF16	< 0.014
MMLU recovery	≥ 99%
RULER@128k recovery	≥ 97%
ASR WER delta	≤ +0.5%

The attention-only approach keeps the audio and voice pipeline entirely at BF16, so ASR and speech generation quality are unaffected by the quantization.

Benchmarks

Metric	Status
Throughput (tok/s)	In progress — T+7d milestone
MMLU delta vs BF16	In progress — T+7d milestone
RULER@128k	In progress — T+30d milestone

No fabricated numbers. Results will be published to this card when measured.

Citation

bibtex

@misc{qwen3technicalreport,
  title  = {Qwen3 Technical Report},
  author = {Qwen Team},
  year   = {2025},
  url    = {https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct}
}

About

88plug AI Lab ships production-grade compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Attention-gold tier — AutoRound W4A16-G32 on attention projections only (iters=200). MoE experts and multimodal towers stay BF16 by design.

Browse all releases → huggingface.co/88plug