INT8 post-training quantization of Qwen/Qwen3-30B-A3B — the 30B omni model with audio, vision, and speech generation. ~33 GB on disk. Runs on a single 48 GB GPU.

At a Glance

Property	Value
Base model	`Qwen/Qwen3-30B-A3B`
Release tier	Provisional (datafree RTN — re-quant scheduled)
Quant method	datafree RTN W8A16 (weight-only INT8)
FLAC status	Not measured (T+7d milestone)
Architecture	Sparse MoE + Whisper audio + ViT vision + speech decoder
Quant format	compressed-tensors (native vLLM)
Quantized	48 attention layers (q/k/v/o_proj) — 192 tensors total
Kept BF16	audio_tower, visual, talker, code2wav
Disk size	~33 GB
Min GPU	1× A6000 48GB or A100 40GB

Why only attention projections?

Qwen3-Omni's MoE experts are implemented via @use_experts_implementation as stacked nn.Parameter tensors — not nn.Linear submodules. They cannot be reached by targets=["Linear"] in any current quantization framework. Only the 192 attention projections (48 layers × q/k/v/o_proj) are quantizable via standard W8A16 RTN.

Memory Requirements

Configuration	BF16	W8A16
Weights	~66 GB	~33 GB
Min GPU	2× A100 40GB	1× A100 40GB / A6000 48GB

Quick Start

Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.

vLLM — text output only

bash

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \
  88plug/Qwen3-Omni-30B-A3B-W8A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

Weights are in compressed-tensors format — no --quantization flag needed. Requires vLLM ≥ v0.21.0. Mainline vLLM returns text only. Audio input works; speech output does not.

vLLM-Omni — full audio output

vLLM-Omni v0.20.0 enables real-time speech output. Required if you need the model to speak.

bash

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-omni:v0.20.0-cu129 vllm serve \
  88plug/Qwen3-Omni-30B-A3B-W8A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

Recommended Sampling Parameters

Mode	Temperature	Top-P	Top-K	Min-P	Use When
Thinking (default)	0.6	0.95	20	0.0	Reasoning, math, code
Non-thinking	0.7	0.8	20	0.0	Chat, creative, fast response

Enable/disable thinking via chat_template_kwargs={"enable_thinking": True/False}. Default is thinking-enabled.

llama.cpp — audio/vision in, text out (no speech output in mainline)

Speech output is not yet in mainline llama.cpp (issue #21956). For full speech out, use the tc-mb/llama.cpp-omni fork. For audio/vision input with text output, mainline b9297 works.

bash

# Convert from BF16 base (audio/vision inputs supported via libmtmd)
python convert_hf_to_gguf.py Qwen/Qwen3-30B-A3B \
  --outfile Qwen3-Omni-30B-BF16.gguf

llama-quantize Qwen3-Omni-30B-BF16.gguf Qwen3-Omni-30B-Q8_0.gguf Q8_0

llama-server \
  --model Qwen3-Omni-30B-Q8_0.gguf \
  --n-gpu-layers 999 \
  --ctx-size 32768 \
  --port 8081

Benchmarks

Metric	Status
Throughput (tok/s)	In progress — T+7d milestone
MMLU delta vs BF16	In progress — T+7d milestone
RULER@128k	In progress — T+30d milestone

No fabricated numbers. Results will be published to this card when measured.

What's Quantized, What's Not

Component	Precision	Reason
Attention q/k/v/o_proj (all 48 layers)	W8A16 INT8	Quantized
MoE experts (routed + shared)	BF16	Not reachable as nn.Linear
`thinker.audio_tower.*`	BF16	Whisper encoder — excluded
`thinker.visual.*`	BF16	ViT — excluded
`talker.*`	BF16	Speech decoder — excluded
`code2wav.*`	BF16	Waveform codec — excluded
Embeddings, LM head, norms	BF16	Standard practice

Quality Targets

Metric	Target
KL divergence vs BF16	< 0.005
MMLU recovery	≥ 99.7%
RULER@128k recovery	≥ 99%

vs. Other Qwen3-Omni-30B Quants

This is the first compressed-tensors W8A16 checkpoint for Qwen3-Omni-30B-A3B. It targets the gap between the full BF16 model and hardware-restricted FP8, enabling single-GPU deployment on 48GB cards.

Quant	Method	Size	GPU Compatibility	Notes
88plug W8A16 (this)	compressed-tensors RTN W8A16	~33 GB	Any Ampere+	First W8A16; native vLLM; attention-only quant
unsloth Qwen3-Omni-30B GGUF	llama.cpp GGUF Q4_K_M	~19 GB	CPU / any GPU	Text+audio in; no speech output in mainline
unsloth Qwen3-Omni-30B GGUF	llama.cpp GGUF Q8_0	~34 GB	Any GPU ≥8 GB	Near-lossless; same speech limitation
BF16 baseline	None	~66 GB	2× A100 40GB	Reference; dual-GPU required

Limitations

Attention-only quantization: Qwen3-Omni's MoE expert weights are stacked nn.Parameter tensors, not nn.Linear submodules — unreachable by any current quantization framework. Only the 192 attention projections are quantized. Expert weights remain BF16.
No speech output in mainline vLLM: talker and code2wav TTS components stay BF16 and are not supported by mainline vLLM. Use vLLM-Omni v0.20.0 for speech output.
Multimodal encoders excluded: audio_tower (Whisper) and visual (ViT) stay BF16 — RTN INT8 not applied.
Benchmark results pending: Throughput and quality benchmarks (MMLU-Pro, GPQA) will be added post-publication.

Citation

bibtex

@misc{qwen3technicalreport,
  title  = {Qwen3 Technical Report},
  author = {Qwen Team},
  year   = {2025},
  url    = {https://huggingface.co/Qwen/Qwen3-30B-A3B}
}

About

88plug AI Lab ships compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Provisional tier — datafree RTN (weight-only rounding, no calibration corpus). A gold AutoRound re-quant is scheduled; 88plug architecture forbids new provisional W4A16 uploads.

Browse all releases → huggingface.co/88plug