Qwen3-Omni: Optimized for AMD ROCm
Qwen3-Omni-30B-A3B-Instruct is a multi-modal LLM supporting text, image, audio, and video, served via
vLLM's OpenAI-compatible HTTP API. This repository packages inference for multi-modal text generation using vLLM, exported and validated for
AMD ROCm so it runs efficiently on AMD GPUs.
This is based on the implementation of Qwen3-Omni found
here.
This repository contains configurations and scripts optimized for
AMD® ROCm™ platforms. You can use the
qwen3_omni AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found
here.
Task Overview
Task: Multi-modal text generation (text, image, audio, video)
Dataset: GSM8K (math, generative, 5-shot)
Output metrics: GSM8K exact-match accuracy, per-modality latency/throughput
Model variants: Default is Qwen3-Omni-30B (bf16) (Qwen/Qwen3-Omni-30B-A3B-Instruct, ~60 GB). Override with the AWQ INT4 variant (cyankiwi/Qwen3-Omni-30B-A3B-Instruct-AWQ-4bit, ~17 GB). Only one variant's weights live on disk at a time; downloading another deletes the previous one's. Both variants share Qwen3-Omni's 32768-token context ceiling.
vLLM note: GPU-only (vLLM server). No CPU or NPU inference path is available. This version of vLLM will not work with kraken2e devices — ROCm does not ship the supporting kernels for gfx1153.
AMD ROCm Optimization
This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm. Key points:
- Validated backend: vLLM (ROCm-enabled, built from source, OpenAI-compatible HTTP API).
- No code changes required versus the upstream Qwen3-Omni implementation — only environment/runtime configuration differs.
- The AWQ variant defaults to
--dtype float16 on ROCm so vLLM selects the Exllama W4A16 kernel.
- Not supported on kraken2e devices (gfx1153) — ROCm lacks supporting kernels for this vLLM version.
| Runtime | Precision | Backend | Hardware | Notes |
|---|
| GPU | bf16 / FP16 (AWQ INT4) | vLLM | AMD GPU (ROCm) | OpenAI-compatible HTTP API; ≥20 GB VRAM for AWQ, ≥64 GB for bf16 |
Getting Started
For setup instructions, evaluation scripts, and custom configuration options, see the
qwen3_omni on GitHub.
Model Details
Model Type: Multi-modal causal language model (text, image, audio, video)
Base Model: Qwen/Qwen3-Omni-30B-A3B-Instruct (default, bf16) — cyankiwi/Qwen3-Omni-30B-A3B-Instruct-AWQ-4bit (AWQ INT4) also supported
Model Stats:
- Model variant: Qwen3-Omni-30B bf16 (default, ~60 GB) — AWQ INT4 (~17 GB) also supported
- Context ceiling: 32768 tokens (shared across both variants)
Accuracy Pipeline
Accuracy is measured with
EleutherAI's lm-evaluation-harness running GSM8K in generative mode against the server's OpenAI-compatible
/v1/completions endpoint. Image, audio, and video accuracy are not covered by this text-only harness.
Metrics Explained
| Metric | Description |
|---|
| exact_match | Exact-match accuracy on GSM8K (math word problems), generative 5-shot evaluation. |
Dig Deeper
Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?
The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Scripts for building vLLM from source, server management, and variant switching
- Per-modality (text/image/audio/video) latency and throughput benchmarking
- GSM8K accuracy evaluation and reproduction instructions
