You can deploy this model and start earning money today!

VikramPal/
Qwen3-Omni-30B-A3B-Thinker-SLURP-DynQuant-3bit

Name: Qwen3-Omni-30B-A3B-Thinker-SLURP-DynQuant-3bit
Availability: InStock

Views

No views yet

Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit

This checkpoint is broken and is published broken on purpose. Do not use it for inference. It scores 25.00% (125/500) on the single task it was built for, against 86.80% for the bf16 model it was quantized from.

If you want something that runs, take VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit, which scores 86.20% at 14.77 GiB and does not separate from the bf16 ceiling. It has the same two caveats as this repo: it is text output only (no speech), and it needs both pip install dynquant and an explicit dynquant.register_hf_quantizer() call before from_pretrained.

It is 11.08 GiB and it answers 25.00% of a 60-way classification correctly, against a chance floor of 1.667%. It is above chance and it is useless. Three of its 500 generations came back unparseable; every other arm in the results table below returned zero.

The reason to publish it is that the failure has a specific, named, reproducible mechanism — a quantization target set below the allocator's own floor budget — and there is no way for anyone else to check that account without the artifact. This repository exists so that floor-override damage can be measured by someone other than the person claiming it. It is a control, not a product.

Read this before you load it

Required, and it is two steps, not one

pip install dynquant # step 1 of 2 -- installing is NOT enough on its own

python
1import dynquant 2dynquant.register_hf_quantizer() # step 2 of 2 -- MUST run before from_pretrained

The checkpoint carries a quantization_config whose quant_method is dynquant. If the dynquant package is not installed — or is installed but dynquant.register_hf_quantizer() is never called — transformers does not raise. It logs a warning (Unknown quantization type, got dynquant ... Hence, we will skip the quantization.), silently sets pre_quantized=False, and from_pretrained returns a randomly initialised model that generates fluent garbage and scores at chance. This exact failure cost two full evaluation runs during the campaign that produced these checkpoints: it showed as 0.0% accuracy across two arms, with 499 and 500 of 500 answers unparseable, and the checkpoints were fine the whole time. Registration is an explicit call, deliberately not an import side effect — there is no transformers entry point that does it for you.

Since this artifact is already known to be broken, that failure would be easy to mistake for the real defect — so do not judge the load by the score. Count the swapped modules, which separates the two cleanly: a correctly loaded 3-bit checkpoint reports 602 DynQuant modules (504 linears + 2 embeddings + 96 expert banks; the 48 routers are dequantized to dense by design), and a silently unquantized one reports 0. The real collapse still answers in parseable indices 99.4% of the time; a random-init model does not.

dynquant 0.4.0 is on PyPI and is the exact version that produced this checkpoint. Beyond it you need transformers>=5.15 and accelerate, which pip install dynquant does not pull:

pip install dynquant "transformers>=5.15,<6" accelerate torch

Source: https://github.com/kambojvikram/dynquant.

Scope: Thinker only, and text output only

Only the Thinker of Qwen3-Omni was trained, merged and quantized. The Talker and code2wav stacks (3,540,613,057 parameters) were nulled out before training and are excluded from the checkpoint entirely.

This model therefore:

accepts audio, images, video and text as input;
emits text only. There is no speech output. This is not a speech-to-speech model.

The class is Qwen3OmniMoeThinkerForConditionalGeneration and model_type is qwen3_omni_moe_thinker. config.json has exactly three sub-configs — audio_config, text_config, vision_config — and no talker.

AutoModelForCausalLM and AutoModel both fail here. qwen3_omni_moe_thinker is absent from MODEL_MAPPING_NAMES and registered only under image-text-to-text, so both raise ValueError: Unrecognized configuration class Qwen3OmniMoeThinkerConfig. Use Qwen3OmniMoeThinkerForConditionalGeneration directly, or AutoModelForImageTextToText. Needs transformers >= 5.0 (produced and scored on 5.15.0; the snippets use the v5 dtype= spelling of torch_dtype=); no trust_remote_code is needed.

Parameter counts: whole Omni 35,259,818,545; Thinker 31,719,205,488; Talker plus code2wav 3,540,613,057. The 96 batched MoE expert banks account for 28,991,029,248 parameters, 91.399% of the Thinker. tie_word_embeddings is false — embed_tokens and lm_head are two separate 152064x2048 tensors, priced and quantized separately.

Why it collapsed

DynQuant allocates bit widths per module against a byte budget, subject to per-role floors. On this architecture the role floors alone cost 3.418 average bits. A 3.00-bit target therefore sits below the floor budget: the floors cannot all be satisfied at any allocation, the soft floors bind, and the allocator starts downgrading modules by lowest return on investment. It reports 400 of 650 modules breached.

At 4.00 bits the floors fit with 0.582 bits to spare, every floor is respected (violations: 0), and the score decides only what gets upgraded. At 3.00 bits the score decides what gets cut. The two arms are not the same experiment run at two budgets: the 3-bit arm measures floor-override damage. A hard-floor allocator would simply have refused a 3.00 target; that this arm ran at all is the soft-floor policy behaving as designed, and 25.00% is the answer to what those cuts cost.

The concrete damage:

42.9% of the parameters land at 2 bits — 13,596,426,240 elements across 125 modules.
lm_head is cut from a floor of 8 bits to 3, the only five-bit drop in the map, in the tensor that emits the token the task is scored on.

No single breach is the cause; 400 modules moved at once and either injury alone might be sufficient. Separating them requires an arm that pins lm_head at its floor and spends the difference elsewhere, which is a change to the floor policy rather than another run of the existing one. That experiment is named here and was not run.

One thing the floor policy did protect: the 48 MoE routers are never breached at either budget. MOE_ROUTER is a structural role, so the tensors that decide which experts a token passes through keep their 8 bits while everything around them is cut. The collapse at 3 bits is not a routing collapse. (What "keeps its 8 bits" means precisely is set out under Quantization detail — routers are quantized to 8 bits and written packed, then dequantized to dense at load.)

Results

Every arm below is the first 500 items of a random.Random(0) shuffle of the SLURP test split — not the first 500 rows as the mirror serves them, which arrive grouped by speaker and scenario. 4 shots drawn from train at seed 0, greedy decoding, max_new_tokens=8, stop at the first newline, add_special_tokens=False, max_prompt_tokens=4096, batch size 8, MoE experts dispatch pinned to eager, against the same 60-intent menu (intents_sha d04b663b407e9f5b5be80c9d11160c391c7b68f516c9da957aaca026138fc86d). Chance is 1.667%. Per-item hits are stored, so every comparison below is an exact McNemar test on paired outcomes.

arm	what it is	accuracy	vs comparator	discordant	p
`omni-base`	whole Omni checkpoint, bf16, no fine-tune	79.40% (397/500)	—	—	—
`omni-sft`	Thinker-only, adapter merged, bf16	86.80% (434/500)	+7.40 vs base	47/10	7.51e-07 separated
`omni-dq4`	DynQuant 4.00-bit map	86.20% (431/500)	−0.60 vs sft	12/15	0.7011 not separated
`omni-dq3` (this repo)	DynQuant 3.00-bit map	25.00% (125/500)	−61.80 vs sft	3/312	1.56e-88 separated

95% confidence intervals on the paired differences: base to sft [+4.51, +10.29]; sft to dq4 [−2.64, +1.44]; sft to dq3 [−66.17, −57.43].

The discordance for this arm is 3/312: it recovered three items the bf16 model missed and lost 312 it had. It returned 3 unparseable generations; every other arm in this table returned 0.

Both packed checkpoints reproduce their in-memory counterpart on all 500 items, 0 discordant, at both widths. That is a round-trip test whose predicted outcome is exact identity — same codes, same scales, same compute dtype means bit-identical weights and therefore identical greedy tokens — so it is a check that the packed path matches the encoder, not an independent replication of the accuracy.

The prompt, exactly

The model answers with an index into a numbered 60-intent menu — an integer such as 37 — not with an intent name and not with a sentence. Anything outside that shape is scored wrong.

Menu. Read scenario and action from SLURP's own annotation at https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl for all three splits (train, devel, test), form f"{scenario}_{action}", take the sorted unique set. That is 60 strings, numbered from 0; sha256("\n".join(menu)) must equal the digest above. Deriving it from one split renumbers the taxonomy and every learned index names its neighbour. The header is:
```
Listen to the spoken command and classify it into one of these intents.
Answer with the number only.
0. alarm_query
...
59. weather_query
```
Block order. The menu and exemplars come first, then the audio, then a trailing Intent:. The exemplars are rendered as text, never as audio, in the form "{transcript} -> {gold_index}" under the line Examples, as text:.
Audio. A bare mono float32 ndarray at 16 kHz inside the {"type": "audio", "audio": ...} block, with sampling_rate=16000 passed beside the block to apply_chat_template. The {"array", "sampling_rate"} dict raises; a wrong rate does not — it silently reads the clip at the wrong speed.
Items. random.Random(0).shuffle(examples), then the first 500 of test.
Shots. sorted(random.Random(0).sample(range(50628), 4)) over the identically shuffled train pool — an index sample, not a prefix. Their gold indices are 27, 15, 34, 22.
Scoring. Strip any reasoning trace, then take a leading integer in [0, 60) if there is one, else the first in-range integer anywhere in the text; anything naming no index in [0, 60) is unparseable and counts as wrong.

The reference implementation is dynquant.eval.slurp, which registers the quantizer itself and raises if a packed checkpoint comes back unpacked:

bash

pip install dynquant "transformers>=5.15,<6" "datasets>=4" torchcodec

dynquant eval VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit \
  --task slurp --split test --limit 500 --shots 4 --shot-seed 0 \
  --model-class Qwen3OmniMoeThinkerForConditionalGeneration \
  --experts-impl eager --device cuda:0 --dtype bfloat16 --out dq3.json

Reproducing it downloads the whole SLURP test split and the whole train split (50,628 recordings, pulled in full to draw four text exemplars), so budget the disk and the network.

Loading it, if you are reproducing the collapse

python

import torch
import dynquant
dynquant.register_hf_quantizer()   # REQUIRED -- see the warning above

from transformers import AutoProcessor, Qwen3OmniMoeThinkerForConditionalGeneration

repo = "VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit"

processor = AutoProcessor.from_pretrained(repo)
assert type(processor).__name__ == "Qwen3OmniMoeProcessor", type(processor).__name__

model = Qwen3OmniMoeThinkerForConditionalGeneration.from_pretrained(
    repo,
    dtype=torch.bfloat16,   # compute dtype; the weights stay packed
    device_map="cuda:0",    # needs `accelerate`
)

# Confirm the load. This is the check that separates the real collapse from a silent
# random-init: 0 means register_hf_quantizer() never ran, any other number means the
# module swap was partial. Do not weaken it to `> 0`.
packed = sum(1 for m in model.modules() if type(m).__name__.startswith("DynQuant"))
assert packed == 602, f"expected 602 packed modules, got {packed}"
# Text output only. There is no Talker in this checkpoint.

Ignore the "Use this model" snippet the Hub renders above this card. It emits a bare AutoModel* call with no dynquant import and no registration — exactly the path that returns a randomly initialised model.

Written and tested against transformers 5.15.0, torch 2.11+cu128, dynquant 0.4.0. Neutral decode fields must be pinned explicitly (do_sample=False, num_beams=1, repetition_penalty=1.0, length_penalty=1.0, no_repeat_ngram_size=0): on transformers 5.x, fields left unset are filled from the checkpoint's own generation_config, so a shipped repetition_penalty survives do_sample=False and moves the score.

This directory ships processor_config.json and no preprocessor_config.json. AutoProcessor resolves Qwen3OmniMoeProcessor from the former; if you see an error naming preprocessor_config.json, that filename is a red herring — the merged bf16 reference has none either and loads fine.

Sizes

Three distinct levels, which should never be mixed. The GiB conversion is attached to the level it belongs to:

	weight-tensor bytes	shard-file bytes	directory (`du -sb`)
bf16 merged Thinker	63,438,410,976 (59.08 GiB)	63,440,876,184	63,454,086,373
DynQuant 4-bit	15,859,386,016 (14.77 GiB)	15,859,696,832	15,871,835,299
DynQuant 3-bit (this repo)	11,894,607,776 (11.08 GiB)	11,894,918,480	11,907,056,499

Like-for-like compression, shard-file against shard-file: 5.33x for this arm, 4.00x for the 4-bit arm.

Resident VRAM, measured with torch.cuda.memory_allocated across a load on an idle card: 11,927,683,584 B for this arm (5.32x less than the bf16 merge's 63,440,876,184 B of shard files — bf16 resident VRAM was not itself measured, so this is not a VRAM-to-VRAM comparison); the 4-bit arm is 15,892,454,912 B, 3.99x less on the same denominator. Residency sits 0.28% above the stored payload for this arm and 0.21% for the 4-bit one — 33,075,808 B and 33,068,896 B. Most of that is the routers, not metadata: all 48 are stored packed at 8 bits (270,336 B each) but restored to dense bfloat16 at load, costing 524,288 B resident each, so 48 x 253,952 B = 12,189,696 B — 36.9% — is that restoration on both arms, identical because MOE_ROUTER is a structural 8-bit floor at either budget. The ~20.9 MB remainder is the runtime's own bookkeeping, not the scales and offsets, which are already counted inside the stored payload.

Those are floors, not requirements: they are weights at load, with no activations, no KV cache, no audio-encoder workspace and no ~4k-token prompt. Peak allocation over a full 500-item run was not recorded in this campaign, so size a card with headroom over the figure above rather than to it.

The saving is real — these are packed checkpoints that dequantize at compute time, not fake-quant checkpoints carrying rounded weights at full width. Had any module fallen back to fp16 the residue would be gigabytes rather than tens of megabytes. None of that redeems the artifact. It is a small 25.00% model.

Quantization detail

Group size 128, asymmetric (per-group bf16 scale plus bf16 offset), layout linear, compute dtype bfloat16. 650 modules in the bit map — 602 written packed, plus the 48 routers, which are restored dense at load — covering 31,712,079,872 elements.

7,125,616 dense parameters (norms, biases, convolutions) stay bf16. The 48 MoE routers (mlp.gate) are a third case, and the one most often misstated: they are in the map at 8 bits — they are 48 of the 48 8-bit modules in the histogram below — and they are written packed at 8 bits (270,336 B each, inside the packed total rather than beside it). Nothing in the packed runtime can stand where a router stands, so the loader dequantizes them back into a dense bfloat16 weight after load, which is why a read-back reports 48 of 48 routers dense and why the assertion above expects 602 rather than 650. Their values therefore carry 8-bit quantization error; only their storage at runtime is dense. What the floor policy guarantees is that a router is never breached.

Achieved average bits for this arm: 2.997055 over quantized elements, 2.999976 over all Thinker parameters. (The 4-bit arm is 3.997249 and 3.999945 respectively.)

Width histogram:

width	this arm (3.00)	the 4-bit arm (4.00)
2-bit	125 modules, 13,596,426,240 elements	9 modules, 1,811,939,328
3-bit	368 modules, 12,595,818,496	67 modules, 5,697,961,984
4-bit	109 modules, 5,507,252,224	524 modules, 23,877,120,000
8-bit	48 modules, 12,582,912	50 modules, 325,058,560
floor violations	400 of 650	0

The element-weighted mean of this arm's histogram is 2.7469 bits, not 2.9971. The difference is the group metadata, which the quoted average includes: a bf16 scale plus a bf16 offset every 128 elements is 0.25 bits per element. Add the 0.25 and you land within 0.0002 of the figure above, the remainder being rows whose length does not divide evenly into groups. The same arithmetic reconciles the 4-bit arm (3.7470 + 0.25).

Training

The adapter was trained with QLoRA over an NF4 base, peft 0.20.0: r=16, lora_alpha=32, lora_dropout=0.05, bias none, targeting q_proj, k_proj, v_proj, o_proj, out_proj, fc1, fc2.

The MoE expert banks were not adapted. LoRA does not reach batched 3-D expert banks, so 91.399% of the parameters were frozen and unadapted throughout. The merge touched 384 weights; all 96 expert banks are unchanged from the base model.

Run with torchrun, world size 2, effective batch 16, 500 optimizer steps, 4,188.1 s wall, on vast.ai: 2x RTX PRO 6000 Blackwell Max-Q (94.97 GiB each, sm_120), torch 2.11+cu128, transformers 5.15.0, dynquant-core 0.4.0. Training used 8,000 of the 50,628 SLURP train recordings and saw 59 distinct intents out of the 60-class menu. Final train_loss 4.031702354431152; the loss curve ran 5.479 to 3.080.

The adapter was trained against NF4 and merged into bf16 — standard for QLoRA, and it means the merged checkpoint is not bit-for-bit the model that produced the signal used to allocate the bit map.

Task and data

SLURP intent classification. The label is the joint scenario_action pair, 60 classes. Audio comes from the Hub dataset marcel-gohsen/slurp; labels come from SLURP's own annotation at https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl.

Two sources are used because the mirrors' own intent column is corrupted: on 1,548 of 72,396 recordings it disagrees with the scenario/action pair, having dropped the scenario, so nine different *_query intents all collapse to query. That yields 91/71/77 classes per split against the real 60. The scenario and action fields are clean, so the joint pair is the label.

SLURP is Bastianelli et al., EMNLP 2020, CC BY 4.0.

Limitations

Beyond the fact that this checkpoint does not work:

The expert banks were never adapted. 91.399% of the Thinker's parameters — the 96 batched MoE banks, 28,991,029,248 parameters — were frozen through fine-tuning and unchanged by the merge. Whatever the fine-tune achieved, it achieved through attention and the dense projections alone. The quantizer does reach them; the trainer did not.

500 evaluation items bound the resolution of every comparison here. That is enough to separate a 61.80-point collapse at any sensible threshold, and it is not enough to resolve small differences; the 4-bit arm's [−2.64, +1.44] interval is the honest form of that limit.

One task, one language. Everything reported is SLURP intent classification in English, under a fixed 4-shot prompt with the full 60-intent menu in context. Nothing here says anything about this quantization's effect on any other task, any other language, or the model's vision and general-purpose capabilities, none of which were measured, and the allocation signal itself was collected on this one task.

SLURP is one collection with one recording protocol. It is not a sample of arbitrary field audio, and accuracy on it does not transfer unexamined to other microphones and acoustic conditions, telephony audio, unfamiliar accents, or spontaneous speech outside this collection.

No baseline quantizers. GPTQ, AWQ and RTN were out of scope for this campaign. There is therefore no matched-bytes competitive claim, and specifically no evidence about whether this collapse is DynQuant's allocation or simply what 3 bits does to this architecture. A uniform-width control at the same byte anchor would answer that in one run and was not run.

The base-to-SFT comparison differs in two things, not one: the base screen ran the whole Omni checkpoint while every post-merge arm runs the Thinker alone, and only the post-merge arms carry the merged adapter. The arms carrying the actual quantization claim (sft against dq4 against dq3) are all Thinker-only and differ in exactly one thing.

No speech output. See the scope section; the Talker was removed before training.

Related repositories

VikramPal/Qwen3-Omni-30B-A3B-Thinker-QLoRA — the QLoRA adapter alone (112,207,855 B, 107 MiB, of which roughly 11.4 MB is tokenizer.json). Its adapter_config.json carries an empty base_model_name_or_path and a null task_type, so nothing in the file tells PeftModel.from_pretrained what to attach to: load the base yourself and attach to .thinker, never to the whole Omni model, whose Talker carries matching q_proj/k_proj/v_proj/o_proj names the adapter was never fitted against.
VikramPal/Qwen3-Omni-30B-A3B-Thinker-bf16 — the merged bf16 reference, 86.80%, 59.08 GiB, the ceiling this arm is measured against.
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit — the one you probably want: 86.20% at 14.77 GiB.
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit — this repository.

DynQuant source, format specification and the allocator that produced this bit map: https://github.com/kambojvikram/dynquant.

License

The base model is Qwen/Qwen3-Omni-30B-A3B-Instruct, released under license: other with license_name: apache-2.0. This derivative mirrors that, and nothing here grants rights beyond the base model's terms. SLURP is CC BY 4.0 and its terms apply to the data, not to these weights.

Citation

SLURP:

bibtex

@inproceedings{bastianelli-etal-2020-slurp,
  title     = {{SLURP}: A Spoken Language Understanding Resource Package},
  author    = {Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2020}
}

This artifact: