⚠️ Experimental / needs more VRAM headroom than a single 24GB GPU provides. See Known limitations before using this in production. If you just want something proven-stable on a 24GB card, use Audex-30B-A3B-W4A16 instead (no shared-expert/embedding quantization, but rock solid).

A second GPTQ quantization pass on top of the v1 recipe, for nvidia/Nemotron-Labs-Audex-30B-A3B. Adds two more quantized groups beyond v1's routed-MoE-only INT4:

Component	Precision	BF16 size → quantized
Routed MoE experts (same as v1)	INT4, group_size=64	58.75GB → ~14GB
Shared expert (new in v2)	INT4, group_size=64	0.92GB → ~0.23GB
Embeddings + lm_head (new in v2)	INT8, channel-wise	2.21GB → stored dense (see below)

Attention/Mamba mixer projections, audio encoder, and router are left untouched at BF16, same reasoning as v1 (most quality-sensitive, least structurally redundant).

Goal of this pass: free enough VRAM to also fit the ~2.4GB speech decoder on the same 24GB GPU as the LLM weights, so TTS synthesis doesn't have to run on CPU (CPU decoder measured at 2.7–3.2x slower than real-time — not viable for interactive use). Net saving from the new groups alone is modest (~0.69GB from the shared expert; the embed/lm_head group did not actually compress — see below), which is why this remains tight on 24GB.

Known limitations

Not stable for sustained real-world inference on a single 24GB GPU. It loads and passes lightweight smoke tests at --gpu-memory-utilization values around 0.88, but under real multi-request traffic it OOMs inside the Mamba mixer's _chunk_state_fwd (SSD scan) — this was reproduced across a GMU sweep from 0.83 to 0.90 and is a hard memory-margin problem, not a config-tuning one. The math: ~20GB quantized weights + 2.4GB decoder leaves too little consistent margin for KV cache + Mamba scan activation memory on a 24GB card.
The embeddings/lm_head group is not actually compressed. vLLM's VocabParallelEmbedding/lm_head layer types for this architecture aren't wired to consume a quant_method, so despite being INT8-quantized during the GPTQ pass, the resulting weights are stored in compressed-tensors' "dense" format (full-size tensors, quantization scale/zero-point metadata computed but unused at inference time). In practice this group contributes close to zero VRAM savings over BF16 today. It's included here in case a future vLLM version wires up that layer type, or if you use a different inference stack that respects the stored quantization scheme.
Given the above, this checkpoint's real usable savings vs v1 are ~0.69GB (shared expert only) — likely not enough headroom to justify the added complexity over v1 on a single 24GB GPU. It may be more useful on a GPU with more headroom (32GB+) where you want the decoder on-GPU with comfortable margin, or as a base for further work (e.g. wiring up embed/lm_head compression properly, or additionally quantizing the decoder itself).
Not independently evaluated for quality regression vs BF16 or vs v1.

Quantization recipe

python

GPTQModifier(
    config_groups={
        "group_0": QuantizationScheme(  # routed experts
            targets=[r"re:.*mixer\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
            weights=QuantizationArgs(num_bits=4, type="int", symmetric=True,
                                      strategy="group", group_size=64),
        ),
        "group_1": QuantizationScheme(  # shared experts
            targets=[r"re:.*mixer\.shared_experts\.(gate_proj|up_proj|down_proj)$"],
            weights=QuantizationArgs(num_bits=4, type="int", symmetric=True,
                                      strategy="group", group_size=64),
        ),
        "group_2": QuantizationScheme(  # embeddings + lm_head
            targets=[r"re:^backbone\.embeddings$", r"re:^lm_head$"],
            weights=QuantizationArgs(num_bits=8, type="int", symmetric=True,
                                      strategy="channel"),
        ),
    },
    ignore=[r"re:.*audio_encoder.*", r"re:.*sound.*", r"re:.*projector.*", r"re:.*mixer\.gate.*"],
)

Calibrated on 512 samples from HuggingFaceH4/ultrachat_200k, max sequence length 2048. Run on Modal (A100-80GB, 96GB host RAM) — the local machine's 32GB RAM was insufficient (severe swap thrashing during weight loading).

The gate_proj pattern is included in the routed/shared expert target regexes even though it isn't separately calibrated/quantized as its own tensor for MoE experts in this architecture — it's required so vLLM's FusedMoE scheme lookup (which checks all three projection names) succeeds; the regex match alone satisfies the "all MoE projections need matching schemes" requirement.

Usage (vLLM)

Same as v1 — requires the audex_30b_a3b_vllm out-of-tree plugin bundled with the original model repo. Drop-in weights replacement for the checkpoint_folder_full/ subfolder layout.

bash

python cascaded_s2s_web_server.py \
  --model /path/to/Audex-30B-A3B-W4A16-v2 \
  --gpu-memory-utilization 0.88 \
  --decoder-device cuda:0   # fits at this GMU on a 24GB card, but not stable under load — see caveats

If you have more VRAM headroom (32GB+, or a second GPU dedicated to the decoder), this checkpoint's weights are still a valid ~1GB-smaller alternative to v1.

wilherman2121/Audex-30B-A3B-W4A16-v2

Audex-30B-A3B-W4A16-v2

Known limitations

Quantization recipe

Usage (vLLM)

wilherman2121/
Audex-30B-A3B-W4A16-v2