Only the routed MoE expert projections (gate_proj/up_proj/down_proj inside each of the 128 experts × 23 layers) are quantized, INT4 with group_size=64, using llmcompressor's GPTQModifier. Everything else — attention/Mamba mixer projections, the audio encoder, embeddings, lm_head, shared expert, and router — is left at full BF16 precision, on the theory that those are more quality-sensitive and less structurally redundant than the routed experts.

This shrinks the routed-expert weights from ~58.75GB to ~14GB, bringing total checkpoint size from 65.26GB down to ~21GB. This is the version that comfortably fits on a single 24GB GPU (e.g. RTX 3090) with room for KV cache, though the TTS speech decoder (~2.4GB) still needs to run on CPU or a second GPU at this size — see Audex-30B-A3B-W4A16-v2 for a further-quantized variant that frees more headroom.

Status: tested and stable. This checkpoint has been used to serve real multi-turn ASR/LLM/TTS traffic without crashes.

Quantization recipe

yaml

GPTQModifier:
  config_groups:
    group_0:
      targets: ['re:.*mixer\.experts\.\d+\.(up_proj|down_proj)$']
      weights:
        num_bits: 4
        type: int
        symmetric: true
        group_size: 64
        strategy: group
        actorder: static
  targets: [Linear]
  ignore: [lm_head, 're:.*embed.*', 're:.*audio.*', 're:.*sound.*',
           're:.*projector.*', 're:.*shared_expert.*', 're:.*\.gate$', 're:.*router.*']
  dampening_frac: 0.01

Calibrated on 512 samples from HuggingFaceH4/ultrachat_200k, max sequence length 2048.

Note: the target regex above matches up_proj/down_proj only, but the resulting checkpoint's config.json targets also include gate_proj in the pattern — this is required so vLLM's FusedMoE scheme lookup succeeds (it checks all three projection names even though gate_proj isn't separately quantized here; the regex match alone satisfies vLLM's "all MoE projections need the same quantization scheme" requirement without requiring a physically separate quantized tensor).

Usage (vLLM)

Requires the audex_30b_a3b_vllm out-of-tree plugin bundled with the original model repo (see its inference_scripts_vllm/ folder) for the NemotronHForCausalLM audio architecture and the cascaded ASR/LLM/TTS web server. This checkpoint is a drop-in weights replacement for that pipeline — same config/tokenizer/audio-preprocessor layout as the original manifest's checkpoint_folder_full/ subfolder.

bash

python cascaded_s2s_web_server.py \
  --model /path/to/Audex-30B-A3B-W4A16 \
  --gpu-memory-utilization 0.90 \
  --decoder-device cpu   # or cuda:0 if you have headroom

Known limitations

Not independently evaluated for ASR/generation quality regression vs the BF16 original — only smoke-tested for stability and round-trip functionality across a handful of languages.
The speech decoder does not fit on-GPU alongside these weights on a 24GB card with healthy KV cache margin; CPU decoder works but is ~2.7–3.2x slower than real-time. See the v2 checkpoint for a variant that frees more VRAM for this.

wilherman2121/Audex-30B-A3B-W4A16

Audex-30B-A3B-W4A16

Quantization recipe

Usage (vLLM)

Known limitations

wilherman2121/
Audex-30B-A3B-W4A16