Nemotron-Labs-Audex-30B-A3B

Introduction

Model Architecture

Templates

Multi-Stage-SFT and Cascaded-RL Pipelines

Quick Start

• Audex-30B-A3B follows the ChatML template and supports both thinking and instruct (non-thinking) modes. Reasoning content is enclosed within <think> and </think> tags. The instruct-mode prefix is task-specific. Prepend one of the following to the assistant's response, i.e. right after the assistant ChatML header:

  Task	Mode	Prefix	Note
  Text	thinking	<think>\n	The model closes the trace with </think> and then answers.
  Text	instruct (non-thinking)	</think>\n
  Audio	instruct (non-thinking)	<think></think>	No newline between </think> and the answer.

• Audex-30B-A3B supports up to a 1M-token context length.
• Audex-30B-A3B follows Nemotron-Cascade-2 on text evaluation.
• Audex-30B-A3B has different recommended inference setups per audio-related task as described below.

Environment

• vLLM inference — text-only reasoning, text-to-speech, text-to-audio, and audio understanding / speech recognition / speech translation: runs on vLLM 0.20.0.
• Hugging Face / transformers inference — requires transformers >= 4.53.0 (tested with 4.53.3) and also works with transformers >= 5.0. This additionally needs mamba-ssm and causal-conv1d. Build against your CUDA toolchain:

Audio QA Inference

• vLLM (recommended) — offline LLM.generate and an OpenAI-compatible audio_url server.
• Hugging Face / transformers — requires transformers >= 4.53.0 (we tested with 4.53.3), and also works with transformers >= 5.

Inputs

Inference recipes

• For audio understanding, we use top_p=0.9 and temperature=0.7.
• For speech recognition and translation, we use greedy sampling.

vLLM (recommended)

• Offline (JSON in → JSONL out):

• OpenAI-compatible server + client:

Hugging Face / transformers

• Example inference script: bash inference_scripts_hf/inference_example.sh.
• Task instruction examples: audio understanding — a question about the audio; speech recognition — Transcribe the speech in the input audio.\n<sound>; speech translation — a translation instruction such as Translate the speech in the input audio into English.\n<sound>.

Audio Generation Inference

Text-to-audio (TTA)

XCODEC1_PATH=/path/to/xcodec1 python3 run_audio_gen_vllm_rvq_logit_mask.py \
        --task tta \
        --model-path $(pwd)/../../checkpoint_folder_audiogen/ \
        --dataset-path /path/to/caption_txt_dir/ \
        --output-dir ../../tta_outputs/dataset_name/ \
        --tensor-parallel-size 8 \
        --temperature 1.0 \
        --top-k 80 \
        --max-tokens 2048 \
        --cfg-scale 3.0 \
        --cfg-pairs-per-batch 2

Text-to-speech (TTS)

./run_tts_vllm.sh --transcription "The weather is so good, and I want to enjoy the beautiful morning in the park." \
    --output-dir ./tts_outputs --utt-id the_weather_is_so_good

Text-Only Reasoning

• Text thinking mode uses the <think>\n prefix. Text instruct (non-thinking) mode uses </think>\n. See run_text_vllm_example.py which implements both (--disable-thinking selects instruct mode).
• chat_template.jinja uses <think></think> when enable_thinking=false which is used for audio tasks. To reproduce the reported text instruct-mode results, construct the prompt directly with </think>\n instead of relying on apply_chat_template.
• To create the checkpoint that completely matches their setup, run python model_conversion_scripts/convert_full_HF_to_textonly_HF.py to remove the audio-related vocabularies.
• Then, the text-only inference completely follows Nemotron-Cascade-2-30B-A3B. See a simple reasoning example with cd inference_scripts_vllm/textonly_scripts/; python run_text_vllm_example.py --model-path $(pwd)/../../checkpoint_folder_textonly.
• Note: you could also avoid model conversion by using sampling_params = SamplingParams(allowed_token_ids=list(range(131072))) in vLLM inference to mask the audio tokens, although we did not thoroughly test this approach.

Demo: speech-to-speech interaction with text reasoning

Reproducibility

• Text-to-speech (TTS): the original non-streaming XCodec2 decoder.
• Text-to-audio (TTA): XCodec1 followed by the enhancement VAE.
• Text: same as Nemotron-Cascade-2.
• Audio understanding: transformers 4.53.3 and Megatron-LM's native inference.

Detailed Benchmark Results

Text Results

Text-To-Speech Results

Text-To-Audio Results

Speech Recognition and Translation Results

Audio Understanding Results

Speech-To-Speech Results

Release Date

License

Citation

@article{Nemotron-Labs-Audex,
  title={Unified Audio Intelligence Without Regressing on Text Intelligence},
  author={Kong, Zhifeng and Lee, Sang-gil and Kim, Jaehyeon and Wang, Boxin and Liu, Zihan and Kim, Sungwon and Chen, Yang and Goel, Arushi and Roy, Rajarshi and Dai, Wenliang and Yang, Zhuolin and Chen, Yangyi and Jiang, Dongfu and Ghosh, Sreyan and Rintamaki, Tuomas and Tao, Andrew and Raiman, Jonathan and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei},
  year={2026}
}