Stable Audio 3 — MNN CUDA Models

Test Environment

Model Summary

Performance (--mnn-int8, 8 steps)

Music

SFX

VRAM = incremental GPU memory used by inference (excluding desktop/display baseline). Decoder uses chunked decoding (~23.8s per chunk), so VRAM is constant regardless of audio length.

Quantization Strategy

T5 Encoder

Model Architecture

Text Prompt → T5Gemma (MNN CPU INT4) → Text Embedding
Duration   → NumberConditioner (MNN CUDA INT8) → Duration Embedding
                                    ↓
                        DiT (MNN CUDA INT8) ← Diffusion Denoising
                                    ↓
                        Decoder (MNN CUDA INT8 FusedWN) → Audio Waveform

Repository Structure

├── music/int8/          # Music variant — INT8 models
│   ├── dit_music_int8.mnn           (445 MB)
│   ├── encoder_int8.mnn             (52 MB)
│   └── number_conditioner_music_int8.mnn  (0.2 MB)
├── music/fp16/          # Music variant — FP16 models (fallback)
│   ├── dit_music_fp16_v2_f32io.mnn
│   ├── encoder.mnn
│   └── number_conditioner_music_fp16.mnn
├── sfx/int8/            # SFX variant — INT8 models
│   ├── dit_sfx_int8.mnn             (445 MB)
│   └── number_conditioner_sfx_int8.mnn    (0.2 MB)
├── sfx/fp16/            # SFX variant — FP16 models (fallback)
│   ├── dit_sfx_fp16_v2_f32io.mnn
│   └── number_conditioner_sfx_fp16.mnn
├── shared/              # Shared across variants
│   ├── decoder_fused_wn.mnn         (53 MB, INT8)
│   ├── text_encoder_int4.mnn        (538 MB, INT4 — default)
│   ├── text_encoder.mnn             (1075 MB, FP32 — fallback)
│   ├── tokenizer.json
│   └── bottleneck_params.json
└── dll/                 # Pre-built Windows DLLs
    ├── MNN.dll
    └── mnn_dit_bridge.dll

Key Features

• INT8 Full Pipeline: T5 INT4 + DiT INT8 + Decoder INT8 + Encoder INT8. Total model size ~1.09 GB, RTF 10-33x on RTX 2080 Ti
• Chunked Decoding: Decoder processes latents in chunks of 256 timesteps, enabling pseudo-streaming output (~23.8s of audio per chunk) and constant VRAM regardless of audio length
• Pre-allocated Memory: Decoder is initialized with chunk_size=256 at load time, eliminating expensive resize during inference
• WeightNorm Pre-fusion: Decoder Conv1d WeightNorm is pre-fused into weights before ONNX→MNN conversion, avoiding FP16 precision issues in L2 normalization
• Softmax Fix: MNN CUDA Softmax kernel patched for small axis sizes (≤64) — see MNN fork

MNN CUDA Bug Fixes

1. Softmax Fix: SOFTMAX_AXIS_REDUCE kernel had a bug where idle threads in block reduction corrupted __shared__ memory for small axis sizes, causing echo artifacts in Decoder attention. Fixed by using simple per-thread SOFTMAX kernel for axis ≤ 64.
2. MatMul PrecisionType Fix: FP16FP32Mix mode incorrectly set precisionType=2 (always FP16), causing precision anomalies. Fixed to use precisionType=0 for FP16FP32Mix/FP32 modes.
3. LoopCreator Op Type Check: CUDALoop::onExecute only supports UnaryOp/MatMul/BinaryOp for non-fused commands. Other op types (LayerNorm, Softmax, Select, etc.) were silently skipped, producing garbage output. LoopCreator now rejects unsupported op types at creation time, falling back to CPU.
4. T5 Forced to CPU: T5 MNN CUDA produces incorrect output (root cause not isolated). T5 always runs on MNN CPU backend, producing correct results verified against ONNX Runtime.

Usage

License