You can deploy this model and start earning money today!

RESMP-DEV/
MiMo-V2.5-ASR-FP8

Name: MiMo-V2.5-ASR-FP8
Availability: InStock

automatic-speech-recognition

yue

XiaomiMiMo/MiMo-V2.5-ASR

finetune

mit

Views

No views yet

MiMo-V2.5-ASR — FP8 (e4m3fn)

FP8-quantized build of XiaomiMiMo/MiMo-V2.5-ASR, the Xiaomi MiMo end-to-end ASR model with native Mandarin/English code-switching, Chinese dialects, song lyrics, noisy/multi-speaker robustness, and native punctuation.

What this is

Weights: float8_e4m3fn, per-output-channel absmax scaling (one fp32 scale per row), baked at save time.
Activations: dynamic per-tensor fp8 quantization each forward pass.
Matmul: torch._scaled_mm (FP8 tensor cores on Ada / Hopper / Blackwell).
Skipped (kept bf16): embeddings, RMSNorm/LayerNorm, biases. 417 Linear layers converted.
The audio encoder/tokenizer (MiMo-Audio-Tokenizer) is not quantized; download it separately for inference.

This roughly halves the LLM weight footprint (~32 GB bf16 → ~16-17 GB on disk).

Important: this is NOT a drop-in `from_pretrained` checkpoint

model.safetensors stores custom FP8Linear buffers (*.weight_fp8, *.weight_scale), not standard HF Linear weights. It must be loaded through the matching FP8Linear modules. Use the loader below.

Usage

bash

git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git
cd MiMo-V2.5-ASR
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1   # required by the audio tokenizer
hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir ./models/MiMo-Audio-Tokenizer
hf download Infatoshi/MiMo-V2.5-ASR-FP8 --local-dir ./MiMo-V2.5-ASR-FP8

Then install the maintained loader from RESMP-DEV/mimo-asr-fp8 and load with its FP8Linear implementation:

python

from quantize_fp8 import load_fp8_model
mimo = load_fp8_model(
    fp8_dir="./MiMo-V2.5-ASR-FP8",
    tokenizer_path="./models/MiMo-Audio-Tokenizer",
    repo_root=".",                     # the cloned MiMo-V2.5-ASR repo
)
print(mimo.asr_sft("audio.wav", audio_tag="<english>"))

Quantization fidelity

Per-output-channel absmax dequant error vs the original fp32 weights, sampled across depth (layers 0/17/35), all attn+mlp projections, lm_head, and the audio local transformer:

relative Frobenius error: ~0.026, uniform across every sampled layer (max 0.027 on lm_head)
no corrupted or outlier layers

This is the expected magnitude for fp8 e4m3 with per-channel scaling (3 mantissa bits).

Requirements

CUDA GPU with FP8 tensor cores (Ada / Hopper / Blackwell), CUDA >= 12.0
torch >= 2.6, safetensors
Blackwell (sm_120, e.g. RTX PRO 6000 / RTX 50xx): use a torch build with CUDA 12.8+ (torch >= 2.7, cu128). torch 2.6 cu124 ships no sm_120 kernels and will fail with "no kernel image is available for execution on the device".

Notes / caveats

FP8 e4m3fn weight-only-style quantization is lossy; expect small WER deltas vs bf16.
Per-tensor dynamic activation scaling is simple and fast but less accurate than per-token scaling on activations with large outliers.

Derivative of an MIT-licensed model; original credit to the Xiaomi MiMo team.

RESMP-DEV/MiMo-V2.5-ASR-FP8