MiDashengLM is an efficient audio-language model that achieves holistic audio understanding through caption-based alignment. It achieves state-of-the-art performance on multiple audio understanding benchmarks while maintaining high inference efficiency—delivering 3.2× throughput speedup and supporting batch sizes up to 512.

📖 For more detailed introduction and technical report, please visit our GitHub repository.

Note that for most applications, we strongly recommend using the BF16 version (mispeech/midashenglm-7b-0804-bf16) for optimal performance and efficiency.

Usage

Load Model

python

from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer

model_id = "mispeech/midashenglm-7b-0804-fp32"  # Only for exact reproduction; otherwise strongly recommend "mispeech/midashenglm-7b-0804-bf16"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

Construct Prompt

python

user_prompt = "Caption the audio."  # You may try any other prompt

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": user_prompt},
            {
                "type": "audio",
                "path": "/path/to/example.wav",
                # or "url": "https://example.com/example.wav"
                # or "audio": np.random.randn(16000)
            },
        ],
    },
]

Generate Output

python

import torch

with torch.no_grad():
    model_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        add_special_tokens=True,
        return_dict=True,
    ).to(device=model.device, dtype=model.dtype)
    generation = model.generate(**model_inputs)
    output = tokenizer.batch_decode(generation, skip_special_tokens=True)  # ["An engine is idling."]

Results

The following evaluation results are based on the model version: mispeech/midashenglm-7b-0804-fp32.

Audio Captioning Results

Domain	Dataset	MiDashengLM	Qwen2.5-Omni-7B	Kimi-Audio-Instruct
Music	MusicCaps	59.71	43.71	35.43
Music	Songdescriber	45.39	45.31	44.63
Sound	AudioCaps	62.18	60.79	49.00
Sound	ClothoV2	49.20	47.55	48.01
Sound	AutoACD	66.52	55.93	44.76

Metrics: FENSE (higher is better).

Audio and Paralinguistic Classification

Dataset	Metric	MiDashengLM	Qwen2.5-Omni-7B	Kimi-Audio-Instruct
VoxCeleb1	ACC↑	92.36	59.71	82.72
VoxLingua107	ACC↑	93.41	51.03	73.65
VoxCeleb-Gender	ACC↑	96.12	99.82	99.69
VGGSound	ACC↑	52.11	0.97	2.20
Cochlscene	ACC↑	74.06	23.88	18.34
NSynth	ACC↑	80.52	60.45	38.09
FMA	ACC↑	63.73	66.77	27.91
FSDKaggle2018	ACC↑	75.25	31.38	24.75
AudioSet	mAP↑	8.86	6.48	3.47
FSD50K	mAP↑	37.58	23.87	27.23

ASR Performance

Dataset	Language	MiDashengLM	Qwen2.5-Omni-7B	Kimi-Audio-Instruct
LibriSpeech test-clean	English	3.7	1.7	1.3
LibriSpeech test-other	English	6.2	3.4	2.4
People's Speech	English	27.8	28.6	22.3
AISHELL2 Mic	Chinese	3.2	2.5	2.7
AISHELL2 iOS	Chinese	2.9	2.6	2.6
AISHELL2 Android	Chinese	3.1	2.7	2.6
GigaSpeech2	Indonesian	20.8	21.2	>100
GigaSpeech2	Thai	36.9	53.8	>100
GigaSpeech2	Viet	18.1	18.6	>100

Metrics: WER/CER (lower is better).

Question Answering Results

Dataset	Subset	Metric	MiDashengLM	Qwen2.5-Omni-7B	Kimi-Audio-Instruct
MuChoMusic		ACC↑	71.35	64.79	67.40
MMAU	Sound	ACC↑	68.47	67.87	74.17
MMAU	Music	ACC↑	66.77	69.16	61.08
MMAU	Speech	ACC↑	63.66	59.76	57.66
MMAU	Average	ACC↑	66.30	65.60	64.30
MusicQA		FENSE↑	62.35	60.60	40.00
AudioCaps-QA		FENSE↑	54.31	53.28	47.34

Metrics: Higher is better.

Citation

MiDashengLM is under the Apache License 2.0, and we encourage its use in both research and business applications.

If you find MiDashengLM useful in your research, please consider citing our work:

bibtex

@techreport{midashenglm7b,
  title      = {MiDashengLM: Efficient Audio Understanding with General Audio Captions},
  author     = {{Horizon Team, MiLM Plus}},
  institution= {Xiaomi Inc.},
  year       = {2025},
  note       = {Contributors: Heinrich Dinkel et al. (listed alphabetically in Appendix B)},
  url        = {https://arxiv.org/abs/2508.03983},
  eprint     = {2508.03983},
}