MGM-Omni is an omni-chatbot capable of processing text, image, video, and speech inputs, and generating both text and speech responses.
MGM-Omni is capable of long-form speech understanding and generation, as well as zero-shot voice cloning in both Chinese and English.
MGM-Omni-TTS-4B is the SpeechLM component of MGM-Omni for speech generation. For the MLLM part, please refer MGM-Omni.
Main Properties
Omni-modality supports: MGM-Omni supports audio, video, image, and text inputs, understands long contexts, and can generate both text and speech outputs, making it a truly versatile multi-modal AI assistant.
Long-form Speech Understanding: Unlike most existing open-source multi-modal models, which typically fail with inputs longer than 15 minutes, MGM-Omni can handle hour-long speech inputs while delivering superior overall and detailed understanding and performance!
Long-form Speech Generation: With a treasure trove of training data and smart Chunk-Based Decoding, MGM-Omni can generate over 10 minutes of smooth, natural speech for continuous storytelling.
Streaming Generation: Thanks to the parallel decoding approach for speech tokens, MGM-Omni enables efficient and smooth streaming audio, making it suitable for live conversations.
Zero-shot Voice Cloning: With MGM-Omni’s extensive and diverse audio training, you can create a customized voice clone by simply recording a short clip (around 10 seconds) and reviewing the results.
Fully Open-source: All the code, models, and training data will be released.
Evaluation
Speech and Audio Understanding
Model
Date
LS-clean↓
LS-other↓
CM-EN↓
CM-ZH↓
AISHELL↓
Mini-Omni2
2024-11
4.7
9.4
-
-
-
Lyra
2024-12
2.0
4.0
-
-
-
VITA-1.5
2025-01
3.4
7.5
-
-
2.2
Qwen2.5-Omni
2025-03
1.6
3.5
7.6
5.2
-
Ola
2025-06
1.9
4.3
-
-
-
MGM-Omni-7B
2025-08
1.7
3.6
8.8
4.5
1.9
MGM-Omni-32B
2025-08
1.5
3.2
8.0
4.0
1.8
This table presents WER and CER results on speech understanding.
Here LS refers to LibriSpeech and CM refers to Common Voice.
Model
Date
Speech↑
Sound↑
Music↑
Mix↑
Average↑
LLaMA-Omni
2024-08
5.2
5.3
4.3
4.0
4.7
Mini-Omni2
2024-11
3.6
3.5
2.6
3.1
3.2
IXC2.5-OmniLive
2024-12
1.6
1.8
1.7
1.6
1.7
VITA-1.5
2025-01
4.8
5.5
4.9
2.9
4.5
Qwen2.5-Omni
2025-03
6.8
5.7
4.8
5.4
5.7
Ola
2025-06
7.3
6.4
5.9
6.0
6.4
MGM-Omni-7B
2025-08
7.3
6.5
6.3
6.1
6.5
MGM-Omni-32B
2025-08
7.1
6.5
6.2
6.2
6.5
This table presents evaluation results on AIR-Bench Chat (speech, sound, music, etc.).
Speech Generation
Model
Date
Model Size
CER↓
SS(ZH)↑
WER↓
SS(EN)↑
CosyVoice2
2024-12
0.5B
1.45
0.748
2.57
0.652
Qwen2.5-Omni-3B
2025-03
0.5B
1.58
0.744
2.51
0.635
Qwen2.5-Omni-7B
2025-03
2B
1.42
0.754
2.33
0.641
MOSS-TTSD-v0
2025-06
2B
2.18
0.594
2.46
0.476
HiggsAudio-v2
2025-07
6B
1.66
0.743
2.44
0.677
MGM-Omni
2025-08
0.6B
1.49
0.749
2.54
0.670
MGM-Omni
2025-08
2B
1.38
0.753
2.28
0.682
MGM-Omni
2025-08
4B
1.34
0.756
2.22
0.684
This table presents evaluation results on speech generation on seed-tts-eval.
For Qwen2.5-Omni, model size refers to the size of the talker.
Citation
If you find this repo useful for your research, we would appreciate it if you could cite our work:
@misc{wang2025mgmomni,
title={MGM-Omni: An Open-source Omni Chatbot},
author={Wang, Chengyao and Zhong, Zhisheng and Peng, Bohao and Yang, Senqiao and Liu, Yuqi and Yu, Bei and Jia, Jiaya},
year={2025},
howpublished={\url{https://mgm-omni.notion.site}},
note={Notion Blog}
}
