🦙🎧 LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis

Authors: Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, Yang Feng*

🔥 News

• [25/05] LLaMA-Omni 2 is accepted at ACL 2025 main conference!

Install

1. Clone this repository.

1. Install packages.

Quick Start

1. Download the Whisper-large-v3 model.

1. Download the flow-matching model and vocoder of CosyVoice 2.

[!Tip] If you’re experiencing unstable connections to Hugging Face from within China, you can try setting the following in your command line:

export HF_ENDPOINT=https://hf-mirror.com

1. Download the LLaMA-Omni2 series models from Hugging Face. LLaMA-Omni2-0.5B/1.5B/3B/7B/14B support English only, while LLaMA-Omni2-0.5B/1.5B/3B/7B/14B/32B-Bilingual support both English and Chinese.

Gradio Demo

1. Launch a controller.
   python -m llama_omni2.serve.controller --host 0.0.0.0 --port 10000
2. Launch a gradio web server.
   python -m llama_omni2.serve.gradio_web_server --controller http://localhost:10000 --port 8000 --vocoder-dir models/cosy2_decoder
3. Launch a model worker.
   python -m llama_omni2.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path models/$model_name --model-name $model_name
4. Visit http://localhost:8000/ and interact with LLaMA-Omni2!

Local Inference

LICENSE

• Non-commercial use: The model may not be used for any commercial purposes.
• Citation: If you use this model in your research, please cite the original work.

Commercial Use Restriction

Acknowledgements

• CosyVoice 2: We use the pretrained speech tokenizer, flow-matching model and vocoder of CosyVoice 2.
• SLAM-LLM: We borrow some code about speech encoder and speech adaptor.

Citation

@inproceedings{
  fang2025llamaomni2,
  title={{LL}a{MA}-{O}mni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis},
  author={Fang, Qingkai and Zhou, Yan and Guo, Shoutao and Zhang, Shaolei and Feng, Yang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics},
  year={2025}
}

@inproceedings{
  fang2025llamaomni,
  title={{LL}a{MA}-{O}mni: Seamless Speech Interaction with Large Language Models},
  author={Qingkai Fang and Shoutao Guo and Yan Zhou and Zhengrui Ma and Shaolei Zhang and Yang Feng},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=PYmrUQmMEw}
}