LLaMA-Omni is a speech-language model built upon Llama-3.1-8B-Instruct. It supports low-latency and high-quality speech interactions, simultaneously generating both text and speech responses based on speech instructions.
💡 Highlights
💪 Built on Llama-3.1-8B-Instruct, ensuring high-quality responses.
🚀 Low-latency speech interaction with a latency as low as 226ms.
🎧 Simultaneous generation of both text and speech responses.
Note: Due to the instability of streaming audio playback in Gradio, we have only implemented streaming audio synthesis without enabling autoplay. If you have a good solution, feel free to submit a PR. Thanks!
Local Inference
To run inference locally, please organize the speech instruction files according to the format in the omni_speech/infer/examples directory, then refer to the following script.
Our code is released under the Apache-2.0 License. Our model is intended for academic research purposes only and may NOT be used for commercial purposes.
You are free to use, modify, and distribute this model in academic settings, provided that the following conditions are met:
Non-commercial use: The model may not be used for any commercial purposes.
Citation: If you use this model in your research, please cite the original work.
Commercial Use Restriction
For any commercial use inquiries or to obtain a commercial license, please contact fengyang@ict.ac.cn.
SLAM-LLM: We borrow some code about speech encoder and speech adaptor.
Citation
If you have any questions, please feel free to submit an issue or contact fangqingkai21b@ict.ac.cn.
If our work is useful for you, please cite as:
@article{fang-etal-2024-llama-omni,
title={LLaMA-Omni: Seamless Speech Interaction with Large Language Models},
author={Fang, Qingkai and Guo, Shoutao and Zhou, Yan and Ma, Zhengrui and Zhang, Shaolei and Feng, Yang},
journal={arXiv preprint arXiv:2409.06666},
year={2024}
}
