Qwen-Audio (Qwen Large Audio Language Model) is the multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-Audio accepts diverse audio (human speech, natural sound, music and song) and text as inputs, outputs text. The contribution of Qwen-Audio include:
Fundamental audio models: Qwen-Audio is a fundamental multi-task audio-language model that supports various tasks, languages, and audio types, serving as a universal audio understanding model. Building upon Qwen-Audio, we develop Qwen-Audio-Chat through instruction fine-tuning, enabling multi-turn dialogues and supporting diverse audio-oriented scenarios.
Multi-task learning framework for all types of audios: To scale up audio-language pre-training, we address the challenge of variation in textual labels associated with different datasets by proposing a multi-task training framework, enabling knowledge sharing and avoiding one-to-many interference. Our model incorporates more than 30 tasks and extensive experiments show the model achieves strong performance.
Strong Performance: Experimental results show that Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Specifically, Qwen-Audio achieves state-of-the-art results on the test set of Aishell1, cochlscene, ClothoAQA, and VocalSound.
Flexible multi-run chat from audio and text input: Qwen-Audio supports multiple-audio analysis, sound understading and reasoning, music appreciation, and tool usage for speech editing.
Qwen-Audio 是阿里云研发的大规模音频语言模型（Large Audio Language Model）。Qwen-Audio 可以以多种音频 (包括说话人语音、自然音、音乐、歌声）和文本作为输入，并以文本作为输出。Qwen-Audio 系列模型的特点包括：
We release Qwen-Audio and Qwen-Audio-Chat, which are pretrained model and Chat model respectively. For more details about Qwen-Audio, please refer to our Github Repo. This repo is the one for Qwen-Audio-Chat.
pytorch 1.12 and above, 2.0 and above are recommended
CUDA 11.4 and above are recommended (this is for GPU users)
FFmpeg
Quickstart
Below, we provide simple examples to show how to use Qwen-Audio with 🤗 Transformers.
Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.
pip install -r requirements.txt
Now you can start with Transformers. For more usage, please refer to tutorial.
🤗 Transformers
To use Qwen-Audio for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, please make sure that you are using the latest code.
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from transformers.generation import GenerationConfig
3import torch
4torch.manual_seed(1234)56# Note: The default behavior now has injection attack prevention off.7tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-Audio-Chat", trust_remote_code=True)89# use bf1610# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-Audio-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval()11# use fp1612# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-Audio-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()13# use cpu only14# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-Audio-Chat", device_map="cpu", trust_remote_code=True).eval()15# use cuda device16model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-Audio-Chat", device_map="cuda", trust_remote_code=True).eval()1718# Specify hyperparameters for generation (No need to do this if you are using transformers>4.32.0)19# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-Audio-Chat", trust_remote_code=True)2021# 1st dialogue turn22query = tokenizer.from_list_format([23{'audio':'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Audio/1272-128104-0000.flac'},# Either a local path or an url24{'text':'what does the person say?'},25])26response, history = model.chat(tokenizer, query=query, history=None)27print(response)28# The person says: "mister quilter is the apostle of the middle classes and we are glad to welcome his gospel".2930# 2nd dialogue turn31response, history = model.chat(tokenizer,'Find the start time and end time of the word "middle classes"', history=history)32print(response)33# The word "middle classes" starts at <|2.33|> seconds and ends at <|3.26|> seconds.
License Agreement
Researchers and developers are free to use the codes and model weights of Qwen-Audio-Chat. We also allow its commercial use. Check our license at LICENSE for more details.
Citation
If you find our paper and code useful in your research, please consider giving a star and citation
BibTeX
1@article{Qwen-Audio,
2 title={Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models},
3 author={Chu, Yunfei and Xu, Jin and Zhou, Xiaohuan and Yang, Qian and Zhang, Shiliang and Yan, Zhijie and Zhou, Chang and Zhou, Jingren},
4 journal={arXiv preprint arXiv:2311.07919},
5 year={2023}
6}
Contact Us
If you are interested to leave a message to either our research team or product team, feel free to send an email to qianwen_opensource@alibabacloud.com.
