Views
No views yet
| Model | HuggingFace | Text Encoder | Language |
|---|---|---|---|
| Dasheng-AudioGen | mispeech/Dasheng-AudioGen | google/flan-t5-large | English |
| Dasheng-AudioGen-Multilingual | mispeech/Dasheng-AudioGen-Multilingual | google/mt5-large | Multilingual |
| Language | Duration (h) | Proportion |
|---|---|---|
| English | 15,367.80 | 58.86% |
| Spanish | 2,740.96 | 10.50% |
| Portuguese | 1,916.24 | 7.34% |
| Russian | 1,217.39 | 4.66% |
| French | 933.91 | 3.58% |
| Japanese | 874.51 | 3.35% |
| Korean | 848.15 | 3.25% |
| German | 842.29 | 3.23% |
| Other | 1,369.16 | 5.24% |
Note: The current multilingual model has notably higher synthesis error rates for all non-English languages. Languages outside the table above are even less reliable. For English-only use cases, the base model (mispeech/Dasheng-AudioGen) is recommended.
pip install torch torchaudio "transformers<5" einopsTested with Python 3.10, torch 2.8.0+cu128, transformers 4.57. Not compatible with transformers 5.x.
<|caption|> tag, which provides the overall scene description. Other tags are optional and can be included as needed.| Tag | Description | Required |
|---|---|---|
<|caption|> | Overall audio scene description | Yes |
<|speech|> | Speaker identity and speaking style | No |
<|asr|> | Spoken transcript / dialogue | No |
<|sfx|> | Sound effects | No |
<|music|> | Background music | No |
<|env|> | Environmental ambience | No |
<|caption|> — prompts without it will be rejected.<|music|> if there is no music).Multilingual prompt convention: All descriptive tags (caption,speech,sfx,music,env) should be written in English. Only the<|asr|>field (the actual spoken content to be synthesized) should use the target language.
caption field is required; all other fields are optional.1import torchaudio
2from transformers import AutoModel
3
4model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen-Multilingual", trust_remote_code=True).cuda()
5
6prompt = model.compose_prompt(
7 caption="A conversation scene on a busy city street.",
8 speech="A young woman speaking softly in Spanish.",
9 env="Rain and distant traffic noise.",
10 asr="Creo que deberíamos irnos ya.",
11)
12audio = model.generate(prompt)
13torchaudio.save("output.wav", audio.cpu(), 16000)prompt parameter. The string must start with <|caption|>.1import torchaudio
2from transformers import AutoModel
3
4model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen-Multilingual", trust_remote_code=True).cuda()
5
6prompt = model.compose_prompt(
7 prompt="<|caption|> A conversation scene on a busy city street. <|speech|> A young woman speaking softly in Spanish. <|asr|> Creo que deberíamos irnos ya. <|env|> Rain and distant traffic noise."
8)
9audio = model.generate(prompt)
10torchaudio.save("output.wav", audio.cpu(), 16000)1import torchaudio
2from transformers import AutoModel
3
4model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen-Multilingual", trust_remote_code=True).cuda()
5
6prompts = [
7 model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
8 model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
9 model.compose_prompt(caption="A piano playing a gentle melody.", music="Soft piano ballad."),
10]
11audios = model.generate(prompts)
12
13for i, audio in enumerate(audios):
14 torchaudio.save(f"output_{i}.wav", audio.unsqueeze(0).cpu(), 16000)1import torchaudio
2from transformers import AutoModel
3
4model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen-Multilingual", trust_remote_code=True).cuda()
5
6prompt = model.compose_prompt(caption="A dog barking in a park")
7audio = model.generate(
8 prompts=prompt,
9 num_steps=25, # number of denoising steps (default: 25)
10 guidance_scale=5.0, # classifier-free guidance scale (default: 5.0)
11 sway_sampling_coef=-1.0, # sway sampling coefficient (default: -1.0, 0 for linear)
12)
13torchaudio.save("output.wav", audio.cpu(), 16000)1@article{mei2026dashengaudiogen,
2 title = {Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text},
3 author = {Jiahao Mei and Heinrich Dinkel and Yadong Niu and Xingwei Sun and Gang Li and Yifan Liao and Jiahao Zhou and Junbo Zhang and Jian Luan and Mengyue Wu},
4 journal = {arXiv preprint arXiv:2605.27838},
5 year = {2026}
6}