Auffusion is a latent diffusion model (LDM) for text-to-audio (TTA) generation. Auffusion can generate realistic audios including human sounds, animal sounds, natural and artificial sounds and sound effects from textual prompts. We introduce Auffusion, a TTA system adapting T2I model frameworks to TTA task, by effectively leveraging their inherent generative strengths and precise cross-modal alignment. Our objective and subjective evaluations demonstrate that Auffusion surpasses previous TTA approaches using limited data and computational resource. We release our model, inference code, and pre-trained checkpoints for the research community.

📣 We are releasing Auffusion-Full-no-adapter which was pre-trained on all datasets described in paper and created for easy use of audio manipulation.

📣 We are releasing Auffusion-Full which was pre-trained on all datasets described in paper.

📣 We are releasing Auffusion which was pre-trained on AudioCaps.

Our code is released here: https://github.com/happylittlecat2333/Auffusion

We uploaded several Auffusion generated samples here: https://auffusion.github.io

Please follow the instructions in the repository for installation, usage and experiments.

First, git clone the repository and install the requirements:

Download the Auffusion model and generate audio from a text prompt:

The auffusion model will be automatically downloaded from huggingface and saved in cache. Subsequent runs will load the model directly from cache.

The generate function uses 100 steps and 7.5 guidance_scale by default to sample from the latent diffusion model. You can also vary parameters for different results.

Please consider citing the following article if you found our work useful: