🎮 Release ControlNet training code & 🎤 Singing2Accompaniment controlnet
🏗️ Architecture
ACE-Step Framework
📝 Abstract
We introduce ACE-Step, a novel open-source foundation model for music generation that overcomes key limitations of existing approaches and achieves state-of-the-art performance through a holistic architectural design. Current methods face inherent trade-offs between generation speed, musical coherence, and controllability. For instance, LLM-based models (e.g., Yue, SongGen) excel at lyric alignment but suffer from slow inference and structural artifacts. Diffusion models (e.g., DiffRhythm), on the other hand, enable faster synthesis but often lack long-range structural coherence.
ACE-Step bridges this gap by integrating diffusion-based generation with Sana’s Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. It further leverages MERT and m-hubert to align semantic representations (REPA) during training, enabling rapid convergence. As a result, our model synthesizes up to 4 minutes of music in just 20 seconds on an A100 GPU—15× faster than LLM-based baselines—while achieving superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics. Moreover, ACE-Step preserves fine-grained acoustic details, enabling advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation (e.g., lyric2vocal, singing2accompaniment).
Rather than building yet another end-to-end text-to-music pipeline, our vision is to establish a foundation model for music AI: a fast, general-purpose, efficient yet flexible architecture that makes it easy to train sub-tasks on top of it. This paves the way for developing powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. In short, we aim to build the Stable Diffusion moment for music.
✨ Features
ACE-Step Framework
🎯 Baseline Quality
🌈 Diverse Styles & Genres
🎸 Supports all mainstream music styles with various description formats including short tags, descriptive text, or use-case scenarios
🎷 Capable of generating music across different genres with appropriate instrumentation and style
🌍 Multiple Languages
🗣️ Supports 19 languages with top 10 well-performing languages including:
⚠️ Due to data imbalance, less common languages may underperform
🎻 Instrumental Styles
🎹 Supports various instrumental music generation across different genres and styles
🎺 Capable of producing realistic instrumental tracks with appropriate timbre and expression for each instrument
🎼 Can generate complex arrangements with multiple instruments while maintaining musical coherence
🎤 Vocal Techniques
🎙️ Capable of rendering various vocal styles and techniques with good quality
🗣️ Supports different vocal expressions including various singing techniques and styles
🎛️ Controllability
🔄 Variations Generation
⚙️ Implemented using training-free, inference-time optimization techniques
🌊 Flow-matching model generates initial noise, then uses trigFlow's noise formula to add additional Gaussian noise
🎚️ Adjustable mixing ratio between original initial noise and new Gaussian noise to control variation degree
🎨 Repainting
🖌️ Implemented by adding noise to the target audio input and applying mask constraints during the ODE process
🔍 When input conditions change from the original generation, only specific aspects can be modified while preserving the rest
🔀 Can be combined with Variations Generation techniques to create localized variations in style, lyrics, or vocals
✏️ Lyric Editing
💡 Innovatively applies flow-edit technology to enable localized lyric modifications while preserving melody, vocals, and accompaniment
🔄 Works with both generated content and uploaded audio, greatly enhancing creative possibilities
ℹ️ Current limitation: can only modify small segments of lyrics at once to avoid distortion, but multiple edits can be applied sequentially
🚀 Applications
🎤 Lyric2Vocal (LoRA)
🔊 Based on a LoRA fine-tuned on pure vocal data, allowing direct generation of vocal samples from lyrics
🛠️ Offers numerous practical applications such as vocal demos, guide tracks, songwriting assistance, and vocal arrangement experimentation
⏱️ Provides a quick way to test how lyrics might sound when sung, helping songwriters iterate faster
📝 Text2Samples (LoRA)
🎛️ Similar to Lyric2Vocal, but fine-tuned on pure instrumental and sample data
🎵 Capable of generating conceptual music production samples from text descriptions
🧰 Useful for quickly creating instrument loops, sound effects, and musical elements for production
🔮 Coming Soon
🎤 RapMachine
🔥 Fine-tuned on pure rap data to create an AI system specialized in rap generation
🏆 Expected capabilities include AI rap battles and narrative expression through rap
📚 Rap has exceptional storytelling and expressive capabilities, offering extraordinary application potential
🎛️ StemGen
🎚️ A controlnet-lora trained on multi-track data to generate individual instrument stems
🎯 Takes a reference track and specified instrument (or instrument reference audio) as input
🎹 Outputs an instrument stem that complements the reference track, such as creating a piano accompaniment for a flute melody or adding jazz drums to a lead guitar
🎤 Singing2Accompaniment
🔄 The reverse process of StemGen, generating a mixed master track from a single vocal track
🎵 Takes a vocal track and specified style as input to produce a complete vocal accompaniment
🎸 Creates full instrumental backing that complements the input vocals, making it easy to add professional-sounding accompaniment to any vocal recording
We've tested ACE-Step on various hardware configurations with the following throughput results:
Device
27 Steps
60 Steps
NVIDIA A100
0.036675
0.0815
MacBook M2 Max
0.44
NVIDIA RTX 4090
0.029
0.064
seconds cost per generated audio (seconds/audio)
For example, to generate a 180-second song, multiply 180 by the seconds cost per generated audio (seconds/audio) for the desired device and step count. This will give you the total time required for the generation process.
ACE-Step enables original music generation across diverse genres, with applications in creative production, education, and entertainment. While designed to support positive and artistic use cases, we acknowledge potential risks such as unintentional copyright infringement due to stylistic similarity, inappropriate blending of cultural elements, and misuse for generating harmful content. To ensure responsible use, we encourage users to verify the originality of generated works, clearly disclose AI involvement, and obtain appropriate permissions when adapting protected styles or materials. By using ACE-Step, you agree to uphold these principles and respect artistic integrity, cultural diversity, and legal compliance. The authors are not responsible for any misuse of the model, including but not limited to copyright violations, cultural insensitivity, or the generation of harmful content.
🙏 Acknowledgements
This project is co-led by ACE Studio and StepFun.
📖 Citation
If you find this project useful for your research, please consider citing:
BibTeX
1@misc{gong2025acestep,
2 title={ACE-Step: A Step Towards Music Generation Foundation Model},
3 author={Junmin Gong, Wenxiao Zhao, Sen Wang, Shengyuan Xu, Jing Guo},
4 howpublished={\url{https://github.com/ace-step/ACE-Step}},
5 year={2025},
6 note={GitHub repository}
7}
