Irodori-TTS-500M-v2-VoiceDesign is a Japanese Text-to-Speech model based on a Rectified Flow Diffusion Transformer (RF-DiT) architecture. Derived from the base v2 model, this variant replaces the reference latent encoder with a caption encoder.
Instead of requiring reference audio for voice cloning, this model features Voice Design—allowing you to fully control and generate the speaker's voice, emotion, and speaking style solely through a descriptive text prompt (caption).
Additionally, the model retains the emoji-based style and sound effect control — by inserting specific emojis into the input text, you can further fine-tune speaking styles, emotions, and sound effects during generation.
🌟 Key Features
Flow Matching TTS: Rectified Flow Diffusion Transformer over continuous DACVAE latents for high-quality Japanese speech synthesis.
Voice Design (Caption Conditioning): Generate diverse voices by describing the speaker's tone, age, gender, and emotion in a text caption. No reference audio is required.
Emoji-based Style Control: Embed emojis directly in the input text for granular control over the delivery. See EMOJI_ANNOTATIONS.md for the full list of supported emojis.
🏗️ Architecture
The model (approximately 500M parameters) consists of three main components:
Text Encoder: Token embeddings initialized from llm-jp/llm-jp-3-150m, followed by self-attention + SwiGLU transformer layers with RoPE.
Caption Encoder: Encodes the style-control text (captions) to define the speaker and acoustic environment, bypassing the need for a reference audio branch.
Diffusion Transformer: Joint-attention DiT blocks with Low-Rank AdaLN (timestep-conditioned adaptive layer normalization), half-RoPE, and SwiGLU MLPs.
Audio is represented as continuous latent sequences via the Aratako/Semantic-DACVAE-Japanese-32dim codec (32-dim), enabling high-quality 48kHz waveform reconstruction.
🎧 Audio Samples
Voice Design via Captions
Examples of controlling the speaker's voice, emotion, and style purely through descriptive text captions. Notice how the same input text can be delivered in entirely different ways.
You can also combine Voice Design captions with emoji annotations embedded in the text. This allows for even finer control over the delivery, adding specific emotional nuances, pauses, or sound effects on top of the base voice design.
The model was trained on a high-quality Japanese speech dataset. To enable the Voice Design functionality, the training data was enriched with comprehensive text captions describing the audio characteristics.
The emoji annotations and initial text captions were generated and labeled using a fine-tuned model based on Qwen/Qwen3-Omni-30B-A3B-Instruct. Subsequently, the text captions were rephrased and refined using Qwen/Qwen3.5-35B-A3B.
⚠️ Limitations
Japanese Only: This model currently supports Japanese text input only.
Prompt Adherence: While the model generally follows the caption's instructions, highly complex or contradictory descriptions might result in inconsistent voice generation.
Emoji Control: While emoji-based style control adds expressiveness, the effect may vary depending on context and is not always perfectly consistent.
Kanji Reading Accuracy: The model's ability to accurately read Kanji is relatively weak compared to other TTS models of a similar size. You may need to convert complex Kanji into Hiragana or Katakana beforehand.
In addition to the license terms, the following ethical restrictions apply:
No Impersonation: Do not use this model's captioning capabilities to intentionally generate voices that impersonate specific real-world individuals (e.g., voice actors, celebrities, public figures) without their explicit consent.
No Misinformation: Do not use this model to generate synthetic speech intended to mislead others or spread misinformation.
Voice Generation Disclaimer: The caption data used to train this model does not include specific speaker names; it only contains descriptive attributes such as gender, pitch, tone, and speaking speed. While it is possible that the generated audio may coincidentally resemble the voice of a real person, this is strictly a probabilistic artifact within the latent space. The model was not trained with the intent of reproducing specific individuals.
Liability Disclaimer: The developers assume no liability for any misuse of this model. Users are solely responsible for ensuring their use of the generated content complies with applicable laws and regulations in their jurisdiction.
🙏 Acknowledgments
This project builds upon the following works:
Echo-TTS — Architecture and training design reference
