Irodori-TTS-500M is a Japanese Text-to-Speech model based on a Rectified Flow Diffusion Transformer (RF-DiT) architecture. The architecture and training design largely follow Echo-TTS, using DACVAE continuous latents as the generation target. It supports zero-shot voice cloning from reference audio.
A unique feature of this model is emoji-based style and sound effect control — by inserting specific emojis into the input text, you can control speaking styles, emotions, and even sound effects in the generated audio.
🌟 Key Features
Flow Matching TTS: Rectified Flow Diffusion Transformer over continuous DACVAE latents for high-quality Japanese speech synthesis.
Voice Cloning: Zero-shot voice cloning from a short reference audio clip.
Emoji-based Style Control: Control speaking styles, emotions, and sound effects by embedding emojis directly in the input text. See EMOJI_ANNOTATIONS.md for the full list of supported emojis and their effects.
🏗️ Architecture
The model (approximately 500M parameters) consists of three main components:
Text Encoder: Token embeddings initialized from llm-jp/llm-jp-3-150m, followed by self-attention + SwiGLU transformer layers with RoPE.
Reference Latent Encoder: Encodes patched reference audio latents for speaker/style conditioning via self-attention + SwiGLU layers.
Diffusion Transformer: Joint-attention DiT blocks with Low-Rank AdaLN (timestep-conditioned adaptive layer normalization), half-RoPE, and SwiGLU MLPs.
Audio is represented as continuous latent sequences via the DACVAE codec (128-dim), enabling high-quality 48kHz waveform reconstruction.
🎧 Audio Samples
1. Standard TTS
Basic Japanese text-to-speech generation (without reference audio).
The model was trained on a high-quality Japanese speech dataset. To enable the emoji-based style control, the training texts were enriched with emoji annotations. These annotations were automatically generated and labeled using a fine-tuned model based on Qwen/Qwen3-Omni-30B-A3B-Instruct.
⚠️ Limitations
Japanese Only: This model currently supports Japanese text input only.
Emoji Control: While emoji-based style control adds expressiveness, the effect may vary depending on context and is not always perfectly consistent.
Audio Quality: Quality depends on training data characteristics. Performance may vary for voices or speaking styles underrepresented in the training data.
Kanji Reading Accuracy: The model's ability to accurately read Kanji is relatively weak compared to other TTS models of a similar size. You may need to convert complex Kanji into Hiragana or Katakana beforehand.
In addition to the license terms, the following ethical restrictions apply:
No Impersonation: Do not use this model to clone or impersonate the voice of any individual (e.g., voice actors, celebrities, public figures) without their explicit consent.
No Misinformation: Do not use this model to generate deepfakes or synthetic speech intended to mislead others or spread misinformation.
Disclaimer: The developers assume no liability for any misuse of this model. Users are solely responsible for ensuring their use of the generated content complies with applicable laws and regulations in their jurisdiction.
🙏 Acknowledgments
This project builds upon the following works:
Echo-TTS — Architecture and training design reference
