Anime-XCodec2-44.1kHz: A 44.1kHz Upsampling Variant of Anime-XCodec2

🔗 Quick Links

• Demo (Gradio / Hugging Face Spaces): https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-44.1kHz-Demo
• This repository (44.1kHz fine-tune): NandemoGHS/Anime-XCodec2-44.1kHz
• Baseline 16kHz model: NandemoGHS/Anime-XCodec2
• Original XCodec2: HKUSTAudio/xcodec2
• Reference Paper (Inworld TTS-1): https://arxiv.org/abs/2507.21138
• Reference Implementation (Inworld TTS): https://github.com/inworld-ai/tts

1) Model Summary

• What it is: A neural speech codec based on Anime-XCodec2 (which is based on XCodec2), fine-tuned to output 44.1kHz high-fidelity Japanese speech (anime/game-style).
• Key Change: Integrates an UpSamplerBlock and utilizes RMS Loss (inspired by Inworld TTS-1) into the decoder architecture.
• Training scope: Decoder-only fine-tuning on ~22,000 hours of Japanese data. Encoder and codebook are frozen.
• Compatibility: Speech tokens are identical to HKUSTAudio/xcodec2 and NandemoGHS/Anime-XCodec2.
• Input Sampling rate: 16 kHz (for encoding, same as XCodec2).
• Output Sampling rate: 44.1 kHz (decoded audio).

2) Intended Use

• Decode XCodec2 speech tokens (e.g., from Llasa or other AR generators) into high-fidelity 44.1kHz Japanese speech (anime/game-style).
• Upgrade existing Anime-XCodec2 (16kHz) pipelines to 44.1kHz output.
• Audio Super-Resolution: As the model accepts 16kHz input and outputs 44.1kHz reconstructed audio, it can also be used as a form of audio super-resolution. However, its performance for this specific purpose is untested/unevaluated.

3) How to Use (Important)

• Installation:
  bash

  1# Install the customi xcodec2 library 2pip install https://huggingface.co/NandemoGHS/Anime-XCodec2-44.1kHz/resolve/main/xcodec2-0.1.6.tar.gz
• Usage: Once the custom library is installed, you can load and use this model just as you would the original XCodec2 or Anime-XCodec2 models. The core inference logic remains the same.

4) Limitations & Trade-offs

• Language scope: Optimized for Japanese. Performance on other languages may degrade.
• Content domain: Tuned toward anime/game-style voices.
• Library Dependency: Requires the custom xcodec2 library linked above. It is not compatible with the original xcodec2 library.

5) Data (High-Level)

• ~22,000 hours of Japanese speech, with a focus on anime/game-style voices.
• Data was prepared for 44.1kHz target output during training.

6) Training Procedure (High-Level)

• Base Model: NandemoGHS/Anime-XCodec2 (16kHz)
• Architecture Modification:
  • Integrated the UpSamplerBlock from the Inworld TTS-1 implementation into the decoder.
  • Upsampler parameters: hop_length=147, upsample_factors=[3, 2], kernel_sizes=[7, 6].
• Loss Function: Adopted RMS Loss (Root Mean Square loss), as introduced in Inworld TTS-1, in addition to the original losses.
• Frozen: Encoder and Codebook (token compatibility preserved).
• Updated (fine-tuned): generator.backbone, generator.head, generator.upsampler, fc_post_a

7) Samples (A/B/C/D Listening)

8) License

• CC-BY-NC 4.0 (inherited from XCodec2 and Anime-XCodec2).
• See: https://creativecommons.org/licenses/by-nc/4.0/

9) Acknowledgements

• HKUSTAudio/xcodec2 (Original model)
• Inworld AI for their work on Inworld TTS-1 (Upsampler architecture and RMS Loss).
• Thanks to contributors and the community around Japanese speech resources.