Seed-VC

Evaluation📊

Installation📥

Usage🛠️

• source is the path to the speech file to convert to reference voice
• target is the path to the speech file as voice reference
• output is the path to the output directory
• diffusion-steps is the number of diffusion steps to use, default is 25, use 30-50 for best quality, use 4-10 for fastest inference
• length-adjust is the length adjustment factor, default is 1.0, set <1.0 for speed-up speech, >1.0 for slow-down speech
• inference-cfg-rate has subtle difference in the output, default is 0.7
• f0-condition is the flag to condition the pitch of the output to the pitch of the source audio, default is False, set to True for singing voice conversion
• auto-f0-adjust is the flag to auto adjust source pitch to target pitch level, default is False, normally not used in singing voice conversion
• semi-tone-shift is the pitch shift in semitones for singing voice conversion, default is 0
• checkpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface.(seed-uvit-whisper-small-wavenet if f0-condition is False else seed-uvit-whisper-base)
• config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingface
• fp16 is the flag to use float16 inference, default is True

• checkpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface. (seed-uvit-whisper-small-wavenet)
• config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingface

• checkpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface. (seed-uvit-whisper-base)
• config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingface

• checkpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface. (seed-uvit-tat-xlsr-tiny)
• config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingface

[!IMPORTANT] It is strongly recommended to use a GPU for real-time voice conversion. Some performance testing has been done on a NVIDIA RTX 3060 Laptop GPU, results and recommended parameter settings are listed below:

• Diffusion Steps is the number of diffusion steps to use, in real-time case usually set to 4~10 for fastest inference;
• Inference CFG Rate has subtle difference in the output, default is 0.7, set to 0.0 gains about 1.5x speed-up;
• Max Prompt Length is the maximum length of the prompt audio, setting to a low value can speed up inference, but may reduce similarity to prompt speech;
• Block Time is the time length of each audio chunk for inference, the higher the value, the higher the latency, note this value must be greater than the inference time per block, set according to your hardware condition;
• Crossfade Length is the time length of crossfade between audio chunks, normally not needed to change;
• Extra context (left) is the time length of extra history context for inference, the higher the value, the higher the inference time, but can increase stability;
• Extra context (right) is the time length of extra future context for inference, the higher the value, the higher the inference time and latency, but can increase stability;

Training🏋️

Fine-tuning on custom data allow the model to clone someone's voice more accurately. It will largely improve speaker similarity on particular speakers, but may slightly increase WER.
A Colab Tutorial is here for you to follow:

1. Prepare your own dataset. It has to satisfy the following:
   • File structure does not matter
   • Each audio file should range from 1 to 30 seconds, otherwise will be ignored
   • All audio files should be in on of the following formats: .wav .flac .mp3 .m4a .opus .ogg
   • Speaker label is not required, but make sure that each speaker has at least 1 utterance
   • Of course, the more data you have, the better the model will perform
   • Training data should be as clean as possible, BGM or noise is not desired
2. Choose a model configuration file from configs/presets/ for fine-tuning, or create your own to train from scratch.
   • For fine-tuning, it should be one of the following:
     • ./configs/presets/config_dit_mel_seed_uvit_xlsr_tiny.yml for real-time voice conversion
     • ./configs/presets/config_dit_mel_seed_uvit_whisper_small_wavenet.yml for offline voice conversion
     • ./configs/presets/config_dit_mel_seed_uvit_whisper_base_f0_44k.yml for singing voice conversion
3. Run the following command to start training:

• config is the path to the model config, choose one of the above for fine-tuning or create your own for training from scratch
• dataset-dir is the path to the dataset directory, which should be a folder containing all the audio files
• run-name is the name of the run, which will be used to save the model checkpoints and logs
• batch-size is the batch size for training, choose depends on your GPU memory.
• max-steps is the maximum number of steps to train, choose depends on your dataset size and training time
• max-epochs is the maximum number of epochs to train, choose depends on your dataset size and training time
• save-every is the number of steps to save the model checkpoint
• num-workers is the number of workers for data loading, set to 0 for Windows

1. If training accidentially stops, you can resume training by running the same command again, the training will continue from the last checkpoint. (Make sure run-name and config arguments are the same so that latest checkpoint can be found)
2. After training, you can use the trained model for inference by specifying the path to the checkpoint and config file.
   • They should be under ./runs/<run-name>/, with the checkpoint named ft_model.pth and config file with the same name as the training config file.
   • You still have to specify a reference audio file of the speaker you'd like to use during inference, similar to zero-shot usage.

TODO📝

• Release code
• Release pretrained models:

  

• Huggingface space demo:

  

• HTML demo page: Demo
• Streaming inference
• Reduce streaming inference latency
• Demo video for real-time voice conversion
• Singing voice conversion
• Noise resiliency for source audio
• Potential architecture improvements
  • U-ViT style skip connections
  • Changed input to OpenAI Whisper
  • Time as Token
• Code for training on custom data
• Few-shot/One-shot speaker fine-tuning
• Changed to BigVGAN from NVIDIA for singing voice decoding
• Whisper version model for singing voice conversion
• Objective evaluation and comparison with RVC/SoVITS for singing voice conversion
• Improve audio quality
• NSF vocoder for better singing voice conversion
• Fix real-time voice conversion artifact while not talking (done by adding a VAD model)
• Colab Notebook for fine-tuning example
• Replace whisper with more advanced linguistic content extractor
• More to be added
• Add Apple Silicon support

Known Issues

• On Mac - running real-time-gui.py might raise an error ModuleNotFoundError: No module named '_tkinter', in this case a new Python version with Tkinter support should be installed. Refer to This Guide on stack overflow for explanation of the problem and a detailed fix.

CHANGELOGS🗒️

• 2025-03-03:
  • Added Mac M Series (Apple Silicon) support
• 2024-11-26:
  • Updated v1.0 tiny version pretrained model, optimized for real-time voice conversion
  • Support one-shot/few-shot single/multi speaker fine-tuning
  • Support using custom checkpoint for webUI & real-time GUI
• 2024-11-19:
  • arXiv paper released
• 2024-10-28:
  • Updated fine-tuned 44k singing voice conversion model with better audio quality
• 2024-10-27:
  • Added real-time voice conversion GUI
• 2024-10-25:
  • Added exhaustive evaluation results and comparisons with RVCv2 for singing voice conversion
• 2024-10-24:
  • Updated 44kHz singing voice conversion model, with OpenAI Whisper as speech content input
• 2024-10-07:
  • Updated v0.3 pretrained model, changed speech content encoder to OpenAI Whisper
  • Added objective evaluation results for v0.3 pretrained model
• 2024-09-22:
  • Updated singing voice conversion model to use BigVGAN from NVIDIA, providing large improvement to high-pitched singing voices
  • Support chunking and streaming output for long audio files in Web UI
• 2024-09-18:
  • Updated f0 conditioned model for singing voice conversion
• 2024-09-14:
  • Updated v0.2 pretrained model, with smaller size and less diffusion steps to achieve same quality, and additional ability to control prosody preservation
  • Added command line inference script
  • Added installation and usage instructions