Views
No views yet
pip install -r requirements.txtpip install -r requirements-mac.txttriton-windows to enable --compile usage, which gains speed up on V2 models:pip install triton-windows==3.2.0.post13| Version | Name | Purpose | Sampling Rate | Content Encoder | Vocoder | Hidden Dim | N Layers | Params | Remarks |
|---|---|---|---|---|---|---|---|---|---|
| v1.0 | seed-uvit-tat-xlsr-tiny (🤗📄) | Voice Conversion (VC) | 22050 | XLSR-large | HIFT | 384 | 9 | 25M | suitable for real-time voice conversion |
| v1.0 | seed-uvit-whisper-small-wavenet (🤗📄) | Voice Conversion (VC) | 22050 | Whisper-small | BigVGAN | 512 | 13 | 98M | suitable for offline voice conversion |
| v1.0 | seed-uvit-whisper-base (🤗📄) | Singing Voice Conversion (SVC) | 44100 | Whisper-small | BigVGAN | 768 | 17 | 200M | strong zero-shot performance, singing voice conversion |
| v2.0 | hubert-bsqvae-small (🤗📄) | Voice & Accent Conversion (VC) | 22050 | ASTRAL-Quantization | BigVGAN | 512 | 13 | 67M(CFM) + 90M(AR) | Best in suppressing source speaker traits |
HF_ENDPOINT=https://hf-mirror.com before every command.1python inference.py --source <source-wav>
2--target <referene-wav>
3--output <output-dir>
4--diffusion-steps 25 # recommended 30~50 for singingvoice conversion
5--length-adjust 1.0
6--inference-cfg-rate 0.7
7--f0-condition False # set to True for singing voice conversion
8--auto-f0-adjust False # set to True to auto adjust source pitch to target pitch level, normally not used in singing voice conversion
9--semi-tone-shift 0 # pitch shift in semitones for singing voice conversion
10--checkpoint <path-to-checkpoint>
11--config <path-to-config>
12 --fp16 Truesource is the path to the speech file to convert to reference voicetarget is the path to the speech file as voice referenceoutput is the path to the output directorydiffusion-steps is the number of diffusion steps to use, default is 25, use 30-50 for best quality, use 4-10 for fastest inferencelength-adjust is the length adjustment factor, default is 1.0, set <1.0 for speed-up speech, >1.0 for slow-down speechinference-cfg-rate has subtle difference in the output, default is 0.7f0-condition is the flag to condition the pitch of the output to the pitch of the source audio, default is False, set to True for singing voice conversionauto-f0-adjust is the flag to auto adjust source pitch to target pitch level, default is False, normally not used in singing voice conversionsemi-tone-shift is the pitch shift in semitones for singing voice conversion, default is 0checkpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface.(seed-uvit-whisper-small-wavenet if f0-condition is False else seed-uvit-whisper-base)config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingfacefp16 is the flag to use float16 inference, default is True1python inference_v2.py --source <source-wav>
2--target <referene-wav>
3--output <output-dir>
4--diffusion-steps 25 # recommended 30~50 for singingvoice conversion
5--length-adjust 1.0 # same as V1
6--intelligibility-cfg-rate 0.7 # controls how clear the output linguistic content is, recommended 0.0~1.0
7--similarity-cfg-rate 0.7 # controls how similar the output voice is to the reference voice, recommended 0.0~1.0
8--convert-style true # whether to use AR model for accent & emotion conversion, set to false will only conduct timbre conversion similar to V1
9--anonymization-only false # set to true will ignore reference audio but only anonymize source speech to an "average voice"
10--top-p 0.9 # controls the diversity of the AR model output, recommended 0.5~1.0
11--temperature 1.0 # controls the randomness of the AR model output, recommended 0.7~1.2
12--repetition-penalty 1.0 # penalizes the repetition of the AR model output, recommended 1.0~1.5
13--cfm-checkpoint-path <path-to-cfm-checkpoint> # path to the checkpoint of the CFM model, leave to blank to auto-download default model from huggingface
14--ar-checkpoint-path <path-to-ar-checkpoint> # path to the checkpoint of the AR model, leave to blank to auto-download default model from huggingfacepython app_vc.py --checkpoint <path-to-checkpoint> --config <path-to-config> --fp16 Truecheckpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface. (seed-uvit-whisper-small-wavenet)config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingfacehttp://localhost:7860/ to use the web interface.python app_svc.py --checkpoint <path-to-checkpoint> --config <path-to-config> --fp16 Truecheckpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface. (seed-uvit-whisper-base)config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingfacepython app_vc_v2.py --cfm-checkpoint-path <path-to-cfm-checkpoint> --ar-checkpoint-path <path-to-ar-checkpoint>cfm-checkpoint-path is the path to the checkpoint of the CFM model, leave to blank to auto-download default model from huggingfacear-checkpoint-path is the path to the checkpoint of the AR model, leave to blank to auto-download default model from huggingface--compile to gain ~x6 speed-up on AR model inferencepython app.py --enable-v1 --enable-v2app_vc.py or app_svc.py as above.--enable-v2 or --enable-v1 to only load one of the model sets.python real-time-gui.py --checkpoint-path <path-to-checkpoint> --config-path <path-to-config>checkpoint is the path to the model checkpoint if you have trained or fine-tuned your own model, leave to blank to auto-download default model from huggingface. (seed-uvit-tat-xlsr-tiny)config is the path to the model config if you have trained or fine-tuned your own model, leave to blank to auto-download default config from huggingface[!IMPORTANT] It is strongly recommended to use a GPU for real-time voice conversion. Some performance testing has been done on a NVIDIA RTX 3060 Laptop GPU, results and recommended parameter settings are listed below:
| Model Configuration | Diffusion Steps | Inference CFG Rate | Max Prompt Length | Block Time (s) | Crossfade Length (s) | Extra context (left) (s) | Extra context (right) (s) | Latency (ms) | Inference Time per Chunk (ms) |
|---|---|---|---|---|---|---|---|---|---|
| seed-uvit-xlsr-tiny | 10 | 0.7 | 3.0 | 0.18s | 0.04s | 2.5s | 0.02s | 430ms | 150ms |
Diffusion Steps is the number of diffusion steps to use, in real-time case usually set to 4~10 for fastest inference;Inference CFG Rate has subtle difference in the output, default is 0.7, set to 0.0 gains about 1.5x speed-up;Max Prompt Length is the maximum length of the prompt audio, setting to a low value can speed up inference, but may reduce similarity to prompt speech;Block Time is the time length of each audio chunk for inference, the higher the value, the higher the latency, note this value must be greater than the inference time per block, set according to your hardware condition;Crossfade Length is the time length of crossfade between audio chunks, normally not needed to change;Extra context (left) is the time length of extra history context for inference, the higher the value, the higher the inference time, but can increase stability;Extra context (right) is the time length of extra future context for inference, the higher the value, the higher the inference time and latency, but can increase stability;Block Time * 2 + Extra context (right), device side delay is usually of ~100ms. The overall delay is the sum of the two.
.wav .flac .mp3 .m4a .opus .oggconfigs/presets/ for fine-tuning, or create your own to train from scratch.
./configs/presets/config_dit_mel_seed_uvit_xlsr_tiny.yml for real-time voice conversion./configs/presets/config_dit_mel_seed_uvit_whisper_small_wavenet.yml for offline voice conversion./configs/presets/config_dit_mel_seed_uvit_whisper_base_f0_44k.yml for singing voice conversion1python train.py
2--config <path-to-config>
3--dataset-dir <path-to-data>
4--run-name <run-name>
5--batch-size 2
6--max-steps 1000
7--max-epochs 1000
8--save-every 500
9--num-workers 0config is the path to the model config, choose one of the above for fine-tuning or create your own for training from scratchdataset-dir is the path to the dataset directory, which should be a folder containing all the audio filesrun-name is the name of the run, which will be used to save the model checkpoints and logsbatch-size is the batch size for training, choose depends on your GPU memory.max-steps is the maximum number of steps to train, choose depends on your dataset size and training timemax-epochs is the maximum number of epochs to train, choose depends on your dataset size and training timesave-every is the number of steps to save the model checkpointnum-workers is the number of workers for data loading, set to 0 for Windows1accelerate launch train_v2.py
2--dataset-dir <path-to-data>
3--run-name <run-name>
4--batch-size 2
5--max-steps 1000
6--max-epochs 1000
7--save-every 500
8--num-workers 0
9--train-cfmrun-name and config arguments are the same so that latest checkpoint can be found)./runs/<run-name>/, with the checkpoint named ft_model.pth and config file with the same name as the training config file.
real-time-gui.py might raise an error ModuleNotFoundError: No module named '_tkinter', in this case a new Python version with Tkinter support should be installed. Refer to This Guide on stack overflow for explanation of the problem and a detailed fix.