Bark-voice-cloning

The models in this repo

• quantifier_hubert_base_ls960.pth (the model trained on literature for 4 epochs)
• quantifier_hubert_base_ls960_14.pth (the model trained on literature for 10 more epochs, based on the previous)
• quantifier_V1_hubert_base_ls960_23.pth (a larger model, trained for more epochs on the same dataset)

Voice cloning

1. Load your wav audio file into your pytorch application
2. For the fine prompt extract discrete representations. (These are used by bark to know about the voice), make sure to .squeeze() the resulting codes.
3. For the coarse prompt do fine_prompt[:2, :], to get the coarse prompt from a fine prompt.
4. For the semantics, load a HuBERT model without Kmeans (I personally use the audiolm-pytorch implementation's hubertwithkmeans, but i edited it to skip kmeans.)
5. Next, to get the actual semantic tokens, run the tokens through this model. Your output will be compatible with bark.
6. Save these files in an npz with numpy.savez(semantic_prompt=semantics, fine_prompt=fine, coarse_prompt=coarse). This is your speaker file containing your cloned voice.

Voice masking

Random

1. Extract semantics from the audio clip using HuBERT and this model
2. Run semantic_to_waveform from bark.api with the extracted semantics
3. The previous step returns the generated audio.

Transfer

1. Create a speaker file using the steps under the voice cloning section
2. Extract the semantics from the clip with the text you want spoken
3. Run semantics_to_waveform from bark.api with the extracted semantics, and the speaker prompt that you created on step 1.
4. The previous step returns the generated audio.

Disclaimer