This discrete vocoder consists of both analysis and synthesis components.
The vocoder operates with 24 kHz audio at a frame rate of 50.
It is designed as a middle ground between the high bitrate of EnCodec
and the lower bitrate alternatives like Mimi (12.5 frames per second) or WaveTokenizer (which uses a single codebook).
1import torch
2import soundfile as sf
3from huggingface_hub import hf_hub_download
4
5device = torch.device('cuda')
6
7# load the model
8encoder_path = hf_hub_download(repo_id="balacoon/vq4_50fps_24khz_vocoder", filename="analysis.jit")
9decoder_path = hf_hub_download(repo_id="balacoon/vq4_50fps_24khz_vocoder", filename="synthesis.jit")
10encoder = torch.jit.load(encoder_path)
11decoder = torch.jit.load(decoder_path)
12
13# read the audio
14orig_audio_npy, sr = sf.read(path, dtype="int16")
15assert sr == 24000
16orig_audio = torch.tensor(orig_audio_npy).to(device).unsqueeze(0) # batch x samples
17# extract audio tokens from the audio
18tokens = encoder(orig_audio) # batch x frames x 4
19# synthesize audio from audio tokens
20resynthesized_audio = decoder(tokens) # batch x samples
See performance of the codec on
vocoder leaderboard:
TTSLeaderboard
