Views
No views yet
XY-Tokenizer is a speech codec that simultaneously models both semantic and acoustic aspects of speech, converting audio into discrete tokens and decoding them back to high-quality audio. It achieves efficient speech representation at only 1kbps with RVQ8 quantization at 12.5Hz frame rate.XY-Tokenizer serves as the underlying neural codec for MOSS-TTSD, our 1.7B Audio Language Model. MOSS-TTSD for advanced text-to-speech and other audio generation tasks on GitHub, Blog, 博客, and Space Demo.XY-Tokenizer with transformers to encode an audio file into discrete tokens and decode it back into a waveform.1import torchaudio
2from transformers import AutoFeatureExtractor, AutoModel
3
4# 1. Load the feature extractor and the codec model
5model_id = "OpenMOSS-Team/XY_Tokenizer_TTSD_V0_hf"
6feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
7codec = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval().to("cuda")
8
9# 2. Load and preprocess the audio
10# The model expects a 16kHz sample rate.
11wav_form, sampling_rate = torchaudio.load("examples/m1.wav")
12if sampling_rate != 16000:
13 wav_form = torchaudio.functional.resample(wav_form, orig_freq=sampling_rate, new_freq=16000)
14
15# 3. Encode the audio into discrete codes
16input_features = feature_extractor(wav_form, sampling_rate=16000, return_attention_mask=True, return_tensors="pt")
17# The 'code' dictionary contains the discrete audio codes
18code = codec.encode(input_features)
19
20# 4. Decode the codes back to an audio waveform
21# The output is high-quality 24kHz audio.
22output_wav = codec.decode(code["audio_codes"], overlap_seconds=10)
23
24# 5. Save the reconstructed audio
25for i, audio in enumerate(output_wav["audio_values"]):
26 torchaudio.save(f"audio_{i}.wav", audio.cpu(), 24000)