Views
No views yet
1from neucodec import NeuCodec
2
3codec = NeuCodec.from_pretrained("neuphonic/neucodec")
4_ = codec.eval().to('cuda')multilingual-tts_audio_* speaker name, such as multilingual-tts_audio_Grace, as used in our TTS benchmark.1import re
2import soundfile as sf
3import torch
4import librosa
5from transformers import AutoTokenizer, AutoModelForCausalLM
6
7model = AutoModelForCausalLM.from_pretrained('Scicom-intl/Multilingual-Expressive-TTS-1.7B')
8tokenizer = AutoTokenizer.from_pretrained('Scicom-intl/Multilingual-Expressive-TTS-1.7B')
9
10speaker = 'DisfluencySpeech'
11text = "Hi nama saya Husein, I am so cute, 我喜欢吃鸡饭, boire du thé glacé, ולהירגע על החוף, وأحب أن أتعرض لبعض أشعة الشمس."
12prompt = f"<|im_start|>{speaker}: {text}<|speech_start|>"
13
14inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to(model.device)
15
16with torch.no_grad():
17 outputs = model.generate(
18 **inputs,
19 max_new_tokens=2048,
20 do_sample=True,
21 temperature=0.8,
22 repetition_penalty=1.15,
23 )
24
25generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)
26audio_tokens = re.findall(r'<\|s_(\d+)\|>', generated_text.split('<|speech_start|>')[1])
27audio_tokens = [int(token) for token in audio_tokens]
28audio_codes = torch.tensor(audio_tokens)[None, None]
29
30with torch.no_grad():
31 audio_waveform = codec.decode_code(audio_codes.cuda())
32
33sf.write('DisfluencySpeech-ms-en-zh-fr-he-ar.mp3', audio_waveform[0, 0].cpu().numpy(), 24000)1speaker = 'genshin-voice_audio_Rahman'
2text = "Hi nama saya Husein, I am so cute, 我喜欢吃鸡饭, boire du thé glacé, ולהירגע על החוף, وأحب أن أتعرض لبعض أشعة الشمس."
3description = """
4Vocal qualities: Very low pitch, clear and steady, with a neutral and composed demeanor.
5Speaking style: Neutral and restrained, with a monotone delivery that lacks significant pitch variation or emotional expression. The speech is methodical and precise, suitable for instructional or educational content.
6Pace: Very slow and deliberate, allowing ample time for each word to be fully articulated and understood.
7Fluency: Consistently fluent throughout, with no pauses, hesitations, or stutters.
8Acoustic environment: The recording has a very confined and enclosed sound, suggesting it was made in a small room or similar close space.
9Audio quality: The audio is generally clear but has a slight background noise, which adds a subtle layer of ambient sound without detracting from the clarity of the speech.
10Content style: Educational or instructional. The tone and pace are well-suited for teaching or explaining a step-by-step process, such as a math problem or a simple task.
11""".strip()
12prompt = f"<|im_start|>{speaker}: {text}<|description|>{description}<|speech_start|>"
13
14inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to(model.device)
15
16with torch.no_grad():
17 outputs = model.generate(
18 **inputs,
19 max_new_tokens=2048,
20 do_sample=True,
21 temperature=0.8,
22 repetition_penalty=1.15,
23 )
24
25generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)
26audio_tokens = re.findall(r'<\|s_(\d+)\|>', generated_text.split('<|speech_start|>')[1])
27audio_tokens = [int(token) for token in audio_tokens]
28audio_codes = torch.tensor(audio_tokens)[None, None]
29
30with torch.no_grad():
31 audio_waveform = codec.decode_code(audio_codes.cuda())
32
33sf.write('Rahman-ms-en-zh-fr-he-ar.mp3', audio_waveform[0, 0].cpu().numpy(), 24000)1speaker = 'genshin-voice_audio_Rahman'
2text = "Hi nama saya Husein, I am so cute, 我喜欢吃鸡饭, boire du thé glacé, ולהירגע על החוף, وأحب أن أتعرض لبعض أشعة الشمس."
3prompt = f"<|im_start|>{speaker}: {text}<|speech_start|>"
4
5inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to(model.device)
6
7with torch.no_grad():
8 outputs = model.generate(
9 **inputs,
10 max_new_tokens=2048,
11 do_sample=True,
12 temperature=0.8,
13 repetition_penalty=1.15,
14 )
15
16generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)
17audio_tokens = re.findall(r'<\|s_(\d+)\|>', generated_text.split('<|speech_start|>')[1])
18audio_tokens = [int(token) for token in audio_tokens]
19audio_codes = torch.tensor(audio_tokens)[None, None]
20
21with torch.no_grad():
22 audio_waveform = codec.decode_code(audio_codes.cuda())
23
24sf.write('Rahman-ms-en-zh-fr-he-ar-nondescription.mp3', audio_waveform[0, 0].cpu().numpy(), 24000)1y, sr = librosa.load('jenny.wav', sr = 16000)
2with torch.no_grad():
3 codes = codec.encode_code(torch.tensor(y)[None, None])
4tokens = ''.join([f'<|s_{i}|>' for i in codes[0, 0]])
5prompt = f"<|im_start|>{tokens}<|description|>"
6
7inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to(model.device)
8
9with torch.no_grad():
10 outputs = model.generate(
11 **inputs,
12 max_new_tokens=2048,
13 do_sample=True,
14 temperature=0.8,
15 repetition_penalty=1.15,
16 )
17tokenizer.decode(outputs[0], skip_special_tokens=False).split('<|description|>')[1]The audio features a young adult female with an East Asian accent, speaking in a neutral tone at a slow pace. Her speech is consistently fluent and very clear, despite the moderate ambient sounds in the background. The environment has a slightly confined quality to it, giving the recording a subtle echo that adds character to her voice. Interestingly, despite the happy emotion she conveys, her speech remains quite monotone, lacking the usual variations in pitch and rhythm that typically accompany positive emotions. This contrast between her cheerful mood and the steady delivery creates an interesting dynamic.\n\nThe low pitch of her voice lends a soothing yet serious quality to the content. Given the poetic nature of the transcript, this audio could be categorized as part of a literary reading or audiobook, perhaps describing a serene morning scene from a poet's work.1y, sr = librosa.load('jenny.wav', sr = 16000)
2with torch.no_grad():
3 codes = codec.encode_code(torch.tensor(y)[None, None])
4tokens = ''.join([f'<|s_{i}|>' for i in codes[0, 0]])
5prompt = f"<|im_start|>{tokens}<|description_category|>"
6
7inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to(model.device)
8
9with torch.no_grad():
10 outputs = model.generate(
11 **inputs,
12 max_new_tokens=2048,
13 do_sample=True,
14 temperature=0.8,
15 repetition_penalty=1.15,
16 )
17tokenizer.decode(outputs[0], skip_special_tokens=False).split('<|description_category|>')[1]Speaker profile: Young adult female with an East Asian accent.
Vocal qualities: Quite low pitch, clear and precise articulation. The voice is steady and controlled, reflecting a sense of calm and maturity.
Speaking style: Neutral and composed, with minimal emotional expression. The delivery is quite monotone, maintaining a consistent tone throughout without much variation in pitch or emphasis.
Pace: Slightly slow, allowing for a deliberate and measured flow of words. This pace enhances clarity and comprehension.
Fluency: Fluent and smooth, with no hesitations or interruptions. The speech is continuous and well-paced.
Acoustic environment: Simulates a very confined indoor space, producing a slightly boxed-in sound that adds to the intimacy of the recording.
Audio quality: Exceptionally clear and clean, with almost no background noise. The recording is crisp and professional-sounding, making it easy to follow along.
Content style: Informative and descriptive, similar to a travel guide or cultural segment. The tone is suitable for providing factual information about natural attractions, such as lakes in Europe.