Parler-TTS Mini: Expresso is a fine-tuned version of Parler-TTS Mini v0.1
on the Expresso dataset. It is a lightweight text-to-speech (TTS)
model that can generate high-quality, natural sounding speech. Compared to the original model, Parler-TTS Expresso provides
superior control over emotions (happy, confused, laughing, sad) and consistent voices (Jerry, Thomas, Elisabeth, Talia).
It is part of the first release from the Parler-TTS project, which aims to
provide the community with TTS training resources and dataset pre-processing code. Details for reproducing this entire
training run are provided in the section Training Procedure.
Usage
Using Expresso is as simple as "bonjour". Simply install the library from source:
You can then use the model with the following inference snippet:
py
1import torch
2from parler_tts import ParlerTTSForConditionalGeneration
3from transformers import AutoTokenizer, set_seed
4import soundfile as sf
56device ="cuda:0"if torch.cuda.is_available()else"cpu"78model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-expresso").to(device)9tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-expresso")1011prompt ="Why do you make me do these examples? They're *so* generic."12description ="Thomas speaks moderately slowly in a sad tone with emphasis and high quality audio."1314input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device)15prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)1617set_seed(42)18generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)19audio_arr = generation.cpu().numpy().squeeze()20sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)
Tips:
Specify the name of a male speaker (Jerry, Thomas) or female speaker (Talia, Elisabeth) for consistent voices
The model can generate in a range of emotions, including: "happy", "confused", "default" (meaning no particular emotion conveyed), "laughing", "sad", "whisper", "emphasis"
Include the term "high quality audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
To emphasise particular words, wrap them in asterisk (e.g. *you* in the example above) and include "emphasis" in the prompt
Training Procedure
Expresso is a high-quality, expressive speech dataset that includes samples from four speakers (two male, two female).
By fine-tuning Parler-TTS Mini v0.1 on this dataset, we can train the model to follow emotion and speaker prompts.
To reproduce this fine-tuning run, we need to perform two steps:
Create text descriptions from the audio samples in the Expresso dataset
Train the model on the (text, audio) pairs
Step 1 is performed using the DataSpeech library, and step 2 using
Parler-TTS. Should you wish to use the pre-annotated dataset from our
experiments, you can jump straight to step 2. For both, you can follow step 0 for
getting set-up.
Step 0: Set-Up
We'll start by creating a fresh Python environment:
You can link your Hugging Face account so that you can push model repositories on the Hub. This will allow you to save
your trained models on the Hub so that you can share them with the community. Simply run the command:
sh
1git config --global credential.helper store
2huggingface-cli login
And then enter an authentication token from https://huggingface.co/settings/tokens. Create a new token if you do not
have one already. You should make sure that this token has "write" privileges.
You also have the option to configure Accelerate by running the following command. Note that you should set the number
of GPUs you wish to use for training/inference, and also the data type (dtype) based on your device (e.g. bfloat16 on
A100 GPUs, float16 on V100 GPUs, etc.):
accelerate config
Optionally, you can also login to Weights and Biases for automatic logging:
wandb login
Step 1: Create Text Descriptions
Creating text descriptions for the dataset comprises three sub-stages from DataSpeech, which we'll cover below.
1.A. Annotate the Expresso dataset
We'll use the main.py file from DataSpeech to label
the following continuous variables:
Note that the script will be faster if you have GPUs at your disposal. It will automatically scale up to every GPU available in your environment. To control which GPUs to run the script on consider indicating via CUDA_VISIBLE_DEVICES environment variable.
The resulting dataset will be pushed to the Hugging Face Hub under your Hugging Face handle. Mine was pushed to reach-vb/expresso-tags.
We can see that the dataset is annotated with continuous features like "speaking_rate" and "snr".
1.B. Map annotations to text bins
The next step involves mapping the continuous variables to discrete ones. This is achieved by binning the continuous variables
into buckets, and assigning each one a text label.
Since the ultimate goal here is to fine-tune the Parler-TTS v0.1 checkpoint
on the Expresso dataset, we want to stay consistent with the text bins of the dataset on which the original model was trained.
To do this, we'll pass v01_bin_edges.json
as an input argument to our script, which holds the bin edges from the original dataset:
Since we leverage the bins from the original dataset, the above script only takes a few seconds. The resulting dataset
will be pushed to the Hugging Face Hub under your Hugging Face handle. Mine was pushed to reach-vb/expresso-tags.
You can notice that text bins such as "slightly noisy", "quite monotone" have been added to the samples.
1.C. Create natural language descriptions from those text bins
Now that we have text bins associated to the Expresso dataset, the next step is to create natural language descriptions.
This involves passing the text bins to a large-language model (LLM), and have it generate corresponding descriptions.
There is a template prompt creation script
in Parler-TTS which can be used to generate descriptions from the features tagged in step 1.A (reverberation, noise, speaking rate, etc).
However, not all of these features are relevant for the Expresso dataset. For instance, Expresso was recorded in a
professional recording studio, so all the samples are high quality. Thus, we chose to create prompts with the following subset of features:
Name: we mapped the speaker ids (ex1, ex2, ex3, ex4) to unique speaker names (Jerry, Elisabeth, Thomas, Talia). This encourages the model to learn specific speakers from the training data
Emotion: we include the emotion provided in the Expresso dataset
Speaking rate: we use the pre-computed text bins from the previous step
In addition, we also hard-coded the quality of the audio to be "very high-quality", given the studio recording conditions.
As an example, if we passed:
Speaker: Jerry
Emotion: confused
Speaking rate: moderate speed
We would expect to generate a sample along the lines of: "Jerry speaks with a confused tone and at a moderate speed with high quality audio."
The modified prompt creation script can be found in this repository. You can download this script with the following Python command:
Note that the Mistral model is gated, so you should ensure you have accepted the terms-of-use from the model card.
You can find the annotated dataset under TODO reach-vb/expresso-tagged-w-speech-mistral,
where you'll find sensible descriptions from the features that we passed.
This step generally demands more resources and times and should use one or many GPUs. Scaling to multiple GPUs using distributed data parallelism (DDP)
is trivial: simply run accelerate config and select the multi-GPU option, specifying the IDs of the GPUs you wish to use. The
above script can then be run using DDP with no code changes.
If you are resource constrained and need to use a smaller model, Gemma 2B
is an excellent choice.
Step 2: Fine-Tune the Model
Fine-tuning is performed using the Parler-TTS training script run_parler_tts_training.py.
It is the same script used to pre-train the model, and can be used for fine-tuning without any code-changes.
To preserve the model's ability to generate speech with generic voice descriptions, such as in the style of
Parler-TTS Mini v0.1, we fine-tuned the model
on a combination of three datasets, including the test split of LibriTTS-R:
On a single 80GB A100 GPU, training took approximately 1.5 hours and returned a final evaluation loss of 4.0. Again, the
script can be configured for multiple GPUs by running accelerate config from the command line; no further
code-changes are required.
Training performance is quite sensitive to learning rate and number of epochs: you should tune these according to your task
and the size of your dataset. In our experiments, we found the best performance to occur after 8 epochs of training
with a learning rate of 8e-5.
If you followed to the end of these steps: congratulations! You should now have a fine-tuned model you can use for your
downstream applications using the inference code-example above. You can try substituting your own dataset, or
run training using a single-speaker dataset, like the Jenny example.
Contrarily to other TTS models, Parler-TTS is a fully open-source release. All datasets, pre-processing, training code and weights are released publicly under permissive license, enabling the community to build on our work and develop their own powerful TTS models.
Parler-TTS was released alongside:
If you found this repository useful, please consider citing this work and also the original Stability AI paper:
@misc{lacombe-etal-2024-parler-tts,
author = {Yoach Lacombe and Vaibhav Srivastav and Sanchit Gandhi},
title = {Parler-TTS},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/huggingface/parler-tts}}
}
@misc{lyth2024natural,
title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
author={Dan Lyth and Simon King},
year={2024},
eprint={2402.01912},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
License
This model is permissively licensed under the Apache 2.0 license.
