FastSpeech2Conformer is a non-autoregressive text-to-speech (TTS) model that combines the strengths of FastSpeech2 and the conformer architecture to generate high-quality speech from text quickly and efficiently.
Model Description
The FastSpeech2Conformer model was proposed with the paper Recent Developments On Espnet Toolkit Boosted By Conformer by Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang. It was first released in this repository. The license used is Apache 2.0.
FastSpeech2 is a non-autoregressive TTS model, which means it can generate speech significantly faster than autoregressive models. It addresses some of the limitations of its predecessor, FastSpeech, by directly training the model with ground-truth targets instead of the simplified output from a teacher model. It also introduces more variation information of speech (e.g., pitch, energy, and more accurate duration) as conditional inputs. Furthermore, the conformer (convolutional transformer) architecture makes use of convolutions inside the transformer blocks to capture local speech patterns, while the attention layer is able to capture relationships in the input that are farther away.
Developed by: Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang.
Run inference via the Transformers modelling code with the model and hifigan separately
python
12from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerModel, FastSpeech2ConformerHifiGan
3import soundfile as sf
45tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer")6inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt")7input_ids = inputs["input_ids"]89model = FastSpeech2ConformerModel.from_pretrained("espnet/fastspeech2_conformer")10output_dict = model(input_ids, return_dict=True)11spectrogram = output_dict["spectrogram"]1213hifigan = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan")14waveform = hifigan(spectrogram)1516sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)
Run inference via the Transformers modelling code with the model and hifigan combined
python
1from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan
2import soundfile as sf
34tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer")5inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt")6input_ids = inputs["input_ids"]78model = FastSpeech2ConformerWithHifiGan.from_pretrained("espnet/fastspeech2_conformer_with_hifigan")9output_dict = model(input_ids, return_dict=True)10waveform = output_dict["waveform"]1112sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)
Run inference with a pipeline and specify which vocoder to use
python
1from transformers import pipeline, FastSpeech2ConformerHifiGan
2import soundfile as sf
34vocoder = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan")5synthesiser = pipeline(model="espnet/fastspeech2_conformer", vocoder=vocoder)67speech = synthesiser("Hello, my dog is cooler than you!")89sf.write("speech.wav", speech["audio"].squeeze(), samplerate=speech["sampling_rate"])
Direct Use
[More Information Needed]
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
