ComSpeech

Authors: Qingkai Fang, Shaolei Zhang, Zhengrui Ma, Min Zhang, Yang Feng*

💡 Highlights

1. ComSpeech is a general composite S2ST model architecture, which can seamlessly integrate any pretrained S2TT and TTS models into a direct S2ST model.
2. ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed.
3. With our proposed training strategy ComSpeech-ZS, we achieve performance comparable to supervised training without using any parallel speech data.

🔥 Quick Start

Requirements

• python==3.8, torch==2.1.2
• Install fairseq:
  bash

  1cd fairseq 2pip install -e .

Data Preparation

[!Note] This section is under construction and will be updated within 3 days.

ComSpeech (Supervised Learning)

[!Note] The following scripts use 4 RTX 3090 GPUs by default. You can adjust --update-freq, --max-tokens-st, --max-tokens, and --batch-size-tts depending on your available GPUs.

1. Pretrain the S2TT model, and the best checkpoint will be saved at ComSpeech/checkpoints/st.cvss.fr-en/checkpoint_best.pt.

1. Pretrain the TTS model, and the best checkpoint will be saved at ComSpeech/checkpoints/tts.fastspeech2.cvss-fr-en/checkpoint_best.pt.

1. Finetune the entire model using the S2ST data, and the chekpoints will be saved at ComSpeech/checkpoints/s2st.fr-en.comspeech.

1. Average the 5 best checkpoints and test the results on the test set.

[!Note] To run inference, you need to download the pretrained HiFi-GAN vocoder from this link and place it in the hifi-gan/ directory.

ComSpeech-ZS (Zero-shot Learning)

1. Pretrain the S2TT model, and the best checkpoint will be saved at ComSpeech/checkpoints/st.cvss.fr-en/checkpoint_best.pt.

1. Pretrain the TTS model, and the best checkpoint will be saved at ComSpeech/checkpoints/tts.fastspeech2.cvss-x-en/checkpoint_best.pt (note: this checkpoint is used for experiments on all language pairs in the zero-shot learning scenario).

1. Finetune the S2TT model and the vocabulary adaptor using S2TT data (stage 1), and the best checkpoint will be saved at ComSpeech/checkpoints/st.cvss.fr-en.ctc/checkpoint_best.pt.

1. Finetune the entire model using both S2TT and TTS data (stage 2), and the checkpoints will be saved at ComSpeech/checkpoints/s2st.fr-en.comspeech-zs.

1. Average the 5 best checkpoints and test the results on the test set.

Checkpoints

Supervised Learning

Zero-shot Learning

🖋 Citation

@inproceedings{fang-etal-2024-can,
    title = {Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?},
    author = {Fang, Qingkai and Zhang, Shaolei and Ma, Zhengrui and Zhang, Min and Feng, Yang},
    booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics},
    year = {2024},
}