The speech tokenizer for EMOVA will continue to be developed as an independent open-source project. Please follow the link:
DSTKThis repo contains the official speech tokenizer used to train the
EMOVA series of models. With a semantic-acoustic disentangled design, it not only facilitates seamless omni-modal alignment among vision, language and speech modalities, but also empowers flexible speech style controls including speakers, emotions and pitches. We summarize its key advantages as:
-
Discrete speech tokenizer: it contains a SPIRAL-based speech-to-unit (S2U) tokenizer to capture both phonetic and tonal information of input speeches, which is then discretized by a finite scalar quantizater (FSQ) into discrete speech units, and a VITS-based unit-to-speech (U2S) de-tokenizer to reconstruct speech signals from speech units.
-
Semantic-acoustic disentanglement: to seamlessly align speech units with the highly semantic embedding space of LLMs, we opt for decoupling the semantic contents and acoustic styles of input speeches, and only the former are utilized to generate the speech tokens.
-
Biligunal tokenization: EMOVA speech tokenizer supports both Chinese and English speech tokenization with the same speech codebook.
-
Flexible speech style control: thanks to the semantic-acoustic disentanglement, EMOVA speech tokenizer supports
24 speech style controls (i.e., 2 speakers, 3 pitches, and 4 emotions). Check the
Usage for more details.
Clone this repo and create the EMOVA virtual environment with conda. Our code has been validated on NVIDIA A800/H20 GPU & Ascend 910B3 NPU servers. Other devices might be available as well.
-
Initialize the conda environment:
1git clone https://github.com/emova-ollm/EMOVA_speech_tokenizer.git
2conda create -n emova python=3.10 -y
3conda activate emova
-
Install the required packages (note that instructions are different from GPUs and NPUs):
1# upgrade pip and setuptools if necessary
2pip install -U pip setuptools
3
4cd emova_speech_tokenizer
5pip install -e . # for NVIDIA GPUs (e.g., A800 and H20)
6pip install -e .[npu] # OR for Ascend NPUS (e.g., 910B3)
EMOVA speech tokenizer can be easily deployed using the 🤗 HuggingFace transformers API!
1import random
2from transformers import AutoModel
3import torch
4
5### Uncomment if you want to use Ascend NPUs
6# import torch_npu
7# from torch_npu.contrib import transfer_to_npu
8
9# load pretrained model
10model = AutoModel.from_pretrained("Emova-ollm/emova_speech_tokenizer_hf", torch_dtype=torch.float32, trust_remote_code=True).eval().cuda()
11
12# S2U
13wav_file = "./examples/s2u/example.wav"
14speech_unit = model.encode(wav_file)
15print(speech_unit)
16
17# U2S
18emotion = random.choice(['angry', 'happy', 'neutral', 'sad'])
19speed = random.choice(['normal', 'fast', 'slow'])
20pitch = random.choice(['normal', 'high', 'low'])
21gender = random.choice(['female', 'male'])
22condition = f'gender-{gender}_emotion-{emotion}_speed-{speed}_pitch-{pitch}'
23
24output_wav_file = f'./examples/u2s/{condition}_output.wav'
25model.decode(speech_unit, condition=condition, output_wav_file=output_wav_file)
If you find our model/code/paper helpful, please consider citing our papers and staring us!
1@article{chen2024emova,
2 title={Emova: Empowering language models to see, hear and speak with vivid emotions},
3 author={Chen, Kai and Gou, Yunhao and Huang, Runhui and Liu, Zhili and Tan, Daxin and Xu, Jing and Wang, Chunwei and Zhu, Yi and Zeng, Yihan and Yang, Kuo and others},
4 journal={arXiv preprint arXiv:2409.18042},
5 year={2024}
6}
