Mimi codec is a state-of-the-art audio neural codec, developped by Kyutai, that combines semantic and acoustic information into audio tokens running at 12.5Hz and a bitrate of 1.1kbps.
Model Details
Model Description
Mimi is a high-fidelity audio codec leveraging neural networks. It introduces a streaming encoder-decoder architecture with quantized latent space, trained in an end-to-end fashion.
It was trained on speech data, which makes it particularly adapted to train speech language models or text-to-speech systems.
Use the following code to get started with the Mimi model using a dummy example from the LibriSpeech dataset (~9MB). First, install the required Python packages:
Then load an audio sample, and run a forward pass of the model:
python
1from datasets import load_dataset, Audio
2from transformers import MimiModel, AutoFeatureExtractor
34# load a demonstration datasets5librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy","clean", split="validation")67# load the model + feature extractor (for pre-processing the audio)8model = MimiModel.from_pretrained("kyutai/mimi")9feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")1011# cast the audio data to the correct sampling rate for the model12librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))13audio_sample = librispeech_dummy[0]["audio"]["array"]1415# pre-process the inputs16inputs = feature_extractor(raw_audio=audio_sample, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt")1718# explicitly encode then decode the audio inputs19encoder_outputs = model.encode(inputs["input_values"])20audio_values = model.decode(encoder_outputs.audio_codes)[0]2122# or the equivalent with a forward pass23audio_values = model(inputs["input_values"]).audio_values
Mimi can be used directly as an audio codec for real-time compression and decompression of speech signals.
It provides high-quality audio compression and efficient decoding.
Out-of-Scope Use
The model is not intended to be used to impersonate other people or any malicious use of any kind.
Bias, Risks, and Limitations
The model has been trained with a few safeguards to try to limit potential toxic usages, however our toxicity analysis shows that it behaves in the middle of existing models with respect to textual generation. It has some bias towards certain domains and topics that are over-represented in the training data. Its capabilities are relatively limited so far and it is trained to produce only one voice to avoid impersonation. Yet, we need the perspective in time to establish the sociotechnical limitations.
Training Details
Training Data
The training data is detailled in the paper.
Training procedure and hyper-parameters
The different stages of the training procedure are detailled in the paper along with the hyper-parameters.
Citation
@techreport{kyutai2024moshi,
author = {Alexandre D\'efossez and Laurent Mazar\'e and Manu Orsini and Am\'elie Royer and Patrick P\'erez and Herv\'e J\'egou and Edouard Grave and Neil Zeghidour},
title = {Moshi: a speech-text foundation model for real-time dialogue},
institution = {Kyutai},
year={2024},
month={September},
url={http://kyutai.org/Moshi.pdf},
}
Model Card Authors
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil Zeghidour, Yoach Lacombe
