Model Card for Ultravox
Ultravox is a multimodal Speech LLM built around a pretrained
Mistral-Nemo-Instruct-2407 and
whisper-large-v3-turbo backbone.
See
https://ultravox.ai for the GitHub repo and more information.
Model Details
Model Description
Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message).
The input to the model is given as a text prompt with a special <|audio|> pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio.
Using the merged embeddings as input, the model will then generate output text as usual.
In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output.
No preference tuning has been applied to this revision of the model.
- Developed by: Fixie.ai
- License: MIT
Model Sources
- Repository: https://ultravox.ai
- Demo: See repo
Usage
Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech-to-speech translation, analysis of spoken audio, etc.
To use the model, try the following:
1# pip install transformers peft librosa
2
3import transformers
4import numpy as np
5import librosa
6
7pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_4_1-mistral-nemo', trust_remote_code=True)
8
9path = "<path-to-input-audio>" # TODO: pass the audio here
10audio, sr = librosa.load(path, sr=16000)
11
12
13turns = [
14 {
15 "role": "system",
16 "content": "You are a friendly and helpful character. You love to answer questions for people."
17 },
18]
19pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)
Training Details
The model uses a pre-trained
Mistral-Nemo-Instruct-2407 backbone as well as the encoder part of
whisper-large-v3-turbo.
Only the multi-modal adapter is trained, while Whisper encoder and Mistral are kept frozen.
We use a knowledge-distillation loss where Ultravox is trying to match the logits of the text-based Mistral backbone.
Training Data
The training dataset is a mix of ASR datasets, extended with continuations generated by Mistral Nemo, and speech translation datasets, which yield a modest improvement in translation evaluations.
Training Procedure
Supervised speech instruction finetuning via knowledge-distillation. For more info, see
training code in Ultravox repo.
Training Hyperparameters
- Training regime: BF16 mixed precision training
- Hardward used: 8x H100 GPUs
Speeds, Sizes, Times
The current version of Ultravox, when invoked with audio content, has a time-to-first-token (TTFT) of approximately 150ms, and a tokens-per-second rate of ~50-100 when using an A100-40GB GPU, all using a Mistral Nemo backbone.
Check out the audio tab on
TheFastest.ai for daily benchmarks and a comparison with other existing models.
Evaluation
| Ultravox 0.4.1 Mistral Nemo |
|---|
| en_ar | 10.36 |
| en_de | 28.39 |
| es_en | 37.49 |
| ru_en | 41.64 |
| en_ca | 26.85 |
| zh_en | 12.65 |
