speech-salamandra-es-en is a speech-language model built on top of salamandra-7b-instruct.
It accepts text and optional speech input and it can perform Automatic Speech Recognition (ASR) and Spoken Question Answering (SQA).
[!WARNING]
DISCLAIMER: This version of Salamandra can expose intermediate reasoning/transcription text. Use skip_thinking=True in the helper method, or processor.decode_response(..., skip_thinking=True), to display only the final response.
speech-salamandra-es-en is a finetuned version of salamandra-7b-instruct coupled with the speech encoder from SeamlessM4T v2. It has been trained to follow general text instructions and to emit an internal thinking sequence followed by the final answer.
Intended Use
The model is intended for both research and commercial use on speech/text instruction-following tasks.
Hardware and Software
Training Framework
The code used to train speech-salamandra-es-en is based on the Transformers library, and will be publicly available soon.
Training was conducted on 4 nodes, each with the following specifications:
4x Nvidia Hopper GPUs with 64GB HBM2 memory
2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
4x NDR200 (BW per node 800Gb/s)
512 GB of Main memory (DDR5)
How to use
[!NOTE] Requirements
To use this model, ensure you have the following Python packages installed: transformers>=4.51.2, torch, torchaudio, numpy, soundfile
The convenience method returns the generated text. Set skip_thinking=True to remove the leading reasoning/transcription span and keep only the final answer:
python
1import soundfile as sf
23audio_path ="/path/to/audio/example.wav"4audio, sr = sf.read(audio_path)5question ="Who is angry in the audio?"67response = model.generate_response(8 processor,9 instruction=question,10 audio=[(audio, sr)],11 max_new_tokens=512,12 skip_thinking=True,13)14print(response)
To inspect the full thinking/transcription sequence, keep skip_thinking=False:
[NOTE] We refer to SQuADv2 as its subset of non-answerable questions. While we refer to SQuAD as the "answerable" subset in its first version.
We have expanded the SQuAD v2 dataset to speech synthesizing the context of each sample using Kyutai's Pocket TTS. To reduce the length of the audios, before synthesizing, we summarize the contexts with Gemma4 31B.
Although the model we are using as backbone is already insructed, we still add IF data during the training phase in order to avoid catastrophic forgetting. We use a subset of the original IF data used in the training of ALIA-40B-Instruct-2605. This subset is composed of:
Dataset
# of samples
WildChat
153392
Aya Dataset
15146
Mentor
12719
Other
39141
Total
220398
Evaluation
Below are the evaluation ASR results on the Common Voice 22.0 and Voxpopuli test sets, and the SQA results on LibriSQA test set and a subset of the SQuAD dataset that we've expanded to synthetic audios.
We use as a baseline Voxtral-Mini to compare our model.
We report the following metrics:
Click to show metrics details
WER: jiwer implementation. Applied the basic normalization from whisper_normalizer before computing WER.
BERTScore: bert_score implementation. We report F1 measure.
Automatic Speech Recognition
Common Voice 22.0 Test
Common Voice 22.0 Test
WER
SpeechSalamandra
Voxtral
en
9.18
11.21
es
6.67
5.44
Voxpopuli Test
Voxpopuli Test
WER
SpeechSalamandra
Voxtral
en
7.78
8.04
es
9.39
6.66
Spoken Question Answering
LibriSQA
LibriSQA
BERTScore
SpeechSalamandra
Voxtral
en
0.68
0.54
SQuAD
SQuAD
BERTScore
SpeechSalamandra
Voxtral
en
0.65
0.68
es
0.58
0.63
Alternative SQuAD evaluation
For the SQA tasks, the answers include information that is not strictly required by the question. For example:
Question: How many authors participated in the paper?
Answer: The number of authors participating in the paper is 5.
Target Answer: The paper has been written by 5 scientists.
Even though the meaning of both answers is the same, the words used are different.
Actually, that is one of the reasons to use BertScore when evaluating this kind of tasks.
One way to make this evaluation more robust is to post-edit the output to make the answer as concise as possible.
After post-editing, the answers would be something like this:
Question: How many authors participated in the paper?
Answer: 5.
Target Answer: 5.
This practice increases the metrics for both SpeechSalamandra and Voxtral, and make their comparison more focused on “what the model says” and not on “how the model says it”:
For further information, please send an email to bsc-lt@bsc.es.
Copyright
Copyright(c) 2026 by AI Institute, Barcelona Supercomputing Center.
Funding
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.
Acknowledgements
The training of the model was possible thanks to the computing time provided by Barcelona Supercomputing Center through MareNostrum 5.
We acknowledge EuroHPC Joint Undertaking for awarding us access to MareNostrum5 as BSC, Spain.
Disclaimer
Be aware that the model may contain biases or other unintended distortions.
When third parties deploy systems or provide services based on this model, or use the model themselves,
they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations,
including those governing the use of Artificial Intelligence.
The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.
If you find our model useful, we would appreciate if you could cite our work as follows:
@misc{bsclt2026speechsalamandra7b ,
title={speech-salamandra-es-en: a Speech LLM model for SQA in English and Spanish.},
author={Speech Salamandra Team},
organization={Barcelona Supercomputing Center},
url={https://huggingface.co/BSC-LT/speech-salamandra-es-en},
year={2026}
}
