Bellver-Soler, J., Guragain, A., Ramos-Varela, S., Córdoba, R., & D’Haro, L.F. (2025) Multilingual Speech Emotion Recognition for Iberian Languages: A Generative AI Approach with LLMs and Data Augmentation Techniques. Universidad Politécnica de Madrid (UPM) 📄 SSRN Preprint

This model combines Whisper-large-v3 as a self-supervised acoustic encoder with attentive pooling and a frozen Bloomz-7B1 LLM classifier, trained for emotion recognition on the IEMOCAP and MEA-Corpus datasets.

🚀 Model Description

Component	Description
Audio Encoder	`openai/whisper-large-v3` – frozen during training
Pooling	Single-head attentive pooling layer
Classifier	Frozen `bigscience/bloomz-7b1` LLM used as classification head
Projection Layer	Linear adapter mapping acoustic embeddings to LLM hidden size
Loss Function	Weighted cross-entropy
Optimizer	AdamW with cosine LR schedule and warm-up
Sampling Rate	16 kHz

🎯 Task

Speech Emotion Recognition (SER) Predict the emotion expressed in an audio waveform among:

['neutral', 'happy', 'sad', 'angry', 'fear', 'surprise', 'disgust']

🧩 Training Details

Dataset	Language	Type	Hours	Notes
IEMOCAP	English	Acted	7.0	Standard benchmark
MEACorpus	Spanish	Natural	13.2	In-the-wild TV data

Stratified speaker-independent splits (train/val/test)
5 random seeds for robustness
Validation metric: F1-macro (95% CI)
Training in bfloat16 on 2× NVIDIA A100 (40GB)

📈 Performance

Dataset	F1 (macro)	Augmentation
IEMOCAP	0.719	Spectrogram masking
MEACorpus	0.786	Mix-up
EMS (Spanish)	0.783	TTS Augmentation
VERBO (Portuguese)	0.765	Mix-up
AhoEmo3 (Basque)	0.994	Mix-up

Frozen Bloomz-7B1 improves F1 by +4.9% over a linear MLP head.

🧬 Architecture

Audio → Whisper Encoder → Attentive Pooling → Linear Projection → Bloomz-7B1 (frozen) → Emotion logits

🧪 Usage Example

python

import torch
import torchaudio

# Load model checkpoint
model = torch.load("whisper-ser-iemocap_meacorpus.ckpt", map_location="cpu")
model.eval()

# Example audio
waveform, sr = torchaudio.load("example.wav")
assert sr == 16000, "Audio must be 16 kHz"

# Forward pass
with torch.no_grad():
    logits = model(waveform)
    probs = torch.nn.functional.softmax(logits, dim=-1)
    emotion = probs.argmax(dim=-1)

print("Predicted emotion:", emotion.item())

🧾 Citation

If you use this model, please cite:

bibtex

@article{bellversoler2025multilingual,
  title={Multilingual Speech Emotion Recognition for Iberian Languages: A Generative AI Approach with LLMs and Data Augmentation Techniques},
  author={Bellver-Soler, Jaime and Guragain, Anmol and Ramos-Varela, Samuel and Córdoba, Ricardo and D’Haro, Luis Fernando},
  journal={Computer Speech & Language},
  year={2025},
  note={Preprint, SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5244228}
}

📂 Related Resources

Resource	Description
🧠 SpeechFactory (GitHub)	Codebase to generate synthetic emotional datasets
🎧 SER-MSPMEA-Spanish (Hugging Face)	Synthetic Spanish emotional dataset generated via FishSpeech-TTS
MSP-MEA	Spanish extension of MSP-Podcast generated with voice cloning
Upcoming: IEMOCAP-MEA dataset	A new cross-lingual dataset combining IEMOCAP and MEACorpus recordings will be released soon, extending the current Whisper-SER model for multilingual benchmarking.

📜 License

Released under CC BY 4.0. Attribution required for derivative works. Note: MEACorpus includes YouTube-sourced content — additional rights may apply.

🙌 Acknowledgements

Supported by:

European Commission – ASTOUND3 (101071191, Horizon Europe)
MCIN/AEI/ERDF – Project BEWORD (PID2021-126061OB-C43)
INNOVATRAD-CM – Comunidad de Madrid (PHS-2024/PH-HUM-52)

Authors: Jaime Bellver-Soler, Anmol Guragain, Samuel Ramos-Varela, Ricardo Córdoba, Luis Fernando D’Haro Speech Technology and Machine Learning Group, Universidad Politécnica de Madrid

jaimebellver/whisper-large-v3-SER

🎙️ Whisper-SER (IEMOCAP + MEACorpus)

🧠 Overview