This repository contains LEFSA model weights for multimodal affective state recognition. LEFSA stands for Label Encoder Fusion Strategy with Averaging and is designed for joint emotion recognition and sentiment recognition from audio, video, and text modalities.

Files

lefsa.pt — LEFSA checkpoint for joint audio-video-text emotion and sentiment recognition.
emoaffectnet.pt — EmoAffectNet model for visual feature extraction. The original EmoAffectNet repository is available here: https://github.com/ElenaRyumina/EMO-AffectNetModel.

What the Model Predicts

The model has two classification heads.

Task	Number of classes	Class order
Emotion recognition	7	`neutral`, `happy`, `sad`, `anger`, `surprise`, `disgust`, `fear`
Sentiment recognition	3	`negative`, `neutral`, `positive`

Use this exact class order when converting logits or probabilities to labels.

Model Overview

Acoustic, visual, and linguistic features are downsampled to a common temporal representation.
The model applies cross-modal transformer blocks to model interactions between modalities.
A label encoder produces unimodal emotion and sentiment predictions and injects this label-level context back into the fusion module.
In LEFSA, unimodal predictions are additionally averaged with multimodal predictions to improve robustness.

Research Corpora

The model family was evaluated in a multilingual and multicorpus setting.

Corpus	Language / domain	Modalities	Tasks
RAMAS	Russian, dyadic semi-spontaneous interactions	Audio, video, text	Emotion, sentiment
MELD	English, scripted TV-series dialogues	Audio, video, text	Emotion, sentiment
CMU-MOSEI	English, in-the-wild YouTube monologues	Audio, video, text	Emotion, sentiment

Emotion labels are mapped to seven classes: neutral, happiness, sadness, anger, surprise, disgust, and fear. Sentiment labels are mapped to three classes: negative, neutral, and positive.

Feature and Fusion Strategy Comparison

MF is macro F1-score. For CMU-MOSEI emotion recognition, mMF is the mean macro F1-score over binary positive/negative emotion classes. The average value is the integral score reported in the paper.

ID	Audio	Video	Text	Fusion	RAMAS Emotion MF7	RAMAS Sentiment MF3	MELD Emotion MF7	MELD Sentiment MF3	CMU-MOSEI Emotion mMF6	CMU-MOSEI Sentiment MF3	Average
1	Wav2Vec2	EmoAffectNet	JINA	BFS	60.57	65.02	38.56	65.94	62.46	62.56	59.18
2	ExHuBERT	EmoAffectNet	JINA	BFS	62.35	64.02	38.40	63.93	62.10	62.32	58.85
3	Wav2Vec2	EmoAffectNet	RoBERTa	BFS	57.64	67.14	35.48	65.03	61.58	60.19	57.84
4	ExHuBERT	EmoAffectNet	RoBERTa	BFS	60.54	64.70	35.05	63.78	60.78	59.25	57.35
5	Wav2Vec2	ResEmoteNet	JINA	BFS	55.88	59.93	37.75	63.81	62.22	63.69	57.21
6	ExHuBERT	ResEmoteNet	JINA	BFS	57.21	61.40	38.25	64.06	61.11	60.81	57.14
7	ExHuBERT	ResEmoteNet	RoBERTa	BFS	49.62	54.44	32.56	62.15	59.88	60.47	53.19
8	Wav2Vec2	ResEmoteNet	RoBERTa	BFS	52.29	54.87	34.08	61.65	58.79	57.42	53.18
9	Wav2Vec2	EmoAffectNet	JINA	LEFS	61.38	66.57	39.79	65.70	62.79	61.59	59.64
10	Wav2Vec2	EmoAffectNet	JINA	LEFSA	62.52	64.96	40.09	67.02	62.30	62.00	59.81

The best LEFSA configuration uses Wav2Vec2 + EmoAffectNet + JINA features with Label Encoder Fusion Strategy with Averaging.

Comparison with Prior Multimodal Approaches

ST means single-task recognition and MT means multitask recognition.

Approach	Corpus	Setup	Emotion A7	Emotion WF7	Sentiment A3	Sentiment WF3
Ours	RAMAS	MT	68.99	67.79	84.11	84.02
Zhang et al.	MELD	MT	41.17	41.22	67.33	67.21
Van et al.	MELD	ST	66.28	65.69	—	—
Hwang et al.	MELD	ST	66.70	65.93	—	—
Tu et al.	MELD	ST	67.85	67.02	—	—
Ours	MELD	MT	62.30	59.79	69.20	69.02

Approach	Corpus	Setup	Emotion mWA6	Emotion mWF6	Sentiment A2	Sentiment WF2
Chauhan et al.	CMU-MOSEI	MT	62.97	79.02	80.37	78.23
Sangwan et al.	CMU-MOSEI	MT	63.16	79.06	80.15	78.30
Hwang et al.	CMU-MOSEI	ST	—	—	87.40	87.30
Zheng et al.	CMU-MOSEI	ST	—	—	85.90	86.00
Ours	CMU-MOSEI	MT	64.78	79.06	84.83	84.90

Related Publications

Markitantov M., Ryumina E., Kaya H., Karpov A. Multi-Modal Multi-Task Affective States Recognition Based on Label Encoder Fusion // In Proc. Interspeech 2025, pp. 3010–3014. https://doi.org/10.21437/Interspeech.2025-2060

bibtex

@inproceedings{markitantov25_interspeech,
  title     = {{Multi-Modal Multi-Task Affective States Recognition Based on Label Encoder Fusion}},
  author    = {Maxim Markitantov and Elena Ryumina and Heysem Kaya and Alexey Karpov},
  year      = {2025},
  booktitle = {{Interspeech 2025}},
  pages     = {3010--3014},
  doi       = {10.21437/Interspeech.2025-2060},
  issn      = {2958-1796}
}

Markitantov M., Ryumina E., Dvoynikova A., Karpov A. Multi-lingual approach for multi-modal emotion and sentiment recognition based on triple fusion // Information Fusion, 2026, vol. 132, article 104207. https://doi.org/10.1016/j.inffus.2026.104207