Early Fusion Emotion Recognition on MELD (Text + Audio)
This repository contains an early fusion multimodal emotion recognition model trained on the MELD dataset, combining textual and acoustic embeddings at the feature level.
Both modalities are encoded independently and fused via embedding concatenation
before classification.
Model Overview
Text encoder:bert-base-uncased (frozen)
Audio encoder: pre-extracted acoustic features (frozen)
Fusion strategy: Early fusion (concatenation)
Classifier: 2-layer MLP
Training strategy:
Both encoders are frozen
Only the fusion classifier is trained
Dataset
Name: MELD (declare-lab/MELD)
Modalities: Text + Audio
Setting: Multi-class emotion classification
Splits: Train / Validation / Test (official MELD splits)
Training Details
Loss: Cross-entropy
Optimizer: Adam
Fusion dimension: 1536
Evaluation metrics:
Accuracy
Macro F1-score
Per-class F1-score
Important Notes
This model does not perform end-to-end multimodal fine-tuning.
Both text and audio encoders act as frozen feature extractors.
The provided weights correspond only to the fusion classifier.
To reproduce results, identical feature extraction pipelines must be used for both modalities.
Intended Use
Multimodal emotion recognition research
Comparison with unimodal baselines
Early vs. late fusion analysis
Educational and academic purposes
Limitations
Temporal context across utterances is not modeled
Speaker identity is not used
Fusion is limited to simple feature concatenation
Performance depends on quality of pre-extracted features
