This model predicts sentiment (positive, neutral, negative) and emotion (joy, sadness, anger, etc.) from video data using audio, text, and visual features.
It was trained on the MELD dataset, which contains dialogue clips from the TV show Friends.
Emotion: anger | disgust | fear | joy | neutral | sadness | surprise
The version shared here is the normalized model (recommended for inference).
📊 Training Metrics
Training was run for 24 epochs. Loss curves show steady improvement.
Split
Final Loss
Train
~2.61
Validation
~2.63
Test
~2.64
You can find the full log of metrics in metrics.json.
🚀 Usage
You can download the model weights directly from this Hub repo and load them in PyTorch:
python
1import torch
23# Load model weights4state_dict = torch.load("model.pth", map_location="cpu")56# If using the provided model class (see GitHub repo)7from models import MultimodalSentimentModel
8model = MultimodalSentimentModel(...)9model.load_state_dict(state_dict)10model.eval()
For full preprocessing pipeline (feature extraction, normalization, and inference), see the GitHub repo.
📌 Notes
• Dataset (MELD.Raw.tar.gz) is not included here due to size; please download it directly from the MELD repo.
• Only the normalized model is provided for easier use.
🙏 Acknowledgements
This project is adapted from the tutorial by Andreas Trolle.
My contribution was to reproduce the training pipeline, understand the design, and adapt it for deployment (e.g., AWS SageMaker).
