CLARA: Enhancing Multimodal Sentiment Analysis via Efficient Vision-Language Fusion
⚠️ IMPORTANT NOTICE
This code is directly related to our manuscript currently under review at The Visual Computer (Springer).
Revision note: the architecture, HFM split handling, and paper-facing
results have been revised after the implementation audit. Use
ravel_revision_results/README.md and
ravel_revision_results/revision_status.md
for the corrected RAVEL artifacts. Legacy CLARA claims and numbers below are
kept for repository history and should not be copied into the revised paper
without checking the updated result tables.
If you use this code in your research, please cite our paper:
1 @article{lam2025clara,
2 title={CLARA: Enhancing Multimodal Sentiment Analysis via Efficient Vision-Language Fusion},
3 author={Lam, Phuong and Phan, Tuoi Thi and Tran, Thien Khai},
4 journal={The Visual Computer},
5 year={2025},
6 publisher={Springer},
7 note={Under Review},
8 doi={10.5281/zenodo.17889083}
9 }
Paper Status:
Journal: The Visual Computer (Springer)Status: Under ReviewSubmitted: December 2025
📋 Table of Contents
🎯 Overview
CLARA (Co-attention Learning for Robust multimodal sentiment Analysis) is a parameter-efficient vision-language framework for multimodal sentiment analysis on social media image-text pairs.
Why CLARA?
CLARA achieves state-of-the-art performance while updating only 7.45% of parameters :
✅ 83.16% accuracy on MVSA-Single (+3.83% over previous SOTA)
✅ 73.51% accuracy on MVSA-Multiple (+0.22% over previous SOTA)
✅ 88.13% accuracy on HFM hate speech detection (+1.03% over previous SOTA)
✅ 51.9% faster training compared to full fine-tuning
✅ 62.5% lower memory usage (6GB vs 16GB peak GPU memory)
🏗️ Architecture
CLARA integrates six key components for efficient and effective multimodal sentiment analysis:
CLARA Architecture Components:
Vision Encoder : CLIP ViT-B/16 with LoRA adapters (r=8, α=16)Text Encoder : DeBERTa-base with hybrid LoRA + selective unfreezingProjection Layers : Map both modalities to shared 512-dimensional spaceCo-Attention Fusion : Two-layer bidirectional multi-head attention (8 heads)Verification Module : Computes modality consensus from unimodal predictionsFeedback Module : Refines final prediction using disagreement signal
Key Innovation:
Unlike traditional fusion approaches, CLARA's bidirectional co-attention creates explicit associations between visual regions and textual spans through symmetric vision↔text and text↔vision processing paths.
✨ Key Features
1. Parameter Efficiency
Only 7.45% trainable parameters (21.27M / 285.47M total)LoRA rank r=8 applied to attention projections92.2% parameter reduction vs full fine-tuning
2. Cross-Modal Reasoning
Bidirectional co-attention prevents text bias8 attention heads for diverse alignment patternsExplicit image-phrase associations
3. Consensus Modeling
Verification module computes agreement signalFeedback mechanism integrates consensusEnhanced neutral class prediction (+8.34% F1)
4. Robust Performance
State-of-the-art on 3 diverse benchmarksStrong generalization across sentiment classesWell-calibrated predictions
📊 Performance
Main Results
Dataset Metric CLARA Previous SOTA Improvement MVSA-Single Accuracy 83.16% 79.33% +3.83% Weighted F1 83.04% 77.51% +5.53% MVSA-Multiple Accuracy 73.51% 73.29% +0.22% Weighted F1 73.45% 70.06% +3.39% HFM Accuracy 88.13% 87.10% +1.03% Macro F1 87.82% 86.62% +1.20%
Efficiency Comparison
Method Trainable Train Time Memory F1 Score Full Fine-tuning 272.0M (100%) 1.35h 16GB 76.50% CLARA (LoRA) 21.27M (7.45%) 0.65h 6GB 83.04% Improvement -92.2% -51.9% -62.5% +6.54%
🚀 Installation
Requirements
Python 3.8+
PyTorch 2.0+
CUDA 11.8+ (for GPU support)
8GB+ GPU memory
Quick Install
1 # Clone repository
2 git clone https://github.com/phuonglamgithub/CLARA.git
3 cd CLARA
4
5 # Install dependencies
6 pip install -r requirements.txt
Detailed Installation
Method 1: Conda (Recommended)
1 # Create conda environment
2 conda create -n clara python = 3.8
3 conda activate clara
4
5 # Install PyTorch with CUDA
6 conda install pytorch torchvision torchaudio pytorch-cuda = 11.8 -c pytorch -c nvidia
7
8 # Install other dependencies
9 pip install transformers == 4.30 .0 peft == 0.4 .0
10 pip install pillow numpy pandas matplotlib seaborn scikit-learn tqdm
Method 2: pip + virtualenv
1 # Create virtual environment
2 python -m venv clara_env
3 source clara_env/bin/activate # Windows: clara_env\Scripts\activate
4
5 # Install dependencies
6 pip install -r requirements.txt
📊 Datasets
Download Datasets
All datasets are available on Google Drive:
Contents:
data/MVSA-Single/ - MVSA-Single datasetdata/MVSA-Multiple/ - MVSA-Multiple datasetdata/HFM/ - Hateful Memes dataset
Dataset Structure
After downloading, organize datasets as follows:
data/
│
├── MVSA-Single/
│ ├── images/
│ │ ├── 1.jpg
│ │ ├── 2.jpg
│ │ └── ...
│ └── labelResultAll.txt
│
├── MVSA-Multiple/
│ ├── images/
│ │ ├── 1.jpg
│ │ ├── 2.jpg
│ │ └── ...
│ └── labelResultAll.txt
│
└── HFM/
├── image/
├── train/
├── val/
├── test/
└── text/
├── train.txt
├── val.txt
└── test.txtDataset Statistics
MVSA-Single
Total: 2,592 samples (after cleaning)Classes: Positive (1,398), Neutral (470), Negative (724)Train/Val/Test: 1,813 / 387 / 392
MVSA-Multiple
Total: 3,555 samples (after filtering)Classes: Positive (2,173), Neutral (1,268), Negative (114)Train/Val/Test: 2,487 / 532 / 536Challenge: Extremely imbalanced (3.2% negative)
HFM (Hateful Memes)
Total: 24,635 samplesClasses: Hateful (13,988), Non-hateful (10,647)Train/Val/Test: 19,816 / 2,410 / 2,409
🎯 Pre-trained Models
Download Model Checkpoints
Pre-trained CLARA models are available on Google Drive in the checkpoints/ folder:
Available Models:
checkpoints/clara_mvsa_single.pt - Trained on MVSA-Singlecheckpoints/clara_mvsa_multi.pt - Trained on MVSA-Multiplecheckpoints/clara_hfm.pt - Trained on HFM
Checkpoint Structure
CLARA/
├── checkpoints/
│ ├── clara_mvsa_single.pt
│ ├── clara_mvsa_multi.pt
│ └── clara_hfm.ptModel Specifications
Model Dataset Size Accuracy F1 Score clara_mvsa_single.ptMVSA-Single ~85 MB 83.16% 83.04% clara_mvsa_multi.ptMVSA-Multiple ~85 MB 73.51% 73.45% clara_hfm.ptHFM ~85 MB 88.13% 87.82%
📸 Results
Confusion Matrices and performance
All confusion matrices and performance visualizations are available in the results/ folder:
Available Visualizations:
results/confusion_matrix_mvsa_single.pngresults/confusion_matrix_mvsa_multiple.pngresults/confusion_matrix_hfm.pngresults/attention_visualization_samples.pngresults/performance_comparison.png
Results Structure
CLARA/
└──results/
├── confusion_matrix_mvsa_single.png
├── confusion_matrix_mvsa_multiple.png
├── confusion_matrix_hfm.png
├── attention_visualization_samples.png
├── performance_comparison.png
└── per_class_performance.png📖 Citation
If you use this code or models in your research, please cite:
Paper Citation
1 @article{lam2025clara,
2 title={CLARA: Enhancing Multimodal Sentiment Analysis via Efficient Vision-Language Fusion},
3 author={Lam, Phuong and Phan, Tuoi Thi and Tran, Thien Khai},
4 journal={The Visual Computer},
5 year={2025},
6 publisher={Springer},
7 note={Under Review},
8 doi={10.5281/zenodo.17862924}
9 }
Code Citation
1 @software{lam2025clara_code,
2 title={CLARA: Implementation},
3 author={Lam, Phuong and Phan, Tuoi Thi and Tran, Thien Khai},
4 year={2025},
5 publisher={Zenodo},
6 version={v1.1.0},
7 doi={10.5281/zenodo.17889083},
8 url={https://github.com/phuonglamgithub/CLARA}
9 }
📞 Contact
Authors
Phuong Lam
Email: lamphuong.ict89@gmail.com
Affiliation: HUFLIT, Vietnam
Tuoi Thi Phan
Email: pttuoi@ntt.edu.vn
Affiliation: Nguyen Tat Thanh University, Vietnam
Thien Khai Tran (Corresponding Author)
Email: thientk@huit.edu.vn
Affiliation: HUIT, Vietnam
Support
For questions about the code or paper:
📄 License
This project is licensed under the MIT License - see the
LICENSE file for details.
🔗 Links & Resources
Repository & Archives
GitHub Repository: https://github.com/phuonglamgithub/CLARA Zenodo (All Versions): https://doi.org/10.5281/zenodo.17862924 (Concept DOI - always points to latest) Zenodo (v1.1.0 Specific): https://doi.org/10.5281/zenodo.17889083 (Version DOI - permanent link) Zenodo (v1.0.0 Specific): https://doi.org/10.5281/zenodo.17862925 (Initial release)
External Resources
Model Checkpoints: Google Drive (~3.3 GB)Datasets: Google Drive (~5.5 GB)Paper: Under review at The Visual Computer (Springer)
Documentation
Made with ❤️ by the CLARA Team
Parameter-Efficient • State-of-the-Art • Open Source
