You can deploy this model and start earning money today!

EsferSami/
meme-sentiment-dual-cross-attention-v1

Name: meme-sentiment-dual-cross-attention-v1
Availability: InStock

Views

No views yet

Meme Sentiment Analysis — Dual Cross-Attention (V1)

Author: EsferSami
Task: Multilingual Multimodal Meme Sentiment Analysis
Dataset: MemoSen — Kaggle
Version: V1 — Frozen Encoders (Pure Pretrained Baseline)

Overview

MemoSen is a multilingual multimodal meme sentiment dataset containing Bengali, English, and code-mixed meme captions paired with meme images. Each sample has a sentiment label: Positive, Negative, or Neutral.

This model resolves cross-modal conflict between meme text and image using a dual cross-attention framework. Meme understanding is challenging because text and image often conflict — especially in sarcastic, ironic, or culturally specific memes.

Model Architecture

Component	Details
Text Encoder	XLM-RoBERTa base (frozen)
Image Encoder	EVA02-CLIP-L/14 from QuanSun (frozen)
Feature Dim	768 projected to 512
Fusion	Dual Cross-Attention + Adaptive Gating
Conflict Detection	Cosine similarity + MLP
Reliability Weighting	Per-modality sigmoid scoring
Classifier	MLP (512 -> 256 -> 128 -> 3)

Dataset

Dataset: MemoSen on Kaggle

Split	Ratio
Train	70%
Validation	15%
Test	15%

Classes: Positive, Negative, Neutral
Languages: Bengali, English, Code-mixed (Banglish)

Training Configuration

Parameter	Value
Epochs	50 (early stopped at 26)
Optimizer	AdamW (lr=1e-4)
Scheduler	CosineAnnealingLR
Loss	Focal Loss + Class Weights
Early Stopping	Patience = 7
Batch Size	32
Max Text Length	128 tokens
Image Size	224 x 224
GPU	T4 (Kaggle)

Results (Validation Set)

Metric	Score
Macro-F1	0.5584

Full test set metrics will be updated after evaluation.
Main metric: Macro-F1

Training Curves

Notes

V1 uses fully frozen encoders — only fusion and classification layers are trained
Trainable parameters are limited to adaptation, cross-attention, conflict detection, reliability weighting, gated fusion, and MLP classifier
V2 will experiment with unfreezing encoders and architecture modifications

Citation

If you use MemoSen dataset, please refer to the original dataset on Kaggle:
https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset

EsferSami/meme-sentiment-dual-cross-attention-v1