You can deploy this model and start earning money today!

Amirhossein75/
siglip2-base-mmhs150k-fusion

Name: siglip2-base-mmhs150k-fusion
Availability: InStock

Views

No views yet

SigLIP2-Base Fusion Model for Multi-Modal Hate Speech Detection

PyTorch

Transformers

License

Dataset

A PyTorch-based multi-modal (image + text) hateful content classification model using SigLIP2 encoder with late fusion architecture, trained on the MMHS150K dataset for detecting hate speech in social media memes and posts.

🎯 Model Description

This model implements a late fusion architecture with gated attention mechanism for detecting hateful content in social media memes and posts. It combines visual and textual features using Google's SigLIP2 (Base-Patch16-224) as the backbone encoder.

The model performs multi-label classification across 5 hate speech categories, making it capable of detecting multiple types of hate in a single post (e.g., content that is both racist and sexist).

🏗️ Architecture

┌─────────────┐     ┌─────────────┐
│   Image     │     │    Text     │
│   Encoder   │     │   Encoder   │
│  (SigLIP2)  │     │  (SigLIP2)  │
└──────┬──────┘     └──────┬──────┘
       │                   │
       ▼                   ▼
┌─────────────┐     ┌─────────────┐
│  Projection │     │  Projection │
│   (Linear)  │     │   (Linear)  │
└──────┬──────┘     └──────┬──────┘
       │                   │
       └─────────┬─────────┘
                 │
                 ▼
         ┌─────────────┐
         │ Gated Fusion│◄── Modality presence flags
         │   Module    │    (handles missing modalities)
         └──────┬──────┘
                │
                ▼
    ┌───────────────────────┐
    │  Interaction Features │
    │  • Fused embedding    │
    │  • Text embedding     │
    │  • Visual embedding   │
    │  • |text - visual|    │
    │  • text ⊙ visual      │
    └───────────┬───────────┘
                │
                ▼
         ┌─────────────┐
         │Classification│
         │  Head (MLP)  │
         │  → 5 classes │
         └─────────────┘

🔑 Key Features

Feature	Description
Backbone	`google/siglip2-base-patch16-224` - Pre-trained SigLIP2 model
Fusion Dimension	512
Max Text Length	64 tokens
Multi-label Output	5 hate speech categories
Gated Attention	Modality-aware fusion with learnable gates
Interaction Features	Rich feature interactions (concatenation, element-wise product, absolute difference)
Missing Modality Handling	Can handle text-only or image-only inputs

🏷️ Output Classes

Index	Class	Description	Dataset Prevalence
0	Racist	Racist content targeting race/ethnicity	32.6%
1	Sexist	Sexist content targeting gender	12.0%
2	Homophobe	Homophobic content targeting sexual orientation	7.6%
3	Religion	Religion-based hate speech	1.5%
4	OtherHate	Other types of hate speech	15.6%

📊 Evaluation Results

Test Set Performance

Metric	Score
F1 Macro	0.507
F1 Micro	0.610
ROC-AUC Macro	0.774
Test Loss	1.530
Throughput	236.3 samples/sec

Per-Class Performance (Validation Set)

Class	Precision	Recall	F1-Score	Support
Racist	0.534	0.874	0.663	1,994
Sexist	0.620	0.585	0.602	875
Homophobe	0.818	0.632	0.713	612
Religion	0.118	0.140	0.128	129
OtherHate	0.479	0.707	0.571	1,195
Micro Avg	0.541	0.729	0.621	4,805
Macro Avg	0.514	0.588	0.535	4,805

⚙️ Optimized Thresholds

The model uses per-class calibrated thresholds for optimal performance (instead of default 0.5):

Class	Threshold
Racist	0.30
Sexist	0.75
Homophobe	0.85
Religion	0.20
OtherHate	0.55

📈 Model Comparison

Model	F1 Macro	F1 Micro	ROC-AUC	Throughput
CLIP Fusion	0.566	0.635	0.783	381.5
CLIP MTL	0.569	0.644	0.783	390.9
SigLIP Fusion (this model)	0.507	0.610	0.774	236.3
CLIP Fusion (Weighted Sampling)	0.557	0.636	0.772	266.4
CLIP Fusion (Bigger Batch)	0.515	0.517	0.804	400.9

🎓 Training Data

MMHS150K Dataset

The model was trained on the MMHS150K (Multi-Modal Hate Speech) dataset, a large-scale multi-modal hate speech dataset collected from Twitter containing 150,000 tweet-image pairs annotated for hate speech detection.

Attribute	Value
Source	Twitter
Total Samples	~150,000
Modalities	Image + Text
Annotation	Multi-label (5 hate categories)
Language	English

Dataset Splits

Split	Samples
Train	~112,500
Validation	~15,000
Test	~22,500

Dataset Reference

Paper: "Exploring Hate Speech Detection in Multimodal Publications" (WACV 2020) Authors: Raul Gomez, Jaume Gibert, Lluis Gomez, Dimosthenis Karatzas

🔧 Training Procedure

Training Configuration

yaml

# Model Configuration
backend: siglip
head: fusion
encoder_name: google/siglip2-base-patch16-224
fusion_dim: 512
max_text_length: 64
freeze_text: false
freeze_image: false

# Training Configuration
num_train_epochs: 6
per_device_train_batch_size: 32
per_device_eval_batch_size: 64
gradient_accumulation_steps: 2

# Learning Rates (Differential)
lr_encoder: 1.0e-5
lr_head: 5.0e-4

# Regularization
weight_decay: 0.02
max_grad_norm: 1.0

# Scheduler
warmup_ratio: 0.05
lr_scheduler_type: cosine

# Loss
loss_type: bce
use_logit_adjustment: false

# Precision
precision: fp16

# Data Augmentation
augment: true
aug_scale_min: 0.8
aug_scale_max: 1.0
horizontal_flip: true
color_jitter: true

# Early Stopping
early_stopping_patience: 3
metric_for_best_model: roc_macro

Training Highlights

Differential Learning Rates: Encoder (1e-5) vs Classification Head (5e-4)
Mixed Precision: FP16 training for efficiency
Data Augmentation: Random scaling, horizontal flip, color jitter
Threshold Calibration: Per-class threshold optimization on validation set
Early Stopping: Patience of 3 epochs based on ROC-AUC macro
Best Checkpoint: Selected based on validation ROC-AUC macro score

Computational Resources

Training Time: ~6 epochs
Best Checkpoint: Step 30,000
Hardware: GPU with FP16 support

🚀 How to Use

Installation

bash

# Clone the training repository
git clone https://github.com/amirhossein-yousefi/multimodal-content-moderation.git
cd multimodal-content-moderation

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt
pip install -e .

Quick Inference with trust_remote_code

python

from transformers import AutoModel, AutoProcessor
from PIL import Image
import torch

# Load model with trust_remote_code
model = AutoModel.from_pretrained(
    "Amirhossein75/siglip2-base-mmhs150k-fusion",
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

# Prepare inputs
image = Image.open("path/to/image.jpg")
text = "sample text from the meme"

inputs = processor(
    text=[text],
    images=[image],
    return_tensors="pt",
    padding=True,
    truncation=True
)

# Inference
model.eval()
with torch.no_grad():
    outputs = model(**inputs)
    probabilities = torch.sigmoid(outputs["logits"])

# Apply optimized thresholds
thresholds = torch.tensor([0.30, 0.75, 0.85, 0.20, 0.55])
predictions = (probabilities > thresholds).int()

class_names = ["racist", "sexist", "homophobe", "religion", "otherhate"]
for i, name in enumerate(class_names):
    print(f"{name}: {bool(predictions[0, i])} (prob: {probabilities[0, i]:.3f})")

Using the predict() Method

python

from transformers import AutoModel, AutoProcessor
from PIL import Image
import torch

# Load model
model = AutoModel.from_pretrained(
    "Amirhossein75/siglip2-base-mmhs150k-fusion",
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

# Prepare inputs
image = Image.open("path/to/image.jpg")
text = "sample text from the meme"

inputs = processor(
    text=[text],
    images=[image],
    return_tensors="pt",
    padding=True,
    truncation=True
)

# Use built-in predict method with calibrated thresholds
result = model.predict(**inputs)

print(result)
# {'predictions': {'racist': False, 'sexist': True, 'homophobe': False, 'religion': False, 'otherhate': False},
#  'probabilities': {'racist': 0.12, 'sexist': 0.78, ...}}

Batch Inference

python

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

# Load model
model = AutoModel.from_pretrained(
    "Amirhossein75/siglip2-base-mmhs150k-fusion",
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

# Prepare batch
images = [Image.open("image1.jpg"), Image.open("image2.jpg")]
texts = ["text for image 1", "text for image 2"]

inputs = processor(
    text=texts,
    images=images,
    return_tensors="pt",
    padding=True,
    truncation=True
)

# Inference
with torch.no_grad():
    outputs = model(**inputs)
    probabilities = torch.sigmoid(outputs["logits"])

# Apply optimized thresholds
thresholds = torch.tensor([0.30, 0.75, 0.85, 0.20, 0.55])
predictions = (probabilities > thresholds).int()

Using with Hugging Face Pipeline (Custom)

python

from huggingface_hub import hf_hub_download
import json

# Download config
config_path = hf_hub_download(
    repo_id="Amirhossein75/siglip2-base-mmhs150k-fusion",
    filename="inference_config.json"
)

with open(config_path) as f:
    config = json.load(f)

print(f"Classes: {config['class_names']}")
print(f"Thresholds: {config['thresholds']}")

📁 Model Files

File	Description
`model.safetensors`	Model weights in safetensors format
`config.json`	Model architecture configuration
`modeling_siglip_fusion.py`	Custom model class for trust_remote_code
`inference_config.json`	Inference settings with thresholds and class names
`label_map.json`	Label name mapping
`test_metrics.json`	Test set evaluation metrics
`val_report.json`	Detailed validation classification report

☁️ AWS SageMaker Deployment

This model is compatible with AWS SageMaker for cloud deployment:

python

from sagemaker.pytorch import PyTorchModel
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

model = PyTorchModel(
    model_data="s3://your-bucket/model.tar.gz",
    role=role,
    entry_point='inference.py',
    source_dir='sagemaker',
    framework_version='2.1.0',
    py_version='py310',
)

predictor = model.deploy(
    instance_type='ml.g4dn.xlarge',
    initial_instance_count=1,
)

# Make prediction
import base64
with open('image.jpg', 'rb') as f:
    image_b64 = base64.b64encode(f.read()).decode('utf-8')

response = predictor.predict({
    'instances': [{
        'text': 'Sample text content',
        'image_base64': image_b64,
    }]
})

See the SageMaker documentation for full deployment guide.

⚠️ Intended Uses & Limitations

✅ Intended Uses

Content moderation for social media platforms
Detecting hateful memes and posts
Research in multi-modal hate speech detection
Building content safety systems
Pre-filtering potentially harmful content for human review

⚠️ Limitations

Limitation	Description
Language	Trained only on English content
Domain	Twitter-specific; may not generalize to other platforms
Class Imbalance	Lower performance on rare categories (Religion: F1=0.128)
Cultural Context	May miss culturally-specific hate speech
Sarcasm/Irony	May struggle with subtle or ironic hateful content
Image-only Hate	Text encoder is important; purely visual hate may be missed

❌ Out-of-Scope Uses

NOT for making final moderation decisions without human review
NOT suitable for legal or compliance purposes without additional validation
NOT for censorship or suppression of legitimate speech
NOT for targeting or profiling individuals

🛡️ Ethical Considerations

This model should be used as a tool to assist human moderators, not replace them
False positives may incorrectly flag legitimate content
False negatives may miss harmful content
Regular evaluation and bias auditing is recommended
Consider the cultural and contextual factors in deployment

📝 Citation

If you use this model, please cite:

bibtex

@misc{yousefi2024multimodal,
  title={Multi-Modal Hateful Content Classification with SigLIP2 Fusion},
  author={Yousefi, Amirhossein},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/Amirhossein75/siglip2-base-mmhs150k-fusion}
}

Dataset Citation

bibtex

@inproceedings{gomez2020exploring,
  title={Exploring Hate Speech Detection in Multimodal Publications},
  author={Gomez, Raul and Gibert, Jaume and Gomez, Lluis and Karatzas, Dimosthenis},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  pages={1470--1478},
  year={2020}
}

SigLIP Citation

bibtex

@article{zhai2023sigmoid,
  title={Sigmoid Loss for Language Image Pre-Training},
  author={Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas},
  journal={arXiv preprint arXiv:2303.15343},
  year={2023}
}

🔗 Links

Resource	Link
GitHub Repository	multimodal-content-moderation
Base Model	google/siglip2-base-patch16-224
CLIP Fusion Model	Amirhossein75/clip-vit-base-mmhs150k-fusion
MMHS150K Dataset	Official Page
SigLIP Paper	arXiv

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please see the GitHub repository for contribution guidelines.