You can deploy this model and start earning money today!

cs5242-hateful-memes/
hateful-memes-model

pytorch

hateful-meme-detection

openai/clip-vit-large-patch14-336

finetune

mit

Views

No views yet

Hateful Meme Detection — CS5242 Group 27 Checkpoints

Trained PyTorch checkpoints for the eight models described in the GitHub repository, plus a 56-point confounder-weighted BCE sweep across the seven frozen CLIP variants.

All checkpoints were trained on biecho/hateful_memes (Facebook Hateful Memes mirror, 8500-sample training set) with seed 42 on the openai/clip-vit-large-patch14-336 backbone.

Quick start

bash

# 1. clone the code repo
git clone https://github.com/biecho/hateful-meme-detection
cd hateful-meme-detection
conda env create -f envs/environment_linux.yml && conda activate hateful-memes
pip install -e .

# 2. dataset (~5 min) + features (~15 min on a single GPU)
python setup_data.py
python extract_features.py --clip-model openai/clip-vit-large-patch14-336

# 3. download all 8 baseline checkpoints (~70 MB) into results/
huggingface-cli download cs5242-hateful-memes/hateful-memes-model \
    --local-dir results/ --include "m1?/*" "m2/*"

# 4. evaluate every checkpoint on test_seen — populates results/m*/results.json
python eval_test.py

For the full 56-point confounder-weight sweep (~500 MB), drop the --include filter.

What's in this repo

Each subfolder is one training run from the unified python train.py --model <id> [--confounder-weight α] CLI:

m1a/best_model.pt          # baseline checkpoint per architecture
m1b/...                    # (8 dirs total: m1a, m1b, m1c, m1d, m1e, m1f, m1g, m2)
m2/best_model.pt           # 2 GB — fine-tuned CLIP + confounder-aware contrastive

m1a_cw2/, m1a_cw3/, ..., m1a_cw50/      # confounder-weight sweep
m1b_cw2/, ...                            # 7 alphas x 7 frozen models = 49 runs
                                         # plus cw=10 from a separate sweep -> 56 total

Each run dir contains:

best_model.pt — torch.save({"model_state_dict": ...}) checkpoint of the best-dev epoch, ready for model.load_state_dict(ckpt["model_state_dict"]).
config.json — exact CLI args used (seed, batch size, lr, etc.).
results.json — best_auroc (dev), final_auroc (dev), final_f1, final_accuracy, elapsed_seconds, plus test_auroc / test_accuracy / test_f1 from eval_test.py.

Headline numbers (single seed 42, ViT-L/14-336, test_seen)

Model	Architecture	Test AUROC
1a	Concat MLP (baseline)	0.8081
1b	Pooled × Pooled CrossAttn	0.7782
1c	Pooled × Token CrossAttn	0.7705
1d	Token × Token CrossAttn	0.7922
1e	Raw Multiply	0.7264
1f	Post-Projection Multiply	0.8180
1g	Pre-Projection Multiply	0.8347
2	Fine-tuned CLIP + Confounder Contrastive	0.8025
1g + cw=10	Pre-Proj + confounder-weighted BCE	0.8455

Architecture progressions

1b → 1c → 1d — pooling-before-attention ablation: pooling destroys token-level structure (1b/1c with pooled queries) versus full token-level attention (1d).
1e → 1f → 1g — fusion-space ablation: raw multiply on CLIP's aligned space (1e) ≪ learned projection on aligned space (1f) ≪ learned projection on raw encoder hidden states (1g).

Citation / authorship

CS5242 Neural Networks and Deep Learning (NUS, 2026), Group 27. Yu Araki · Umar Bin Moiz · Sherene Thai Shi Yin · Diego Meyer.

For methodology, evaluation details, and reproduction instructions, see the GitHub repo.