This is the non-ensemble deep model - the best single configuration from Team VANGUARD's submission to EXIST 2026 Task 2. It is a five-stream human-centered multimodal framework for sexism detection in memes that fuses textual, visual, demographic, and physiological modalities through a FiLM-conditioned cross-attention architecture.
Rather than predicting a single ground-truth label, the model is trained to learn from annotator disagreement, conditioning its predictions on who saw the meme and how they physically reacted to it. Meme text and visual descriptions are extracted by Gemma 4 (via Ollama). Text and image representations are produced by LoRA-adapted XLM-RoBERTa and CLIP ViT-B/32 encoders, fused via multi-head cross-attention, and then modulated by a 56-dimensional human-context vector (annotator demographics + physiological sensor embedding) via Feature-wise Linear Modulation (FiLM). Subtask 2.1 is framed as a label distribution learning problem using soft-label KL divergence over the full annotator distribution.
This configuration - no augmentation, no SVM ensemble - achieves the best overall validation and All-split test scores, outperforming every variant that adds augmentation or the ensemble.
Figure 1: Overview of the five-stream FiLM-conditioned cross-attention architecture. Five input streams, VLM-extracted embedded text, VLM-generated visual description, meme RGB image, annotator demographics, and physiological sensor vector, are fused through cross-attention and FiLM modulation into a shared 512-dimensional fused state, producing predictions for all three subtasks.
Files
File
Description
best_model.pt
Trained model weights (~1.4 GB). Contains LoRA adapters for both XLM-RoBERTa and CLIP, plus all fusion, FiLM, and task head parameters
config.json
Model hyperparameters and architecture configuration
best_thresholds.json
Per-subtask decision thresholds, grid-searched on validation to maximize macro F1
sensor_scaler.joblib
StandardScaler fitted on the training split for the 4-D physiological feature vector; must be applied before inference
Note:best_model.pt stores LoRA adapter weights and all custom components. The frozen backbones (FacebookAI/xlm-roberta-base and openai/clip-vit-base-patch32) are loaded separately from Hugging Face at inference time.
The model expects a 4-dimensional physiological feature vector per meme — [log_reaction_time, fixation_count, saccade_count, hr_std] — standardized using the StandardScaler fitted on the training split. Without real annotator biometric data, pass zeros directly; FiLM conditioning is a mathematical identity in this case since its weights are zero-initialized.
python
1import numpy as np
23# With real biometric data:4# raw = np.array([[log_reaction_time, fixation_count, saccade_count, hr_std]], dtype=np.float32)5# sensorial = torch.tensor(sensor_scaler.transform(raw), dtype=torch.float32).to(device)67# Without biometric data:8sensorial = torch.zeros(1,4).to(device)
Full inference example
For a complete inference example using the pipeline from the repository, see runnable/evaluate.py in the GitHub repo.
VLM enrichment (preprocessing)
Meme text and visual descriptions are extracted by Gemma 4 (gemma4:e4b via Ollama, temperature 0.0, top_p 0.1) before training and inference. This step is not embedded in the model weights, the repo's runnable/preprocess.py handles it.
Intended Use
Intended: Research on human-centered multimodal sexism detection; reproducing EXIST 2026 Task 2 results; studying label distribution learning and biometric conditioning in subjective NLP tasks.
Out of scope: Production content moderation without further safety validation; domains outside meme analysis; tasks other than EXIST 2026 Task 2 subtasks without fine-tuning.
Limitations
Physiological signals are meme-level averages - features are averaged across all 16 annotators per meme, discarding individual-level variation. A viewer-specific model could better capture subjectivity.
Sensor multicollinearity - fixation count and saccade count are nearly perfectly correlated (ρ ≈ 1.0), so the 4-D sensor vector likely carries less independent information than its dimensionality suggests.
EEG signals - not individually significant in statistical testing; they enter only through the classical SVM ensemble (not this model), leaving open whether richer temporal EEG modeling could unlock additional signal.
No single component survives multiple-comparison control - ablations over 5 seeds with Holm-Bonferroni correction across 63 tests find no individually statistically detectable benefit from FiLM conditioning, the image stream, or the contrastive loss. System gains appear to arise from integration rather than any single module. We report this openly.
English-Spanish gap - hard ICM-Norm on test is 0.5496 (EN) vs. 0.4306 (ES), likely because XLM-RoBERTa and CLIP pretraining better captures English idioms of online sexism.
Subtask 2.3 collapse - the 6-class fine-grained head collapses to a near-trivial solution under multi-task training; only single-task training on 2.3 breaks this pattern.
Citation
bibtex
1@InProceedings{Mocanu_2026_EXIST_CLEF,
2 author = {Mocanu, Ana-Maria Luisa and Mocanu, Sebastian and Truică, Ciprian-Octavian and Apostol, Elena-Simona},
3 title = {Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detection},
4 booktitle = {Conference and Labs of the Evaluation Forum (CLEF), EXIST 2026 Lab, Task 2},
5 month = {September},
6 year = {2026}
7}
Links
Resource
Link
Paper
WIP — will be updated when proceedings are published
