You can deploy this model and start earning money today!

FatimahEmadEldin/
imageeval2026-task2-craibench

Name: imageeval2026-task2-craibench
Availability: InStock

Views

No views yet

ImageEval 2026 · Task 2 — CRAI-Bench

System artefacts for CRAI-Bench: given a reference image of a Qatari cultural scene, the caption used to prompt a text-to-image model, and the generated image, predict five cultural accuracy dimensions.

CRAI_composite = 0.30*CEA + 0.20*CC + 0.20*CS + 0.20*CI - 0.10*HP

Ranking metric is Spearman on the composite; MAE is the tiebreaker.

Results — dev (n=40), fit on train, held out

system	Spearman ↑	MAE ↓
Organiser GPT-4 baseline (as reported)	0.6250	0.2248
GPT-4 raw, scoring the shipped `gold_llm.tsv`	0.2153	0.3666
Qwen3-VL-8B raw	0.5482	0.2550
Qwen3-VL-8B calibrated + version	0.7064	0.1960
version one-hot only — no image, no judge	0.6823	0.2091

Per-dimension Spearman for the submitted system: CEA 0.716, CC 0.711, CS 0.726, CI 0.730, HP -0.095.

Method

A VLM judge scores the five dimensions from (reference image, caption, generated image); a per-dimension ridge maps [5 raw dims + caption-version one-hot] to the human scores, fit on the 120 train instances. The composite is always recomputed with the official formula, never predicted.

Two facts drive this:

The organisers note GPT-4 "systematically overestimates cultural accuracy". It does, on every dimension — bias +0.142 (CEA) to +0.247 (CI), +0.182 on the composite.
Spearman is rank-based, so monotone recalibration cannot change it. Calibration buys MAE only; improving the ranking needs new signal. Caption version supplies it.

Caveats stated plainly

n = 40. Spearman's standard error is roughly ±0.16. Our 0.7064 versus the version-only control's 0.6823 is not a significant gap.
Train CV was optimistic: 0.8159 under 6-fold grouped CV versus 0.7064 on dev.
A model that never sees an image scores 0.6823. Caption version is close to a label — a benchmark design artefact that any honest system paper must report.
The organisers' GPT-4 baseline did not reproduce: their shipped gold_llm.tsv scores 0.2153 on dev, not the reported 0.6250.
Humans almost never apply a hallucination penalty (mean HP 0.008 vs GPT-4's 0.173), and GPT-4's HP ranking is anti-correlated with theirs (rho -0.114).

Data and licence

Built on the organisers' CRAI-Bench release, distributed via their Google Drive and subject to their terms; the task repository asks that data not be committed publicly. This repo contains no dataset content — no images, no captions, no gold labels. Contact: imageeval2026@gmail.com

FatimahEmadEldin/imageeval2026-task2-craibench