calnugget/

adversemed-500

AdverseMed-500 is a hand-verified false-premise medical multiple-choice benchmark for measuring miscalibration in medical language models. All 500 items embed a false clinical premise, and the correct action is to abstain / flag the premise, not to choose an A–D answer.

Paper: AdverseMed-500: A Hand-Verified False-Premise Benchmark for Measuring Miscalibration in Medical Language Models. Vallamsetty (2026). Preregistration: OSF osf.io/mehu4 Repo (code +… See the full description on the dataset page: https://huggingface.co/datasets/calnugget/adversemed-500.