Health/Science Misinformation Verdict Model (MuRIL, Hindi/Punjabi)
MuRIL fine-tuned to classify a Hindi/Punjabi health/science claim's
verdict: accurate, misleading, false, unverifiable. AutoScientist
Challenge Part 2, Science category.
Scope note
Deliberately excludes political/communal content. India's most
prominent Hindi fact-check archives (Alt News, Vishvas News) are
currently dominated by politically sensitive material (deepfakes of
politicians, protest-related claims) that's legally murky to
redistribute and inappropriate for a public training dataset. This
model stays to health/science claims only, sourced from BOOM Hindi
(confirmed via robots.txt to have no anthropic-ai block, unlike
Vishvas News, which explicitly disallows it) plus IndicCorpV2
health-domain text.
Results
Accuracy 56.0%, Macro-F1 0.298 (held-out 20% split, 50 rows).
| Class | Precision | Recall | F1 | Support |
|---|
| unverifiable | 0.65 | 0.58 | 0.61 | 26 |
| accurate | 0.48 | 0.72 | 0.58 | 18 |
| misleading | 0.00 | 0.00 | 0.00 | 4 |
| false | 0.00 | 0.00 | 0.00 | 2 |
Known limitation, disclosed honestly: misleading and false have
too few examples in this dataset for reliable classification. See the
source project's PART2_SUBMISSION.md for the full data-sourcing
writeup, including why only 8 of 248 rows are real verified
fact-checks rather than model-synthesized labels.
