tojpaj/science-misinfo-model

Health/Science Misinformation Verdict Model (MuRIL, Hindi/Punjabi)

MuRIL fine-tuned to classify a Hindi/Punjabi health/science claim's verdict: accurate, misleading, false, unverifiable. AutoScientist Challenge Part 2, Science category.

Scope note

Deliberately excludes political/communal content. India's most prominent Hindi fact-check archives (Alt News, Vishvas News) are currently dominated by politically sensitive material (deepfakes of politicians, protest-related claims) that's legally murky to redistribute and inappropriate for a public training dataset. This model stays to health/science claims only, sourced from BOOM Hindi (confirmed via robots.txt to have no anthropic-ai block, unlike Vishvas News, which explicitly disallows it) plus IndicCorpV2 health-domain text.

Results

Accuracy 56.0%, Macro-F1 0.298 (held-out 20% split, 50 rows).

Class	Precision	Recall	F1	Support
unverifiable	0.65	0.58	0.61	26
accurate	0.48	0.72	0.58	18
misleading	0.00	0.00	0.00	4
false	0.00	0.00	0.00	2

Known limitation, disclosed honestly: misleading and false have too few examples in this dataset for reliable classification. See the source project's PART2_SUBMISSION.md for the full data-sourcing writeup, including why only 8 of 248 rows are real verified fact-checks rather than model-synthesized labels.