Typed in Roman, known in Devanagari: probes and results

Entry for the Lexsi AI Safety and Alignment Hackathon, Track 2b, "Hallucination detection and abstention". Model studied: CohereLabs/tiny-aya-global (3.35B), no fine-tuning.

Code, data and the full write-up: https://github.com/aryan-gupta7/tinyaya-script-router

What this repo holds

Linear correctness probes (logistic regression on the last-prompt-token hidden state, layers 19 and 31, corrected Roman view), result JSON files, summary.md with every table, and the figures. The same files are in the GitHub repo under probes/ and results/.

Headline results (2,400 Mintaka Hindi questions, 1,281 question groups)

  • Devanagari 15.6% against Roman 10.1% accuracy (exact McNemar p = 2e-12; 248 questions right only in Devanagari, 115 only in Roman). The gap exceeds the typo-noise gap by 3.8 points (95% interval 2.1 to 5.2, group bootstrap). On non-yes/no questions: 14.9% against 6.3%.
  • Out-of-fold probe AUROC for predicting a wrong Roman answer: 0.84 against 0.80 for first-token entropy on all questions, but entropy is better on non-yes/no questions (0.87 against 0.82). We claim comparable, not better.
  • Router (answer, rewrite to Devanagari, abstain): no better than an entropy gate or random routing at matched coverage. Self-rewrite lifts accuracy from 10.1% to 11.0% at about three times the tokens. Negative result.
  • 150 questions typed by teammates from the English question scored 22.0% (Devanagari 12.7% on the same questions): machine romanization is not the same as real typing.

Limits

Low closed-book accuracy, possible contamination, one language and one model, answers scored by a strict string scorer that was checked only by the AI assistant (3 correct sentence-style answers were under-scored in a 90-answer read-through), probes are correlational.

Licence

Tiny Aya is CC-BY-NC with an acceptable use policy; keep use non-commercial. Mintaka is CC-BY-4.0.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including s1nn3rx69/script-router-probes