Test Report and Comments on IIT Roorkee - Modi Script AI Model
Arun Joshi, Indic-Scripts Research Forum
On July 18, 2025, IndiaToday.in announced “World’s first AI model cracks Modi script, Thanks to IIT Roorkee – A significant step in AI-driven heritage preservation, supporting Digital India, BharatGPT, and global cultural digitization initiatives”Our team at Indic-Scripts Research Forum (https://IndicScripts.in) has been building a dataset using Modi-script documents and Modi-transliterated Devanagari Marathi documents with the help of Bharat Itihas Sanshodhak Mandal’s contribution to the project. We were elated to read the above news. We applauded the efforts of the IIT Roorkee team for building a custom vision AI model and announcing their success globally.
We were also interested in deciphering their research report titled “Historic Scripts to Modern Vision: A Novel Dataset and A VLM Framework for Transliteration of Modi Script to Devanagari".
As part of their work, the authors curated a novel dataset, which they have made publicly accessible via the Hugging Face platform. Their proposed methodology involved the use of two AI models within a knowledge distillation framework: the first, a pre-trained model referred to as the teacher model, which was explicitly fine-tuned for the Modi transliteration task; and the second, referred to as the student model, that learns from the teacher model to enhance transliteration accuracy. Our AI team and collaborating Modi-script experts in Pune were excited to see both the models and their novel datasets. We found that the first teacher model has been released through a repository titled "historyHulk/ModiTrans-12B-Gemma-Teacher" on Hugging Face.
As this was the only AI model announced publicly as having cracked Modi-script to transliterate successfully into Devnagri script, our team decided to examine it by feeding it many Modi sentences. However, when the Gemma-based teacher model was used to infer images of handwritten Modi script sentences, the results were found to be largely inaccurate.
Non-technical users or beginners attempting to transliterate Modi into Devanagari would not achieve any success. They would thus believe that the announcement and the results are flawed due to poor model performance.
To align with the expected input format, we used Modi sentences from verified manuscripts. We tested each sentence image five times. As the announced model by IIT Roorkee is auto-regressive, we expected slightly different results in every run. However, in all five attempts per image, none of the Devnagari outputs were accurate or legible.
Below are a few results when the model was used for Devnagari transliteration using this model as available on the HuggingFace repository:
ModiTrans-12B-Gemma-Teacher Model Outputs:-
1. Input Modi Sentence
Devnagari Transliteration by Modi expert: दस्तक स्वारी राजश्री रामचद्र
AI Model tried five times to give these results:
- दिलाप्रतापचिं आंबेर नपट
- प्रायीन संमीनर यावर द
- दाहालप्राचीन सम्सीक करणप
- व्यास्तप्रस्थानभूईस सम्मील कवयते
- दालाप्रस्थान परिवर्तन स्वामी खड़े
2. Input Modi Sentence
Devnagari Transliteration: चौधरी किले पुरंधर व वज्रगड त||
AI Model tried five times to give these results:
- किसान आपला ग्राम देऊटेन गावात हजर आम्हांरा
- कर्मसी केले बाबत मनुयावर आनंदाचा वर्षाव होईल
- आ जाने मित्रास उतरदूर भाषाओं अगहा
- चाहे मृत्युमांपासून ह्या दस्तावेज प्रत्यक्ष लोकांना दाखवणावा
- बोद्धसौर रे शुरू गायीचे ठिकाणा
3. Input Modi Sentence
Devnagri Transliternation: कमाविसदारान व चोकीदारान व र-
AI Model tried five times to give these results:
- सेठी हे करून तयारी केली घरांकडे वेळ होत नाही तर आम्ही दूर जावे म्हणूनच
- अंत पुरुषनी चालनेपरित्या पितरा आल होत तत्कलापरोक्त
- कीद करही मजकूर केलेली यांचा करण्याची कार्यवाही करावें तरी संपादणीचे ताब्यास देऊ ती
- समुल्पानावरून ताहुली पुंजोंस प्रत्यक्ष वरिला पारकर जे
- सजलको यासुफअहे वडेसि जात आहे
4. Input Modi sentence
Devnagri Transliteration: हदारान व बाजै लोकान व मोकदमान
AI Model tried five times to give these results:
- व्यायामास बोथे सावकाश यदि बोलावे खेलने मोठे प्रजापतता
- तुम्ही चिंता करू नये तुमचेत्व आत्मप्रतिषेक करून घ्या
- मिळावयाराने तेथों जखमोप्रकरणे उद्यत करून मोठ
- जाहा जाय काय नावे पो हे साफे करार केला
- हपर चा बाणेये चक जीवन सीतने व्यंजन प्रमाण
5. Input Modi Sentence
Devnagri Transliteration: जकाती महालानिहाय सु।। सित
AI Model tried five times to give these results:
- नारायनराव सरकार यांची प्रकाशणी व विक्री
- स बाजीचा ताबा येतो हीनत चिल्हाडागो हाजिर
- यारबास व्यवस्थांनी विचार कराचीगाई
- तुमही त्याचप्रमाणे रविवार येईल तेंदिला
- व्हाया स्वामी चैतन्य लाखाटी
IIT Roorkee approach to building the MoDeTrans dataset, containing 2,043 images of Modi-script sentences along with their Devanagari transliterations, is questionable. It appears that 2043 Modi-script lines used as a dataset to train the model are insufficient, and probably because of that, the Teacher and Student AI models are not showing accurate transliteration.
The Indic Scripts Research Forum is dedicating the required time and effort to generate a paired words dataset using numerous historical books and documents, with the assistance of Modi experts.