README.md

---
library_name: transformers
license: mit
pipeline_tag: text-classification
tags:
  - language-identification
  - indian-languages
  - multilingual
  - muril
---

# ILID: Native Script Language Identification for Indian Languages

The model `yashingle-ai/ILID` is a MuRIL-based model fine-tuned for the language identification task, capable of identifying English and all 22 official Indian languages. It was presented in the paper [ILID: Native Script Language Identification for Indian Languages](https://huggingface.co/papers/2507.11832).

**Project Page:** [https://yashingle-ai.github.io/ILID/](https://yashingle-ai.github.io/ILID/)
**Code Repository:** [https://github.com/yashingle-ai/TextLangDetect](https://github.com/yashingle-ai/TextLangDetect)

## Model Details

### Model Description
This model is a fine-tuned version of `google/muril-base-cased` on the language identification task, capable of distinguishing between English and all 22 official Indian languages. It addresses the challenges of distinguishing languages in noisy, short, and code-mixed environments, particularly relevant for the diverse linguistic landscape of India.

-   **Developed by:** Pruthwik Mishra, Yash Ingle
-   **Model type:** BERT-based for Sequence Classification
-   **Language(s) (NLP):** Multilingual (22 official Indian languages + English)
-   **Finetuned from model:** `google/muril-base-cased`

## Uses

The model can be directly used for English and Indian language identification, serving as a preprocessing step for applications like multilingual machine translation, information retrieval, and question answering.

### Out-of-Scope Use
The model is not designed for languages other than English and the official Indian languages. Performance may vary on very low-resource Indian languages.

## Bias, Risks, and Limitations
The model may not perform optimally on very resource-poor languages such as Manipuri (in Meitei script), Sindhi, or Maithili, as highlighted in the original paper.

### Recommendations
Users should be aware of the model's limitations and potential biases when applying it to specific use cases or languages outside its primary training scope.

## How to Get Started with the Model

You can use the model directly with the `transformers` library for text classification:

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "yashingle-ai/ILID" # Replace with actual model ID if different
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example usage (Hindi text)
text_to_classify_hi = "\u0928\u092e\u0938\u094d\u0924\u0947, \u092f\u0939 \u090f\u0915 \u092a\u0930\u0940\u0915\u094d\u0937\u0923 \u0935\u093e\u0915\u094d\u092f \u0939\u0948\u0964"
inputs_hi = tokenizer(text_to_classify_hi, return_tensors="pt")

with torch.no_grad():
    logits_hi = model(**inputs_hi).logits

predicted_class_id_hi = logits_hi.argmax().item()
predicted_label_hi = model.config.id2label[predicted_class_id_hi] # Maps to LABEL_X

print(f"Text: '{text_to_classify_hi}'")
print(f"Predicted language label: {predicted_label_hi}")

# Example usage (English text)
text_to_classify_en = "Hello, this is a test sentence."
inputs_en = tokenizer(text_to_classify_en, return_tensors="pt")

with torch.no_grad():
    logits_en = model(**inputs_en).logits

predicted_class_id_en = logits_en.argmax().item()
predicted_label_en = model.config.id2label[predicted_class_id_en]

print(f"Text: '{text_to_classify_en}'")
print(f"Predicted language label: {predicted_label_en}")
```

## Training Details

### Training Data
The model was trained on the newly created ILID (Indian Language Identification Dataset).

### Training Hyperparameters
The model was fine-tuned from `google/muril-base-cased`.

## Evaluation

The model was evaluated on the created ILID corpus and the Bhasha-Abhijnaanam benchmark.

### Metrics
The primary evaluation metric used was F1-score.

### Results
The model achieved an average F1-score of 0.96.

## Citation
If you find our work helpful or inspiring, please feel free to cite it.

```bibtex
@misc{ingle2025ilidnativescriptlanguage,
      title={ILID: Native Script Language Identification for Indian Languages},
      author={Yash Ingle and Pruthwik Mishra},
      year={2025},
      eprint={2507.11832},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11832},
}
```