Text Classification
Transformers
Safetensors
bert

Improve model card: Add pipeline tag, usage example, and project link

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +62 -146
README.md CHANGED
@@ -1,197 +1,113 @@
1
  ---
2
  library_name: transformers
3
- tags: []
 
 
 
 
 
 
4
  ---
5
 
6
- # Model Card for Model ID
7
 
8
- <!-- Provide a quick summary of what the model is/does. -->
9
 
10
- The model is a MuRIL based finetuned model on the language identification task. The model can identify English and all 22 official Indian languaages.
 
11
 
12
  ## Model Details
13
 
14
  ### Model Description
 
15
 
16
- <!-- Provide a longer summary of what this model is. -->
17
-
18
- This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
-
20
- - **Developed by:** Pruthwik Mishra, Yash Ingle
21
- - **Funded by [optional]:** SVNIT, Surat
22
- - **Shared by [optional]:** [More Information Needed]
23
- - **Model type:** [More Information Needed]
24
- - **Language(s) (NLP):** [More Information Needed]
25
- - **License:** MIT
26
- - **Finetuned from model [optional]:** google/muril-base-cased
27
-
28
- ### Model Sources [optional]
29
-
30
- <!-- Provide the basic links for the model. -->
31
-
32
- - **Repository:** [https://github.com/yashingle-ai/TextLangDetect]
33
- - **Paper [optional]:** [https://arxiv.org/abs/2507.11832]
34
- - **Demo [optional]:** [More Information Needed]
35
 
36
  ## Uses
37
 
38
- The model can be directly used for English and Indian language identification.
39
- ### Direct Use
40
-
41
- The model can be directly used for English and Indian language identification.
42
-
43
- ### Downstream Use [optional]
44
-
45
- Can be integrated into any pipeline that requires language identification for the concerned languages.
46
-
47
- [More Information Needed]
48
 
49
  ### Out-of-Scope Use
50
-
51
- The model may not work for languages other than English and Indian languages.
52
- [More Information Needed]
53
 
54
  ## Bias, Risks, and Limitations
55
-
56
- The model may not perform well on very resource poor languages such as Manipuri (in Meitei script), Sindhi, Maithili.
57
- [More Information Needed]
58
 
59
  ### Recommendations
60
-
61
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
62
-
63
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
64
 
65
  ## How to Get Started with the Model
66
 
67
- Use the code below to get started with the model.
68
-
69
- [More Information Needed]
70
 
71
- ## Training Details
 
 
72
 
73
- ### Training Data
 
 
 
74
 
75
- <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
 
76
 
77
- [More Information Needed]
 
78
 
79
- ### Training Procedure
 
80
 
81
- <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
82
 
83
- #### Preprocessing [optional]
 
 
84
 
85
- [More Information Needed]
 
86
 
 
 
87
 
88
- #### Training Hyperparameters
 
 
89
 
90
- - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
91
 
92
- #### Speeds, Sizes, Times [optional]
 
93
 
94
- 250K Dataset named ILID (Indian Language Identification Dataset)
95
- [More Information Needed]
96
 
97
  ## Evaluation
98
 
99
- The models are evaluated on the created corpora and the Bhasha-Abhijnaanam benchmark.
100
- ### Testing Data, Factors & Metrics
101
-
102
- #### Testing Data
103
-
104
- <!-- This should link to a Dataset Card if possible. -->
105
-
106
- [More Information Needed]
107
 
108
- #### Factors
109
-
110
- <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
111
-
112
- [More Information Needed]
113
-
114
- #### Metrics
115
-
116
- F1-score
117
- [More Information Needed]
118
 
119
  ### Results
 
120
 
121
- 0.96 F1 on an average
122
- #### Summary
123
-
124
-
125
-
126
- ## Model Examination [optional]
127
-
128
- <!-- Relevant interpretability work for the model goes here -->
129
-
130
- [More Information Needed]
131
-
132
- ## Environmental Impact
133
-
134
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
135
-
136
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
137
-
138
- - **Hardware Type:** [More Information Needed]
139
- - **Hours used:** [More Information Needed]
140
- - **Cloud Provider:** [More Information Needed]
141
- - **Compute Region:** [More Information Needed]
142
- - **Carbon Emitted:** [More Information Needed]
143
-
144
- ## Technical Specifications [optional]
145
-
146
- ### Model Architecture and Objective
147
-
148
- [More Information Needed]
149
-
150
- ### Compute Infrastructure
151
-
152
- [More Information Needed]
153
-
154
- #### Hardware
155
-
156
- [More Information Needed]
157
-
158
- #### Software
159
-
160
- [More Information Needed]
161
-
162
- ## Citation [optional]
163
-
164
- <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
165
-
166
- **BibTeX:**
167
 
 
168
  @misc{ingle2025ilidnativescriptlanguage,
169
- title={ILID: Native Script Language Identification for Indian Languages},
170
  author={Yash Ingle and Pruthwik Mishra},
171
  year={2025},
172
  eprint={2507.11832},
173
  archivePrefix={arXiv},
174
  primaryClass={cs.CL},
175
- url={https://arxiv.org/abs/2507.11832},
176
  }
177
- **APA:**
178
-
179
- [More Information Needed]
180
-
181
- ## Glossary [optional]
182
-
183
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
184
-
185
- [More Information Needed]
186
-
187
- ## More Information [optional]
188
-
189
- [More Information Needed]
190
-
191
- ## Model Card Authors [optional]
192
-
193
- [More Information Needed]
194
-
195
- ## Model Card Contact
196
-
197
- [More Information Needed]
 
1
  ---
2
  library_name: transformers
3
+ license: mit
4
+ pipeline_tag: text-classification
5
+ tags:
6
+ - language-identification
7
+ - indian-languages
8
+ - multilingual
9
+ - muril
10
  ---
11
 
12
+ # ILID: Native Script Language Identification for Indian Languages
13
 
14
+ The model `yashingle-ai/ILID` is a MuRIL-based model fine-tuned for the language identification task, capable of identifying English and all 22 official Indian languages. It was presented in the paper [ILID: Native Script Language Identification for Indian Languages](https://huggingface.co/papers/2507.11832).
15
 
16
+ **Project Page:** [https://yashingle-ai.github.io/ILID/](https://yashingle-ai.github.io/ILID/)
17
+ **Code Repository:** [https://github.com/yashingle-ai/TextLangDetect](https://github.com/yashingle-ai/TextLangDetect)
18
 
19
  ## Model Details
20
 
21
  ### Model Description
22
+ This model is a fine-tuned version of `google/muril-base-cased` on the language identification task, capable of distinguishing between English and all 22 official Indian languages. It addresses the challenges of distinguishing languages in noisy, short, and code-mixed environments, particularly relevant for the diverse linguistic landscape of India.
23
 
24
+ - **Developed by:** Pruthwik Mishra, Yash Ingle
25
+ - **Model type:** BERT-based for Sequence Classification
26
+ - **Language(s) (NLP):** Multilingual (22 official Indian languages + English)
27
+ - **Finetuned from model:** `google/muril-base-cased`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
 
29
  ## Uses
30
 
31
+ The model can be directly used for English and Indian language identification, serving as a preprocessing step for applications like multilingual machine translation, information retrieval, and question answering.
 
 
 
 
 
 
 
 
 
32
 
33
  ### Out-of-Scope Use
34
+ The model is not designed for languages other than English and the official Indian languages. Performance may vary on very low-resource Indian languages.
 
 
35
 
36
  ## Bias, Risks, and Limitations
37
+ The model may not perform optimally on very resource-poor languages such as Manipuri (in Meitei script), Sindhi, or Maithili, as highlighted in the original paper.
 
 
38
 
39
  ### Recommendations
40
+ Users should be aware of the model's limitations and potential biases when applying it to specific use cases or languages outside its primary training scope.
 
 
 
41
 
42
  ## How to Get Started with the Model
43
 
44
+ You can use the model directly with the `transformers` library for text classification:
 
 
45
 
46
+ ```python
47
+ from transformers import AutoTokenizer, AutoModelForSequenceClassification
48
+ import torch
49
 
50
+ # Load model and tokenizer
51
+ model_name = "yashingle-ai/ILID" # Replace with actual model ID if different
52
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
53
+ model = AutoModelForSequenceClassification.from_pretrained(model_name)
54
 
55
+ # Example usage (Hindi text)
56
+ text_to_classify_hi = "\u0928\u092e\u0938\u094d\u0924\u0947, \u092f\u0939 \u090f\u0915 \u092a\u0930\u0940\u0915\u094d\u0937\u0923 \u0935\u093e\u0915\u094d\u092f \u0939\u0948\u0964"
57
+ inputs_hi = tokenizer(text_to_classify_hi, return_tensors="pt")
58
 
59
+ with torch.no_grad():
60
+ logits_hi = model(**inputs_hi).logits
61
 
62
+ predicted_class_id_hi = logits_hi.argmax().item()
63
+ predicted_label_hi = model.config.id2label[predicted_class_id_hi] # Maps to LABEL_X
64
 
65
+ print(f"Text: '{text_to_classify_hi}'")
66
+ print(f"Predicted language label: {predicted_label_hi}")
67
 
68
+ # Example usage (English text)
69
+ text_to_classify_en = "Hello, this is a test sentence."
70
+ inputs_en = tokenizer(text_to_classify_en, return_tensors="pt")
71
 
72
+ with torch.no_grad():
73
+ logits_en = model(**inputs_en).logits
74
 
75
+ predicted_class_id_en = logits_en.argmax().item()
76
+ predicted_label_en = model.config.id2label[predicted_class_id_en]
77
 
78
+ print(f"Text: '{text_to_classify_en}'")
79
+ print(f"Predicted language label: {predicted_label_en}")
80
+ ```
81
 
82
+ ## Training Details
83
 
84
+ ### Training Data
85
+ The model was trained on the newly created ILID (Indian Language Identification Dataset).
86
 
87
+ ### Training Hyperparameters
88
+ The model was fine-tuned from `google/muril-base-cased`.
89
 
90
  ## Evaluation
91
 
92
+ The model was evaluated on the created ILID corpus and the Bhasha-Abhijnaanam benchmark.
 
 
 
 
 
 
 
93
 
94
+ ### Metrics
95
+ The primary evaluation metric used was F1-score.
 
 
 
 
 
 
 
 
96
 
97
  ### Results
98
+ The model achieved an average F1-score of 0.96.
99
 
100
+ ## Citation
101
+ If you find our work helpful or inspiring, please feel free to cite it.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
102
 
103
+ ```bibtex
104
  @misc{ingle2025ilidnativescriptlanguage,
105
+ title={ILID: Native Script Language Identification for Indian Languages},
106
  author={Yash Ingle and Pruthwik Mishra},
107
  year={2025},
108
  eprint={2507.11832},
109
  archivePrefix={arXiv},
110
  primaryClass={cs.CL},
111
+ url={https://arxiv.org/abs/2507.11832},
112
  }
113
+ ```