| --- |
| language: |
| - mai |
| license: cc-by-4.0 |
| tags: |
| - maithili |
| - roberta |
| - pretrained-language-model |
| - low-resource |
| - indic |
| - devanagari |
| - maithili-bert |
| - fill-mask |
| datasets: |
| - AiventraLab/maithili-corpus |
| pipeline_tag: fill-mask |
| mask_token: "<mask>" |
| widget: |
| - text: "भगवान् <mask>क ध्यान" |
| - text: "मैथिली <mask> भाषा छी" |
| - text: "राम <mask> गेलाह" |
| - text: "हम <mask> जाइत छी" |
| --- |
| |
| # MaithiliBERT: First RoBERTa Model for the Maithili Language |
|
|
| **MaithiliBERT** is a **RoBERTa-base** language model (124M parameters) pretrained from scratch on **MaithiliCorpus** — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the **first RoBERTa** ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal. |
|
|
| ## Model Architecture |
|
|
| | Property | Value | |
| |----------|-------| |
| | Architecture | RoBERTa-base | |
| | Parameters | 124,493,648 | |
| | Vocabulary | Byte-level BPE, 50,000 tokens | |
| | Context Window | 512 tokens | |
| | Hidden Size | 768 | |
| | Intermediate Size | 3072 | |
| | Attention Heads | 12 | |
| | Hidden Layers | 12 | |
| | Training Data | MaithiliCorpus gold_deduped (47,882 documents, 39.8M words) | |
| | Training Steps | 14,210 (10 epochs) | |
| | Effective Batch Size | 32 | |
| | Optimizer | AdamW (lr=5e-4) | |
| | Precision | FP16 | |
| | License | CC-BY-4.0 | |
| |
| ## Tokenizer |
| |
| The tokenizer is a **byte-level BPE** trained from scratch on MaithiliCorpus with a vocabulary of **50,000 tokens** — 60% smaller than mBERT (119K) and 80% smaller than XLM-R/IndicBERTv2 (250K). This reduces embedding parameters from 192M to 38.4M, shrinking the embedding layer from 61% to 31% of total model parameters. |
| |
| | Property | Value | |
| |----------|-------| |
| | Algorithm | Byte-level BPE | |
| | Vocabulary Size | 50,000 | |
| | Training Data | MaithiliCorpus (39.8M words) | |
| | Training | SentencePiece, `--model_type=bpe` | |
| | Special Tokens | `<s>`, `</s>`, `<unk>`, `<pad>`, `<mask>` | |
| | UNK Rate (on Maithili) | 0.00% | |
|
|
| ## Acknowledgments |
|
|
| The authors gratefully acknowledge [AnK Komputing](https://ank-world.com) for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services. |
|
|
| ## Intended Use |
|
|
| - Masked language modeling (fill-mask) |
| - Fine-tuning for text classification, NER, and other NLU tasks |
| - Feature extraction for downstream Maithili NLP |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import pipeline |
| fill = pipeline("fill-mask", model="AiventraLab/maithili-roberta") |
| results = fill("भगवान् <mask>क ध्यान") |
| ``` |
|
|
