File size: 2,793 Bytes
98cbd91 daa0a88 43c2549 98cbd91 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
language:
- mai
license: cc-by-4.0
tags:
- maithili
- roberta
- pretrained-language-model
- low-resource
- indic
- devanagari
- maithili-bert
- fill-mask
datasets:
- AiventraLab/maithili-corpus
pipeline_tag: fill-mask
mask_token: "<mask>"
widget:
- text: "भगवान् <mask>क ध्यान"
- text: "मैथिली <mask> भाषा छी"
- text: "राम <mask> गेलाह"
- text: "हम <mask> जाइत छी"
---
# MaithiliBERT: First RoBERTa Model for the Maithili Language
**MaithiliBERT** is a **RoBERTa-base** language model (124M parameters) pretrained from scratch on **MaithiliCorpus** — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the **first RoBERTa** ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.
## Model Architecture
| Property | Value |
|----------|-------|
| Architecture | RoBERTa-base |
| Parameters | 124,493,648 |
| Vocabulary | Byte-level BPE, 50,000 tokens |
| Context Window | 512 tokens |
| Hidden Size | 768 |
| Intermediate Size | 3072 |
| Attention Heads | 12 |
| Hidden Layers | 12 |
| Training Data | MaithiliCorpus gold_deduped (47,882 documents, 39.8M words) |
| Training Steps | 14,210 (10 epochs) |
| Effective Batch Size | 32 |
| Optimizer | AdamW (lr=5e-4) |
| Precision | FP16 |
| License | CC-BY-4.0 |
## Tokenizer
The tokenizer is a **byte-level BPE** trained from scratch on MaithiliCorpus with a vocabulary of **50,000 tokens** — 60% smaller than mBERT (119K) and 80% smaller than XLM-R/IndicBERTv2 (250K). This reduces embedding parameters from 192M to 38.4M, shrinking the embedding layer from 61% to 31% of total model parameters.
| Property | Value |
|----------|-------|
| Algorithm | Byte-level BPE |
| Vocabulary Size | 50,000 |
| Training Data | MaithiliCorpus (39.8M words) |
| Training | SentencePiece, `--model_type=bpe` |
| Special Tokens | `<s>`, `</s>`, `<unk>`, `<pad>`, `<mask>` |
| UNK Rate (on Maithili) | 0.00% |
## Acknowledgments
The authors gratefully acknowledge [AnK Komputing](https://ank-world.com) for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.
## Intended Use
- Masked language modeling (fill-mask)
- Fine-tuning for text classification, NER, and other NLU tasks
- Feature extraction for downstream Maithili NLP
## Usage
```python
from transformers import pipeline
fill = pipeline("fill-mask", model="AiventraLab/maithili-roberta")
results = fill("भगवान् <mask>क ध्यान")
```
|