---
language:
- mai
license: cc-by-4.0
tags:
- maithili
- roberta
- pretrained-language-model
- low-resource
- indic
- devanagari
- maithili-bert
- fill-mask
datasets:
- AiventraLab/maithili-corpus
pipeline_tag: fill-mask
mask_token: "<mask>"
widget:
- text: "भगवान् <mask>क ध्यान"
- text: "मैथिली <mask> भाषा छी"
- text: "राम <mask> गेलाह"
- text: "हम <mask> जाइत छी"
---

# MaithiliBERT: First RoBERTa Model for the Maithili Language

**MaithiliBERT** is a **RoBERTa-base** language model (124M parameters) pretrained from scratch on **MaithiliCorpus** — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the **first RoBERTa** ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.

## Model Architecture

| Property | Value |
|----------|-------|
| Architecture | RoBERTa-base |
| Parameters | 124,493,648 |
| Vocabulary | Byte-level BPE, 50,000 tokens |
| Context Window | 512 tokens |
| Hidden Size | 768 |
| Intermediate Size | 3072 |
| Attention Heads | 12 |
| Hidden Layers | 12 |
| Training Data | MaithiliCorpus gold_deduped (47,882 documents, 39.8M words) |
| Training Steps | 14,210 (10 epochs) |
| Effective Batch Size | 32 |
| Optimizer | AdamW (lr=5e-4) |
| Precision | FP16 |
| License | CC-BY-4.0 |

## Tokenizer

The tokenizer is a **byte-level BPE** trained from scratch on MaithiliCorpus with a vocabulary of **50,000 tokens** — 60% smaller than mBERT (119K) and 80% smaller than XLM-R/IndicBERTv2 (250K). This reduces embedding parameters from 192M to 38.4M, shrinking the embedding layer from 61% to 31% of total model parameters.

| Property | Value |
|----------|-------|
| Algorithm | Byte-level BPE |
| Vocabulary Size | 50,000 |
| Training Data | MaithiliCorpus (39.8M words) |
| Training | SentencePiece, `--model_type=bpe` |
| Special Tokens | `<s>`, `</s>`, `<unk>`, `<pad>`, `<mask>` |
| UNK Rate (on Maithili) | 0.00% |

## Acknowledgments

The authors gratefully acknowledge [AnK Komputing](https://ank-world.com) for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.

## Intended Use

- Masked language modeling (fill-mask)
- Fine-tuning for text classification, NER, and other NLU tasks
- Feature extraction for downstream Maithili NLP

## Usage

```python
from transformers import pipeline
fill = pipeline("fill-mask", model="AiventraLab/maithili-roberta")
results = fill("भगवान् <mask>क ध्यान")
```