README.md

---
language:
- mai
license: cc-by-4.0
tags:
- maithili
- causal-language-model
- pretrained-language-model
- low-resource
- indic
- devanagari
- text-generation
datasets:
- AiventraLab/maithili-corpus
pipeline_tag: text-generation
widget:
- text: "मैथिली भाषा एक"
- text: "आजकल बिहार में"
- text: "मोर गीत"
---

# MaithiliCLM: First Causal Language Model for the Maithili Language

**MaithiliCLM** is a **decoder-only causal language model** (109.5M parameters) pretrained from scratch on **MaithiliCorpus** — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the **first causal/generative language model** ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.

Unlike encoder-only models (BERT, RoBERTa) that only support understanding tasks, MaithiliCLM enables **text generation**, making it suitable for creative writing, dialogue systems, summarization, and other generative applications in Maithili.

## Model Architecture

| Property | Value |
|----------|-------|
| Architecture | Decoder-only Transformer (GPT-family) |
| Parameters | 109,545,984 |
| Vocabulary | Byte-level BPE, 32,000 tokens |
| Context Window | 1,024 tokens |
| Hidden Size | 768 |
| Intermediate Size | 3,072 |
| Attention Heads | 12 |
| Hidden Layers | 12 |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Position Encoding | Rotary Position Embeddings (RoPE) |
| Dropout | 0.1 |
| Training Data | MaithiliCorpus LID-filtered (48,064 documents, 27.5M words) |
| Training Tokens | 983M |
| Training Steps | 30,000 |
| Effective Batch Size | 32 |
| Optimizer | AdamW (lr=3e-4, weight_decay=0.1) |
| LR Schedule | Cosine with warmup |
| Precision | FP16 mixed |
| License | CC-BY-4.0 |

### Architecture Details

MaithiliCLM implements a modernized GPT-2 family architecture with the following enhancements:

- **Rotary Position Embeddings (RoPE)**: Enables better length generalization than learned absolute positional embeddings
- **RMSNorm**: More efficient than LayerNorm, used in LLaMA and recent models
- **SwiGLU**: Improved activation function over GeLU, used in PaLM and LLaMA
- **Causal Self-Attention**: Standard autoregressive masking for language modeling

## Tokenizer

The tokenizer is a **byte-level BPE** trained from scratch on MaithiliCorpus with a vocabulary of **32,000 tokens**. This compact vocabulary reduces embedding parameters from 153.6M to 24.6M, shrinking the embedding layer from 55% to 22% of total model parameters.

| Property | Value |
|----------|-------|
| Algorithm | Byte-level BPE |
| Vocabulary Size | 32,000 |
| Training Data | MaithiliCorpus (39.8M words) |
| Tokens per Word | 3.53 |
| Special Tokens | `<s>`, `</s>`, `<unk>`, `<pad>` |

## Training

### Data Preparation

The training corpus underwent rigorous cleaning and quality filtering:

1. **Source**: AiventraLab/maithili-corpus (49,629 raw documents)
2. **Cleaning**: HTML removal, encoding normalization, language verification
3. **LID Filtering**: Retained documents with ≥50% Maithili sentences (48,064 documents)
4. **Train/Val Split**: 47,103 training / 961 validation documents
5. **Tokenization**: Byte-level BPE encoding, packed into 1,024-token blocks

### Training Configuration

- **Duration**: ~12 hours
- **Tokens Seen**: 983M (~10.3 epochs)
- **Peak VRAM**: 6.46 GB
- **Throughput**: ~26,500 tokens/second

### Training Dynamics

| Metric | Step 0 | Step 10K | Step 20K | Step 30K |
|--------|--------|----------|----------|----------|
| Train Loss | 10.32 | 3.65 | 3.18 | 3.04 |
| Val Loss | 10.31 | 3.42 | 3.18 | 3.12 |
| Val PPL | 29,674 | 30.56 | 24.05 | 22.68 |

Training plateaued after ~30K steps with diminishing returns (<1 ppl improvement per 5K steps).


## Generation Quality

The model generates coherent Maithili text across diverse prompts. Generated continuations are syntactically correct and semantically plausible, though coherence degrades with longer sequences due to the limited training data.

## Intended Use

- **Text Generation**: Creative writing, article completion, dialogue systems
- **Fine-tuning Base**: Adapt for classification, NER, QA, summarization
- **Research**: Foundation for Maithili NLP, low-resource language modeling
- **Education**: Teaching Maithili language and computational linguistics

### How to Use

```python
from transformers import AutoTokenizer
import torch

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("AiventraLab/maithili-clm")

# Load model (requires custom model class - see model.py)
# model = MaithiliGPT.from_pretrained("AiventraLab/maithili-clm")

# Generate text
prompt = "मैथिली भाषा एक"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Autoregressive generation
output = model.generate(
    input_ids,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    do_sample=True
)

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
```

## Acknowledgments

The authors gratefully acknowledge [AnK Komputing](https://ank-world.com) for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.

## Citation

```bibtex
@article{maithili2026clm,
  title={MaithiliCLM: First Causal Language Model for the Maithili Language},
  author={[Authors]},
  journal={arXiv preprint},
  year={2026}
}
```

## License

CC-BY 4.0 applies to the model weights, tokenizer, and documentation. The training data (MaithiliCorpus) retains its original license terms.

## See Also

- [MaithiliCorpus](https://huggingface.co/AiventraLab/maithili-corpus) - Training data
- [MaithiliBERT](https://huggingface.co/AiventraLab/maithili-roberta) - Encoder-only counterpart