README.md

---
language:
  - mai
  - hi
  - sa
tags:
  - maithili
  - low-resource
  - morphology
  - corpus
  - indic
  - devanagari
license: cc-by-4.0
pretty_name: MaithiliCorpus
size_categories:
  - 10M<n<100M
task_categories:
  - text-generation
  - text-classification
  - other
task_ids:
  - language-modeling
  - other-morphological-analysis
---

# MaithiliCorpus

An auditable 65.6M-word Maithili text resource compiled from 347 sources with source-level provenance, four automatically assigned domains, and separately versioned LID-filtered and deduplicated views.

## Dataset Description

MaithiliCorpus is the largest open-source Maithili text corpus, addressing the severe resource gap for a language spoken by approximately 34 million people. The corpus is released in three versioned views to support different research needs:

| View | Documents | Words | Description |
|------|-----------|-------|-------------|
| **Cleaned** | 74,989 | 65.6M | Tier-1 cleaned + Danda-normalized + IndicLID-annotated |
| **LID-filtered** | 49,629 | 44.0M | Majority-Maithili documents (mai_fraction ≥ 0.5) |
| **LID-filtered + dedup** | 47,882 | 39.8M | Exact duplicate removal applied |

### Key Features

- **347 unique sources** across 4 domains (Literary 55.9%, Social Science 37.2%, Folklore 6.2%, Children's Literature 0.7%)
- **Language verification** via AI4Bharat IndicLID-FTN (real `mai_Deva` label)
- **Full quality suite**: MATTR, MTLD, Zipf adherence, entropy, duplication metrics
- **Provenance metadata**: source, URL, domain labels, LID scores on every record

## Dataset Structure

### Data Fields

| Field | Type | Description |
|-------|------|-------------|
| `source` | string | Website/source identifier |
| `url` | string | Original article URL |
| `title` | string | Article title (when available) |
| `content` | string | Main text content in Maithili |
| `category` | string | Domain label (literary, social_science, folklore, children_literature) |
| `date` | string | Publication date (when available) |
| `author` | string | Author name (when available) |
| `scraped_at` | string | ISO timestamp when scraped |

### Additional Fields (LID-filtered and dedup views)

| Field | Type | Description |
|-------|------|-------------|
| `mai_fraction` | float | Fraction of sentences predicted as Maithili by IndicLID-FTN |
| `lid_majority` | string | Majority predicted language label |
| `lid_n_sentences` | int | Number of sentences analyzed |

### Loading the Dataset

```python
import json

# Load the LID-filtered view (recommended)
with open('maithili_corpus_gold.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        record = json.loads(line)
        print(f"Source: {record['source']}")
        print(f"Domain: {record.get('category', 'unknown')}")
        print(f"Words: {len(record['content'].split())}")
        print(f"Maithili fraction: {record.get('mai_fraction', 'N/A')}")
```

## Dataset Statistics

| Metric | Cleaned | LID-filtered | LID-filtered + dedup |
|--------|---------|--------------|----------------------|
| Documents | 74,989 | 49,629 | 47,882 |
| Words | 65.6M | 44.0M | 39.8M |
| Unique types | 2,010,038 | 1,265,221 | — |
| MATTR (mean) | 0.5911 | 0.5950 | — |
| MTLD (mean) | 34.77 | 34.37 | — |
| Zipf R² | 0.9814 | 0.9843 | — |
| Devanagari ratio | 96.0% | 97.1% | — |
| Exact-sentence dup | 0.49% | — | — |
| MinHash near-dup (τ=0.7) | 21.03% | 19.95% | — |

## Domain Distribution

| Domain | Records | Words | % Words |
|--------|---------|-------|---------|
| Literary | 12,818 | 37.1M | 55.9% |
| Social Science | 55,585 | 24.7M | 37.2% |
| Folklore | 7,019 | 4.1M | 6.2% |
| Children's Literature | 450 | 455K | 0.7% |

## Language Identification

IndicLID-FTN was applied to all 74,989 cleaned documents (1,555,513 sentences):

- Sentence-level Maithili: 62.94%
- Doc-majority Maithili: 69.44%
- Retained (≥50% Maithili): 49,629 documents

The lower sentence-level percentage reflects systematic confusion with close Devanagari sister languages (Hindi 23.8%, Sanskrit 4.2%, Marathi 2.0%, Nepali 1.6%), not actual non-Maithili content.

## Recommended Usage

**For language model pretraining**: Use the LID-filtered view (`maithili_corpus_gold.jsonl`) — every document is majority-Maithili.

**For recall-oriented research**: Use the Cleaned view (`maithili_corpus_annotated.jsonl`) — includes all documents with LID scores for custom filtering.

**For deduplicated training**: Use the LID-filtered + dedup view (`maithili_corpus_gold_deduped.jsonl`) — removes exact duplicates.

## Citation

```bibtex
@article{maithili2026corpus,
  title={MaithiliCorpus: An Auditable 65.6M-Word Resource and a Validation Study of Raw-Text Morphological Rule Induction},
  author={[Authors]},
  journal={arXiv preprint},
  year={2026}
}
```

## License

CC-BY 4.0 applies to the compilation, annotations, and metadata. Constituent texts retain their original copyright terms. Source attribution (`source`, `url`) must be preserved on redistribution.

## Limitations

1. **Source concentration**: Social Science domain is 74.9% from three sources
2. **Domain imbalance**: Children's literature contributes only 0.7% of words
3. **Language mixing**: ~13% of documents have 0% Maithili sentences (removed in LID-filtered view)
4. **Domain labels are automatic**: Accepted by native speakers in 70.6% of annotation rows
5. **Near-duplicate flags**: 21.03% of documents flagged (concentrated in short news items)

## See Also

- [Gold Standard Review](gold_standard/GOLD_STANDARD_REVIEW.md)
- [Corpus Documentation](CORPUS_DOCUMENTATION.md)
- [Quality Metrics](QUALITY_METRICS.md)