AiventraLab/maithili-corpus

MaithiliCorpus

An auditable 65.6M-word Maithili text resource compiled from 347 sources with source-level provenance, four automatically assigned domains, and separately versioned LID-filtered and deduplicated views.

Dataset Description

MaithiliCorpus is the largest open-source Maithili text corpus, addressing the severe resource gap for a language spoken by approximately 34 million people. The corpus is released in three versioned views to support different research needs:

View	Documents	Words	Description
Cleaned	74,989	65.6M	Tier-1 cleaned + Danda-normalized + IndicLID-annotated
LID-filtered	49,629	44.0M	Majority-Maithili documents (mai_fraction ≥ 0.5)
LID-filtered + dedup	47,882	39.8M	Exact duplicate removal applied

Key Features

• 347 unique sources across 4 domains (Literary 55.9%, Social Science 37.2%, Folklore 6.2%, Children's Literature 0.7%)
• Language verification via AI4Bharat IndicLID-FTN (real mai_Deva label)
• Full quality suite: MATTR, MTLD, Zipf adherence, entropy, duplication metrics
• Provenance metadata: source, URL, domain labels, LID scores on every record

Dataset Structure

Data Fields

Field	Type	Description
source	string	Website/source identifier
url	string	Original article URL
title	string	Article title (when available)
content	string	Main text content in Maithili
category	string	Domain label (literary, social_science, folklore, children_literature)
date	string	Publication date (when available)
author	string	Author name (when available)
scraped_at	string	ISO timestamp when scraped

Additional Fields (LID-filtered and dedup views)

Field	Type	Description
mai_fraction	float	Fraction of sentences predicted as Maithili by IndicLID-FTN
lid_majority	string	Majority predicted language label
lid_n_sentences	int	Number of sentences analyzed

Loading the Dataset

import json

# Load the LID-filtered view (recommended)
with open('maithili_corpus_gold.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        record = json.loads(line)
        print(f"Source: {record['source']}")
        print(f"Domain: {record.get('category', 'unknown')}")
        print(f"Words: {len(record['content'].split())}")
        print(f"Maithili fraction: {record.get('mai_fraction', 'N/A')}")

Dataset Statistics

Metric	Cleaned	LID-filtered	LID-filtered + dedup
Documents	74,989	49,629	47,882
Words	65.6M	44.0M	39.8M
Unique types	2,010,038	1,265,221	—
MATTR (mean)	0.5911	0.5950	—
MTLD (mean)	34.77	34.37	—
Zipf R²	0.9814	0.9843	—
Devanagari ratio	96.0%	97.1%	—
Exact-sentence dup	0.49%	—	—
MinHash near-dup (τ=0.7)	21.03%	19.95%	—

Domain Distribution

Domain	Records	Words	% Words
Literary	12,818	37.1M	55.9%
Social Science	55,585	24.7M	37.2%
Folklore	7,019	4.1M	6.2%
Children's Literature	450	455K	0.7%

Language Identification

IndicLID-FTN was applied to all 74,989 cleaned documents (1,555,513 sentences):

• Sentence-level Maithili: 62.94%
• Doc-majority Maithili: 69.44%
• Retained (≥50% Maithili): 49,629 documents

The lower sentence-level percentage reflects systematic confusion with close Devanagari sister languages (Hindi 23.8%, Sanskrit 4.2%, Marathi 2.0%, Nepali 1.6%), not actual non-Maithili content.

Recommended Usage

For language model pretraining: Use the LID-filtered view (maithili_corpus_gold.jsonl) — every document is majority-Maithili.

For recall-oriented research: Use the Cleaned view (maithili_corpus_annotated.jsonl) — includes all documents with LID scores for custom filtering.

For deduplicated training: Use the LID-filtered + dedup view (maithili_corpus_gold_deduped.jsonl) — removes exact duplicates.

Citation

@article{maithili2026corpus,
  title={MaithiliCorpus: An Auditable 65.6M-Word Resource and a Validation Study of Raw-Text Morphological Rule Induction},
  author={[Authors]},
  journal={arXiv preprint},
  year={2026}
}

License

CC-BY 4.0 applies to the compilation, annotations, and metadata. Constituent texts retain their original copyright terms. Source attribution (source, url) must be preserved on redistribution.

Limitations

1. Source concentration: Social Science domain is 74.9% from three sources
2. Domain imbalance: Children's literature contributes only 0.7% of words
3. Language mixing: ~13% of documents have 0% Maithili sentences (removed in LID-filtered view)
4. Domain labels are automatic: Accepted by native speakers in 70.6% of annotation rows
5. Near-duplicate flags: 21.03% of documents flagged (concentrated in short news items)

See Also

• Gold Standard Review
• Corpus Documentation
• Quality Metrics