Lexei1297/mari_mbert

Mari mBERT: Fine-tuning Multilingual BERT for the Mari Language

This repository contains the code and methodology for fine-tuning bert-base-multilingual-cased (mBERT) on a monolingual Mari language corpus using Masked Language Modeling (MLM). The project was developed as part of a Master's thesis in Linguistics at Volga State University of Technology (VSTU), within the research area of AI-assisted language revitalization.

Background

The Mari language is a Finno-Ugric language of the Volga region classified as low-resource: despite an active speaker community, it remains underrepresented in digital NLP infrastructure. This project addresses that gap by adapting a multilingual transformer model to Mari through targeted fine-tuning, following a methodology comparable to the LakotaBERT project.

The central technical contribution is the extension of the mBERT tokenizer vocabulary with Mari-specific graphemes absent from the standard token set, combined with MLM fine-tuning on a large monolingual corpus. The result is a reproducible, open-source baseline model for Mari NLP tasks.

Dataset

• Corpus: mari-lab/mari-monolingual-corpus (Mari Laboratory, Hugging Face)
• Size: approximately 1.4 million lines of Mari text
• Split: 90% training / 10% validation
• Format: one sentence per line (JSONL)

Model

• Base: google-bert/bert-base-multilingual-cased
• Task: Masked Language Modeling (MLM), 15% masking probability
• Tokenizer extension: 13 Mari graphemes added (ҥ ӧ ӱ ӹ ӓ ӆ ӊ and uppercase variants)
• Hardware: NVIDIA A100 80 GB
• Precision: bfloat16 (native A100 Tensor Core support)

Training Configuration

Parameter	Value
Sequence length	200 tokens
Batch size	128
Gradient accumulation	1
Epochs	5
Learning rate	3e-5
MLM probability	15%
Optimizer	AdamW (fused)
Weight decay	0.01
Warmup ratio	0.06

Evaluation Results

Evaluation was performed on 2,000 sampled sequences (13,292 masked positions), following the LakotaBERT Table 4 protocol.

Metric	Mari mBERT (this work)	LakotaBERT	mBERT baseline
Accuracy	53.57%	51.48%	54.42%
Precision	0.54	0.56	0.50
F1-Score	0.54	0.49	0.54
MRR	0.63	0.51	0.51
CER	0.32	0.43	0.38
Hit@10	0.81	0.31	0.24
BLEU	0.41	0.09	0.09

The model shows strong ranking performance (MRR, Hit@10) relative to comparable low-resource projects, attributable in part to the typological proximity of Mari to other Finno-Ugric languages already represented in the mBERT pretraining corpus (Finnish, Estonian, Hungarian).

Output Artifacts

After training, the following files are saved to the output directory:

• model.safetensors — fine-tuned model weights
• tokenizer_config.json — extended tokenizer configuration
• metrics.json — full evaluation metrics
• training_dynamics.png — train/validation loss curves
• metrics_summary.png — bar chart of evaluation metrics

Requirements

torch >= 2.0
transformers
datasets
nltk
numpy
pandas
matplotlib
tqdm

Install dependencies:

pip install torch transformers datasets nltk numpy pandas matplotlib tqdm

Usage

Prepare the corpus as a JSONL file with one field text per line, then run:

python mari_mbert_a100.py

Key environment variables for configuration:

Variable	Default	Description
SEQ	200	Sequence length
BS	128	Batch size per device
EP	5	Number of training epochs
LR	3e-5	Learning rate
WORKERS	16	DataLoader worker count
METRICS_SAMPLE	2000	Sequences sampled for evaluation
USE_COMPILE	0	Enable torch.compile (experimental)

Intended Applications

The fine-tuned model is intended as a foundation for downstream Mari NLP tools, including spell-checking and autocorrection systems, predictive text input for mobile devices and messengers, fill-in-the-blank and grammar exercises for language learners, and further fine-tuning for POS tagging and syntactic parsing.

Limitations

• The training corpus is monolingual; parallel Mari-Russian data was not used
• Dialectal variation between Hill Mari and Meadow Mari is not separated in the corpus
• Automated metrics do not substitute for evaluation by native speakers

Reproducibility

All code is published under an open license. The training corpus is publicly available on Hugging Face. Any researcher can reproduce the full experiment using the provided script and corpus.

Citation

If you use this work, please cite:

Zadvornykh, A. S. Use of AI systems to preserve and revitalize languages. Yoshkar-Ola, 24 June 2026. Yoshkar-Ola: VSTU, 2026.

Lexei1297/mari_mbert

Acknowledgments

• Mari Laboratory for the open monolingual Mari corpus
• Google Research for the bert-base-multilingual-cased model
• The LakotaBERT project for the evaluation methodology