HmarBERT (Proof of Concept)

HmarBERT is an early proof-of-concept model for the Hmar language (hmr, ISO 639-3).

We adapted robzchhangte/MizBERT by swapping 303 cognate words in the vocabulary and fine-tuning on ~101k Hmar sentences with whole-word masking.

It worked well as a feasibility test, reaching an evaluation loss of 2.48 (perplexity 12.0). However, because it started from a Mizo base, the model still leans toward Mizo sentence patterns on less common words. For a completely clean, from-scratch model with zero Mizo priors, check out HmarBERT-mini.

---

Model Details

• Base model: robzchhangte/MizBERT (BERT-base, 110M params)
• Vocabulary: 30,522 tokens with 303 in-place Mizo $\rightarrow$ Hmar cognate swaps
• Training data: hmar-heritage-org/dolma-pretrain (101,867 sentences, ~2.5M words)
• Training setup: 20 epochs across 3 stages using whole-word masking

---

Benchmark Results

Model / Stage	Curriculum	Eval Loss	Perplexity	Notes
MizBERT Base	Raw MizBERT (zero-shot)	2.71	15.05	Before adaptation
HmarBERT Stage 2	6 epochs (15% WWM, 303 swaps)	2.55	12.78	After vocab swap
HmarBERT Stage 3	8 epochs (dual cycle WWM)	2.48	12.00	Best checkpoint

---

Interactive Demo

Try the model in your browser: Dolma — Powered by HmarBERT

---

Usage

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="azinamotoe/HmarBERT")
result = fill_mask("Hrietthiem [MASK] rel el tala, ka um dan hi!")
for r in result[:3]:
    print(f"{r['token_str']:<15} (score: {r['score']:.4f}) -> {r['sequence']}")