mns-spm-32k — SentencePiece unigram tokenizer for Mongol bichig

32,000 pieces, trained on 412,576 unique lines of traditional Mongolian (real OCR labels + a synthetic corpus), all passed through one canonical normalizer first.

tok/char on real OCR labels	0.2716
exact round-trip	0 failures over 19,101 blocks, 7,836 composites, 412,576 corpus lines

Why it is trained the way it is

Traditional Mongolian carries meaning in characters you cannot see, and the usual SentencePiece defaults destroy them:

• normalization_rule_name="identity" — the default nmt_nfkc rewrites NNBSP (U+202F) to a plain space, which erases the case-suffix glue.
• remove_extra_whitespaces=False — same distinction, from the other side.
• user_defined_symbols=["\n"] — otherwise the newline is UNK and every multi-line label decodes to ⁇.

Preserved exactly: NNBSP (case-suffix glue), MVS U+180E (final vowel separator), FVS U+180B–U+180D (lexical variant selectors), ZWJ U+200D, and Mongolian digits U+1810–U+1819.

Use

import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.load("mns_spm_32k.model")
sp.encode("ᠮᠣᠩᠭᠣᠯ ᠤᠨ ᠲᠡᠦᠬᠡ", out_type=str)

The .model file is the artifact to use. A XLMRobertaTokenizer wrapper is included for convenience, but that class ends convert_tokens_to_string with .strip(), so it silently drops leading and trailing whitespace.

Normalize before you tokenize

Text must be in the same convention the tokenizer was trained on, or the suffix pieces will not match. Text typed on a Bolorsoft Tungaamal keyboard needs U+1888 → U+182C and U+1889 → U+182D, and the FVS it inserts inside case suffixes removed.