QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages
Paper • 2606.23943 • Published
QuechuaBERT (PRPE)
Small BertForMaskedLM for Southern Quechua, tokenized with the morphological PRPE segmenter from QuechuaTok.
Training note
CPU run: ~20k Llamacha lines, 5000 MLM steps, tiny BERT (hidden=256, layers=4). Final train loss ≈ 6.19. Early checkpoint for visibility, not a finished SOTA model.
Paper
Contreras, M. (2026). QuechuaTok. arXiv:2606.23943 — https://arxiv.org/abs/2606.23943
Usage
Install QuechuaTok, then load the model weights from this repo. Custom PRPE tokenizer lives in quechuatok.hf_tokenizer.
See QuechuaTok issue #8 and scripts/train_quechuabert.py.
- Downloads last month
- -