Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,56 @@
+---
+library_name: tokenizers
+tags:
+  - tokenizer
+  - multilingual
+  - byte-level-bpe
+  - indigenous-languages
+  - tachiwin
+---
+
+# Tachiwin multilingual tokenizer
+
+A multilingual ByteLevel-BPE tokenizer trained with Hugging Face Tokenizers.
+
+## Corpus weighting
+
+| Component | Target |
+|---|---:|
+| Modern + old exotic-language data | 70% |
+| English | 10% |
+| Spanish | 10% |
+| Code | 10% |
+
+The complete available exotic-language corpus is used as the 70% anchor.
+Its existing modern/old composition is preserved.
+
+## Tokenizer
+
+- Model: BPE
+- Vocabulary target: 256,000
+- Initial alphabet: complete ByteLevel alphabet
+- ByteLevel GPT-2 regex: disabled
+- Unicode normalizer: none
+- Special tokens: 94
+- Human-language tags: 60
+
+## Corpus size
+
+Total materialized corpus:
+
+50,027,186 bytes
+(0.047 GiB)
+
+## Important training note
+
+The Hugging Face BPE trainer does not expose an internal resumable merge-state
+checkpoint. The recipe therefore treats the completed `tokenizer.json` as the
+training checkpoint:
+
+- corpus preparation is resumable;
+- recipe/statistics/checksums are stored in `recipe/`;
+- if `tokenizer.json` already exists, subsequent runs skip BPE training;
+- an interrupted BPE computation itself must be restarted.
+
+This avoids changing the training procedure merely to obtain artificial
+checkpointing.