Add tokenizer parity tests
Browse files- tokenizer-tests.md +30 -0
tokenizer-tests.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Tokenizer parity tests (Soprano)
|
| 2 |
+
|
| 3 |
+
This repo includes GGUF files converted from `ekwek/Soprano-1.1-80M`.
|
| 4 |
+
|
| 5 |
+
The upstream Hugging Face tokenizer (`tokenizer.json`) uses:
|
| 6 |
+
- normalizer: Lowercase + whitespace collapse (`\s+ -> " "`)
|
| 7 |
+
- pre_tokenizer: Digits(individual_digits=True) + Split(Regex: `\s+|\w+|[^\w\s]+`, Isolated)
|
| 8 |
+
- model: BPE
|
| 9 |
+
|
| 10 |
+
To ensure llama.cpp tokenization matches Hugging Face exactly, llama.cpp was extended with a dedicated pre-tokenizer: `tokenizer.ggml.pre = soprano`.
|
| 11 |
+
|
| 12 |
+
## Parity test cases
|
| 13 |
+
|
| 14 |
+
All cases below were tested by comparing:
|
| 15 |
+
- HF ids: `tokenizers.Tokenizer.from_file(tokenizer.json).encode(text).ids`
|
| 16 |
+
- GGUF ids: `llama-tokenize -m Soprano-1.1-80M.F16.gguf --ids --no-bos -p <text>`
|
| 17 |
+
|
| 18 |
+
Result: **ALL_OK = True** (all ids match).
|
| 19 |
+
|
| 20 |
+
| # | text (repr) | hf_len | gg_len |
|
| 21 |
+
|---|------------|--------|--------|
|
| 22 |
+
| 1 | `"Hello, world! 12345 測試 test..."` | 23 | 23 |
|
| 23 |
+
| 2 | `" Leading spaces\nnew\tline 99 bottles "` | 27 | 27 |
|
| 24 |
+
| 3 | `"abc123def 000 3.14"` | 16 | 16 |
|
| 25 |
+
| 4 | `"MIXED_case With_Punct!!! and---dashes"` | 26 | 26 |
|
| 26 |
+
| 5 | `"emoji ✅😶🌫️🚀 + symbols $+<=>^`|"` | 30 | 30 |
|
| 27 |
+
| 6 | `"CJK: 我想在apple工作1314151天～"` | 23 | 23 |
|
| 28 |
+
| 7 | `"Quotes: I've been 'told he's there, 'RE you sure?"` | 33 | 33 |
|
| 29 |
+
| 8 | `"Whitespace: \r\n\n\t end"` | 10 | 10 |
|
| 30 |
+
|