fix: correct the single-token coverage claim to 98.74%
Browse files- 392 of 397 distinct characters over all 29,600 Mon validation lines
- record lines_measured, so the sampling is auditable
The card said 100% with sampling labelled "whole split, no cap", but the
figure came from the first 5,000 lines. tokenizer.json is unchanged and
still hashes to 34d181532eee7e6754bfbad693753ee4660bdda00eb36a2ccc7cd2f372c71282,
byte-identical to the packaged copy; only the card was re-measured.
- README.md +5 -1
- model_card.json +5 -3
README.md
CHANGED
|
@@ -60,7 +60,11 @@ Vocabulary 64,256 · Unigram · trained on the train split of an 893,936-line /
|
|
| 60 |
| English | 4.112 | 34.6 | — *(n=0)* | **100%** | 0.02% |
|
| 61 |
| mixed script | 3.804 | 24.6 | 0.81% *(n=28,133)* | **100%** | 0.19% |
|
| 62 |
|
| 63 |
-
**
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
Three notes on reading these:
|
| 66 |
|
|
|
|
| 60 |
| English | 4.112 | 34.6 | — *(n=0)* | **100%** | 0.02% |
|
| 61 |
| mixed script | 3.804 | 24.6 | 0.81% *(n=28,133)* | **100%** | 0.19% |
|
| 62 |
|
| 63 |
+
**98.74% of the distinct characters in the Mon validation split are single
|
| 64 |
+
tokens** — 392 of 397, over all 29,600 lines. The remaining five (a combining
|
| 65 |
+
diaeresis below, a Greek capital pi, two Sinhala letters and one emoji; seven
|
| 66 |
+
occurrences in 2.28M characters) round-trip through byte fallback at one to four
|
| 67 |
+
tokens each.
|
| 68 |
|
| 69 |
Three notes on reading these:
|
| 70 |
|
model_card.json
CHANGED
|
@@ -81,8 +81,9 @@
|
|
| 81 |
"unreconstructable": 11
|
| 82 |
},
|
| 83 |
"coverage": {
|
| 84 |
-
"distinct":
|
| 85 |
-
"single_token":
|
|
|
|
| 86 |
}
|
| 87 |
},
|
| 88 |
"notes": {
|
|
@@ -93,6 +94,7 @@
|
|
| 93 |
},
|
| 94 |
"eval": {
|
| 95 |
"split": "val",
|
| 96 |
-
"sampling": "whole split, no cap"
|
|
|
|
| 97 |
}
|
| 98 |
}
|
|
|
|
| 81 |
"unreconstructable": 11
|
| 82 |
},
|
| 83 |
"coverage": {
|
| 84 |
+
"distinct": 397,
|
| 85 |
+
"single_token": 0.9874,
|
| 86 |
+
"lines_measured": 29600
|
| 87 |
}
|
| 88 |
},
|
| 89 |
"notes": {
|
|
|
|
| 94 |
},
|
| 95 |
"eval": {
|
| 96 |
"split": "val",
|
| 97 |
+
"sampling": "whole split, no cap",
|
| 98 |
+
"coverage_lines_measured": 29600
|
| 99 |
}
|
| 100 |
}
|