janakhpon commited on
Commit
b2eb67d
·
1 Parent(s): 73b284c

fix: correct the single-token coverage claim to 98.74%

Browse files

- 392 of 397 distinct characters over all 29,600 Mon validation lines
- record lines_measured, so the sampling is auditable

The card said 100% with sampling labelled "whole split, no cap", but the
figure came from the first 5,000 lines. tokenizer.json is unchanged and
still hashes to 34d181532eee7e6754bfbad693753ee4660bdda00eb36a2ccc7cd2f372c71282,
byte-identical to the packaged copy; only the card was re-measured.

Files changed (2) hide show
  1. README.md +5 -1
  2. model_card.json +5 -3
README.md CHANGED
@@ -60,7 +60,11 @@ Vocabulary 64,256 · Unigram · trained on the train split of an 893,936-line /
60
  | English | 4.112 | 34.6 | — *(n=0)* | **100%** | 0.02% |
61
  | mixed script | 3.804 | 24.6 | 0.81% *(n=28,133)* | **100%** | 0.19% |
62
 
63
- **100% of the characters in the Mon validation split are single tokens.**
 
 
 
 
64
 
65
  Three notes on reading these:
66
 
 
60
  | English | 4.112 | 34.6 | — *(n=0)* | **100%** | 0.02% |
61
  | mixed script | 3.804 | 24.6 | 0.81% *(n=28,133)* | **100%** | 0.19% |
62
 
63
+ **98.74% of the distinct characters in the Mon validation split are single
64
+ tokens** — 392 of 397, over all 29,600 lines. The remaining five (a combining
65
+ diaeresis below, a Greek capital pi, two Sinhala letters and one emoji; seven
66
+ occurrences in 2.28M characters) round-trip through byte fallback at one to four
67
+ tokens each.
68
 
69
  Three notes on reading these:
70
 
model_card.json CHANGED
@@ -81,8 +81,9 @@
81
  "unreconstructable": 11
82
  },
83
  "coverage": {
84
- "distinct": 199,
85
- "single_token": 1.0
 
86
  }
87
  },
88
  "notes": {
@@ -93,6 +94,7 @@
93
  },
94
  "eval": {
95
  "split": "val",
96
- "sampling": "whole split, no cap"
 
97
  }
98
  }
 
81
  "unreconstructable": 11
82
  },
83
  "coverage": {
84
+ "distinct": 397,
85
+ "single_token": 0.9874,
86
+ "lines_measured": 29600
87
  }
88
  },
89
  "notes": {
 
94
  },
95
  "eval": {
96
  "split": "val",
97
+ "sampling": "whole split, no cap",
98
+ "coverage_lines_measured": 29600
99
  }
100
  }