Hugging Face's logo

baobabtech
/
millid-medium-v1

Text Classification
PyTorch
ONNX
Safetensors
multilingual
language-identification
byte-level
Eval Results (legacy)
🇪🇺 Region: EU
Model card Files
xet
 Community
4

Model card for the public release

#2
by byamasupatrick - opened
base: refs/heads/main
←
from: refs/pr/2
Discussion Files changed
+220
-230
Files changed (1) hide show
  1. README.md +220 -230
README.md CHANGED
@@ -51,13 +51,15 @@ A byte-level, encoder-only transformer that names the language of a passage, fro
51
 
52
  ## The MillID models
53
 
54
- | Model | Size | Macro F1, `test` / `udhr` | Runs best on |
55
- |---|---|---|---|
56
- | [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M parameters, ONNX 11 MB | 0.9554 / 0.8764 | a CPU, or a web page |
57
- | **MillID medium v1** (this model) | 115M parameters, ONNX 455 MB | 0.9576 / 0.8780 | a GPU |
58
- | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M parameters, ONNX 764 MB | 0.9646 / 0.8812 | a GPU: the most accurate |
59
 
60
- All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 / 0.8873 on the same sets.
 
 
 
 
 
 
61
 
62
  ## Use it
63
 
@@ -99,261 +101,249 @@ UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-
99
 
100
  ## Evaluation against GlotLID and fastText
101
 
102
- `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. `glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every model on the passages whose language all of them know: the head-to-head on equal terms.
103
-
104
- | Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
105
- |---|---|---|---|---|---|---|---|---|---|
106
- | validation | millid | 0.9613 | 0.9910 | 0.9949 | 0.9573 | 0.9621 | 0.9584 | 0.9850 | 1626 |
107
- | test | millid | 0.9616 | 0.9910 | 0.9945 | 0.9564 | 0.9614 | 0.9576 | 0.9841 | 1621 |
108
- | test | glotlid | 0.9776 | 0.9977 | 0.9984 | 0.9783 | 0.9752 | 0.9748 | 1.0000 | 1621 |
109
- | test | glotlid@millid | 0.9783 | 0.9978 | 0.9985 | 0.9778 | 0.9765 | 0.9755 | 1.0000 | 1621 |
110
- | test | fasttext | 0.2511 | 0.2610 | 0.2625 | 0.0520 | 0.1205 | 0.0657 | 0.0000 | 1621 |
111
- | test | fasttext@millid | 0.2515 | 0.2611 | 0.2625 | 0.0521 | 0.1209 | 0.0659 | 0.0000 | 1621 |
112
- | test-common | millid | 0.9425 | 0.9882 | 0.9924 | 0.9732 | 0.9361 | 0.9527 | 0.9738 | 213 |
113
- | test-common | glotlid | 0.9738 | 0.9925 | 0.9946 | 0.9924 | 0.9771 | 0.9835 | 0.9983 | 213 |
114
- | test-common | glotlid@millid | 0.9749 | 0.9927 | 0.9948 | 0.9923 | 0.9790 | 0.9846 | 0.9983 | 213 |
115
- | test-common | fasttext | 0.9339 | 0.9707 | 0.9762 | 0.9357 | 0.9169 | 0.9149 | 0.9680 | 213 |
116
- | test-common | fasttext@millid | 0.9352 | 0.9710 | 0.9763 | 0.9354 | 0.9201 | 0.9173 | 0.9680 | 213 |
117
- | udhr | millid | 0.8524 | 0.9194 | 0.9340 | 0.9187 | 0.8769 | 0.8780 | 0.9910 | 358 |
118
- | udhr | glotlid | 0.8598 | 0.9280 | 0.9421 | 0.9261 | 0.8826 | 0.8865 | 0.9915 | 358 |
119
- | udhr | glotlid@millid | 0.8614 | 0.9294 | 0.9433 | 0.9258 | 0.8844 | 0.8873 | 0.9915 | 358 |
120
- | udhr | fasttext | 0.4625 | 0.4857 | 0.4903 | 0.3152 | 0.4479 | 0.3497 | 0.0000 | 358 |
121
- | udhr | fasttext@millid | 0.4629 | 0.4857 | 0.4904 | 0.3152 | 0.4483 | 0.3499 | 0.0000 | 358 |
122
- | udhr-common | millid | 0.9121 | 0.9633 | 0.9688 | 0.9770 | 0.9382 | 0.9480 | 0.9955 | 170 |
123
- | udhr-common | glotlid | 0.9336 | 0.9678 | 0.9732 | 0.9888 | 0.9576 | 0.9657 | 1.0000 | 170 |
124
- | udhr-common | glotlid@millid | 0.9338 | 0.9683 | 0.9736 | 0.9888 | 0.9579 | 0.9659 | 1.0000 | 170 |
125
- | udhr-common | fasttext | 0.9187 | 0.9647 | 0.9741 | 0.9474 | 0.9431 | 0.9323 | 1.0000 | 170 |
126
- | udhr-common | fasttext@millid | 0.9195 | 0.9647 | 0.9742 | 0.9474 | 0.9441 | 0.9329 | 1.0000 | 170 |
127
- | udhr-all | millid | 0.7301 | 0.7875 | 0.8000 | 0.7415 | 0.7318 | 0.7184 | 0.9672 | 429 |
128
- | udhr-all | glotlid | 0.7502 | 0.8145 | 0.8282 | 0.7695 | 0.7529 | 0.7454 | 0.9796 | 429 |
129
- | udhr-all | glotlid@millid | 0.7378 | 0.7960 | 0.8079 | 0.7425 | 0.7381 | 0.7245 | 0.9748 | 429 |
130
- | udhr-all | fasttext | 0.3961 | 0.4160 | 0.4200 | 0.2479 | 0.3737 | 0.2795 | 0.0000 | 429 |
131
- | udhr-all | fasttext@millid | 0.3964 | 0.4160 | 0.4200 | 0.2479 | 0.3741 | 0.2797 | 0.0000 | 429 |
132
 
133
  Top-1 accuracy by passage length:
134
 
135
- | Set | Model | short | sentence | paragraph |
136
  |---|---|---|---|---|
137
- | validation | millid | 0.9363 | 0.9818 | 0.9858 |
138
- | test | millid | 0.9382 | 0.9809 | 0.9846 |
139
- | test | glotlid | 0.9664 | 0.9853 | 0.9911 |
140
- | test | glotlid@millid | 0.9674 | 0.9858 | 0.9912 |
141
- | test | fasttext | 0.2760 | 0.2501 | 0.1958 |
142
- | test | fasttext@millid | 0.2764 | 0.2506 | 0.1959 |
143
- | test-common | millid | 0.9169 | 0.9696 | 0.9737 |
144
- | test-common | glotlid | 0.9594 | 0.9876 | 0.9947 |
145
- | test-common | glotlid@millid | 0.9609 | 0.9885 | 0.9949 |
146
- | test-common | fasttext | 0.9112 | 0.9562 | 0.9651 |
147
- | test-common | fasttext@millid | 0.9125 | 0.9581 | 0.9655 |
148
-
149
- ### Leaderboard: test
150
-
151
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **110** languages, worse on **776** and within 0.01 on 735; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is `reports/test/leaderboard.csv`.
152
 
153
  The 20 languages with the most passages:
154
 
155
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
156
- |---|---|---|---|---|---|---|---|
157
- | `fra_Latn` | French | 1652 | 0.952 | 0.928 | 0.928 | 0.808 | 0.808 |
158
- | `eng_Latn` | English | 1444 | 0.916 | 0.858 | 0.859 | 0.653 | 0.648 |
159
- | `spa_Latn` | Spanish | 1136 | 0.942 | 0.951 | 0.951 | 0.574 | 0.574 |
160
- | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
161
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.829 | 0.785 | 0.785 | 0.513 | 0.512 |
162
- | `por_Latn` | Portuguese | 657 | 0.955 | 0.947 | 0.949 | 0.756 | 0.755 |
163
- | `kan_Knda` | Kannada | 601 | 0.982 | 0.993 | 0.993 | 0.922 | 0.922 |
164
- | `zul_Latn` | Zulu | 594 | 0.833 | 0.896 | 0.896 | 0.539 | 0.539 |
165
- | `swh_Latn` | Swahili | 592 | 0.858 | 0.917 | 0.916 | 0.346 | 0.346 |
166
- | `mar_Deva` | Marathi | 589 | 0.981 | 0.989 | 0.990 | 0.926 | 0.926 |
167
- | `tso_Latn` | Tsonga | 581 | 0.970 | 0.956 | 0.956 | 0.425 | 0.425 |
168
- | `hun_Latn` | Hungarian | 576 | 0.990 | 0.999 | 0.999 | 0.232 | 0.232 |
169
- | `afr_Latn` | Afrikaans | 569 | 0.974 | 0.985 | 0.985 | 0.861 | 0.860 |
170
- | `tel_Telu` | Telugu | 555 | 0.994 | 0.994 | 0.994 | 0.951 | 0.951 |
171
- | `nbl_Latn` | South Ndebele | 528 | 0.895 | 0.948 | 0.949 | 0.000 | 0.000 |
172
- | `rus_Cyrl` | Russian | 528 | 0.925 | 0.949 | 0.949 | 0.462 | 0.462 |
173
- | `ory_Orya` | Odia (individual language) | 511 | 0.993 | 0.991 | 0.994 | 0.976 | 0.976 |
174
- | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
175
- | `nso_Latn` | Northern Sotho | 479 | 0.960 | 0.969 | 0.969 | 0.915 | 0.914 |
176
- | `kua_Latn` | Kuanyama | 476 | 0.979 | 0.986 | 0.986 | 0.000 | 0.000 |
177
 
178
  Where MillID gains most over GlotLID:
179
 
180
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
181
- |---|---|---|---|---|---|---|---|
182
- | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 | 0.000 | 0.000 |
183
- | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.000 |
184
- | `ajp_Arab` | South Levantine Arabic | 28 | 0.902 | 0.581 | 0.581 | 0.000 | 0.000 |
185
- | `und_Laoo` | und | 20 | 0.857 | 0.621 | 0.621 | 0.000 | 0.000 |
186
- | `sgh_Cyrl` | Shughni | 9 | 0.900 | 0.667 | 0.667 | 0.000 | 0.000 |
187
- | `und_Thaa` | und | 29 | 0.873 | 0.651 | 0.651 | 0.000 | 0.000 |
188
- | `icr_Latn` | Islander Creole English | 24 | 0.870 | 0.649 | 0.684 | 0.000 | 0.000 |
189
- | `cek_Latn` | Eastern Khumi Chin | 67 | 0.853 | 0.680 | 0.684 | 0.000 | 0.000 |
190
- | `dag_Latn` | Dagbani | 149 | 0.817 | 0.662 | 0.660 | 0.000 | 0.000 |
191
- | `und_Syrc` | und | 43 | 0.977 | 0.822 | 0.822 | 0.000 | 0.000 |
192
- | `tuk_Arab` | Turkmen | 135 | 0.903 | 0.761 | 0.761 | 0.000 | 0.000 |
193
- | `orv_Cyrl` | Old Russian | 239 | 0.935 | 0.800 | 0.800 | 0.000 | 0.000 |
194
- | `hae_Latn` | Eastern Oromo | 75 | 0.786 | 0.657 | 0.657 | 0.000 | 0.000 |
195
- | `und_Kana` | und | 136 | 0.989 | 0.862 | 0.862 | 0.000 | 0.000 |
196
- | `kby_Latn` | Manga Kanuri | 7 | 0.452 | 0.326 | 0.326 | 0.000 | 0.000 |
197
 
198
  Where MillID loses most to GlotLID:
199
 
200
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
201
- |---|---|---|---|---|---|---|---|
202
- | `ccp_Latn` | Chakma | 3 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
203
- | `ttj_Latn` | Tooro | 28 | 0.490 | 0.946 | 0.946 | 0.000 | 0.000 |
204
- | `hak_Hani` | Hakka Chinese | 29 | 0.415 | 0.824 | 0.824 | 0.000 | 0.000 |
205
- | `mus_Latn` | Muscogee | 2 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
206
- | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
207
- | `knc_Arab` | Central Kanuri | 68 | 0.509 | 0.823 | 0.825 | 0.007 | 0.007 |
208
- | `bdq_Latn` | Bahnar | 9 | 0.500 | 0.800 | 0.800 | 0.000 | 0.000 |
209
- | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.500 | 0.783 | 0.783 | 0.000 | 0.000 |
210
- | `und_Ethi` | und | 14 | 0.741 | 1.000 | 1.000 | 0.000 | 0.000 |
211
- | `hrv_Latn` | Croatian | 168 | 0.687 | 0.944 | 0.944 | 0.178 | 0.178 |
212
- | `tui_Latn` | Tupuri | 4 | 0.750 | 0.667 | 1.000 | 0.000 | 0.000 |
213
- | `nse_Latn` | Nsenga | 12 | 0.500 | 0.750 | 0.750 | 0.000 | 0.000 |
214
- | `dks_Latn` | Southeastern Dinka | 3 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
215
- | `mbd_Latn` | Dibabawon Manobo | 14 | 0.757 | 1.000 | 1.000 | 0.000 | 0.000 |
216
- | `lnd_Latn` | Lundayeh | 5 | 0.769 | 1.000 | 1.000 | 0.000 | 0.000 |
217
-
218
- ### Leaderboard: udhr
219
-
220
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **63** languages, worse on **95** and within 0.01 on 200; against fastText restricted to MillID's labels, MillID is better on **304** languages, worse on **13** and within 0.01 on 41. The full table, with precision and recall, is `reports/udhr/leaderboard.csv`.
221
 
222
  The 20 languages with the most passages:
223
 
224
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
225
- |---|---|---|---|---|---|---|---|
226
- | `roh_Latn` | Romansh | 399 | 0.991 | 0.999 | 0.999 | 0.946 | 0.946 |
227
- | `oci_Latn` | Occitan | 398 | 0.331 | 0.376 | 0.376 | 0.435 | 0.435 |
228
- | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.886 | 0.868 | 0.868 | 0.468 | 0.468 |
229
- | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
230
- | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
231
- | `acu_Latn` | Achuar-Shiwiar | 180 | 0.743 | 0.854 | 0.854 | 0.000 | 0.000 |
232
- | `hus_Latn` | Huastec | 175 | 0.991 | 0.986 | 0.986 | 0.000 | 0.000 |
233
- | `hau_Latn` | Hausa | 155 | 0.997 | 1.000 | 1.000 | 0.876 | 0.876 |
234
- | `auc_Latn` | Waorani | 154 | 0.050 | 0.000 | 0.000 | 0.000 | 0.000 |
235
- | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.775 | 0.877 | 0.877 | 0.000 | 0.000 |
236
- | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
237
- | `snn_Latn` | Siona | 136 | 0.773 | 0.514 | 0.514 | 0.000 | 0.000 |
238
- | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.769 | 0.680 | 0.680 | 0.000 | 0.000 |
239
- | `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.992 | 0.000 | 0.000 |
240
- | `hat_Latn` | Haitian Creole | 117 | 0.991 | 0.979 | 0.979 | 0.699 | 0.699 |
241
- | `twi_Latn` | Twi | 116 | 0.987 | 0.996 | 0.996 | 0.399 | 0.396 |
242
- | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.857 | 0.000 | 0.000 |
243
- | `kmb_Latn` | Kimbundu | 116 | 0.979 | 1.000 | 1.000 | 0.877 | 0.877 |
244
- | `nya_Latn` | Nyanja | 116 | 0.987 | 1.000 | 1.000 | 0.806 | 0.806 |
245
- | `chr_Cher` | Cherokee | 115 | 0.715 | 0.663 | 0.663 | 0.000 | 0.000 |
246
 
247
  Where MillID gains most over GlotLID:
248
 
249
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
250
- |---|---|---|---|---|---|---|---|
251
- | `mic_Latn` | Mi'kmaw | 59 | 1.000 | 0.427 | 0.427 | 0.000 | 0.000 |
252
- | `tet_Latn` | Tetum | 58 | 0.699 | 0.167 | 0.167 | 0.000 | 0.000 |
253
- | `skr_Arab` | Saraiki | 63 | 0.923 | 0.500 | 0.500 | 0.000 | 0.000 |
254
- | `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.578 | 0.189 | 0.189 |
255
- | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.824 | 0.462 | 0.462 | 0.000 | 0.000 |
256
- | `fuf_Latn` | Pular | 58 | 0.816 | 0.483 | 0.483 | 0.000 | 0.000 |
257
- | `tzm_Tfng` | Central Atlas Tamazight | 1 | 0.333 | 0.065 | 0.065 | 0.033 | 0.033 |
258
- | `snn_Latn` | Siona | 136 | 0.773 | 0.514 | 0.514 | 0.000 | 0.000 |
259
- | `bum_Latn` | Bulu | 58 | 0.319 | 0.098 | 0.098 | 0.000 | 0.000 |
260
- | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.215 | 0.000 | 0.000 | 0.000 | 0.000 |
261
- | `pnb_Arab` | Western Panjabi | 61 | 0.931 | 0.739 | 0.739 | 0.000 | 0.000 |
262
- | `lld_Latn` | Ladin | 59 | 0.899 | 0.731 | 0.731 | 0.000 | 0.000 |
263
- | `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.812 | 0.966 | 0.966 |
264
- | `pbb_Latn` | Páez | 56 | 0.849 | 0.713 | 0.713 | 0.000 | 0.000 |
265
- | `eve_Cyrl` | Even | 49 | 0.990 | 0.886 | 0.886 | 0.000 | 0.000 |
266
 
267
  Where MillID loses most to GlotLID:
268
 
269
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
270
- |---|---|---|---|---|---|---|---|
271
- | `gyr_Latn` | Guarayu | 49 | 0.339 | 0.990 | 0.990 | 0.000 | 0.000 |
272
- | `cbu_Latn` | Candoshi-Shapra | 57 | 0.371 | 0.882 | 0.874 | 0.000 | 0.000 |
273
- | `shp_Latn` | Shipibo-Conibo | 62 | 0.410 | 0.912 | 0.912 | 0.000 | 0.000 |
274
- | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.434 | 0.015 | 0.015 |
275
- | `qvm_Latn` | Margos-Yarowilca-Lauricocha Quechua | 60 | 0.094 | 0.500 | 0.500 | 0.000 | 0.000 |
276
- | `cpu_Latn` | Pichis Ashéninka | 60 | 0.571 | 0.974 | 0.974 | 0.000 | 0.000 |
277
- | `hin_Deva` | Hindi | 59 | 0.607 | 0.957 | 0.957 | 0.975 | 0.975 |
278
- | `ell_Grek` | Greek | 115 | 0.655 | 1.000 | 1.000 | 1.000 | 1.000 |
279
- | `bho_Deva` | Bhojpuri | 56 | 0.643 | 0.933 | 0.933 | 0.832 | 0.832 |
280
- | `fuv_Latn` | Nigerian Fulfulde | 108 | 0.405 | 0.681 | 0.689 | 0.475 | 0.475 |
281
- | `cni_Latn` | Asháninka | 52 | 0.708 | 0.963 | 0.963 | 0.000 | 0.000 |
282
- | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.460 | 0.000 | 0.000 |
283
- | `ind_Latn` | Indonesian | 60 | 0.674 | 0.903 | 0.903 | 0.565 | 0.565 |
284
- | `maz_Latn` | Central Mazahua | 59 | 0.742 | 0.939 | 0.939 | 0.000 | 0.000 |
285
- | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.857 | 0.000 | 0.000 |
286
-
287
- ### Leaderboard: test-common
288
-
289
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **142** and within 0.01 on 65; against fastText restricted to MillID's labels, MillID is better on **93** languages, worse on **47** and within 0.01 on 73. The full table, with precision and recall, is `reports/test-common/leaderboard.csv`.
290
 
291
  The 20 languages with the most passages:
292
 
293
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
294
- |---|---|---|---|---|---|---|---|
295
- | `fra_Latn` | French | 1652 | 0.974 | 0.955 | 0.955 | 0.963 | 0.963 |
296
- | `eng_Latn` | English | 1444 | 0.941 | 0.876 | 0.877 | 0.857 | 0.857 |
297
- | `spa_Latn` | Spanish | 1136 | 0.953 | 0.958 | 0.958 | 0.941 | 0.941 |
298
- | `tam_Taml` | Tamil | 754 | 0.998 | 1.000 | 1.000 | 0.997 | 0.997 |
299
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.886 | 0.957 | 0.954 |
300
- | `por_Latn` | Portuguese | 657 | 0.963 | 0.954 | 0.956 | 0.950 | 0.949 |
301
- | `kan_Knda` | Kannada | 601 | 0.987 | 0.994 | 0.994 | 0.999 | 0.999 |
302
- | `zul_Latn` | Zulu | 594 | 0.871 | 0.904 | 0.904 | 0.970 | 0.970 |
303
- | `swh_Latn` | Swahili | 592 | 0.897 | 0.954 | 0.953 | 0.945 | 0.945 |
304
- | `mar_Deva` | Marathi | 589 | 0.981 | 0.989 | 0.990 | 0.987 | 0.987 |
305
- | `tso_Latn` | Tsonga | 581 | 0.984 | 0.956 | 0.956 | 0.986 | 0.986 |
306
- | `hun_Latn` | Hungarian | 576 | 0.991 | 1.000 | 1.000 | 0.980 | 0.980 |
307
- | `afr_Latn` | Afrikaans | 569 | 0.976 | 0.985 | 0.985 | 0.986 | 0.986 |
308
- | `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.998 | 0.999 | 0.999 |
309
- | `rus_Cyrl` | Russian | 528 | 0.929 | 0.985 | 0.985 | 0.927 | 0.927 |
310
- | `ory_Orya` | Odia (individual language) | 511 | 0.996 | 0.997 | 1.000 | 0.996 | 0.996 |
311
- | `sot_Latn` | Southern Sotho | 491 | 0.967 | 0.977 | 0.978 | 0.856 | 0.856 |
312
- | `nso_Latn` | Northern Sotho | 479 | 0.961 | 0.970 | 0.970 | 0.967 | 0.967 |
313
- | `asm_Beng` | Assamese | 467 | 0.995 | 0.999 | 0.999 | 0.997 | 0.997 |
314
- | `ben_Beng` | Bangla | 466 | 0.987 | 0.997 | 0.997 | 0.989 | 0.989 |
315
 
316
  Where MillID gains most over GlotLID:
317
 
318
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
319
- |---|---|---|---|---|---|---|---|
320
- | `ewo_Latn` | Ewondo | 134 | 0.965 | 0.879 | 0.883 | 0.860 | 0.860 |
321
- | `eng_Latn` | English | 1444 | 0.941 | 0.876 | 0.877 | 0.857 | 0.857 |
322
- | `arb_Latn` | Standard Arabic | 163 | 0.944 | 0.906 | 0.913 | 0.852 | 0.852 |
323
- | `tso_Latn` | Tsonga | 581 | 0.984 | 0.956 | 0.956 | 0.986 | 0.986 |
324
- | `fra_Latn` | French | 1652 | 0.974 | 0.955 | 0.955 | 0.963 | 0.963 |
325
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.886 | 0.957 | 0.954 |
326
- | `knc_Latn` | Central Kanuri | 293 | 0.959 | 0.952 | 0.952 | 0.981 | 0.981 |
327
- | `por_Latn` | Portuguese | 657 | 0.963 | 0.954 | 0.956 | 0.950 | 0.949 |
328
- | `san_Deva` | Sanskrit | 239 | 0.996 | 0.990 | 0.990 | 0.971 | 0.971 |
329
- | `bug_Latn` | Buginese | 78 | 0.987 | 0.981 | 0.981 | 0.938 | 0.938 |
330
- | `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.863 | 0.863 |
331
- | `mlt_Latn` | Maltese | 291 | 0.997 | 0.993 | 0.993 | 0.961 | 0.961 |
332
- | `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.998 | 0.999 | 0.999 |
333
- | `sin_Sinh` | Sinhala | 232 | 1.000 | 1.000 | 1.000 | 0.998 | 0.998 |
334
- | `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 1.000 | 0.990 | 0.990 |
335
 
336
  Where MillID loses most to GlotLID:
337
 
338
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
339
- |---|---|---|---|---|---|---|---|
340
- | `knc_Arab` | Central Kanuri | 68 | 0.562 | 0.857 | 0.860 | 0.029 | 0.029 |
341
- | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.645 | 0.934 | 0.934 | 0.751 | 0.751 |
342
- | `hrv_Latn` | Croatian | 168 | 0.717 | 0.994 | 0.994 | 0.856 | 0.856 |
343
- | `bos_Latn` | Bosnian | 198 | 0.727 | 1.000 | 1.000 | 0.897 | 0.897 |
344
- | `zsm_Latn` | Standard Malay | 244 | 0.773 | 0.998 | 0.998 | 0.908 | 0.908 |
345
- | `taq_Tfng` | Tamasheq | 104 | 0.613 | 0.805 | 0.805 | 0.000 | 0.000 |
346
- | `ind_Latn` | Indonesian | 335 | 0.807 | 0.991 | 0.991 | 0.887 | 0.887 |
347
- | `kin_Latn` | Kinyarwanda | 210 | 0.818 | 0.981 | 0.981 | 0.854 | 0.854 |
348
- | `run_Latn` | Rundi | 291 | 0.843 | 0.985 | 0.985 | 0.855 | 0.855 |
349
- | `nob_Latn` | Norwegian Bokmål | 265 | 0.820 | 0.961 | 0.959 | 0.894 | 0.894 |
350
- | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.797 | 0.877 | 0.892 | 0.859 | 0.859 |
351
- | `mag_Deva` | Magahi | 97 | 0.905 | 1.000 | 1.000 | 0.772 | 0.772 |
352
- | `vec_Latn` | Venetian | 49 | 0.766 | 0.861 | 0.861 | 0.620 | 0.620 |
353
- | `cjk_Latn` | Chokwe | 24 | 0.909 | 1.000 | 1.000 | 0.807 | 0.807 |
354
- | `taq_Latn` | Tamasheq | 106 | 0.909 | 1.000 | 1.000 | 0.838 | 0.838 |
355
-
356
- Per-language precision, recall (sensitivity), F1 and specificity for every model are in `reports/<set>/`.
357
 
358
  ## Training
359
 
 
51
 
52
  ## The MillID models
53
 
54
+ Three sizes, this one in bold. Macro F1 on `test` and `udhr`, the sets below:
 
 
 
 
55
 
56
+ | Model | Parameters | ONNX | Test F1 | UDHR F1 | Runs on |
57
+ |---|---|---|---|---|---|
58
+ | [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 11 MB | 0.9554 | 0.8764 | CPU, web page |
59
+ | **MillID medium v1** | 115M | 455 MB | 0.9576 | 0.8780 | GPU |
60
+ | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M | 764 MB | 0.9646 | 0.8812 | GPU |
61
+
62
+ All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 and 0.8873.
63
 
64
  ## Use it
65
 
 
101
 
102
  ## Evaluation against GlotLID and fastText
103
 
104
+ `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels) are each restricted to MillID's labels, their best answer among them: the like-for-like comparison. fastText's answers are mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. fastText cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every system on the passages whose language all of them know: the head-to-head on equal terms.
105
+
106
+ Macro F1 on each set, every system on the same passages; GlotLID and fastText restricted to MillID's labels:
107
+
108
+ | Set | Languages | MillID | GlotLID | fastText |
109
+ |---|---|---|---|---|
110
+ | `test` | 1,621 | 0.9576 | 0.9755 | 0.0659 |
111
+ | `test-common` | 213 | 0.9527 | 0.9846 | 0.9173 |
112
+ | `udhr` | 358 | 0.8780 | 0.8873 | 0.3499 |
113
+ | `udhr-common` | 170 | 0.9480 | 0.9659 | 0.9329 |
114
+ | `udhr-all` | 429 | 0.7184 | 0.7245 | 0.2797 |
115
+
116
+ MillID on each set:
117
+
118
+ | Set | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 |
119
+ |---|---|---|---|---|---|---|---|
120
+ | `validation` | 0.9613 | 0.9910 | 0.9949 | 0.9573 | 0.9621 | 0.9584 | 0.9850 |
121
+ | `test` | 0.9616 | 0.9910 | 0.9945 | 0.9564 | 0.9614 | 0.9576 | 0.9841 |
122
+ | `test-common` | 0.9425 | 0.9882 | 0.9924 | 0.9732 | 0.9361 | 0.9527 | 0.9738 |
123
+ | `udhr` | 0.8524 | 0.9194 | 0.9340 | 0.9187 | 0.8769 | 0.8780 | 0.9910 |
124
+ | `udhr-common` | 0.9121 | 0.9633 | 0.9688 | 0.9770 | 0.9382 | 0.9480 | 0.9955 |
125
+ | `udhr-all` | 0.7301 | 0.7875 | 0.8000 | 0.7415 | 0.7318 | 0.7184 | 0.9672 |
 
 
 
 
 
 
 
 
126
 
127
  Top-1 accuracy by passage length:
128
 
129
+ | Set | System | Short | Sentence | Paragraph |
130
  |---|---|---|---|---|
131
+ | `validation` | MillID | 0.9363 | 0.9818 | 0.9858 |
132
+ | `test` | MillID | 0.9382 | 0.9809 | 0.9846 |
133
+ | `test` | GlotLID | 0.9674 | 0.9858 | 0.9912 |
134
+ | `test` | fastText | 0.2764 | 0.2506 | 0.1959 |
135
+ | `test-common` | MillID | 0.9169 | 0.9696 | 0.9737 |
136
+ | `test-common` | GlotLID | 0.9609 | 0.9885 | 0.9949 |
137
+ | `test-common` | fastText | 0.9125 | 0.9581 | 0.9655 |
138
+
139
+ ### Per language: test
140
+
141
+ F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **110** languages, worse on **776** and within 0.01 on 735; against fastText, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in `reports/test/leaderboard.csv`.
 
 
 
 
142
 
143
  The 20 languages with the most passages:
144
 
145
+ | Label | Language | Passages | MillID | GlotLID | fastText |
146
+ |---|---|---|---|---|---|
147
+ | `fra_Latn` | French | 1652 | 0.952 | 0.928 | 0.808 |
148
+ | `eng_Latn` | English | 1444 | 0.916 | 0.859 | 0.648 |
149
+ | `spa_Latn` | Spanish | 1136 | 0.942 | 0.951 | 0.574 |
150
+ | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.965 |
151
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.829 | 0.785 | 0.512 |
152
+ | `por_Latn` | Portuguese | 657 | 0.955 | 0.949 | 0.755 |
153
+ | `kan_Knda` | Kannada | 601 | 0.982 | 0.993 | 0.922 |
154
+ | `zul_Latn` | Zulu | 594 | 0.833 | 0.896 | 0.539 |
155
+ | `swh_Latn` | Swahili | 592 | 0.858 | 0.916 | 0.346 |
156
+ | `mar_Deva` | Marathi | 589 | 0.981 | 0.990 | 0.926 |
157
+ | `tso_Latn` | Tsonga | 581 | 0.970 | 0.956 | 0.425 |
158
+ | `hun_Latn` | Hungarian | 576 | 0.990 | 0.999 | 0.232 |
159
+ | `afr_Latn` | Afrikaans | 569 | 0.974 | 0.985 | 0.860 |
160
+ | `tel_Telu` | Telugu | 555 | 0.994 | 0.994 | 0.951 |
161
+ | `nbl_Latn` | South Ndebele | 528 | 0.895 | 0.949 | 0.000 |
162
+ | `rus_Cyrl` | Russian | 528 | 0.925 | 0.949 | 0.462 |
163
+ | `ory_Orya` | Odia (individual language) | 511 | 0.993 | 0.994 | 0.976 |
164
+ | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.978 | 0.660 |
165
+ | `nso_Latn` | Northern Sotho | 479 | 0.960 | 0.969 | 0.914 |
166
+ | `kua_Latn` | Kuanyama | 476 | 0.979 | 0.986 | 0.000 |
167
 
168
  Where MillID gains most over GlotLID:
169
 
170
+ | Label | Language | Passages | MillID | GlotLID | fastText |
171
+ |---|---|---|---|---|---|
172
+ | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 |
173
+ | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 |
174
+ | `ajp_Arab` | South Levantine Arabic | 28 | 0.902 | 0.581 | 0.000 |
175
+ | `und_Laoo` | und | 20 | 0.857 | 0.621 | 0.000 |
176
+ | `sgh_Cyrl` | Shughni | 9 | 0.900 | 0.667 | 0.000 |
177
+ | `und_Thaa` | und | 29 | 0.873 | 0.651 | 0.000 |
178
+ | `icr_Latn` | Islander Creole English | 24 | 0.870 | 0.684 | 0.000 |
179
+ | `cek_Latn` | Eastern Khumi Chin | 67 | 0.853 | 0.684 | 0.000 |
180
+ | `dag_Latn` | Dagbani | 149 | 0.817 | 0.660 | 0.000 |
181
+ | `und_Syrc` | und | 43 | 0.977 | 0.822 | 0.000 |
182
+ | `tuk_Arab` | Turkmen | 135 | 0.903 | 0.761 | 0.000 |
183
+ | `orv_Cyrl` | Old Russian | 239 | 0.935 | 0.800 | 0.000 |
184
+ | `hae_Latn` | Eastern Oromo | 75 | 0.786 | 0.657 | 0.000 |
185
+ | `und_Kana` | und | 136 | 0.989 | 0.862 | 0.000 |
186
+ | `kby_Latn` | Manga Kanuri | 7 | 0.452 | 0.326 | 0.000 |
187
 
188
  Where MillID loses most to GlotLID:
189
 
190
+ | Label | Language | Passages | MillID | GlotLID | fastText |
191
+ |---|---|---|---|---|---|
192
+ | `ccp_Latn` | Chakma | 3 | 0.333 | 1.000 | 0.000 |
193
+ | `ttj_Latn` | Tooro | 28 | 0.490 | 0.946 | 0.000 |
194
+ | `hak_Hani` | Hakka Chinese | 29 | 0.415 | 0.824 | 0.000 |
195
+ | `mus_Latn` | Muscogee | 2 | 0.667 | 1.000 | 0.000 |
196
+ | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 0.000 |
197
+ | `knc_Arab` | Central Kanuri | 68 | 0.509 | 0.825 | 0.007 |
198
+ | `bdq_Latn` | Bahnar | 9 | 0.500 | 0.800 | 0.000 |
199
+ | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.500 | 0.783 | 0.000 |
200
+ | `und_Ethi` | und | 14 | 0.741 | 1.000 | 0.000 |
201
+ | `hrv_Latn` | Croatian | 168 | 0.687 | 0.944 | 0.178 |
202
+ | `tui_Latn` | Tupuri | 4 | 0.750 | 1.000 | 0.000 |
203
+ | `nse_Latn` | Nsenga | 12 | 0.500 | 0.750 | 0.000 |
204
+ | `dks_Latn` | Southeastern Dinka | 3 | 0.750 | 1.000 | 0.000 |
205
+ | `mbd_Latn` | Dibabawon Manobo | 14 | 0.757 | 1.000 | 0.000 |
206
+ | `lnd_Latn` | Lundayeh | 5 | 0.769 | 1.000 | 0.000 |
207
+
208
+ ### Per language: udhr
209
+
210
+ F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **63** languages, worse on **95** and within 0.01 on 200; against fastText, MillID is better on **304** languages, worse on **13** and within 0.01 on 41. Every language, with precision, recall and the unrestricted scores, is in `reports/udhr/leaderboard.csv`.
211
 
212
  The 20 languages with the most passages:
213
 
214
+ | Label | Language | Passages | MillID | GlotLID | fastText |
215
+ |---|---|---|---|---|---|
216
+ | `roh_Latn` | Romansh | 399 | 0.991 | 0.999 | 0.946 |
217
+ | `oci_Latn` | Occitan | 398 | 0.331 | 0.376 | 0.435 |
218
+ | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.886 | 0.868 | 0.468 |
219
+ | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.000 |
220
+ | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.000 | 0.000 | 0.000 |
221
+ | `acu_Latn` | Achuar-Shiwiar | 180 | 0.743 | 0.854 | 0.000 |
222
+ | `hus_Latn` | Huastec | 175 | 0.991 | 0.986 | 0.000 |
223
+ | `hau_Latn` | Hausa | 155 | 0.997 | 1.000 | 0.876 |
224
+ | `auc_Latn` | Waorani | 154 | 0.050 | 0.000 | 0.000 |
225
+ | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.775 | 0.877 | 0.000 |
226
+ | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 0.944 |
227
+ | `snn_Latn` | Siona | 136 | 0.773 | 0.514 | 0.000 |
228
+ | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.769 | 0.680 | 0.000 |
229
+ | `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.000 |
230
+ | `hat_Latn` | Haitian Creole | 117 | 0.991 | 0.979 | 0.699 |
231
+ | `twi_Latn` | Twi | 116 | 0.987 | 0.996 | 0.396 |
232
+ | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.000 |
233
+ | `kmb_Latn` | Kimbundu | 116 | 0.979 | 1.000 | 0.877 |
234
+ | `nya_Latn` | Nyanja | 116 | 0.987 | 1.000 | 0.806 |
235
+ | `chr_Cher` | Cherokee | 115 | 0.715 | 0.663 | 0.000 |
236
 
237
  Where MillID gains most over GlotLID:
238
 
239
+ | Label | Language | Passages | MillID | GlotLID | fastText |
240
+ |---|---|---|---|---|---|
241
+ | `mic_Latn` | Mi'kmaw | 59 | 1.000 | 0.427 | 0.000 |
242
+ | `tet_Latn` | Tetum | 58 | 0.699 | 0.167 | 0.000 |
243
+ | `skr_Arab` | Saraiki | 63 | 0.923 | 0.500 | 0.000 |
244
+ | `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.189 |
245
+ | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.824 | 0.462 | 0.000 |
246
+ | `fuf_Latn` | Pular | 58 | 0.816 | 0.483 | 0.000 |
247
+ | `tzm_Tfng` | Central Atlas Tamazight | 1 | 0.333 | 0.065 | 0.033 |
248
+ | `snn_Latn` | Siona | 136 | 0.773 | 0.514 | 0.000 |
249
+ | `bum_Latn` | Bulu | 58 | 0.319 | 0.098 | 0.000 |
250
+ | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.215 | 0.000 | 0.000 |
251
+ | `pnb_Arab` | Western Panjabi | 61 | 0.931 | 0.739 | 0.000 |
252
+ | `lld_Latn` | Ladin | 59 | 0.899 | 0.731 | 0.000 |
253
+ | `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.966 |
254
+ | `pbb_Latn` | Páez | 56 | 0.849 | 0.713 | 0.000 |
255
+ | `eve_Cyrl` | Even | 49 | 0.990 | 0.886 | 0.000 |
256
 
257
  Where MillID loses most to GlotLID:
258
 
259
+ | Label | Language | Passages | MillID | GlotLID | fastText |
260
+ |---|---|---|---|---|---|
261
+ | `gyr_Latn` | Guarayu | 49 | 0.339 | 0.990 | 0.000 |
262
+ | `cbu_Latn` | Candoshi-Shapra | 57 | 0.371 | 0.874 | 0.000 |
263
+ | `shp_Latn` | Shipibo-Conibo | 62 | 0.410 | 0.912 | 0.000 |
264
+ | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.015 |
265
+ | `qvm_Latn` | Margos-Yarowilca-Lauricocha Quechua | 60 | 0.094 | 0.500 | 0.000 |
266
+ | `cpu_Latn` | Pichis Ashéninka | 60 | 0.571 | 0.974 | 0.000 |
267
+ | `hin_Deva` | Hindi | 59 | 0.607 | 0.957 | 0.975 |
268
+ | `ell_Grek` | Greek | 115 | 0.655 | 1.000 | 1.000 |
269
+ | `bho_Deva` | Bhojpuri | 56 | 0.643 | 0.933 | 0.832 |
270
+ | `fuv_Latn` | Nigerian Fulfulde | 108 | 0.405 | 0.689 | 0.475 |
271
+ | `cni_Latn` | Asháninka | 52 | 0.708 | 0.963 | 0.000 |
272
+ | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.000 |
273
+ | `ind_Latn` | Indonesian | 60 | 0.674 | 0.903 | 0.565 |
274
+ | `maz_Latn` | Central Mazahua | 59 | 0.742 | 0.939 | 0.000 |
275
+ | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.000 |
276
+
277
+ ### Per language: test-common
278
+
279
+ F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **6** languages, worse on **142** and within 0.01 on 65; against fastText, MillID is better on **93** languages, worse on **47** and within 0.01 on 73. Every language, with precision, recall and the unrestricted scores, is in `reports/test-common/leaderboard.csv`.
280
 
281
  The 20 languages with the most passages:
282
 
283
+ | Label | Language | Passages | MillID | GlotLID | fastText |
284
+ |---|---|---|---|---|---|
285
+ | `fra_Latn` | French | 1652 | 0.974 | 0.955 | 0.963 |
286
+ | `eng_Latn` | English | 1444 | 0.941 | 0.877 | 0.857 |
287
+ | `spa_Latn` | Spanish | 1136 | 0.953 | 0.958 | 0.941 |
288
+ | `tam_Taml` | Tamil | 754 | 0.998 | 1.000 | 0.997 |
289
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.954 |
290
+ | `por_Latn` | Portuguese | 657 | 0.963 | 0.956 | 0.949 |
291
+ | `kan_Knda` | Kannada | 601 | 0.987 | 0.994 | 0.999 |
292
+ | `zul_Latn` | Zulu | 594 | 0.871 | 0.904 | 0.970 |
293
+ | `swh_Latn` | Swahili | 592 | 0.897 | 0.953 | 0.945 |
294
+ | `mar_Deva` | Marathi | 589 | 0.981 | 0.990 | 0.987 |
295
+ | `tso_Latn` | Tsonga | 581 | 0.984 | 0.956 | 0.986 |
296
+ | `hun_Latn` | Hungarian | 576 | 0.991 | 1.000 | 0.980 |
297
+ | `afr_Latn` | Afrikaans | 569 | 0.976 | 0.985 | 0.986 |
298
+ | `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.999 |
299
+ | `rus_Cyrl` | Russian | 528 | 0.929 | 0.985 | 0.927 |
300
+ | `ory_Orya` | Odia (individual language) | 511 | 0.996 | 1.000 | 0.996 |
301
+ | `sot_Latn` | Southern Sotho | 491 | 0.967 | 0.978 | 0.856 |
302
+ | `nso_Latn` | Northern Sotho | 479 | 0.961 | 0.970 | 0.967 |
303
+ | `asm_Beng` | Assamese | 467 | 0.995 | 0.999 | 0.997 |
304
+ | `ben_Beng` | Bangla | 466 | 0.987 | 0.997 | 0.989 |
305
 
306
  Where MillID gains most over GlotLID:
307
 
308
+ | Label | Language | Passages | MillID | GlotLID | fastText |
309
+ |---|---|---|---|---|---|
310
+ | `ewo_Latn` | Ewondo | 134 | 0.965 | 0.883 | 0.860 |
311
+ | `eng_Latn` | English | 1444 | 0.941 | 0.877 | 0.857 |
312
+ | `arb_Latn` | Standard Arabic | 163 | 0.944 | 0.913 | 0.852 |
313
+ | `tso_Latn` | Tsonga | 581 | 0.984 | 0.956 | 0.986 |
314
+ | `fra_Latn` | French | 1652 | 0.974 | 0.955 | 0.963 |
315
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.954 |
316
+ | `knc_Latn` | Central Kanuri | 293 | 0.959 | 0.952 | 0.981 |
317
+ | `por_Latn` | Portuguese | 657 | 0.963 | 0.956 | 0.949 |
318
+ | `san_Deva` | Sanskrit | 239 | 0.996 | 0.990 | 0.971 |
319
+ | `bug_Latn` | Buginese | 78 | 0.987 | 0.981 | 0.938 |
320
+ | `kam_Latn` | Kamba | 272 | 0.980 | 0.974 | 0.863 |
321
+ | `mlt_Latn` | Maltese | 291 | 0.997 | 0.993 | 0.961 |
322
+ | `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.999 |
323
+ | `sin_Sinh` | Sinhala | 232 | 1.000 | 1.000 | 0.998 |
324
+ | `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 0.990 |
325
 
326
  Where MillID loses most to GlotLID:
327
 
328
+ | Label | Language | Passages | MillID | GlotLID | fastText |
329
+ |---|---|---|---|---|---|
330
+ | `knc_Arab` | Central Kanuri | 68 | 0.562 | 0.860 | 0.029 |
331
+ | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.645 | 0.934 | 0.751 |
332
+ | `hrv_Latn` | Croatian | 168 | 0.717 | 0.994 | 0.856 |
333
+ | `bos_Latn` | Bosnian | 198 | 0.727 | 1.000 | 0.897 |
334
+ | `zsm_Latn` | Standard Malay | 244 | 0.773 | 0.998 | 0.908 |
335
+ | `taq_Tfng` | Tamasheq | 104 | 0.613 | 0.805 | 0.000 |
336
+ | `ind_Latn` | Indonesian | 335 | 0.807 | 0.991 | 0.887 |
337
+ | `kin_Latn` | Kinyarwanda | 210 | 0.818 | 0.981 | 0.854 |
338
+ | `run_Latn` | Rundi | 291 | 0.843 | 0.985 | 0.855 |
339
+ | `nob_Latn` | Norwegian Bokmål | 265 | 0.820 | 0.959 | 0.894 |
340
+ | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.797 | 0.892 | 0.859 |
341
+ | `mag_Deva` | Magahi | 97 | 0.905 | 1.000 | 0.772 |
342
+ | `vec_Latn` | Venetian | 49 | 0.766 | 0.861 | 0.620 |
343
+ | `cjk_Latn` | Chokwe | 24 | 0.909 | 1.000 | 0.807 |
344
+ | `taq_Latn` | Tamasheq | 106 | 0.909 | 1.000 | 0.838 |
345
+
346
+ Per-language precision, recall (sensitivity), F1 and specificity for every system, restricted and as it is, are in `reports/<set>/`, and the totals in `reports/summary.json`.
347
 
348
  ## Training
349