Hugging Face's logo

baobabtech
/
millid

Text Classification
PyTorch
language-identification
leaderboard
🇪🇺 Region: EU
Model card Files
xet
 Community
2
millid / README.md
byamasupatrick's picture
byamasupatrick
MillID leaderboard against GlotLID and fastText
e2f16de verified
preview code
|
Raw History Blame
29.1 kB
metadata 
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
tags:
  - language-identification
  - leaderboard
datasets:
  - baobabtech/millid-dataset

MillID leaderboard

MillID (MultiLingual Language IDentifier) against GlotLID and Meta's fastText language identifier (NLLB). F1 is per language, then averaged with every language weighted equally (macro).

The models: MillID tiny v1.5, MillID medium v1, MillID large v1.5. Code, and running them in Python or JavaScript without a server: baobab-tech/millid.

Every score is on the test split of the build the models were trained on. The public baobabtech/millid-dataset is that build without Save the Children's and Nal'ibali's text, which is not ours to redistribute: the same splits, and a test split 1.3% smaller. udhr is the same everywhere.

  • MillID knows 1,641 labels, GlotLID over 2,000, fastText 218. Each external model is shown restricted to MillID's labels (its best answer among them), the like-for-like comparison; its as-is scores are below.
  • On test and udhr, fastText is mostly scored on languages it cannot answer. test-common and udhr-common keep the passages whose language all three know (about 210 languages): the head-to-head on equal terms.
  • Read GlotLID's scores with their bias in mind: the dataset's FinePDFs and FineWeb-2 passages were labelled by GlotLID, and GlotLID was trained on UDHR, so both favour it.
  • fastText's labels are mapped onto MillID's spelling: Simplified and Traditional Han as Hani, pes/prs as fas, est as ekk, tgl as fil.

Macro F1, every set

Set Languages MillID tiny v1.5 MillID medium v1 MillID large v1.5 GlotLID fastText
test: every language, passages from documents never seen in training 1621 0.9554 0.9576 0.9646 0.9755 0.0659
test-common: the languages all three systems know 213 0.9513 0.9527 0.9594 0.9846 0.9173
udhr: the Declaration of Human Rights, languages MillID knows 358 0.8764 0.8780 0.8812 0.8873 0.3499
udhr-common: the Declaration, languages all three know 170 0.9513 0.9480 0.9529 0.9659 0.9329
udhr-all: the Declaration, every language in it 429 0.7149 0.7184 0.7201 0.7245 0.2797

The best score on each set is in bold.

test

Every language, passages from documents never seen in training.

Model Size Top-1 Top-3 Top-5 Macro P Macro R Macro F1 Median F1 Languages F1 ≥ 0.9
GlotLID, restricted to MillID's labels fastText, 1.7 GB 0.9783 0.9978 0.9985 0.9778 0.9765 0.9755 1.0000 1490 of 1621
GlotLID fastText, 1.7 GB 0.9776 0.9977 0.9984 0.9783 0.9752 0.9748 1.0000 1488 of 1621
MillID large v1.5 192.6M 0.9674 0.9924 0.9956 0.9655 0.9663 0.9646 0.9894 1478 of 1621
MillID medium v1 115.0M 0.9616 0.9910 0.9945 0.9564 0.9614 0.9576 0.9841 1451 of 1621
MillID tiny v1.5 2.9M 0.9587 0.9890 0.9932 0.9550 0.9588 0.9554 0.9825 1446 of 1621
fastText, restricted to MillID's labels fastText, 1.2 GB 0.2515 0.2611 0.2625 0.0521 0.1209 0.0659 0.0000 26 of 1621
fastText (Meta, NLLB) fastText, 1.2 GB 0.2511 0.2610 0.2625 0.0520 0.1205 0.0657 0.0000 25 of 1621

test-common

The languages all three systems know.

Model Size Top-1 Top-3 Top-5 Macro P Macro R Macro F1 Median F1 Languages F1 ≥ 0.9
GlotLID, restricted to MillID's labels fastText, 1.7 GB 0.9749 0.9927 0.9948 0.9923 0.9790 0.9846 0.9983 203 of 213
GlotLID fastText, 1.7 GB 0.9738 0.9925 0.9946 0.9924 0.9771 0.9835 0.9983 201 of 213
MillID large v1.5 192.6M 0.9525 0.9898 0.9938 0.9757 0.9460 0.9594 0.9791 193 of 213
MillID medium v1 115.0M 0.9425 0.9882 0.9924 0.9732 0.9361 0.9527 0.9738 191 of 213
MillID tiny v1.5 2.9M 0.9396 0.9857 0.9909 0.9709 0.9353 0.9513 0.9726 188 of 213
fastText, restricted to MillID's labels fastText, 1.2 GB 0.9352 0.9710 0.9763 0.9354 0.9201 0.9173 0.9680 166 of 213
fastText (Meta, NLLB) fastText, 1.2 GB 0.9339 0.9707 0.9762 0.9357 0.9169 0.9149 0.9680 165 of 213

udhr

The Declaration of Human Rights, languages MillID knows.

Model Size Top-1 Top-3 Top-5 Macro P Macro R Macro F1 Median F1 Languages F1 ≥ 0.9
GlotLID, restricted to MillID's labels fastText, 1.7 GB 0.8614 0.9294 0.9433 0.9258 0.8844 0.8873 0.9915 291 of 358
GlotLID fastText, 1.7 GB 0.8598 0.9280 0.9421 0.9261 0.8826 0.8865 0.9915 287 of 358
MillID large v1.5 192.6M 0.8579 0.9183 0.9309 0.9171 0.8825 0.8812 0.9914 283 of 358
MillID medium v1 115.0M 0.8524 0.9194 0.9340 0.9187 0.8769 0.8780 0.9910 282 of 358
MillID tiny v1.5 2.9M 0.8492 0.9117 0.9257 0.9148 0.8736 0.8764 0.9907 274 of 358
fastText, restricted to MillID's labels fastText, 1.2 GB 0.4629 0.4857 0.4904 0.3152 0.4483 0.3499 0.0000 61 of 358
fastText (Meta, NLLB) fastText, 1.2 GB 0.4625 0.4857 0.4903 0.3152 0.4479 0.3497 0.0000 60 of 358

udhr-common

The Declaration, languages all three know.

Model Size Top-1 Top-3 Top-5 Macro P Macro R Macro F1 Median F1 Languages F1 ≥ 0.9
GlotLID, restricted to MillID's labels fastText, 1.7 GB 0.9338 0.9683 0.9736 0.9888 0.9579 0.9659 1.0000 156 of 170
GlotLID fastText, 1.7 GB 0.9336 0.9678 0.9732 0.9888 0.9576 0.9657 1.0000 155 of 170
MillID large v1.5 192.6M 0.9168 0.9630 0.9687 0.9800 0.9436 0.9529 1.0000 152 of 170
MillID tiny v1.5 2.9M 0.9123 0.9606 0.9655 0.9794 0.9393 0.9513 1.0000 149 of 170
MillID medium v1 115.0M 0.9121 0.9633 0.9688 0.9770 0.9382 0.9480 0.9955 153 of 170
fastText, restricted to MillID's labels fastText, 1.2 GB 0.9195 0.9647 0.9742 0.9474 0.9441 0.9329 1.0000 142 of 170
fastText (Meta, NLLB) fastText, 1.2 GB 0.9187 0.9647 0.9741 0.9474 0.9431 0.9323 1.0000 141 of 170

udhr-all

The Declaration, every language in it.

Model Size Top-1 Top-3 Top-5 Macro P Macro R Macro F1 Median F1 Languages F1 ≥ 0.9
GlotLID fastText, 1.7 GB 0.7502 0.8145 0.8282 0.7695 0.7529 0.7454 0.9796 277 of 429
GlotLID, restricted to MillID's labels fastText, 1.7 GB 0.7378 0.7960 0.8079 0.7425 0.7381 0.7245 0.9748 272 of 429
MillID large v1.5 192.6M 0.7348 0.7865 0.7973 0.7392 0.7364 0.7201 0.9643 266 of 429
MillID medium v1 115.0M 0.7301 0.7875 0.8000 0.7415 0.7318 0.7184 0.9672 266 of 429
MillID tiny v1.5 2.9M 0.7274 0.7809 0.7928 0.7340 0.7290 0.7149 0.9649 254 of 429
fastText, restricted to MillID's labels fastText, 1.2 GB 0.3964 0.4160 0.4200 0.2479 0.3741 0.2797 0.0000 55 of 429
fastText (Meta, NLLB) fastText, 1.2 GB 0.3961 0.4160 0.4200 0.2479 0.3737 0.2795 0.0000 54 of 429

Per language, test

Against GlotLID restricted, the best MillID is better on 165 languages and worse on 498; against fastText restricted, the best MillID is better on 1616 languages and worse on 0 (F1 differing by more than 0.01), of 1621. Every language is in leaderboard_test.csv.

The 30 languages with the most passages:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
fra_Latn French 1652 0.928 0.928 0.808 0.808 0.929 0.952 0.960
eng_Latn English 1444 0.858 0.859 0.653 0.648 0.917 0.916 0.934
spa_Latn Spanish 1136 0.951 0.951 0.574 0.574 0.930 0.942 0.955
tam_Taml Tamil 754 0.993 0.993 0.965 0.965 0.997 0.997 0.999
arb_Arab Arabic (Modern Standard) 745 0.785 0.785 0.513 0.512 0.814 0.829 0.833
por_Latn Portuguese 657 0.947 0.949 0.756 0.755 0.947 0.955 0.965
kan_Knda Kannada 601 0.993 0.993 0.922 0.922 0.969 0.982 0.965
zul_Latn Zulu 594 0.896 0.896 0.539 0.539 0.846 0.833 0.878
swh_Latn Swahili 592 0.917 0.916 0.346 0.346 0.848 0.858 0.870
mar_Deva Marathi 589 0.989 0.990 0.926 0.926 0.965 0.981 0.976
tso_Latn Tsonga 581 0.956 0.956 0.425 0.425 0.965 0.970 0.978
hun_Latn Hungarian 576 0.999 0.999 0.232 0.232 0.987 0.990 0.991
afr_Latn Afrikaans 569 0.985 0.985 0.861 0.860 0.967 0.974 0.979
tel_Telu Telugu 555 0.994 0.994 0.951 0.951 0.991 0.994 0.988
rus_Cyrl Russian 528 0.949 0.949 0.462 0.462 0.932 0.925 0.949
nbl_Latn South Ndebele 528 0.948 0.949 0.000 0.000 0.903 0.895 0.927
ory_Orya Odia (individual language) 511 0.991 0.994 0.976 0.976 0.988 0.993 0.993
sot_Latn Southern Sotho 491 0.977 0.978 0.660 0.660 0.966 0.966 0.972
nso_Latn Northern Sotho 479 0.969 0.969 0.915 0.914 0.956 0.960 0.962
kua_Latn Kuanyama 476 0.986 0.986 0.000 0.000 0.962 0.979 0.975
asm_Beng Assamese 467 0.999 0.999 0.990 0.990 0.997 0.994 0.996
ben_Beng Bangla 466 0.997 0.997 0.718 0.718 0.977 0.980 0.979
pan_Guru Punjabi 460 1.000 1.000 0.990 0.990 1.000 1.000 1.000
amh_Ethi Amharic 458 1.000 1.000 0.807 0.807 0.981 0.984 0.986
tsn_Latn Tswana 456 0.961 0.961 0.652 0.652 0.945 0.939 0.947
vro_Latn Võro 448 0.990 0.990 0.000 0.000 0.947 0.958 0.973
gaz_Latn Oromo (West Central) 441 0.938 0.938 0.407 0.407 0.910 0.943 0.928
som_Latn Somali 439 0.994 0.994 0.487 0.487 0.982 0.988 0.990
uzn_Latn Northern Uzbek 436 0.994 0.994 0.449 0.449 0.976 0.978 0.982
tha_Thai Thai 431 0.980 0.980 0.650 0.650 0.979 0.970 0.981

Where MillID gains most over GlotLID:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
lbj_Tibt Ladakhi 66 0.000 0.000 0.000 0.000 0.943 0.943 0.923
ain_Latn Ainu 1 0.000 0.000 0.000 0.000 0.667 0.667 0.667
ajp_Arab South Levantine Arabic 28 0.581 0.581 0.000 0.000 0.946 0.902 0.902
und_Laoo und 20 0.621 0.621 0.000 0.000 0.923 0.857 0.788
sgh_Cyrl Shughni 9 0.667 0.667 0.000 0.000 0.857 0.900 0.889
und_Thaa und 29 0.651 0.651 0.000 0.000 0.881 0.873 0.863
icr_Latn Islander Creole English 24 0.649 0.684 0.000 0.000 0.698 0.870 0.700
und_Geor und 32 0.694 0.694 0.000 0.000 0.877 0.706 0.792
tdt_Latn Tetun Dili 57 0.603 0.603 0.000 0.000 0.693 0.656 0.782
cek_Latn Eastern Khumi Chin 67 0.680 0.684 0.000 0.000 0.785 0.853 0.825
und_Syrc und 43 0.822 0.822 0.000 0.000 0.966 0.977 0.989
neb_Latn Toura (Côte d'Ivoire) 11 0.625 0.842 0.000 0.000 0.957 0.957 1.000
dag_Latn Dagbani 149 0.662 0.660 0.000 0.000 0.767 0.817 0.814
tuk_Arab Turkmen 135 0.761 0.761 0.000 0.000 0.905 0.903 0.917
orv_Cyrl Old Russian 239 0.800 0.800 0.000 0.000 0.931 0.935 0.938

Where MillID loses most to GlotLID:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
mus_Latn Muscogee 2 1.000 1.000 0.000 0.000 0.333 0.667 0.667
ttj_Latn Tooro 28 0.946 0.946 0.000 0.000 0.485 0.490 0.622
knc_Arab Central Kanuri 68 0.823 0.825 0.007 0.007 0.462 0.509 0.532
hrv_Latn Croatian 168 0.944 0.944 0.178 0.178 0.622 0.687 0.707
esi_Latn North Alaskan Inupiatun 9 0.783 0.783 0.000 0.000 0.455 0.500 0.556
hak_Hani Hakka Chinese 29 0.824 0.824 0.000 0.000 0.621 0.415 0.581
bos_Latn Bosnian 198 0.902 0.902 0.181 0.181 0.649 0.673 0.700
plw_Latn Brooke's Point Palawano 8 1.000 1.000 0.000 0.000 0.727 0.800 0.762
ccp_Latn Chakma 3 1.000 1.000 0.000 0.000 0.800 0.333 0.750
mbd_Latn Dibabawon Manobo 14 1.000 1.000 0.000 0.000 0.596 0.757 0.800
taq_Tfng Tamasheq 104 0.718 0.718 0.000 0.000 0.529 0.535 0.527
zsm_Latn Standard Malay 244 0.957 0.957 0.321 0.319 0.770 0.746 0.781
biu_Latn Biete 7 1.000 1.000 0.000 0.000 0.667 0.667 0.833
ldi_Latn Laari 62 0.992 0.992 0.000 0.000 0.777 0.831 0.765
tzm_Tfng Central Atlas Tamazight 157 0.816 0.816 0.569 0.569 0.659 0.620 0.614

Per language, test-common

Against GlotLID restricted, the best MillID is better on 13 languages and worse on 114; against fastText restricted, the best MillID is better on 111 languages and worse on 31 (F1 differing by more than 0.01), of 213. Every language is in leaderboard_test-common.csv.

The 30 languages with the most passages:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
fra_Latn French 1652 0.955 0.955 0.963 0.963 0.943 0.974 0.983
eng_Latn English 1444 0.876 0.877 0.857 0.857 0.932 0.941 0.953
spa_Latn Spanish 1136 0.958 0.958 0.941 0.941 0.937 0.953 0.972
tam_Taml Tamil 754 1.000 1.000 0.997 0.997 0.997 0.998 0.999
arb_Arab Arabic (Modern Standard) 745 0.886 0.886 0.957 0.954 0.871 0.905 0.905
por_Latn Portuguese 657 0.954 0.956 0.950 0.949 0.959 0.963 0.973
kan_Knda Kannada 601 0.994 0.994 0.999 0.999 0.973 0.987 0.967
zul_Latn Zulu 594 0.904 0.904 0.970 0.970 0.883 0.871 0.909
swh_Latn Swahili 592 0.954 0.953 0.945 0.945 0.893 0.897 0.909
mar_Deva Marathi 589 0.989 0.990 0.987 0.987 0.968 0.981 0.979
tso_Latn Tsonga 581 0.956 0.956 0.986 0.986 0.979 0.984 0.985
hun_Latn Hungarian 576 1.000 1.000 0.980 0.980 0.989 0.991 0.993
afr_Latn Afrikaans 569 0.985 0.985 0.986 0.986 0.972 0.976 0.983
tel_Telu Telugu 555 0.998 0.998 0.999 0.999 0.997 1.000 0.999
rus_Cyrl Russian 528 0.985 0.985 0.927 0.927 0.945 0.929 0.956
ory_Orya Odia (individual language) 511 0.997 1.000 0.996 0.996 0.994 0.996 0.995
sot_Latn Southern Sotho 491 0.977 0.978 0.856 0.856 0.969 0.967 0.974
nso_Latn Northern Sotho 479 0.970 0.970 0.967 0.967 0.958 0.961 0.964
asm_Beng Assamese 467 0.999 0.999 0.997 0.997 0.997 0.995 0.997
ben_Beng Bangla 466 0.997 0.997 0.989 0.989 0.988 0.987 0.993
pan_Guru Punjabi 460 1.000 1.000 0.990 0.990 1.000 1.000 1.000
amh_Ethi Amharic 458 1.000 1.000 0.998 0.998 0.984 0.987 0.986
tsn_Latn Tswana 456 0.961 0.961 0.870 0.870 0.948 0.940 0.947
gaz_Latn Oromo (West Central) 441 0.974 0.974 0.973 0.973 0.935 0.955 0.941
som_Latn Somali 439 0.995 0.995 0.991 0.991 0.984 0.992 0.991
uzn_Latn Northern Uzbek 436 0.999 0.999 0.985 0.985 0.984 0.984 0.986
tha_Thai Thai 431 0.980 0.980 0.981 0.981 0.980 0.970 0.981
ydd_Hebr Eastern Yiddish 430 1.000 1.000 0.999 0.999 0.993 0.998 0.994
tur_Latn Turkish 430 0.999 0.999 0.980 0.980 0.968 0.974 0.973
nno_Latn Norwegian Nynorsk 427 0.985 0.985 0.954 0.954 0.913 0.919 0.927

Where MillID gains most over GlotLID:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
ewo_Latn Ewondo 134 0.879 0.883 0.860 0.860 0.959 0.965 0.978
eng_Latn English 1444 0.876 0.877 0.857 0.857 0.932 0.941 0.953
arb_Latn Standard Arabic 163 0.906 0.913 0.852 0.852 0.942 0.944 0.949
tso_Latn Tsonga 581 0.956 0.956 0.986 0.986 0.979 0.984 0.985
fra_Latn French 1652 0.955 0.955 0.963 0.963 0.943 0.974 0.983
arb_Arab Arabic (Modern Standard) 745 0.886 0.886 0.957 0.954 0.871 0.905 0.905
por_Latn Portuguese 657 0.954 0.956 0.950 0.949 0.959 0.963 0.973
hin_Deva Hindi 392 0.877 0.877 0.863 0.863 0.855 0.817 0.893
fuv_Latn Nigerian Fulfulde 164 0.877 0.892 0.859 0.859 0.907 0.797 0.844
kam_Latn Kamba 272 0.972 0.974 0.863 0.863 0.980 0.980 0.989
knc_Latn Central Kanuri 293 0.952 0.952 0.981 0.981 0.967 0.959 0.967
spa_Latn Spanish 1136 0.958 0.958 0.941 0.941 0.937 0.953 0.972
bug_Latn Buginese 78 0.981 0.981 0.938 0.938 0.981 0.987 0.994
bjn_Arab Banjar 116 0.834 0.987 0.453 0.960 0.992 0.983 0.996
san_Deva Sanskrit 239 0.990 0.990 0.971 0.971 0.990 0.996 0.998

Where MillID loses most to GlotLID:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
knc_Arab Central Kanuri 68 0.857 0.860 0.029 0.029 0.522 0.562 0.598
bos_Latn Bosnian 198 1.000 1.000 0.897 0.897 0.685 0.727 0.762
tzm_Tfng Central Atlas Tamazight 157 0.934 0.934 0.751 0.751 0.697 0.645 0.649
hrv_Latn Croatian 168 0.994 0.994 0.856 0.856 0.659 0.717 0.757
zsm_Latn Standard Malay 244 0.998 0.998 0.908 0.908 0.789 0.773 0.809
taq_Tfng Tamasheq 104 0.805 0.805 0.000 0.000 0.654 0.613 0.604
ind_Latn Indonesian 335 0.991 0.991 0.887 0.887 0.842 0.807 0.848
nob_Latn Norwegian Bokmål 265 0.961 0.959 0.894 0.894 0.821 0.820 0.832
kin_Latn Kinyarwanda 210 0.981 0.981 0.854 0.854 0.849 0.818 0.864
run_Latn Rundi 291 0.985 0.985 0.855 0.855 0.883 0.843 0.879
mag_Deva Magahi 97 1.000 1.000 0.772 0.772 0.890 0.905 0.897
cjk_Latn Chokwe 24 1.000 1.000 0.807 0.807 0.857 0.909 0.837
taq_Latn Tamasheq 106 1.000 1.000 0.838 0.838 0.902 0.909 0.922
dzo_Tibt Dzongkha 346 0.999 0.999 0.696 0.696 0.906 0.926 0.916
lim_Latn Limburgish 196 1.000 1.000 0.956 0.956 0.896 0.915 0.930

Per language, udhr

Against GlotLID restricted, the best MillID is better on 75 languages and worse on 57; against fastText restricted, the best MillID is better on 313 languages and worse on 8 (F1 differing by more than 0.01), of 358. Every language is in leaderboard_udhr.csv.

The 30 languages with the most passages:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
roh_Latn Romansh 399 0.999 0.999 0.946 0.946 0.994 0.991 0.998
oci_Latn Occitan 398 0.376 0.376 0.435 0.435 0.277 0.331 0.338
cmn_Hani Chinese (Mandarin) 319 0.868 0.868 0.468 0.468 0.892 0.886 0.875
krl_Latn Karelian 183 0.011 0.011 0.000 0.000 0.000 0.000 0.000
qug_Latn Chimborazo Highland Quichua 182 0.000 0.000 0.000 0.000 0.011 0.000 0.000
acu_Latn Achuar-Shiwiar 180 0.854 0.854 0.000 0.000 0.828 0.743 0.801
hus_Latn Huastec 175 0.986 0.986 0.000 0.000 0.977 0.991 0.988
hau_Latn Hausa 155 1.000 1.000 0.876 0.876 0.990 0.997 1.000
auc_Latn Waorani 154 0.000 0.000 0.000 0.000 0.000 0.050 0.025
kwi_Latn Awa-Cuaiquer 145 0.877 0.877 0.000 0.000 0.880 0.775 0.863
jpn_Jpan Japanese 140 1.000 1.000 0.944 0.944 1.000 1.000 1.000
snn_Latn Siona 136 0.514 0.514 0.000 0.000 0.630 0.773 0.467
ktu_Latn Kituba (Democratic Republic of Congo) 118 0.680 0.680 0.000 0.000 0.765 0.769 0.773
lat_Latn Latin 118 0.992 0.992 0.000 0.000 0.996 0.996 0.996
hat_Latn Haitian Creole 117 0.979 0.979 0.699 0.699 0.992 0.991 0.987
nya_Latn Nyanja 116 1.000 1.000 0.806 0.806 0.991 0.987 0.978
kmb_Latn Kimbundu 116 1.000 1.000 0.877 0.877 0.970 0.979 0.991
rmn_Latn Balkan Romani 116 0.857 0.857 0.000 0.000 0.678 0.667 0.704
twi_Latn Twi 116 0.996 0.996 0.399 0.396 0.983 0.987 0.991
chr_Cher Cherokee 115 0.663 0.663 0.000 0.000 0.670 0.715 0.945
por_Latn Portuguese 115 0.996 0.996 0.793 0.793 0.955 0.996 1.000
lvs_Latn Standard Latvian 115 1.000 1.000 0.991 0.991 0.987 0.996 1.000
pov_Latn Upper Guinea Crioulo 115 0.833 0.833 0.000 0.000 0.821 0.838 0.827
kng_Latn Koongo 115 0.193 0.193 0.457 0.457 0.076 0.047 0.047
ell_Grek Greek 115 1.000 1.000 1.000 1.000 0.670 0.655 0.670
tha_Thai Thai 114 1.000 1.000 0.996 0.996 1.000 1.000 1.000
tso_Latn Tsonga 114 0.781 0.781 0.691 0.691 0.860 0.882 0.774
cjk_Latn Chokwe 114 1.000 1.000 0.533 0.533 0.871 0.949 0.959
fas_Arab Persian 114 1.000 1.000 0.934 0.934 0.991 0.991 0.982
lin_Latn Lingala 110 1.000 1.000 0.986 0.986 0.991 1.000 0.996

Where MillID gains most over GlotLID:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
mic_Latn Mi'kmaw 59 0.427 0.427 0.000 0.000 0.992 1.000 1.000
tet_Latn Tetum 58 0.167 0.167 0.000 0.000 0.638 0.699 0.708
fuf_Latn Pular 58 0.483 0.483 0.000 0.000 0.906 0.816 0.936
skr_Arab Saraiki 63 0.500 0.500 0.000 0.000 0.867 0.923 0.875
sag_Latn Sango 59 0.578 0.578 0.189 0.189 0.992 0.992 0.992
zgh_Tfng Standard Moroccan Tamazight 60 0.462 0.462 0.000 0.000 0.750 0.824 0.652
qvh_Latn Huamalíes-Dos de Mayo Huánuco Quechua 58 0.154 0.154 0.000 0.000 0.468 0.215 0.215
chr_Cher Cherokee 115 0.663 0.663 0.000 0.000 0.670 0.715 0.945
tzm_Tfng Central Atlas Tamazight 1 0.065 0.065 0.033 0.033 0.167 0.333 0.100
snn_Latn Siona 136 0.514 0.514 0.000 0.000 0.630 0.773 0.467
fkv_Latn Kven Finnish 58 0.667 0.667 0.000 0.000 0.635 0.697 0.906
bum_Latn Bulu 58 0.098 0.098 0.000 0.000 0.215 0.319 0.188
mxv_Latn Metlatónoc Mixtec 58 0.000 0.000 0.000 0.000 0.129 0.215 0.034
pnb_Arab Western Panjabi 61 0.739 0.739 0.000 0.000 0.882 0.931 0.884
lld_Latn Ladin 59 0.731 0.731 0.000 0.000 0.909 0.899 0.881

Where MillID loses most to GlotLID:

Label Language Passages GlotLID GlotLID restricted fastText fastText restricted MillID tiny v1.5 MillID medium v1 MillID large v1.5
gyr_Latn Guarayu 49 0.990 0.990 0.000 0.000 0.537 0.339 0.393
dyu_Latn Dyula 58 0.434 0.434 0.015 0.015 0.000 0.000 0.000
ell_Grek Greek 115 1.000 1.000 1.000 1.000 0.670 0.655 0.670
cbu_Latn Candoshi-Shapra 57 0.882 0.874 0.000 0.000 0.500 0.371 0.557
shp_Latn Shipibo-Conibo 62 0.912 0.912 0.000 0.000 0.469 0.410 0.607
cpu_Latn Pichis Ashéninka 60 0.974 0.974 0.000 0.000 0.729 0.571 0.571
qvn_Latn North Junín Quechua 57 0.460 0.460 0.000 0.000 0.219 0.219 0.034
hin_Deva Hindi 59 0.957 0.957 0.975 0.975 0.723 0.607 0.727
cni_Latn Asháninka 52 0.963 0.963 0.000 0.000 0.773 0.708 0.717
hrv_Latn Croatian 58 0.727 0.727 0.474 0.474 0.500 0.537 0.559
rmn_Latn Balkan Romani 116 0.857 0.857 0.000 0.000 0.678 0.667 0.704
koi_Cyrl Komi-Permyak 57 1.000 1.000 0.000 0.000 0.837 0.837 0.849
qvm_Latn Margos-Yarowilca-Lauricocha Quechua 60 0.500 0.500 0.000 0.000 0.356 0.094 0.356
kea_Latn Kabuverdianu 58 0.935 0.935 0.125 0.124 0.792 0.794 0.773
ind_Latn Indonesian 60 0.903 0.903 0.565 0.565 0.768 0.674 0.645