Upload README.md
Browse files
README.md
CHANGED
|
@@ -18,32 +18,32 @@ datasets:
|
|
| 18 |
- **Model type:** Text classification (language identification)
|
| 19 |
- **Language(s) (NLP):** en
|
| 20 |
- **License:** gpl-3.0
|
| 21 |
-
- **Resources for more information:** [OpenLID paper](https://aclanthology.org/2023.acl-short.75/)
|
| 22 |
|
| 23 |
## Model description
|
| 24 |
|
| 25 |
OpenLID-v2 is a high-coverage, high-performance language identification model. It is an improved version of [OpenLID](https://huggingface.co/laurievb/OpenLID).
|
| 26 |
|
| 27 |
-
The original model and training data are described in [Burchell et al. (2023)](https://aclanthology.org/2023.acl-short.75/). The changes made to produce OpenLID-v2
|
| 28 |
|
| 29 |
|
| 30 |
### How to use
|
| 31 |
|
| 32 |
-
Here is how to use this model to detect the language of a given text. For best results, text should be cleaned and normalised with [
|
| 33 |
|
| 34 |
```python
|
| 35 |
>>> import fasttext
|
| 36 |
-
>>> from
|
| 37 |
>>> from huggingface_hub import hf_hub_download
|
| 38 |
|
| 39 |
>>> model_path = hf_hub_download(repo_id="laurievb/OpenLID-v2", filename="openlid_v2.bin")
|
| 40 |
>>> model = fasttext.load_model(model_path)
|
| 41 |
-
>>> input_text =
|
| 42 |
>>> model.predict(input_text)
|
| 43 |
|
| 44 |
(('__label__eng_Latn',), array([0.81148803]))
|
| 45 |
|
| 46 |
-
>>> model.predict("Hello, world!", k=5)
|
| 47 |
|
| 48 |
(('__label__eng_Latn', '__label__vie_Latn', '__label__nld_Latn', '__label__pol_Latn', '__label__deu_Latn'),
|
| 49 |
array([0.61224753, 0.21323682, 0.09696738, 0.01359863, 0.01319415]))
|
|
@@ -51,13 +51,15 @@ Here is how to use this model to detect the language of a given text. For best r
|
|
| 51 |
|
| 52 |
### Limitations and bias
|
| 53 |
|
| 54 |
-
The dataset and model
|
|
|
|
|
|
|
| 55 |
|
| 56 |
Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
|
| 57 |
|
| 58 |
## Training data
|
| 59 |
|
| 60 |
-
The model was trained on the [OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2).
|
| 61 |
|
| 62 |
## Training procedure
|
| 63 |
|
|
@@ -76,7 +78,9 @@ The model was trained using fastText with the following hyperparameters set. All
|
|
| 76 |
|
| 77 |
### Evaluation datasets
|
| 78 |
|
| 79 |
-
|
|
|
|
|
|
|
| 80 |
|
| 81 |
### BibTeX entry and citation info
|
| 82 |
|
|
@@ -102,4 +106,209 @@ The model was evaluated using the FLORES-200 benchmark provided by Costa-jussà
|
|
| 102 |
pages = "865--879",
|
| 103 |
abstract = "Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033{\%} across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, which we audit manually to ensure reliability. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model{'}s performance, both in comparison to existing open models and by language class.",
|
| 104 |
}
|
| 105 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
- **Model type:** Text classification (language identification)
|
| 19 |
- **Language(s) (NLP):** en
|
| 20 |
- **License:** gpl-3.0
|
| 21 |
+
- **Resources for more information:** [OpenLID paper](https://aclanthology.org/2023.acl-short.75/)
|
| 22 |
|
| 23 |
## Model description
|
| 24 |
|
| 25 |
OpenLID-v2 is a high-coverage, high-performance language identification model. It is an improved version of [OpenLID](https://huggingface.co/laurievb/OpenLID).
|
| 26 |
|
| 27 |
+
The original model and training data are described in [Burchell et al. (2023)](https://aclanthology.org/2023.acl-short.75/). The changes made to produce OpenLID-v2 are described in [the OpenLID-v2 dataset repo](https://huggingface.co/datasets/laurievb/OpenLID-v2).
|
| 28 |
|
| 29 |
|
| 30 |
### How to use
|
| 31 |
|
| 32 |
+
Here is how to use this model to detect the language of a given text. For best results, text should be cleaned and normalised with [openlid_normer.clean_line](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/tools/openlid_normer.py) prior to classification.
|
| 33 |
|
| 34 |
```python
|
| 35 |
>>> import fasttext
|
| 36 |
+
>>> from openlid_normer import clean_line
|
| 37 |
>>> from huggingface_hub import hf_hub_download
|
| 38 |
|
| 39 |
>>> model_path = hf_hub_download(repo_id="laurievb/OpenLID-v2", filename="openlid_v2.bin")
|
| 40 |
>>> model = fasttext.load_model(model_path)
|
| 41 |
+
>>> input_text = clean_line("Hello, world!")
|
| 42 |
>>> model.predict(input_text)
|
| 43 |
|
| 44 |
(('__label__eng_Latn',), array([0.81148803]))
|
| 45 |
|
| 46 |
+
>>> model.predict("Hello, world!", k=5) # lower score for eng_Latn without cleaning
|
| 47 |
|
| 48 |
(('__label__eng_Latn', '__label__vie_Latn', '__label__nld_Latn', '__label__pol_Latn', '__label__deu_Latn'),
|
| 49 |
array([0.61224753, 0.21323682, 0.09696738, 0.01359863, 0.01319415]))
|
|
|
|
| 51 |
|
| 52 |
### Limitations and bias
|
| 53 |
|
| 54 |
+
The dataset and model cover 200 language varieties. However, some language varieties (e.g. Arabic dialects) are very hard to distinguish and in practice, it may only be possible to classify a input at the macrolanguage level.
|
| 55 |
+
|
| 56 |
+
The FLORES+ test set consists of sentences from a single domain (wiki articles), and so performance on this test set may not reflect how well our classifier works in other domains.
|
| 57 |
|
| 58 |
Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
|
| 59 |
|
| 60 |
## Training data
|
| 61 |
|
| 62 |
+
The model was trained on the [OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2). The data was normalised and classes were up/downsampled with temperature sampling prior to training; code to do this can be found [in the `scripts` directory](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/make_training_openlid.py) in the OpenLID-v2 dataset repository.
|
| 63 |
|
| 64 |
## Training procedure
|
| 65 |
|
|
|
|
| 78 |
|
| 79 |
### Evaluation datasets
|
| 80 |
|
| 81 |
+
We evaluate the model using the [FLORES+ evaluation benchmark](https://huggingface.co/datasets/openlanguagedata/flores_plus), normalising text prior to classification with [openlid_normer.clean_line](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/tools/openlid_normer.py). Full results are available below.
|
| 82 |
+
|
| 83 |
+
The original OpenLID model was evaluated using the FLORES-200 benchmark provided by Costa-jussà et al. (2022), with further information available in the [OpenLID paper](https://aclanthology.org/2023.acl-short.75/).
|
| 84 |
|
| 85 |
### BibTeX entry and citation info
|
| 86 |
|
|
|
|
| 106 |
pages = "865--879",
|
| 107 |
abstract = "Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033{\%} across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, which we audit manually to ensure reliability. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model{'}s performance, both in comparison to existing open models and by language class.",
|
| 108 |
}
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
## Evaluation results
|
| 112 |
+
|
| 113 |
+
| Language code | Line count | F1 score |
|
| 114 |
+
|---------------|------------|----------|
|
| 115 |
+
| ace_Arab | 6360 | 0.971029 |
|
| 116 |
+
| ace_Latn | 16845 | 0.998517 |
|
| 117 |
+
| acm_Arab | 5455 | 0.025121 |
|
| 118 |
+
| acq_Arab | 1831 | 0.001974 |
|
| 119 |
+
| aeb_Arab | 20541 | 0.488032 |
|
| 120 |
+
| afr_Latn | 1032866 | 0.999012 |
|
| 121 |
+
| als_Latn | 341372 | 1.0 |
|
| 122 |
+
| amh_Ethi | 810989 | 0.999506 |
|
| 123 |
+
| apc_Arab | 97293 | 0.386029 |
|
| 124 |
+
| arb_Arab | 7100646 | 0.33617 |
|
| 125 |
+
| ars_Arab | 25771 | 0.025373 |
|
| 126 |
+
| ary_Arab | 27376 | 0.579467 |
|
| 127 |
+
| arz_Arab | 69832 | 0.481471 |
|
| 128 |
+
| asm_Beng | 121242 | 1.0 |
|
| 129 |
+
| ast_Latn | 64998 | 0.991605 |
|
| 130 |
+
| awa_Deva | 8425 | 0.655352 |
|
| 131 |
+
| ayr_Latn | 140086 | 1.0 |
|
| 132 |
+
| azb_Arab | 10801 | 0.915957 |
|
| 133 |
+
| azj_Latn | 457599 | 0.998026 |
|
| 134 |
+
| bak_Cyrl | 63553 | 1.0 |
|
| 135 |
+
| bam_Latn | 9389 | 0.619494 |
|
| 136 |
+
| ban_Latn | 15202 | 0.977353 |
|
| 137 |
+
| bel_Cyrl | 83859 | 1.0 |
|
| 138 |
+
| bem_Latn | 378301 | 0.979612 |
|
| 139 |
+
| ben_Beng | 491942 | 0.996032 |
|
| 140 |
+
| bho_Deva | 53666 | 0.904134 |
|
| 141 |
+
| bjn_Arab | 6289 | 0.968215 |
|
| 142 |
+
| bjn_Latn | 20264 | 0.985665 |
|
| 143 |
+
| bod_Tibt | 2468 | 0.854072 |
|
| 144 |
+
| bos_Latn | 196005 | 0.69401 |
|
| 145 |
+
| bug_Latn | 7495 | 0.99504 |
|
| 146 |
+
| bul_Cyrl | 596120 | 1.0 |
|
| 147 |
+
| cat_Latn | 113745 | 0.99802 |
|
| 148 |
+
| ceb_Latn | 991957 | 0.998519 |
|
| 149 |
+
| ces_Latn | 424303 | 0.998026 |
|
| 150 |
+
| cjk_Latn | 35645 | 0.928159 |
|
| 151 |
+
| ckb_Arab | 24989 | 0.999506 |
|
| 152 |
+
| cmn_Hans | 1043000 | 0.986693 |
|
| 153 |
+
| cmn_Hant | 2011585 | 0.89396 |
|
| 154 |
+
| crh_Latn | 17398 | 0.992541 |
|
| 155 |
+
| cym_Latn | 97264 | 1.0 |
|
| 156 |
+
| dan_Latn | 2460965 | 0.989066 |
|
| 157 |
+
| deu_Latn | 652883 | 1.0 |
|
| 158 |
+
| dik_Latn | 25833 | 0.999011 |
|
| 159 |
+
| dyu_Latn | 16861 | 0.053309 |
|
| 160 |
+
| dzo_Tibt | 6903 | 0.886842 |
|
| 161 |
+
| ekk_Latn | 2984641 | 0.999506 |
|
| 162 |
+
| ell_Grek | 2977115 | 0.999506 |
|
| 163 |
+
| eng_Latn | 7514770 | 0.990206 |
|
| 164 |
+
| epo_Latn | 332895 | 0.999506 |
|
| 165 |
+
| eus_Latn | 613564 | 1.0 |
|
| 166 |
+
| ewe_Latn | 578181 | 0.998028 |
|
| 167 |
+
| fao_Latn | 38378 | 0.997036 |
|
| 168 |
+
| fij_Latn | 355285 | 1.0 |
|
| 169 |
+
| fil_Latn | 1178464 | 0.999013 |
|
| 170 |
+
| fin_Latn | 2299900 | 1.0 |
|
| 171 |
+
| fon_Latn | 30895 | 0.99802 |
|
| 172 |
+
| fra_Latn | 586064 | 0.99703 |
|
| 173 |
+
| fur_Latn | 53980 | 0.999506 |
|
| 174 |
+
| fuv_Latn | 13921 | 0.98191 |
|
| 175 |
+
| gaz_Latn | 331430 | 1.0 |
|
| 176 |
+
| gla_Latn | 49218 | 0.999506 |
|
| 177 |
+
| gle_Latn | 195791 | 1.0 |
|
| 178 |
+
| glg_Latn | 41582 | 0.994557 |
|
| 179 |
+
| gug_Latn | 78880 | 0.99852 |
|
| 180 |
+
| guj_Gujr | 834918 | 1.0 |
|
| 181 |
+
| hat_Latn | 294042 | 0.992643 |
|
| 182 |
+
| hau_Latn | 340263 | 0.989247 |
|
| 183 |
+
| heb_Hebr | 987305 | 0.999506 |
|
| 184 |
+
| hin_Deva | 1071332 | 0.799519 |
|
| 185 |
+
| hne_Deva | 52536 | 0.927026 |
|
| 186 |
+
| hrv_Latn | 785563 | 0.741921 |
|
| 187 |
+
| hun_Latn | 2559216 | 0.999506 |
|
| 188 |
+
| hye_Armn | 357578 | 1.0 |
|
| 189 |
+
| ibo_Latn | 484363 | 0.999013 |
|
| 190 |
+
| ilo_Latn | 966361 | 0.995573 |
|
| 191 |
+
| ind_Latn | 1682898 | 0.925908 |
|
| 192 |
+
| isl_Latn | 43332 | 0.998519 |
|
| 193 |
+
| ita_Latn | 478358 | 0.995547 |
|
| 194 |
+
| jav_Latn | 64377 | 0.988235 |
|
| 195 |
+
| jpn_Jpan | 886638 | 0.99852 |
|
| 196 |
+
| kab_Latn | 50772 | 0.829508 |
|
| 197 |
+
| kac_Latn | 11156 | 1.0 |
|
| 198 |
+
| kam_Latn | 51265 | 0.866741 |
|
| 199 |
+
| kan_Knda | 355427 | 1.0 |
|
| 200 |
+
| kas_Arab | 6225 | 0.979324 |
|
| 201 |
+
| kas_Deva | 6738 | 0.968925 |
|
| 202 |
+
| kat_Geor | 412072 | 1.0 |
|
| 203 |
+
| kaz_Cyrl | 50643 | 0.999506 |
|
| 204 |
+
| kbp_Latn | 52382 | 1.0 |
|
| 205 |
+
| kea_Latn | 5505 | 0.965764 |
|
| 206 |
+
| khk_Cyrl | 166505 | 1.0 |
|
| 207 |
+
| khm_Khmr | 75713 | 0.999506 |
|
| 208 |
+
| kik_Latn | 94116 | 0.963281 |
|
| 209 |
+
| kin_Latn | 439856 | 0.799766 |
|
| 210 |
+
| kir_Cyrl | 366840 | 1.0 |
|
| 211 |
+
| kmb_Latn | 90314 | 0.95809 |
|
| 212 |
+
| kmr_Latn | 15084 | 0.997041 |
|
| 213 |
+
| knc_Arab | 6337 | 0.702564 |
|
| 214 |
+
| knc_Latn | 6254 | 0.998516 |
|
| 215 |
+
| kor_Hang | 350945 | 1.0 |
|
| 216 |
+
| ktu_Latn | 206325 | 0.985352 |
|
| 217 |
+
| lao_Laoo | 24712 | 1.0 |
|
| 218 |
+
| lij_Latn | 27454 | 0.997531 |
|
| 219 |
+
| lim_Latn | 47490 | 0.994563 |
|
| 220 |
+
| lin_Latn | 538130 | 0.997041 |
|
| 221 |
+
| lit_Latn | 2360462 | 0.999506 |
|
| 222 |
+
| lmo_Latn | 33288 | 0.99505 |
|
| 223 |
+
| ltg_Latn | 14203 | 0.997033 |
|
| 224 |
+
| ltz_Latn | 36810 | 0.999506 |
|
| 225 |
+
| lua_Latn | 288714 | 0.996536 |
|
| 226 |
+
| lug_Latn | 245216 | 0.995569 |
|
| 227 |
+
| luo_Latn | 134777 | 0.998517 |
|
| 228 |
+
| lus_Latn | 191617 | 0.99802 |
|
| 229 |
+
| lvs_Latn | 2533501 | 0.997531 |
|
| 230 |
+
| mag_Deva | 6330 | 0.966281 |
|
| 231 |
+
| mai_Deva | 33093 | 0.988574 |
|
| 232 |
+
| mal_Mlym | 378020 | 1.0 |
|
| 233 |
+
| mar_Deva | 1006184 | 0.997536 |
|
| 234 |
+
| min_Latn | 31047 | 0.995547 |
|
| 235 |
+
| mkd_Cyrl | 393081 | 0.999506 |
|
| 236 |
+
| mlt_Latn | 2011002 | 0.996063 |
|
| 237 |
+
| mni_Beng | 47076 | 0.996063 |
|
| 238 |
+
| mos_Latn | 193219 | 0.976227 |
|
| 239 |
+
| mri_Latn | 47736 | 0.999506 |
|
| 240 |
+
| mya_Mymr | 547113 | 1.0 |
|
| 241 |
+
| nld_Latn | 2609642 | 0.994573 |
|
| 242 |
+
| nno_Latn | 98176 | 0.980779 |
|
| 243 |
+
| nob_Latn | 1749713 | 0.971935 |
|
| 244 |
+
| npi_Deva | 229595 | 0.995069 |
|
| 245 |
+
| nso_Latn | 552404 | 0.989237 |
|
| 246 |
+
| nus_Latn | 6294 | 1.0 |
|
| 247 |
+
| nya_Latn | 780066 | 0.994106 |
|
| 248 |
+
| oci_Latn | 239737 | 0.997289 |
|
| 249 |
+
| ory_Orya | 92475 | 1.0 |
|
| 250 |
+
| pag_Latn | 287179 | 0.998024 |
|
| 251 |
+
| pan_Guru | 354236 | 1.0 |
|
| 252 |
+
| pap_Latn | 397355 | 0.978703 |
|
| 253 |
+
| pbt_Arab | 276372 | 0.997041 |
|
| 254 |
+
| pes_Arab | 2810268 | 0.662182 |
|
| 255 |
+
| plt_Latn | 47052 | 1.0 |
|
| 256 |
+
| pol_Latn | 3035767 | 0.996553 |
|
| 257 |
+
| por_Latn | 3623950 | 0.992134 |
|
| 258 |
+
| prs_Arab | 31038 | 0.577474 |
|
| 259 |
+
| quy_Latn | 152002 | 1.0 |
|
| 260 |
+
| ron_Latn | 436311 | 0.998028 |
|
| 261 |
+
| run_Latn | 454887 | 0.850575 |
|
| 262 |
+
| rus_Cyrl | 6688484 | 1.0 |
|
| 263 |
+
| sag_Latn | 251562 | 0.999506 |
|
| 264 |
+
| san_Deva | 46056 | 0.990524 |
|
| 265 |
+
| sat_Olck | 29033 | 1.0 |
|
| 266 |
+
| scn_Latn | 39233 | 0.996059 |
|
| 267 |
+
| shn_Mymr | 22187 | 1.0 |
|
| 268 |
+
| sin_Sinh | 423966 | 1.0 |
|
| 269 |
+
| slk_Latn | 2815971 | 0.999012 |
|
| 270 |
+
| slv_Latn | 2684050 | 0.997044 |
|
| 271 |
+
| smo_Latn | 361969 | 0.998519 |
|
| 272 |
+
| sna_Latn | 754901 | 0.995084 |
|
| 273 |
+
| snd_Arab | 47901 | 0.998026 |
|
| 274 |
+
| som_Latn | 187966 | 0.998028 |
|
| 275 |
+
| sot_Latn | 1941 | 0.963115 |
|
| 276 |
+
| spa_Latn | 676635 | 0.993083 |
|
| 277 |
+
| srd_Latn | 46037 | 0.997531 |
|
| 278 |
+
| srp_Cyrl | 308075 | 0.999506 |
|
| 279 |
+
| ssw_Latn | 112237 | 0.989537 |
|
| 280 |
+
| sun_Latn | 46337 | 0.993076 |
|
| 281 |
+
| swe_Latn | 2429547 | 1.0 |
|
| 282 |
+
| swh_Latn | 226377 | 0.92972 |
|
| 283 |
+
| szl_Latn | 32177 | 0.996533 |
|
| 284 |
+
| tam_Taml | 550090 | 1.0 |
|
| 285 |
+
| taq_Latn | 10262 | 0.731371 |
|
| 286 |
+
| taq_Tfng | 6290 | 0.959677 |
|
| 287 |
+
| tat_Cyrl | 253516 | 1.0 |
|
| 288 |
+
| tel_Telu | 276262 | 1.0 |
|
| 289 |
+
| tgk_Cyrl | 131708 | 1.0 |
|
| 290 |
+
| tha_Thai | 728313 | 1.0 |
|
| 291 |
+
| tir_Ethi | 473470 | 0.999506 |
|
| 292 |
+
| tpi_Latn | 457544 | 0.999011 |
|
| 293 |
+
| tsn_Latn | 775066 | 0.974458 |
|
| 294 |
+
| tso_Latn | 747226 | 0.9941 |
|
| 295 |
+
| tuk_Latn | 157610 | 1.0 |
|
| 296 |
+
| tum_Latn | 233136 | 0.994584 |
|
| 297 |
+
| tur_Latn | 598819 | 0.992636 |
|
| 298 |
+
| twi_Latn | 538421 | 0.998516 |
|
| 299 |
+
| uig_Arab | 81940 | 1.0 |
|
| 300 |
+
| ukr_Cyrl | 1123812 | 1.0 |
|
| 301 |
+
| umb_Latn | 215640 | 0.983655 |
|
| 302 |
+
| urd_Arab | 487265 | 0.98062 |
|
| 303 |
+
| uzn_Latn | 1463925 | 0.99852 |
|
| 304 |
+
| vec_Latn | 41746 | 0.995074 |
|
| 305 |
+
| vie_Latn | 864979 | 0.999506 |
|
| 306 |
+
| war_Latn | 278265 | 1.0 |
|
| 307 |
+
| wol_Latn | 26985 | 0.996047 |
|
| 308 |
+
| xho_Latn | 907281 | 0.985309 |
|
| 309 |
+
| ydd_Hebr | 923 | 0.999506 |
|
| 310 |
+
| yor_Latn | 524493 | 0.996553 |
|
| 311 |
+
| yue_Hant | 59348 | 0.874099 |
|
| 312 |
+
| zgh_Tfng | 9485 | 0.96124 |
|
| 313 |
+
| zsm_Latn | 401337 | 0.954902 |
|
| 314 |
+
| zul_Latn | 941301 | 0.970106 |
|