Hugging Face's logo

laurievb
/
OpenLID-v2

Text Classification
fastText
language-identification
Model card Files
xet
Community
2
laurievb commited on
Commit
35a8ff1
·
verified ·
1 Parent(s): e807787

hplt (#3)

Browse files

- Delete scripts (9d1827993ae2f4d14ea4813e1069ffc79e1a677e)
- Upload README.md (46b2879c11af2de08a16df5c187e22557d30658d)
- Upload openlid_v2.bin (01d8e4586b42376e0fd1c25dda2fb3ec140c912e)

Files changed (5) hide show
  1. README.md +219 -10
  2. openlid_v2.bin +2 -2
  3. scripts/openlid.py +0 -87
  4. scripts/prepare_openlid_v2_for_model_training.sh +0 -17
  5. scripts/sample_with_temperature.py +0 -98
README.md CHANGED
@@ -18,32 +18,32 @@ datasets:
18
  - **Model type:** Text classification (language identification)
19
  - **Language(s) (NLP):** en
20
  - **License:** gpl-3.0
21
- - **Resources for more information:** [OpenLID paper](https://aclanthology.org/2023.acl-short.75/) and [OpenLID v2 blog](https://laurieburchell.github.io/2024/11/12/OpenLID-v2.html)
22
 
23
  ## Model description
24
 
25
  OpenLID-v2 is a high-coverage, high-performance language identification model. It is an improved version of [OpenLID](https://huggingface.co/laurievb/OpenLID).
26
 
27
- The original model and training data are described in [Burchell et al. (2023)](https://aclanthology.org/2023.acl-short.75/). The changes made to produce OpenLID-v2 and the rationale behind them are described in [this blog post](https://laurieburchell.github.io/2024/11/12/OpenLID-v2.html).
28
 
29
 
30
  ### How to use
31
 
32
- Here is how to use this model to detect the language of a given text. For best results, text should be cleaned and normalised with [openlid.clean_text()](scripts/openlid.py) prior to classification:
33
 
34
  ```python
35
  >>> import fasttext
36
- >>> from openlid import clean_text
37
  >>> from huggingface_hub import hf_hub_download
38
 
39
  >>> model_path = hf_hub_download(repo_id="laurievb/OpenLID-v2", filename="openlid_v2.bin")
40
  >>> model = fasttext.load_model(model_path)
41
- >>> input_text = clean_text("Hello, world!")
42
  >>> model.predict(input_text)
43
 
44
  (('__label__eng_Latn',), array([0.81148803]))
45
 
46
- >>> model.predict("Hello, world!", k=5)
47
 
48
  (('__label__eng_Latn', '__label__vie_Latn', '__label__nld_Latn', '__label__pol_Latn', '__label__deu_Latn'),
49
  array([0.61224753, 0.21323682, 0.09696738, 0.01359863, 0.01319415]))
@@ -51,13 +51,15 @@ Here is how to use this model to detect the language of a given text. For best r
51
 
52
  ### Limitations and bias
53
 
54
- The dataset and model only covers 189 languages. In addition, because the FLORES-200 test set consists of sentences from a single domain (wiki articles), performance on this test set may not reflect how well our classifier works in other domains.
 
 
55
 
56
  Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
57
 
58
  ## Training data
59
 
60
- The model was trained on the [OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2). Classes were up/downsampled with temperature sampling prior to training; code to do this can be found [in the `scripts` directory](scripts/prepare_openlid_v2_for_model_training.sh).
61
 
62
  ## Training procedure
63
 
@@ -76,7 +78,9 @@ The model was trained using fastText with the following hyperparameters set. All
76
 
77
  ### Evaluation datasets
78
 
79
- The model was evaluated using the FLORES-200 benchmark provided by Costa-jussà et al. (2022) using [normalised language labels](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/relabel_data.py). Further information is available in the [OpenLID paper](https://aclanthology.org/2023.acl-short.75/) and [OpenLID v2 blog](https://laurieburchell.github.io/2024/11/12/OpenLID-v2.html).
 
 
80
 
81
  ### BibTeX entry and citation info
82
 
@@ -102,4 +106,209 @@ The model was evaluated using the FLORES-200 benchmark provided by Costa-jussà
102
  pages = "865--879",
103
  abstract = "Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033{\%} across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, which we audit manually to ensure reliability. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model{'}s performance, both in comparison to existing open models and by language class.",
104
  }
105
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  - **Model type:** Text classification (language identification)
19
  - **Language(s) (NLP):** en
20
  - **License:** gpl-3.0
21
+ - **Resources for more information:** [OpenLID paper](https://aclanthology.org/2023.acl-short.75/)
22
 
23
  ## Model description
24
 
25
  OpenLID-v2 is a high-coverage, high-performance language identification model. It is an improved version of [OpenLID](https://huggingface.co/laurievb/OpenLID).
26
 
27
+ The original model and training data are described in [Burchell et al. (2023)](https://aclanthology.org/2023.acl-short.75/). The changes made to produce OpenLID-v2 are described in [the OpenLID-v2 dataset repo](https://huggingface.co/datasets/laurievb/OpenLID-v2).
28
 
29
 
30
  ### How to use
31
 
32
+ Here is how to use this model to detect the language of a given text. For best results, text should be cleaned and normalised with [openlid_normer.clean_line](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/tools/openlid_normer.py) prior to classification.
33
 
34
  ```python
35
  >>> import fasttext
36
+ >>> from openlid_normer import clean_line
37
  >>> from huggingface_hub import hf_hub_download
38
 
39
  >>> model_path = hf_hub_download(repo_id="laurievb/OpenLID-v2", filename="openlid_v2.bin")
40
  >>> model = fasttext.load_model(model_path)
41
+ >>> input_text = clean_line("Hello, world!")
42
  >>> model.predict(input_text)
43
 
44
  (('__label__eng_Latn',), array([0.81148803]))
45
 
46
+ >>> model.predict("Hello, world!", k=5) # lower score for eng_Latn without cleaning
47
 
48
  (('__label__eng_Latn', '__label__vie_Latn', '__label__nld_Latn', '__label__pol_Latn', '__label__deu_Latn'),
49
  array([0.61224753, 0.21323682, 0.09696738, 0.01359863, 0.01319415]))
 
51
 
52
  ### Limitations and bias
53
 
54
+ The dataset and model cover 200 language varieties. However, some language varieties (e.g. Arabic dialects) are very hard to distinguish and in practice, it may only be possible to classify a input at the macrolanguage level.
55
+
56
+ The FLORES+ test set consists of sentences from a single domain (wiki articles), and so performance on this test set may not reflect how well our classifier works in other domains.
57
 
58
  Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
59
 
60
  ## Training data
61
 
62
+ The model was trained on the [OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2). The data was normalised and classes were up/downsampled with temperature sampling prior to training; code to do this can be found [in the `scripts` directory](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/make_training_openlid.py) in the OpenLID-v2 dataset repository.
63
 
64
  ## Training procedure
65
 
 
78
 
79
  ### Evaluation datasets
80
 
81
+ We evaluate the model using the [FLORES+ evaluation benchmark](https://huggingface.co/datasets/openlanguagedata/flores_plus), normalising text prior to classification with [openlid_normer.clean_line](https://huggingface.co/datasets/laurievb/OpenLID-v2/blob/main/scripts/tools/openlid_normer.py). Full results are available below.
82
+
83
+ The original OpenLID model was evaluated using the FLORES-200 benchmark provided by Costa-jussà et al. (2022), with further information available in the [OpenLID paper](https://aclanthology.org/2023.acl-short.75/).
84
 
85
  ### BibTeX entry and citation info
86
 
 
106
  pages = "865--879",
107
  abstract = "Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033{\%} across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, which we audit manually to ensure reliability. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model{'}s performance, both in comparison to existing open models and by language class.",
108
  }
109
+ ```
110
+
111
+ ## Evaluation results
112
+
113
+ | Language code | Line count | F1 score |
114
+ |---------------|------------|----------|
115
+ | ace_Arab | 6360 | 0.971029 |
116
+ | ace_Latn | 16845 | 0.998517 |
117
+ | acm_Arab | 5455 | 0.025121 |
118
+ | acq_Arab | 1831 | 0.001974 |
119
+ | aeb_Arab | 20541 | 0.488032 |
120
+ | afr_Latn | 1032866 | 0.999012 |
121
+ | als_Latn | 341372 | 1.0 |
122
+ | amh_Ethi | 810989 | 0.999506 |
123
+ | apc_Arab | 97293 | 0.386029 |
124
+ | arb_Arab | 7100646 | 0.33617 |
125
+ | ars_Arab | 25771 | 0.025373 |
126
+ | ary_Arab | 27376 | 0.579467 |
127
+ | arz_Arab | 69832 | 0.481471 |
128
+ | asm_Beng | 121242 | 1.0 |
129
+ | ast_Latn | 64998 | 0.991605 |
130
+ | awa_Deva | 8425 | 0.655352 |
131
+ | ayr_Latn | 140086 | 1.0 |
132
+ | azb_Arab | 10801 | 0.915957 |
133
+ | azj_Latn | 457599 | 0.998026 |
134
+ | bak_Cyrl | 63553 | 1.0 |
135
+ | bam_Latn | 9389 | 0.619494 |
136
+ | ban_Latn | 15202 | 0.977353 |
137
+ | bel_Cyrl | 83859 | 1.0 |
138
+ | bem_Latn | 378301 | 0.979612 |
139
+ | ben_Beng | 491942 | 0.996032 |
140
+ | bho_Deva | 53666 | 0.904134 |
141
+ | bjn_Arab | 6289 | 0.968215 |
142
+ | bjn_Latn | 20264 | 0.985665 |
143
+ | bod_Tibt | 2468 | 0.854072 |
144
+ | bos_Latn | 196005 | 0.69401 |
145
+ | bug_Latn | 7495 | 0.99504 |
146
+ | bul_Cyrl | 596120 | 1.0 |
147
+ | cat_Latn | 113745 | 0.99802 |
148
+ | ceb_Latn | 991957 | 0.998519 |
149
+ | ces_Latn | 424303 | 0.998026 |
150
+ | cjk_Latn | 35645 | 0.928159 |
151
+ | ckb_Arab | 24989 | 0.999506 |
152
+ | cmn_Hans | 1043000 | 0.986693 |
153
+ | cmn_Hant | 2011585 | 0.89396 |
154
+ | crh_Latn | 17398 | 0.992541 |
155
+ | cym_Latn | 97264 | 1.0 |
156
+ | dan_Latn | 2460965 | 0.989066 |
157
+ | deu_Latn | 652883 | 1.0 |
158
+ | dik_Latn | 25833 | 0.999011 |
159
+ | dyu_Latn | 16861 | 0.053309 |
160
+ | dzo_Tibt | 6903 | 0.886842 |
161
+ | ekk_Latn | 2984641 | 0.999506 |
162
+ | ell_Grek | 2977115 | 0.999506 |
163
+ | eng_Latn | 7514770 | 0.990206 |
164
+ | epo_Latn | 332895 | 0.999506 |
165
+ | eus_Latn | 613564 | 1.0 |
166
+ | ewe_Latn | 578181 | 0.998028 |
167
+ | fao_Latn | 38378 | 0.997036 |
168
+ | fij_Latn | 355285 | 1.0 |
169
+ | fil_Latn | 1178464 | 0.999013 |
170
+ | fin_Latn | 2299900 | 1.0 |
171
+ | fon_Latn | 30895 | 0.99802 |
172
+ | fra_Latn | 586064 | 0.99703 |
173
+ | fur_Latn | 53980 | 0.999506 |
174
+ | fuv_Latn | 13921 | 0.98191 |
175
+ | gaz_Latn | 331430 | 1.0 |
176
+ | gla_Latn | 49218 | 0.999506 |
177
+ | gle_Latn | 195791 | 1.0 |
178
+ | glg_Latn | 41582 | 0.994557 |
179
+ | gug_Latn | 78880 | 0.99852 |
180
+ | guj_Gujr | 834918 | 1.0 |
181
+ | hat_Latn | 294042 | 0.992643 |
182
+ | hau_Latn | 340263 | 0.989247 |
183
+ | heb_Hebr | 987305 | 0.999506 |
184
+ | hin_Deva | 1071332 | 0.799519 |
185
+ | hne_Deva | 52536 | 0.927026 |
186
+ | hrv_Latn | 785563 | 0.741921 |
187
+ | hun_Latn | 2559216 | 0.999506 |
188
+ | hye_Armn | 357578 | 1.0 |
189
+ | ibo_Latn | 484363 | 0.999013 |
190
+ | ilo_Latn | 966361 | 0.995573 |
191
+ | ind_Latn | 1682898 | 0.925908 |
192
+ | isl_Latn | 43332 | 0.998519 |
193
+ | ita_Latn | 478358 | 0.995547 |
194
+ | jav_Latn | 64377 | 0.988235 |
195
+ | jpn_Jpan | 886638 | 0.99852 |
196
+ | kab_Latn | 50772 | 0.829508 |
197
+ | kac_Latn | 11156 | 1.0 |
198
+ | kam_Latn | 51265 | 0.866741 |
199
+ | kan_Knda | 355427 | 1.0 |
200
+ | kas_Arab | 6225 | 0.979324 |
201
+ | kas_Deva | 6738 | 0.968925 |
202
+ | kat_Geor | 412072 | 1.0 |
203
+ | kaz_Cyrl | 50643 | 0.999506 |
204
+ | kbp_Latn | 52382 | 1.0 |
205
+ | kea_Latn | 5505 | 0.965764 |
206
+ | khk_Cyrl | 166505 | 1.0 |
207
+ | khm_Khmr | 75713 | 0.999506 |
208
+ | kik_Latn | 94116 | 0.963281 |
209
+ | kin_Latn | 439856 | 0.799766 |
210
+ | kir_Cyrl | 366840 | 1.0 |
211
+ | kmb_Latn | 90314 | 0.95809 |
212
+ | kmr_Latn | 15084 | 0.997041 |
213
+ | knc_Arab | 6337 | 0.702564 |
214
+ | knc_Latn | 6254 | 0.998516 |
215
+ | kor_Hang | 350945 | 1.0 |
216
+ | ktu_Latn | 206325 | 0.985352 |
217
+ | lao_Laoo | 24712 | 1.0 |
218
+ | lij_Latn | 27454 | 0.997531 |
219
+ | lim_Latn | 47490 | 0.994563 |
220
+ | lin_Latn | 538130 | 0.997041 |
221
+ | lit_Latn | 2360462 | 0.999506 |
222
+ | lmo_Latn | 33288 | 0.99505 |
223
+ | ltg_Latn | 14203 | 0.997033 |
224
+ | ltz_Latn | 36810 | 0.999506 |
225
+ | lua_Latn | 288714 | 0.996536 |
226
+ | lug_Latn | 245216 | 0.995569 |
227
+ | luo_Latn | 134777 | 0.998517 |
228
+ | lus_Latn | 191617 | 0.99802 |
229
+ | lvs_Latn | 2533501 | 0.997531 |
230
+ | mag_Deva | 6330 | 0.966281 |
231
+ | mai_Deva | 33093 | 0.988574 |
232
+ | mal_Mlym | 378020 | 1.0 |
233
+ | mar_Deva | 1006184 | 0.997536 |
234
+ | min_Latn | 31047 | 0.995547 |
235
+ | mkd_Cyrl | 393081 | 0.999506 |
236
+ | mlt_Latn | 2011002 | 0.996063 |
237
+ | mni_Beng | 47076 | 0.996063 |
238
+ | mos_Latn | 193219 | 0.976227 |
239
+ | mri_Latn | 47736 | 0.999506 |
240
+ | mya_Mymr | 547113 | 1.0 |
241
+ | nld_Latn | 2609642 | 0.994573 |
242
+ | nno_Latn | 98176 | 0.980779 |
243
+ | nob_Latn | 1749713 | 0.971935 |
244
+ | npi_Deva | 229595 | 0.995069 |
245
+ | nso_Latn | 552404 | 0.989237 |
246
+ | nus_Latn | 6294 | 1.0 |
247
+ | nya_Latn | 780066 | 0.994106 |
248
+ | oci_Latn | 239737 | 0.997289 |
249
+ | ory_Orya | 92475 | 1.0 |
250
+ | pag_Latn | 287179 | 0.998024 |
251
+ | pan_Guru | 354236 | 1.0 |
252
+ | pap_Latn | 397355 | 0.978703 |
253
+ | pbt_Arab | 276372 | 0.997041 |
254
+ | pes_Arab | 2810268 | 0.662182 |
255
+ | plt_Latn | 47052 | 1.0 |
256
+ | pol_Latn | 3035767 | 0.996553 |
257
+ | por_Latn | 3623950 | 0.992134 |
258
+ | prs_Arab | 31038 | 0.577474 |
259
+ | quy_Latn | 152002 | 1.0 |
260
+ | ron_Latn | 436311 | 0.998028 |
261
+ | run_Latn | 454887 | 0.850575 |
262
+ | rus_Cyrl | 6688484 | 1.0 |
263
+ | sag_Latn | 251562 | 0.999506 |
264
+ | san_Deva | 46056 | 0.990524 |
265
+ | sat_Olck | 29033 | 1.0 |
266
+ | scn_Latn | 39233 | 0.996059 |
267
+ | shn_Mymr | 22187 | 1.0 |
268
+ | sin_Sinh | 423966 | 1.0 |
269
+ | slk_Latn | 2815971 | 0.999012 |
270
+ | slv_Latn | 2684050 | 0.997044 |
271
+ | smo_Latn | 361969 | 0.998519 |
272
+ | sna_Latn | 754901 | 0.995084 |
273
+ | snd_Arab | 47901 | 0.998026 |
274
+ | som_Latn | 187966 | 0.998028 |
275
+ | sot_Latn | 1941 | 0.963115 |
276
+ | spa_Latn | 676635 | 0.993083 |
277
+ | srd_Latn | 46037 | 0.997531 |
278
+ | srp_Cyrl | 308075 | 0.999506 |
279
+ | ssw_Latn | 112237 | 0.989537 |
280
+ | sun_Latn | 46337 | 0.993076 |
281
+ | swe_Latn | 2429547 | 1.0 |
282
+ | swh_Latn | 226377 | 0.92972 |
283
+ | szl_Latn | 32177 | 0.996533 |
284
+ | tam_Taml | 550090 | 1.0 |
285
+ | taq_Latn | 10262 | 0.731371 |
286
+ | taq_Tfng | 6290 | 0.959677 |
287
+ | tat_Cyrl | 253516 | 1.0 |
288
+ | tel_Telu | 276262 | 1.0 |
289
+ | tgk_Cyrl | 131708 | 1.0 |
290
+ | tha_Thai | 728313 | 1.0 |
291
+ | tir_Ethi | 473470 | 0.999506 |
292
+ | tpi_Latn | 457544 | 0.999011 |
293
+ | tsn_Latn | 775066 | 0.974458 |
294
+ | tso_Latn | 747226 | 0.9941 |
295
+ | tuk_Latn | 157610 | 1.0 |
296
+ | tum_Latn | 233136 | 0.994584 |
297
+ | tur_Latn | 598819 | 0.992636 |
298
+ | twi_Latn | 538421 | 0.998516 |
299
+ | uig_Arab | 81940 | 1.0 |
300
+ | ukr_Cyrl | 1123812 | 1.0 |
301
+ | umb_Latn | 215640 | 0.983655 |
302
+ | urd_Arab | 487265 | 0.98062 |
303
+ | uzn_Latn | 1463925 | 0.99852 |
304
+ | vec_Latn | 41746 | 0.995074 |
305
+ | vie_Latn | 864979 | 0.999506 |
306
+ | war_Latn | 278265 | 1.0 |
307
+ | wol_Latn | 26985 | 0.996047 |
308
+ | xho_Latn | 907281 | 0.985309 |
309
+ | ydd_Hebr | 923 | 0.999506 |
310
+ | yor_Latn | 524493 | 0.996553 |
311
+ | yue_Hant | 59348 | 0.874099 |
312
+ | zgh_Tfng | 9485 | 0.96124 |
313
+ | zsm_Latn | 401337 | 0.954902 |
314
+ | zul_Latn | 941301 | 0.970106 |
openlid_v2.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:aeed00d143f342be8a4883ee821f5e9db616cedcfa613ef3b408f0d849440b0c
3
- size 1226652168
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f01b47c6182909291a3394f77f682031aaeade346e443c4e496e94bd2a9fa703
3
+ size 1217494683
scripts/openlid.py DELETED
@@ -1,87 +0,0 @@
1
- import unicodedata
2
- import emoji
3
- import sys
4
-
5
- class Demojizer:
6
- """
7
- based on:
8
- https://github.com/carpedm20/emoji/blob/d8bbfe455c6fcd12b96ed1dce6e0978fe7a47431/emoji/core.py#L141
9
- """
10
-
11
- def _get_search_tree(self):
12
- _SEARCH_TREE = {}
13
- for emj in emoji.unicode_codes.EMOJI_DATA:
14
- sub_tree = _SEARCH_TREE
15
- lastidx = len(emj) - 1
16
- for i, char in enumerate(emj):
17
- if char not in sub_tree:
18
- sub_tree[char] = {}
19
- sub_tree = sub_tree[char]
20
- if i == lastidx:
21
- sub_tree["data"] = emoji.unicode_codes.EMOJI_DATA[emj]
22
- return _SEARCH_TREE
23
-
24
- def __init__(self) -> None:
25
- self.search_tree = self._get_search_tree()
26
-
27
- def __call__(self, string: str, replace_str: str):
28
- result = []
29
- i = 0
30
- length = len(string)
31
- state = 0
32
- while i < length:
33
- consumed = False
34
- char = string[i]
35
- if char in self.search_tree:
36
- j = i + 1
37
- sub_tree = self.search_tree[char]
38
- while j < length and string[j] in sub_tree:
39
- sub_tree = sub_tree[string[j]]
40
- j += 1
41
- if "data" in sub_tree:
42
- state = 1
43
- consumed = True
44
- result.append(replace_str)
45
- i = j - 1
46
- else:
47
- state = 0
48
- elif state == 1:
49
- if char.isspace():
50
- consumed = True
51
- else:
52
- state = 0
53
-
54
- if not consumed and char != "\ufe0e" and char != "\ufe0f":
55
- result.append(char)
56
- i += 1
57
-
58
- return "".join(result)
59
-
60
-
61
- def _get_replacer(replace_by: str = " ") -> str:
62
- non_printable_map = {
63
- ord(c): replace_by
64
- for c in (chr(i) for i in range(sys.maxunicode + 1))
65
- # same as \p{C} in perl
66
- # see https://www.unicode.org/reports/tr44/#General_Category_Values
67
- if unicodedata.category(c) in {"C", "Cc", "Cf", "Cs", "Co", "Cn"}
68
- }
69
-
70
- def replace_non_printing_char(line) -> str:
71
- return line.translate(non_printable_map)
72
-
73
- return replace_non_printing_char
74
-
75
-
76
- def clean_text(input_text: str) -> str:
77
- """cleans input text prior to LID"""
78
- replace_nonprint = _get_replacer(" ")
79
- demoji = Demojizer()
80
-
81
- clean = replace_nonprint(input_text)
82
- clean = unicodedata.normalize("NFKC", clean)
83
- clean = demoji(clean, "")
84
-
85
- return clean
86
-
87
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/prepare_openlid_v2_for_model_training.sh DELETED
@@ -1,17 +0,0 @@
1
- #!/bin/bash
2
- # author: laurie
3
- # script to sample OpenLID-v2 prior to training
4
- # usage: bash prepare_opelid_v2_for_model_training.sh PATH_TO_OPENLID-V2
5
- set -eo pipefail
6
-
7
- START_DIR=${PWD}
8
- echo "starting dir is ${START_DIR}"
9
- INPUT_DATA=$1 # should be openlid-v2 dataset
10
- echo "using openlid-v2 data from ${1}"
11
-
12
- echo "generating counts in stats/"
13
- mkdir -p stats
14
- cut -f2 -d$'\t' $INPUT_DATA | uniq -c > stats/openlid-v2-unsampled.counts
15
-
16
- echo "applying temperature sampling..."
17
- python scripts/sample_with_temperature.py $INPUT_DATA stats/openlid-v2-unsampled.counts > openlid-v2-sampled.tsv
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/sample_with_temperature.py DELETED
@@ -1,98 +0,0 @@
1
- """samples with temperature, grouping by language code. assumes input files is sorted by language group"""
2
-
3
- import argparse
4
- import logging
5
- import random
6
- import sys
7
-
8
- def parse_args():
9
- parser = argparse.ArgumentParser()
10
- parser.add_argument("corpus_filepath", type=str, help="path to input corpus to sample")
11
- parser.add_argument("linecounts_filepath", type=str, help="path to file containing line counts of input corpus (from 'uniq -c')")
12
- return parser.parse_args()
13
-
14
- # def count_lines(file):
15
- # def blocks(files, size=65536):
16
- # while True:
17
- # b = files.read(size)
18
- # if not b: break
19
- # yield b
20
- # with open(file, "r",encoding="utf-8",errors='ignore') as f:
21
- # return (sum(bl.count("\n") for bl in blocks(f)))
22
-
23
- def main():
24
- logging.basicConfig(
25
- level=logging.INFO,
26
- filename='sampling.log',
27
- filemode='w',
28
- format='%(asctime)s %(levelname)s: %(message)s',
29
- datefmt='%m/%d/%Y %I:%M:%S %p')
30
- logger = logging.getLogger(__name__)
31
-
32
- args = parse_args()
33
-
34
- logger.info(f"creating counts lookup dict from {args.linecounts_filepath}")
35
- with open(args.linecounts_filepath) as f:
36
- total_raw_lines = 0
37
- lc_lookup = dict()
38
- for line in f:
39
- count, lang = line.strip().split(' ')
40
- count = int(count)
41
- lc_lookup[lang] = {"raw_lines": count}
42
- total_raw_lines += count
43
-
44
- logger.info(f"lookup dict finished ({len(lc_lookup)} entries)")
45
- logger.info(f"dataset contains {total_raw_lines} lines")
46
-
47
- # calculate lines to keep with (((raw_lines_in_lang / total_line_count) ** 0.3) / total_proprotions) * total lines
48
-
49
- # calculate proportions
50
- logger.info("calculating sampling factors")
51
- total_sampling_factors = 0
52
- for lang in lc_lookup:
53
- # we sample lines proportional to this so smaller langs are upsampled and larger langs are downsampled
54
- sampling_factor = (lc_lookup[lang]['raw_lines'] / total_raw_lines) ** 0.3
55
- lc_lookup[lang]["sampling_factor"] = sampling_factor
56
- total_sampling_factors += sampling_factor
57
-
58
- logger.info(f"sampling factor total is {total_sampling_factors}")
59
- logger.info(f"calculating number of lines to sample")
60
- total_lines_to_sample = 0
61
- for lang in lc_lookup:
62
- lines_to_sample = round(lc_lookup[lang]["sampling_factor"]/total_sampling_factors * total_raw_lines)
63
- lc_lookup[lang]['lines_to_sample'] = lines_to_sample
64
- total_lines_to_sample += lines_to_sample
65
- prop_size_difference = abs((total_raw_lines - total_lines_to_sample)/total_lines_to_sample)
66
- assert prop_size_difference < 0.01 # sense check that sampled corpus is right size
67
- logger.info(
68
- f"total raw lines is {total_raw_lines}, total sampled lines is {total_lines_to_sample} ({prop_size_difference:.3%} difference)")
69
-
70
- # assume input file is sorted by group
71
- logger.info(f"sampling from {args.corpus_filepath}")
72
- with open(args.corpus_filepath, "r") as f:
73
- single_lang_line_store = []
74
- langcode = ""
75
- while line := f.readline():
76
- line = line.strip()
77
- _, nextlang, _ = line.split('\t')
78
- if langcode == nextlang or langcode == "": # same language
79
- single_lang_line_store.append(line)
80
- else: # language change, time to sample and write out
81
- raw_lines_in_lang = len(single_lang_line_store)
82
- assert raw_lines_in_lang == lc_lookup[langcode]["raw_lines"] # sanity check it's same data
83
- num_lines_to_keep = lc_lookup[langcode]["lines_to_sample"]
84
- logger.info(f"finished reading {langcode}: read in {raw_lines_in_lang}, writing {num_lines_to_keep}")
85
- if raw_lines_in_lang > num_lines_to_keep:
86
- sampled_lines_gc = (x for x in random.sample(single_lang_line_store, num_lines_to_keep))
87
- else: # need to oversample, so now use sampling with replacement
88
- sampled_lines_gc = (x for x in random.choices(single_lang_line_store, k=num_lines_to_keep))
89
- for out in sampled_lines_gc:
90
- sys.stdout.write(f"{out}\n")
91
- logger.info(f"finished writing {langcode} to stdout, now collecting lines for {nextlang}")
92
- single_lang_line_store = [line]
93
- langcode = nextlang
94
- logger.info("sampling complete!")
95
-
96
-
97
- if __name__ == "__main__":
98
- main()