Update README.md
Browse files
README.md
CHANGED
|
@@ -26,7 +26,7 @@ Here is how to use this model to detect the language of a given text. For best r
|
|
| 26 |
|
| 27 |
```shell
|
| 28 |
pip install fasttext==0.9.3 huggingface-hub==0.35.3 numpy==1.23.5 regex==2024.4.28
|
| 29 |
-
|
| 30 |
|
| 31 |
```python
|
| 32 |
import fasttext
|
|
@@ -62,13 +62,13 @@ print(
|
|
| 62 |
|
| 63 |
### Limitations and bias
|
| 64 |
|
| 65 |
-
The dataset and model cover 194 language varieties. However, some language varieties (e.g. Arabic dialects) are very hard to distinguish and in practice, it may only be possible to classify
|
| 66 |
|
| 67 |
Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
|
| 68 |
|
| 69 |
## Training data
|
| 70 |
|
| 71 |
-
The model was trained on the samples from [OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2), [glotlid-corpus](https://huggingface.co/datasets/cis-lmu/glotlid-corpus) and [Wikipedia](https://dumps.wikimedia.org/backup-index.html). The data was normalised and classes were up/downsampled with temperature sampling prior to training; code to do this can be found [the OpenLID-v3 repository](https://github.com/hplt-project/openlid).
|
| 72 |
|
| 73 |
## Training procedure
|
| 74 |
|
|
|
|
| 26 |
|
| 27 |
```shell
|
| 28 |
pip install fasttext==0.9.3 huggingface-hub==0.35.3 numpy==1.23.5 regex==2024.4.28
|
| 29 |
+
```
|
| 30 |
|
| 31 |
```python
|
| 32 |
import fasttext
|
|
|
|
| 62 |
|
| 63 |
### Limitations and bias
|
| 64 |
|
| 65 |
+
The dataset and model cover 194 language varieties. However, some language varieties (e.g. Arabic dialects) are very hard to distinguish and in practice, it may only be possible to classify an input at the macrolanguage level.
|
| 66 |
|
| 67 |
Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
|
| 68 |
|
| 69 |
## Training data
|
| 70 |
|
| 71 |
+
The model was trained on the samples from [OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2), [glotlid-corpus](https://huggingface.co/datasets/cis-lmu/glotlid-corpus) and [Wikipedia](https://dumps.wikimedia.org/backup-index.html). The data was normalised and classes were up/downsampled with temperature sampling prior to training; code to do this can be found in [the OpenLID-v3 repository](https://github.com/hplt-project/openlid).
|
| 72 |
|
| 73 |
## Training procedure
|
| 74 |
|