| datasets: | |
| - cis-lmu/glotlid-corpus | |
| metrics: | |
| - exact_match | |
| - f1 | |
| base_model: | |
| - cis-lmu/glotlid | |
| pipeline_tag: text-classification | |
| library_name: fasttext | |
| tags: | |
| - LID | |
| # Model Card for Model ID | |
| <!-- Provide a quick summary of what the model is/does. --> | |
| This model is a reproduction of GlotLID on 125 languages using the Latn script, trained on the original GlotLID-C dataset for these languages, enriched by 1 million word-level examples per language. | |
| The word-level examples were obtained from splitting sentences from the dataset. It has also been trained with a bigger hashmap than GlotLID (2e6 instead of 1e6).. | |
| ## Model Details | |
| ### Model Description | |
| <!-- Provide a longer summary of what this model is. --> | |
| - **Developed by:** Joanna Radoła | |
| - **Model type:** fasttext architecture | |
| - **Language(s):** fon, fra, fur, fuv, gaz, gla, gle, glg, gug, hat, hau,ace, afr, als, ast, ayr, azj, bam, ban, bem, bjn, bug, cat, ceb, ces, cjk, crh, cym, dan, deu, dik, dyu, ekk, eng, epo, eus, ewe, fao, fij, fil, fin, fon, fra, fur, fuv, gaz, gla, gle, glg, gug, hat, hau, hin, hun, ibo, ilo, ind, isl, ita, jav, kab, kac, kam, kbp, kea, kik, kin, kmb, kmr, knc, kng, lij, lim, lin, lit, lmo, ltg, ltz, lua, lug, luo, lus, lvs, min, mlt, mos, mri, nld, nno, nob, npi, nso, nus, nya, oci, pag, pap, plt, pol, por, quy, ron, run, sag, scn, slk, slv, smo, sna, som, sot, spa, srd, ssw, sun, swe, swh, szl, taq, tpi, tsn, tso, tuk, tum, tur, twi, umb, uzn, vec, vie, war, wol, xho, yor, zsm, zul | |
| ## How to Get Started with the Model | |
| <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> | |
| ``` | |
| import fasttext | |
| from huggingface_hub import hf_hub_download | |
| model_path = hf_hub_download(repo_id="paruwka/LiteLID", filename="wordlid_v3.ftz", cache_dir=None) | |
| model = fasttext.load_model(model_path) | |
| model.predict(['predicting', 'language'], k=3) # this will return a tuple: (list of lists of top-k language labels, list of lists of their respective probabilities) | |
| ``` | |
| #### Training Hyperparameters | |
| lr=0.8, epochs=1, dim=256, minn=2, maxn=5, bucket=2000000, loss='softmax' | |
| ## Evaluation | |
| ... | |
| <!-- This section describes the evaluation protocols and provides the results. --> | |