library_name: transformers
license: mit
base_model: xlm-roberta-base
tags:
- generated_from_trainer
metrics:
- precision
- recall
- f1
- accuracy
language:
- multilingual
- af
- am
- ar
- as
- ba
- be
- bg
- bn
- bo
- br
- bs
- ca
- ce
- ckb
- cs
- cy
- da
- de
- dv
- el
- en
- eo
- es
- et
- eu
- fa
- fi
- fr
- ga
- gd
- gl
- gu
- he
- hi
- hr
- hu
- hy
- id
- is
- it
- ja
- jv
- ka
- kk
- km
- kn
- ko
- ku
- ky
- la
- lb
- lo
- lt
- lv
- mg
- mk
- ml
- mn
- mr
- ms
- mt
- my
- ne
- nl
- 'no'
- ny
- oc
- om
- or
- pa
- pl
- ps
- pt
- rm
- ro
- ru
- sd
- si
- sk
- sl
- so
- sq
- sr
- su
- sv
- sw
- ta
- te
- tg
- th
- ti
- tl
- tr
- tt
- ug
- uk
- ur
- uz
- vi
- yo
- zh
- zu
model-index:
- name: polyglot-tagger
results: []
datasets:
- wikimedia/wikipedia
- HuggingFaceFW/finetranslations
- google/smol
- DerivedFunction/nlp-noise-snippets
- DerivedFunction/wikipedia-language-snippets-filtered
- DerivedFunction/finetranslations-filtered
- DerivedFunction/lang-ner-v2
- DerivedFunction/tatoeba-filtered
pipeline_tag: token-classification
Fine-tuned xlm-roberta-base for sentence-level language tagging across 100 languages.
The model predicts BIO-style language tags over tokens, which makes it useful for
language identification, code-switch detection, and multilingual document analysis.
Model description
Introducing Polyglot Tagger, a new way to classify multi-lingual documents. By training specifically on token classification on individual sentences, the model generalizes well on a variety of languages, while also behaves as a multi-label classifier, and extracts sentences based on its language.
Intended uses & limitations
This model can be treated as a base model for further fine-tuning on specific language identification extraction tasks. Note that as a general language tagging model, it can potentially get confused from shared language families or from short texts. For example, English and German, Spanish and Portuguese, and Russian and Ukrainian.
The model is trained on a sentence with a minimum of four tokens, so it may not accurately classify very short and ambigous statements. Note that this model is experimental and may produce unexpected results compared to generic text classifiers. It is trained on cleaned text, therefore, "messy" text may unexpectedly produce different results.
Note that Romanized versions of any language may only have minor representation in the training set, such as Romanized Russian, and Hindi.
Training and Evaluation Data
A synthetic training row consists of 1-4 individual and mostly independent sentences extracted from various sources. The actual training and evaluation data, as well as coverage
is found in DerivedFunction/lang-ner-v2.
It achieves the following results on the evaluation set:
- Loss: 0.0427
- Precision: 0.8949
- Recall: 0.9144
- F1: 0.9046
- Accuracy: 0.9892
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 72
- eval_batch_size: 36
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 144
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 2
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|---|---|---|
| 0.0919 | 0.0894 | 2500 | 0.1243 | 0.7388 | 0.8336 | 0.7833 | 0.9712 |
| 0.0798 | 0.1788 | 5000 | 0.0950 | 0.7928 | 0.8607 | 0.8254 | 0.9774 |
| 0.0738 | 0.2682 | 7500 | 0.0857 | 0.8173 | 0.8722 | 0.8438 | 0.9785 |
| 0.0611 | 0.3575 | 10000 | 0.0797 | 0.8247 | 0.8767 | 0.8499 | 0.9812 |
| 0.0588 | 0.4469 | 12500 | 0.0732 | 0.8336 | 0.8843 | 0.8582 | 0.9822 |
| 0.0542 | 0.5363 | 15000 | 0.0665 | 0.8560 | 0.8922 | 0.8737 | 0.9838 |
| 0.0557 | 0.6257 | 17500 | 0.0613 | 0.8607 | 0.8949 | 0.8775 | 0.9845 |
| 0.0486 | 0.7151 | 20000 | 0.0590 | 0.8567 | 0.8953 | 0.8755 | 0.9851 |
| 0.0474 | 0.8045 | 22500 | 0.0601 | 0.8660 | 0.8971 | 0.8813 | 0.9854 |
| 0.0545 | 0.8938 | 25000 | 0.0574 | 0.8675 | 0.9003 | 0.8836 | 0.9857 |
| 0.0485 | 0.9832 | 27500 | 0.0566 | 0.8723 | 0.9018 | 0.8868 | 0.9858 |
| 0.0440 | 1.0726 | 30000 | 0.0522 | 0.8769 | 0.9042 | 0.8904 | 0.9867 |
| 0.0396 | 1.1620 | 32500 | 0.0509 | 0.8761 | 0.9046 | 0.8901 | 0.9873 |
| 0.0383 | 1.2514 | 35000 | 0.0489 | 0.8788 | 0.9057 | 0.8921 | 0.9879 |
| 0.0370 | 1.3408 | 37500 | 0.0486 | 0.8842 | 0.9087 | 0.8963 | 0.9877 |
| 0.0350 | 1.4302 | 40000 | 0.0489 | 0.8769 | 0.9054 | 0.8909 | 0.9874 |
| 0.0330 | 1.5195 | 42500 | 0.0478 | 0.8842 | 0.9091 | 0.8965 | 0.9879 |
| 0.0308 | 1.6089 | 45000 | 0.0458 | 0.8897 | 0.9122 | 0.9008 | 0.9888 |
| 0.0317 | 1.6983 | 47500 | 0.0454 | 0.8873 | 0.9114 | 0.8992 | 0.9887 |
| 0.0322 | 1.7877 | 50000 | 0.0447 | 0.8900 | 0.9117 | 0.9007 | 0.9888 |
| 0.0310 | 1.8771 | 52500 | 0.0439 | 0.8910 | 0.9126 | 0.9017 | 0.9888 |
| 0.0294 | 1.9665 | 55000 | 0.0427 | 0.8949 | 0.9144 | 0.9046 | 0.9892 |
Framework versions
- Transformers 5.0.0
- Pytorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2