README.md

---
library_name: transformers
license: mit
base_model: xlm-roberta-base
tags:
- language-detection
- language-identification
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: polyglot-tagger
  results: []
datasets:
- wikimedia/wikipedia
- HuggingFaceFW/finetranslations
- google/smol
- DerivedFunction/nlp-noise-snippets
- DerivedFunction/wikipedia-language-snippets-filtered
- DerivedFunction/finetranslations-filtered
- DerivedFunction/additional-language-snippets
pipeline_tag: token-classification
language:
- en
- es
- fr
- de
- it
- pt
- nl
- vi
- tr
- la
- id
- ms
- af
- sq
- is
- no
- sv
- da
- fi
- hu
- pl
- cs
- ro
- ru
- bg
- uk
- sr
- be
- kk
- mk
- mn
- zh
- ja
- ko
- hi
- ur
- bn
- ta
- te
- mr
- gu
- kn
- ml
- pa
- as
- or
- ar
- fa
- ps
- sd
- ug
- el
- he
- hy
- ka
- am
- km
- lo
- my
- th
- si
- bo
- dv
- ti
- sw
- eu
---


# Polyglot Tagger: 67L (Experimental)

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base).
It achieves the following results on the evaluation set:
- Loss: 0.0404
- Precision: 0.8848
- Recall: 0.9012
- F1: 0.8929
- Accuracy: 0.9909

## Model description

Introducing Polyglot Tagger 66L, a new way to classify multi-lingual documents. By training specifically on token classification on individual sentences, the model 
generalizes well on a variety of languages, while also behaves as a multi-label classifier, and extracts sentences based on its language.

## Intended uses & limitations
This model can be treated as a base model for further fine-tuning on specific language identification extraction tasks. 
Note that as a general language tagging model, it can potentially get confused from shared language families or from short texts. For example, English and German, Spanish and Portuguese, and Russian and Ukrainian.

The model is trained on a sentence with a minimum of four tokens, so it may not accurately classify very short and ambigous statements. Note that this model is experimental
and may produce unexpected results compared to generic text classifiers. It is trained on cleaned text, therefore, "messy" text may unexpectedly produce different results.

### Training and Evaluation Data
The model was trained on a synthetic dataset of roughly **3 million samples**, covering 67 languages across diverse script families 
(Latin, Cyrillic, Indic, Arabic, Han, etc.), from `wikimedia/wikipedia` (up to 200,000 individual sentences, 120,000 reserve from up to 100,000 unique articles, 
by taking the first half of Wikipedia after filtering for stubs), `google/smol` (up to 1000 individual sentences), `HuggingFaceFW/finetranslations` (up to 50,000 sentences, 30,000 reserve from up to 50,000 unique rows), 
and additional sentences from various sources for major languages (`en`, `es`, `pt`, `ru`, `hi`, `de`, `fr`, etc) (up to 50,000 sentences, 30,000 reserve from up to 100,000 unique rows).
in which it is split into a reserve set for pure documents, and a main set for everything else. 

A synthetic training row consists of 1-4 individual and mostly independent sentences extracted from various sources.

The data composition follows a strategic curriculum:

* **60% Pure Documents:** Single-language sequences to establish strong baseline profiles for each language.
* **30% Homogenous Mixed:** Documents containing one main language, and clear transitions between two or more languages to train boundary detection.
* **10% Mixed with Noise:** Integration of "neutral" spans including code snippets, mathematical notation, emojis, symbols, and `rot_13` text tagged as `O` or their respective source to reduce hallucination.

### Supported Languages and Limitations (66)
The model supports the following ISO-coded languages: 
`af, am, ar, as, be, bg, bn, bo, cs, da, de, dv, el, en, es, eu, fa, fi, fr, gu, he, hi, 
hu, hy, id, is, it, ja, ka, kk, km, kn, ko, la, lo, ml, mk, mn, mr, ms, my, nl, no, 
or, pa, pl, ps, pt, ro, ru, sd, si, sq, sr, sv, sw, ta, te, th, ti, tr, ug, uk, ur, vi, zh`

> Note that Romanized versions of any language is not included in the training set, such as Romanized Russian, and Hindi.


### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 72
- eval_batch_size: 36
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 144
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 2
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step  | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:------:|:-----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.0404        | 0.1206 | 2500  | 0.0649          | 0.7944    | 0.8616 | 0.8266 | 0.9868   |
| 0.0394        | 0.2412 | 5000  | 0.0538          | 0.8181    | 0.8696 | 0.8430 | 0.9893   |
| 0.0345        | 0.3618 | 7500  | 0.0456          | 0.8355    | 0.8781 | 0.8563 | 0.9906   |
| 0.0280        | 0.4824 | 10000 | 0.0493          | 0.8404    | 0.8836 | 0.8614 | 0.9897   |
| 0.0286        | 0.6030 | 12500 | 0.0515          | 0.8425    | 0.8805 | 0.8611 | 0.9889   |
| 0.0275        | 0.7236 | 15000 | 0.0423          | 0.8371    | 0.8852 | 0.8605 | 0.9905   |
| 0.0209        | 0.8442 | 17500 | 0.0429          | 0.8671    | 0.8908 | 0.8788 | 0.9911   |
| 0.0265        | 0.9648 | 20000 | 0.0379          | 0.8550    | 0.8881 | 0.8712 | 0.9919   |
| 0.0223        | 1.0854 | 22500 | 0.0371          | 0.8665    | 0.8967 | 0.8814 | 0.9918   |
| 0.0220        | 1.2060 | 25000 | 0.0344          | 0.8687    | 0.8954 | 0.8818 | 0.9926   |
| 0.0225        | 1.3266 | 27500 | 0.0332          | 0.8776    | 0.9011 | 0.8892 | 0.9928   |
| 0.0186        | 1.4472 | 30000 | 0.0390          | 0.8711    | 0.9018 | 0.8862 | 0.9920   |
| 0.0200        | 1.5678 | 32500 | 0.0315          | 0.8840    | 0.9046 | 0.8942 | 0.9931   |
| 0.0170        | 1.6884 | 35000 | 0.0313          | 0.8867    | 0.9066 | 0.8965 | 0.9932   |
| 0.0170        | 1.8090 | 37500 | 0.0305          | 0.8804    | 0.9034 | 0.8918 | 0.9933   |
| 0.0176        | 1.9296 | 40000 | 0.0305          | 0.8866    | 0.9058 | 0.8961 | 0.9935   |


### Framework versions

- Transformers 5.0.0
- Pytorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2