Update README.md

efce9dd verified 2 days ago

6.43 kB

library_name: transformers
license: mit
base_model: xlm-roberta-base
tags:
  - language-detection
  - language-identification
metrics:
  - precision
  - recall
  - f1
  - accuracy
model-index:
  - name: polyglot-tagger
    results: []
datasets:
  - wikimedia/wikipedia
  - HuggingFaceFW/finetranslations
  - google/smol
  - DerivedFunction/nlp-noise-snippets
  - DerivedFunction/wikipedia-language-snippets-filtered
  - DerivedFunction/finetranslations-filtered
  - DerivedFunction/additional-language-snippets
pipeline_tag: token-classification
language:
  - en
  - es
  - fr
  - de
  - it
  - pt
  - nl
  - vi
  - tr
  - la
  - id
  - ms
  - af
  - sq
  - is
  - 'no'
  - sv
  - da
  - fi
  - hu
  - pl
  - cs
  - ro
  - ru
  - bg
  - uk
  - sr
  - be
  - kk
  - mk
  - mn
  - zh
  - ja
  - ko
  - hi
  - ur
  - bn
  - ta
  - te
  - mr
  - gu
  - kn
  - ml
  - pa
  - as
  - or
  - ar
  - fa
  - ps
  - sd
  - ug
  - el
  - he
  - hy
  - ka
  - am
  - km
  - lo
  - my
  - th
  - si
  - bo
  - dv
  - ti
  - sw
  - eu

Polyglot Tagger: 67L (Experimental)

This model is a fine-tuned version of xlm-roberta-base. It achieves the following results on the evaluation set:

Loss: 0.0404
Precision: 0.8848
Recall: 0.9012
F1: 0.8929
Accuracy: 0.9909

Model description

Introducing Polyglot Tagger 66L, a new way to classify multi-lingual documents. By training specifically on token classification on individual sentences, the model generalizes well on a variety of languages, while also behaves as a multi-label classifier, and extracts sentences based on its language.

Intended uses & limitations

This model can be treated as a base model for further fine-tuning on specific language identification extraction tasks. Note that as a general language tagging model, it can potentially get confused from shared language families or from short texts. For example, English and German, Spanish and Portuguese, and Russian and Ukrainian.

The model is trained on a sentence with a minimum of four tokens, so it may not accurately classify very short and ambigous statements. Note that this model is experimental and may produce unexpected results compared to generic text classifiers. It is trained on cleaned text, therefore, "messy" text may unexpectedly produce different results.

Training and Evaluation Data

The model was trained on a synthetic dataset of roughly 3 million samples, covering 67 languages across diverse script families (Latin, Cyrillic, Indic, Arabic, Han, etc.), from wikimedia/wikipedia (up to 200,000 individual sentences, 120,000 reserve from up to 100,000 unique articles, by taking the first half of Wikipedia after filtering for stubs), google/smol (up to 1000 individual sentences), HuggingFaceFW/finetranslations (up to 50,000 sentences, 30,000 reserve from up to 50,000 unique rows), and additional sentences from various sources for major languages (en, es, pt, ru, hi, de, fr, etc) (up to 50,000 sentences, 30,000 reserve from up to 100,000 unique rows). in which it is split into a reserve set for pure documents, and a main set for everything else.

A synthetic training row consists of 1-4 individual and mostly independent sentences extracted from various sources.

The data composition follows a strategic curriculum:

60% Pure Documents: Single-language sequences to establish strong baseline profiles for each language.
30% Homogenous Mixed: Documents containing one main language, and clear transitions between two or more languages to train boundary detection.
10% Mixed with Noise: Integration of "neutral" spans including code snippets, mathematical notation, emojis, symbols, and rot_13 text tagged as O or their respective source to reduce hallucination.

Supported Languages and Limitations (66)

The model supports the following ISO-coded languages: af, am, ar, as, be, bg, bn, bo, cs, da, de, dv, el, en, es, eu, fa, fi, fr, gu, he, hi, hu, hy, id, is, it, ja, ka, kk, km, kn, ko, la, lo, ml, mk, mn, mr, ms, my, nl, no, or, pa, pl, ps, pt, ro, ru, sd, si, sq, sr, sv, sw, ta, te, th, ti, tr, ug, uk, ur, vi, zh

Note that Romanized versions of any language is not included in the training set, such as Romanized Russian, and Hindi.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 72
eval_batch_size: 36
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 144
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 2
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
0.0404	0.1206	2500	0.0649	0.7944	0.8616	0.8266	0.9868
0.0394	0.2412	5000	0.0538	0.8181	0.8696	0.8430	0.9893
0.0345	0.3618	7500	0.0456	0.8355	0.8781	0.8563	0.9906
0.0280	0.4824	10000	0.0493	0.8404	0.8836	0.8614	0.9897
0.0286	0.6030	12500	0.0515	0.8425	0.8805	0.8611	0.9889
0.0275	0.7236	15000	0.0423	0.8371	0.8852	0.8605	0.9905
0.0209	0.8442	17500	0.0429	0.8671	0.8908	0.8788	0.9911
0.0265	0.9648	20000	0.0379	0.8550	0.8881	0.8712	0.9919
0.0223	1.0854	22500	0.0371	0.8665	0.8967	0.8814	0.9918
0.0220	1.2060	25000	0.0344	0.8687	0.8954	0.8818	0.9926
0.0225	1.3266	27500	0.0332	0.8776	0.9011	0.8892	0.9928
0.0186	1.4472	30000	0.0390	0.8711	0.9018	0.8862	0.9920
0.0200	1.5678	32500	0.0315	0.8840	0.9046	0.8942	0.9931
0.0170	1.6884	35000	0.0313	0.8867	0.9066	0.8965	0.9932
0.0170	1.8090	37500	0.0305	0.8804	0.9034	0.8918	0.9933
0.0176	1.9296	40000	0.0305	0.8866	0.9058	0.8961	0.9935

Framework versions

Transformers 5.0.0
Pytorch 2.10.0+cu128
Datasets 4.0.0
Tokenizers 0.22.2