MixLID

Token-Level Language Identification in Mixed-Language Text

Demo: Hugging Face Space

Paper: Coming soon.

MixLID identifies the language of individual tokens, including short expressions embedded in another language. It supports 111 language labels and returns both language spans and one predominant language per input. Numbers, punctuation, and symbols have separate labels: NUM, PUNCT, and SYM.

Highlights

• Identify languages within mixed text: locate language spans, including short expressions embedded in another language, across 111 supported language labels.
• Two prediction modes: identify the language of individual tokens or the predominant language of an entire input with the same model.

Property	MixLID
Backbone	mmBERT-small
Architecture	Encoder with a token classification head
Supported inputs	Monolingual and mixed-language text
Language coverage	111 language labels
Auxiliary labels	NUM, PUNCT, SYM
Prediction modes	Token-level predictions and predominant input language
Output details	Language labels, character offsets, and prediction scores
Context window	8,192 tokens

Potential applications

Application	Potential use of MixLID
Multilingual data filtering	Select text in target languages and locate other-language spans within documents.
Reinforcement learning and generation evaluation	Measure whether responses follow the requested language, and explore language consistency as a reward component, including unwanted language mixing.
Speech and audio-text research	Check transcript languages, annotate mixed-language transcripts, and compare text-side labels with audio-language predictions during corpus preparation.

Usage

pip install torch==2.9.1 transformers==4.57.0 safetensors==0.7.0

Transformers

Load the downloaded model directory, or replace "./MixLID" with the published Hugging Face model ID:

from transformers import AutoModel

model = AutoModel.from_pretrained("./MixLID", trust_remote_code=True)
# To use a GPU: model = model.to("cuda:0")

text = "The exhibition brings together photographs and handwritten letters."
sentence = model.predict_sentence(text)
print(sentence["language"], sentence["score"])

result = model.predict_tokens(text)
for span in result["spans"]:
    print(span["text"], span["label"], span["start"], span["end"], span["score"])

trust_remote_code=True loads the prediction methods included in this repository. When using a Hub model ID, Transformers downloads the required Python files automatically.

Local MixLID module

Download the repository, including mixlid.py, and run Python from that directory:

from mixlid import MixLID

model = MixLID.from_pretrained()  # Loads the checkpoint beside mixlid.py.
# To use a GPU: model = MixLID.from_pretrained(device="cuda:0")

text = "The exhibition brings together photographs and handwritten letters."
sentence = model.predict_sentence(text)
print(sentence["language"], sentence["score"])

result = model.predict_tokens(text)
for span in result["spans"]:
    print(span["text"], span["label"], span["start"], span["end"], span["score"])

This method loads the built-in Transformers architecture without enabling remote code. Both interfaces provide the same predictions: predict_sentence() returns the predominant language of each input, and predict_tokens() returns subword predictions and merged language spans. Both accept a string or a list of strings. predict_sentence() does not split paragraphs into sentences.

Examples

The outputs below were checked with the released checkpoint.

Korean, English, and French

text = '연구팀은 대학원생들과 함께 1,000개 문서를 검토했고, the initial results를 바탕으로 오류 유형을 정리했다. 프랑스 측은 «Nous allons poursuivre cette recherche avec une équipe internationale.»라고 밝혔다. 평균 점수는 3.14였으며, 다음 분석에서는 a larger sample을 사용할 예정이다.'

result = model.predict_tokens(text)
result["spans"]  # Each span contains text, start, end, label, and score.

Label	Predicted spans
kor_Hang	연구팀은 대학원생들과 함께 · 개 문서를 검토했고 · 를 바탕으로 오류 유형을 정리했다 · 프랑스 측은 · 라고 밝혔다 · 평균 점수는 · 였으며 · 다음 분석에서는 · 을 사용할 예정이다
NUM	1,000 · 3.14
PUNCT	, · . · « · .» · . · , · .
eng_Latn	the initial results · a larger sample
fra_Latn	Nous allons poursuivre cette recherche avec une équipe internationale

Turkish, German, and English

text = 'Yeni sergiye nach der Arbeit gitmeyi düşünüyoruz; girişte free admission yazıyor ama atölyeler için eine Anmeldung gerekiyor. Çocuklara yönelik bölümde hands-on activities var, bu yüzden mit der ganzen Familie gitmek daha keyifli olabilir.'

result = model.predict_tokens(text)
result["spans"]  # Each span contains text, start, end, label, and score.

Label	Predicted spans
tur_Latn	Yeni sergiye · gitmeyi düşünüyoruz · girişte · yazıyor ama atölyeler için · gerekiyor · Çocuklara yönelik bölümde · var · bu yüzden · gitmek daha keyifli olabilir
deu_Latn	nach der Arbeit · eine Anmeldung · mit der ganzen Familie
PUNCT	; · . · - · , · .
eng_Latn	free admission · hands · on activities

Performance

Monolingual identification

Each baseline and MixLID are evaluated on their shared language set; #L gives its label count. ACC is overall accuracy, and Macro F1 averages per-language F1. Scores are percentages, with the higher score in each comparison in bold. Comparison sets differ across baselines.

For fastText lid.176, CommonLingua, and Polyglot Tagger, script eligibility follows the evaluation mapping where explicit official script coverage is unavailable.

Mixed-language identification

Each cell reports ACC / Macro F1 (%). All models are evaluated on the same eligible tokens. Scores are computed per example and averaged within each language pair; Avg is the arithmetic mean across pairs. Non-language, mixed, and ambiguous reference labels, and tokens consisting only of whitespace, numbers, punctuation, or symbols, are excluded consistently.

ACS

Language pair	CLD3	MaskLID	Polyglot Tagger	MixLID
Danish–English	56.1 / 32.6	38.7 / 20.7	62.5 / 37.9	83.3 / 73.5
German–English	56.4 / 33.9	35.9 / 22.9	78.2 / 62.0	98.0 / 96.9
Spanish–English	48.2 / 28.2	22.9 / 15.3	74.6 / 56.9	96.1 / 94.1
French–English	54.8 / 32.1	24.4 / 15.5	71.0 / 47.7	93.1 / 90.4
Indonesian–English	31.7 / 18.4	19.3 / 12.1	38.0 / 24.2	92.1 / 89.1
Italian–English	56.3 / 33.3	29.6 / 19.3	74.3 / 55.7	95.6 / 93.6
Dutch–English	52.2 / 30.9	35.8 / 21.9	66.1 / 45.1	92.6 / 90.0
Swedish–English	56.6 / 32.9	41.7 / 24.9	71.7 / 48.7	92.3 / 87.1
Turkish–German	58.1 / 35.1	53.1 / 37.2	86.6 / 80.5	93.9 / 81.5
Turkish–English	44.6 / 25.8	31.2 / 18.9	69.9 / 45.6	91.6 / 87.4
Chinese–English	86.2 / 69.5	37.5 / 25.2	94.7 / 89.2	97.0 / 93.0
Avg	54.6 / 33.9	33.6 / 21.3	71.6 / 53.9	93.2 / 88.8

CodeMixBench

Language pair	CLD3	MaskLID	Polyglot Tagger	MixLID
German–English	86.3 / 61.1	55.3 / 29.2	90.6 / 65.8	95.2 / 82.8
Spanish–English	55.9 / 47.3	25.8 / 17.6	82.3 / 71.2	92.8 / 88.7
Chinese–English	80.5 / 46.0	61.6 / 31.0	83.9 / 55.3	95.5 / 90.3
Avg	74.2 / 51.5	47.6 / 25.9	85.6 / 64.1	94.5 / 87.3

Output format

Function	Returns
predict_sentence(text)	text, language, score
predict_tokens(text)	text, tokens, spans

• Tokens: subword predictions with text, start, end, label, and score.
• Spans: consecutive predictions with the same label, merged across intervening ordinary spaces. These are language spans, not word segmentation.
• Input language: majority vote over retained subword language predictions, excluding NUM, PUNCT, and SYM. Its score is the winning vote share, not a calibrated probability.
• Offsets: zero-based character positions in the original Python string; end is exclusive. Input text is not normalized.

Special tokens, padding, and ordinary space-only ranges are omitted. Tabs and newlines retain model predictions. Empty input or input without language votes returns und with score 0; this is a helper fallback, not a classifier label. Token scores are softmax probabilities, and span scores average the constituent token scores.

Supported languages

MixLID predicts 111 language labels, listed below with their full names. Labels distinguish scripts where applicable; for example, Mandarin Chinese has separate simplified and traditional Chinese labels. The additional labels are NUM (numbers), PUNCT (punctuation), and SYM (symbols).

Show all 111 language labels

Language	Label	Language	Label
Afrikaans	afr_Latn	Marathi	mar_Deva
Amharic	amh_Ethi	Modern Greek	ell_Grek
Arabic	ara_Arab	Nepali	npi_Deva
Armenian	hye_Armn	Nigerian Fulfulde	fuv_Latn
Assamese	asm_Beng	North Azerbaijani	azj_Latn
Bambara	bam_Latn	Northern Uzbek	uzn_Latn
Basque	eus_Latn	Norwegian Bokmål	nob_Latn
Bengali	ben_Beng	Nyanja	nya_Latn
Bulgarian	bul_Cyrl	Odia	ory_Orya
Burmese	mya_Mymr	Panjabi	pan_Guru
Catalan	cat_Latn	Paraguayan Guaraní	gug_Latn
Cebuano	ceb_Latn	Pedi	nso_Latn
Central Khmer	khm_Khmr	Persian	fas_Arab
Central Kurdish	ckb_Arab	Plateau Malagasy	plt_Latn
Croatian	hrv_Latn	Polish	pol_Latn
Czech	ces_Latn	Portuguese	por_Latn
Danish	dan_Latn	Romanian	ron_Latn
Dutch	nld_Latn	Russian	rus_Cyrl
English	eng_Latn	Serbian	srp_Cyrl
Filipino	fil_Latn	Shan	shn_Mymr
Finnish	fin_Latn	Shona	sna_Latn
French	fra_Latn	Sindhi	snd_Arab
Ganda	lug_Latn	Sinhala	sin_Sinh
Georgian	kat_Geor	Slovak	slk_Latn
German	deu_Latn	Slovenian	slv_Latn
Gujarati	guj_Gujr	Somali	som_Latn
Haitian	hat_Latn	Southern Pashto	pbt_Arab
Halh Mongolian	khk_Cyrl	Southern Sotho	sot_Latn
Hausa	hau_Latn	Spanish	spa_Latn
Hebrew	heb_Hebr	Standard Estonian	ekk_Latn
Hindi	hin_Deva	Standard Latvian	lvs_Latn
Hungarian	hun_Latn	Standard Malay	zsm_Latn
Icelandic	isl_Latn	Sundanese	sun_Latn
Igbo	ibo_Latn	Swahili	swh_Latn
Iloko	ilo_Latn	Swati	ssw_Latn
Indonesian	ind_Latn	Swedish	swe_Latn
Italian	ita_Latn	Tajik	tgk_Cyrl
Japanese	jpn_Jpan	Tamil	tam_Taml
Javanese	jav_Latn	Telugu	tel_Telu
Kabuverdianu	kea_Latn	Thai	tha_Thai
Kachin	kac_Latn	Tibetan	bod_Tibt
Kannada	kan_Knda	Tigrinya	tir_Ethi
Kazakh	kaz_Cyrl	Tosk Albanian	als_Latn
Kinyarwanda	kin_Latn	Tsonga	tso_Latn
Kirghiz	kir_Cyrl	Tswana	tsn_Latn
Korean	kor_Hang	Turkish	tur_Latn
Lao	lao_Laoo	Ukrainian	ukr_Cyrl
Lingala	lin_Latn	Urdu	urd_Arab
Lithuanian	lit_Latn	Vietnamese	vie_Latn
Luo	luo_Latn	Waray	war_Latn
Macedonian	mkd_Cyrl	West Central Oromo	gaz_Latn
Malayalam	mal_Mlym	Wolof	wol_Latn
Maltese	mlt_Latn	Xhosa	xho_Latn
Mandarin Chinese	cmn_Hans	Yoruba	yor_Latn
Mandarin Chinese	cmn_Hant	Zulu	zul_Latn
Maori	mri_Latn