germanic-lid

A 6.3 MB fastText language identifier for English and the Germanic languages. It has one class for other Germanic varieties and one other class for everything else. It is built for filtering large corpora on CPU and labels about 25,000 lines per second on one core.

This is version 2.1 (24 Oct 2026). Every fastText file was retrained on a uniformly shuffled copy of the same training data, with the same settings. The first release (v2.0, 9 Oct 2026) was trained on a file whose label mix drifted along it. That cost it accuracy on Bokmål against Nynorsk: v2.0 labelled 3.3% of the Bokmål FLORES-200 sentences as Nynorsk, and v2.1 labels 0.8%. v2.0 stays available at revision v2.0. TrustLLM D5.2 used pareto/germanic-lid-d128.bin from v2.0.

The 16 labels are en de nl af fy lb gsw yi is fo da sv nb nn gem_other other:

• gsw: Alemannic / Swiss German.
• gem_other: other Germanic varieties (Low German, Limburgish, Bavarian, North and Saterland Frisian, West Flemish, Zeelandic, Pennsylvania German, Kölsch, Palatine German). In a filter, treat gem_other as other, unless you want those varieties.
• other: everything non-Germanic, plus non-language noise.

Files

File	What
germanic-lid.ftz	quantised fastText model, 64 dimensions, 6.3 MB (the default)
labels.txt	the 16 labels
germanic_lid.py	wrapper: GermanicLID(path).predict(lines, threshold) and a CLI to predict, filter or annotate JSONL and parquet
results/	metrics JSON and tables for v2.1 (per-label P/R/F1 with bootstrap CIs, confusion matrices, hard pairs, threshold curves)
pareto/germanic-lid-d32.ftz	2.6 MB, 31,000 lines/s/core
pareto/germanic-lid-d32.bin	162 MB, unquantised, 45,000 lines/s/core: the best choice for bulk filtering
pareto/germanic-lid-d128.bin	1.1 GB, our most accurate (macro-F1 0.985 on the independent sets)
pareto/PARETO.md, pareto/pareto.png, pareto/pareto.json	accuracy, speed and size of 20 systems and 4 cascades
pareto/archive/	the d64 .bin and d128 .ftz of v2.1 with their training records; the d64 word-bigram model, char NB/LR and a CLD3-style network (trained in the v2.0 file order and not retrained); the superseded v1

Usage

from germanic_lid import GermanicLID
lid = GermanicLID("germanic-lid.ftz")
lid.predict(["Hvar er næsta strætóstoppistöð?", "Dette er ein test."], threshold=0.7)

python germanic_lid.py filter --model germanic-lid.ftz --input in.jsonl --field text --keep is,fo --threshold 0.7 --output out.jsonl
python germanic_lid.py filter --model germanic-lid.ftz --input in.parquet --field text --keep nb,nn --threshold 0.7 --output out.parquet

The wrapper normalises the text before prediction: NFKC, lowercasing, URLs and e-mail addresses removed, digits mapped to 0. It then applies a script gate: a Germanic label needs mostly Latin letters (Hebrew for yi), and any other line becomes other. If you call fastText directly, apply the same normalisation.

Recipe

• Training lines shuffled uniformly at random (NumPy permutation, seed 0).
• fastText supervised, softmax; dim 64; char n-grams 2-5 plus words (minCount 50); 1M buckets; 3 epochs; lr 0.8; seed 0; 3 threads.
• Quantised with qnorm, cutoff 150k, dsub 2 and 1 retrain epoch at lr 0.2.
• d32: as above with cutoff 100k and 2 threads. d128: char n-grams 1-6, 2M buckets, cutoff 400k, 2 threads.
• Trained on 7.95M lines: 400k-600k per large class; small classes upsampled to at most 3×; 2.6M lines for other.
• Dedup by normalised-line hash; lines that occur under two labels are dropped; splits by document.

Training data

Source	Used for	Licence
OpenLID-v3 dataset (HPLT)	most labels; other (180 languages + noise)	per-source; all allow non-commercial use
Wikipedia 20231101	every Germanic label and gem_other; close confusables in other (Sami, Greenlandic, Celtic, Romance minority languages, Sorbian, Finnic minority languages)	CC BY-SA 4.0
FineWeb-2 train	fo, fy, lb, gsw, yi; a line is kept only if GlotLID agrees on that single line	ODC-By 1.0
Our cleaned Icelandic corpus (private)	is	mixed source licences

Not used: glotlid-corpus (gated, no access); FLORES+ (gated, no access); Scots Wikipedia (quality problem).

Results

Our model (6.3 MB) against OpenLID-v3 (1.23 GB, with the preprocessing from its model card) and GlotLID v3 (1.69 GB), with the baselines' labels mapped to ours. Macro-F1 is over the 12 labels all three support. Accuracy covers all 16 labels. Speed is measured through Python on one core of an Apple M4 Max and includes our normalisation and script gate.

Set	Ours	OpenLID-v3	GlotLID
FLORES-200 devtest, macro-F1	0.996	0.995	0.997*
UDHR, macro-F1	0.973	0.888	0.990
Icelandic held-out documents, share labelled is	0.944	0.918	0.898
same, accuracy against native-speaker labels	0.974	0.952	0.944
In-domain held-out, macro-F1	0.942	0.882	0.917
Under 20 characters, accuracy	0.768	0.626	0.706
20-50 characters, accuracy	0.916	0.752	0.879
Synthetic web noise, accuracy	0.908	0.754	0.801
FineWeb-2 web lines (agreement with GlotLID-derived labels)	0.908	0.793	0.900
Lines/s, one core	24,900	16,800	4,400

* GlotLID was trained on FLORES-200 dev.

Public test sets (accuracy, %):

Set	Ours, 6.3 MB	Ours, d32 .bin	GlotLID	OpenLID-v3
Nordic DSL (15K test, six labels)	95.2	95.3	95.0	93.8
SLIDE, loose / exact	96.1 / 92.2	96.2 / 92.3	97.2 / 93.4	94.5 / 90.5
FastSpell (Danish, Bokmål, Nynorsk)	82.3	82.9	81.1	83.5
NorDial	96.1	96.1	95.8	95.4
Tatoeba, our 14 Germanic languages	91.6	91.7	93.2	84.2

Hard pairs on FLORES (share of the sentences of A labelled B):

Pair	Ours	OpenLID-v3	GlotLID
nb→nn / nn→nb	0.8% / 1.4%	1.9% / 1.7%	0.5% / 2.4%
nb→da / da→nb	0.5% / 0.4%	0.8% / 0.7%	0.3% / 0.5%
is→fo / fo→is	0.0% / 0.3%	0.1% / 0.0%	0.1% / 0.0%
nl↔af, de↔lb	0.1% or less	0.0%	0.0%

For gsw, fy and gem_other, see the per-label tables in results/; OpenLID-v3 has none of them. 95% bootstrap CIs for every number are in results/tables_v2.1.md.

A threshold of 0.7 raises in-domain Germanic precision from 0.945 to 0.978 and costs 2.7 points of recall. On FLORES, precision is then 0.999 at 99.0% recall.

Choosing a model

Every fastText file works with the same wrapper (--model <file>).

• Bulk filtering: pareto/germanic-lid-d32.bin with a threshold of 0.7.
• Small and portable: germanic-lid.ftz or pareto/germanic-lid-d32.ftz.
• Short text or maximum accuracy: pareto/germanic-lid-d128.bin.
• See pareto/PARETO.md.

Caveats

• The label set is closed, so unseen Latin-script languages can receive a Germanic label, most often gem_other, en or gsw. Use a threshold.
• On human-labelled web lines in English, German, Dutch, Frisian and Afrikaans (CommonLID), GlotLID and heliport are clearly more precise: macro-F1 over English, German, Dutch and Afrikaans is 0.834 and 0.828 against 0.731 for this model (0.786 with a threshold of 0.7). English and German lines often get a small-language label, and Hebrew lines often get yi. For those languages, prefer GlotLID; this model is built for the Nordic languages.
• Bokmål against Nynorsk is the hardest pair for every system we tested; ours is level with GlotLID on it. Lines under 20 characters reach only about 77% accuracy (all systems struggle there).
• gsw means Alemannic broadly (Wikipedia als includes Alsatian and Swabian).
• The FineWeb-2 portion inherits GlotLID's line-level decisions for fo, fy, lb, gsw and yi.
• The weights are released under Apache-2.0. Some training sources allow only non-commercial use (see the table above), so check their terms before commercial reuse.

Citation

@misc{einarsson2026germaniclid,
  author       = {Einarsson, Hafsteinn},
  title        = {germanic-lid: a fastText language identifier for English and the Germanic languages},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/hafsteinn/germanic-lid}}
}