unilid-1940 (calibrated)
UNILID language-identification model covering 1,940 language-script combinations, trained on 60M samples from the GlotLID-C corpus. This repository distributes the calibrated release: a version-2 .unilid file containing the trained per-language token probability tables plus the bundled calibration artifact (per-language decision thresholds, training-line counts, and the calibration constants).
Code: https://github.com/Ahmetcanyvz/UNILID (calibrated inference is added by PR #1; until it merges, install from the calibration-release branch of https://github.com/cimeister/UNILID).
Files
| File | Size | Content |
|---|---|---|
unilid-1940-calibrated.unilid |
780 MB | Base weights (float32, 1,940 languages x 100k vocabulary) + bundled calibration (version-2 container) |
calibration.json |
160 KB | The same calibration artifact as a standalone file |
Usage
from unilid import load_model
model = load_model("unilid-1940-calibrated.unilid") # calibrated inference (default)
lang, tokens, score = model.predict("The quick brown fox jumps over the lazy dog.")
print(lang) # 'eng_Latn'
# Base (uncalibrated) behavior of the original release:
base_model = load_model("unilid-1940-calibrated.unilid", calibrated=False)
Requires the UNILID package with its Rust tokenizers extension built (installation instructions in the repository README).
Calibration
Calibrated inference applies two corrections, specified in the UNILID paper: a shared constant c = -21 for unseen-token log-probabilities (one-sided, no renormalization), and re-examination of low-margin predictions into two groups of languages (languages with fewer than 18,000 training samples, and a four-language high-entropy group), with reassignment to a candidate in ranks 2-5 that has at least 100,000 training samples and a score within 21 natural-log units of the top score.
Measured effect:
| Evaluation | Base | Calibrated |
|---|---|---|
| GlotLID-C test pool (45.4M lines), macro F1 | 0.929 | 0.957 |
| UDHR (parallel, near-equal per-language sample counts), macro F1 | 0.859 | 0.838 |
| CommonLID (out-of-domain web text, 109 labels), macrolanguage-aware accuracy | 0.845 | 0.860 |
| CommonLID, tag-level macro F1 | 0.723 | 0.715 |
On UDHR and on CommonLID's tag-level macro F1, re-examination also moves some correct low-margin predictions, which lowers those numbers; on CommonLID it also lowers the number of lines predicted as languages outside the 109-label set from 32,901 to 25,884, which raises accuracy. The gains appear on test data whose per-language line counts follow a collection's natural imbalance, over a label set that includes under-resourced languages. Pass calibrated=False where the base behavior is wanted.
New languages can be added to this model without retraining anything else (unilid-add-language); the repository README documents the workflow and its three caveats.
License and notices
Apache-2.0. The weights represent aggregated statistical patterns (per-language token probability estimates) derived from the GlotLID-C corpus, which aggregates publicly available sources that may include materials subject to copyright or other rights. The weights do not store or reproduce expressive text content, and no training data is included in this distribution. If you are a rights holder and believe your content was included in the training data and wish to request its exclusion from future versions, contact the authors. Users are responsible for ensuring their use complies with applicable laws and the licensing requirements of any data they process.