--- license: bsd-3-clause library_name: metldr pipeline_tag: text-classification tags: - language-identification - langid - int8 - quantized language: - multilingual --- # py3langid-compact The py3langid language identification model (langid.py by Marco Lui and Timothy Baldwin, as shipped in [rust-py3langid](https://github.com/markusmobius/rust-py3langid) 0.4.1, BSD-3-Clause; 142 labels) in a compact form for the browser: int8 weights with one scale per feature, and the byte automaton stored as each row's differences from the start row (a 256-bit mask plus its targets). In memory it takes 25 MB instead of 97 MB, and this gzip file is 5.2 MB. It is the language detector of the metldr browser extension, read by its vendored `py3langid` Rust crate. ## Format (`py3langid-compact.lidc.gz`, gzip of a little-endian file) Magic `LIDC1\0\0\0`; num_features, num_languages, num_states, num_rows, num_targets, history bytes (u32 each); the start row (256 u32); per row a 256-bit mask (4 u64); per row its first target (u32); the targets (u32); each state's row (u16); each state's output feature (i32, -1 for none); priors (f32 per language); scales (f32 per feature); weights (i8, features by languages); labels (u16 length and UTF-8 bytes each). Converted from the upstream `py3langid.lidg` (SHA-256 `da6860a9218a6122bcf26ba6e8752946336dc4f9c0816c6eb4fa790c7444dec2`) by `pack-langid in.lidg out.lidc`. ## Evaluation (papluca/language-identification test, 10,000 texts, 20 languages) | detector | sentence accuracy | four-word accuracy | time per text | memory | | --- | --- | --- | --- | --- | | **this model** | **99.14%** | **93.62%** | 26 µs | 25 MB | | py3langid upstream | 99.14% | 93.65% | 23 µs | 97 MB | | lingua, high accuracy | 98.80% | 93.49% | 1,165 µs | 275 MB of models | | fastText lid.176.ftz | 97.63% | 89.42% | 11 µs | 1 MB | | langidentify lite | 94.34% | 91.93% | 27 µs | | | whatlang | 90.59% | 77.81% | 39 µs | | It agrees with upstream on 19,981 of 20,000 texts (each sentence and its first four words). ## License BSD-3-Clause, as the upstream model.