py3langid-compact / README.md
WhiteHades's picture
py3langid in compact form for metldr
21f3535 verified
|
Raw History Blame Contribute Delete
2.11 kB
metadata
license: bsd-3-clause
library_name: metldr
pipeline_tag: text-classification
tags:
  - language-identification
  - langid
  - int8
  - quantized
language:
  - multilingual

py3langid-compact

The py3langid language identification model (langid.py by Marco Lui and Timothy Baldwin, as shipped in rust-py3langid 0.4.1, BSD-3-Clause; 142 labels) in a compact form for the browser: int8 weights with one scale per feature, and the byte automaton stored as each row's differences from the start row (a 256-bit mask plus its targets). In memory it takes 25 MB instead of 97 MB, and this gzip file is 5.2 MB. It is the language detector of the metldr browser extension, read by its vendored py3langid Rust crate.

Format (py3langid-compact.lidc.gz, gzip of a little-endian file)

Magic LIDC1\0\0\0; num_features, num_languages, num_states, num_rows, num_targets, history bytes (u32 each); the start row (256 u32); per row a 256-bit mask (4 u64); per row its first target (u32); the targets (u32); each state's row (u16); each state's output feature (i32, -1 for none); priors (f32 per language); scales (f32 per feature); weights (i8, features by languages); labels (u16 length and UTF-8 bytes each).

Converted from the upstream py3langid.lidg (SHA-256 da6860a9218a6122bcf26ba6e8752946336dc4f9c0816c6eb4fa790c7444dec2) by pack-langid in.lidg out.lidc.

Evaluation (papluca/language-identification test, 10,000 texts, 20 languages)

detector sentence accuracy four-word accuracy time per text memory
this model 99.14% 93.62% 26 µs 25 MB
py3langid upstream 99.14% 93.65% 23 µs 97 MB
lingua, high accuracy 98.80% 93.49% 1,165 µs 275 MB of models
fastText lid.176.ftz 97.63% 89.42% 11 µs 1 MB
langidentify lite 94.34% 91.93% 27 µs
whatlang 90.59% 77.81% 39 µs

It agrees with upstream on 19,981 of 20,000 texts (each sentence and its first four words).

License

BSD-3-Clause, as the upstream model.