|
Download README.md from WhiteHades/py3langid-compact: direct link, hf CLI and curl.
- Browser
- Download file 2.11 kB
-
https://huggingface.co/WhiteHades/py3langid-compact/resolve/main/README.md
- Command line
-
hf download hf://WhiteHades/py3langid-compact/README.md
-
curl -L -o README.md https://huggingface.co/WhiteHades/py3langid-compact/resolve/main/README.md
2.11 kB
| license: bsd-3-clause | |
| library_name: metldr | |
| pipeline_tag: text-classification | |
| tags: | |
| - language-identification | |
| - langid | |
| - int8 | |
| - quantized | |
| language: | |
| - multilingual | |
| # py3langid-compact | |
| The py3langid language identification model (langid.py by Marco Lui and Timothy Baldwin, as shipped in | |
| [rust-py3langid](https://github.com/markusmobius/rust-py3langid) 0.4.1, BSD-3-Clause; 142 labels) in a | |
| compact form for the browser: int8 weights with one scale per feature, and the byte automaton stored as each | |
| row's differences from the start row (a 256-bit mask plus its targets). In memory it takes 25 MB instead of | |
| 97 MB, and this gzip file is 5.2 MB. It is the language detector of the metldr browser extension, read by | |
| its vendored `py3langid` Rust crate. | |
| ## Format (`py3langid-compact.lidc.gz`, gzip of a little-endian file) | |
| Magic `LIDC1\0\0\0`; num_features, num_languages, num_states, num_rows, num_targets, history bytes (u32 | |
| each); the start row (256 u32); per row a 256-bit mask (4 u64); per row its first target (u32); the targets | |
| (u32); each state's row (u16); each state's output feature (i32, -1 for none); priors (f32 per language); | |
| scales (f32 per feature); weights (i8, features by languages); labels (u16 length and UTF-8 bytes each). | |
| Converted from the upstream `py3langid.lidg` (SHA-256 | |
| `da6860a9218a6122bcf26ba6e8752946336dc4f9c0816c6eb4fa790c7444dec2`) by | |
| `pack-langid in.lidg out.lidc`. | |
| ## Evaluation (papluca/language-identification test, 10,000 texts, 20 languages) | |
| | detector | sentence accuracy | four-word accuracy | time per text | memory | | |
| | --- | --- | --- | --- | --- | | |
| | **this model** | **99.14%** | **93.62%** | 26 µs | 25 MB | | |
| | py3langid upstream | 99.14% | 93.65% | 23 µs | 97 MB | | |
| | lingua, high accuracy | 98.80% | 93.49% | 1,165 µs | 275 MB of models | | |
| | fastText lid.176.ftz | 97.63% | 89.42% | 11 µs | 1 MB | | |
| | langidentify lite | 94.34% | 91.93% | 27 µs | | | |
| | whatlang | 90.59% | 77.81% | 39 µs | | | |
| It agrees with upstream on 19,981 of 20,000 texts (each sentence and its first four words). | |
| ## License | |
| BSD-3-Clause, as the upstream model. | |