| --- |
| license: mit |
| base_model: minishlab/potion-multilingual-128M |
| tags: |
| - model2vec |
| - static-embeddings |
| - embeddings |
| - multilingual |
| - quantized |
| library_name: model2vec |
| --- |
| |
| # potion-multilingual-128M-i8-tokenade |
|
|
| An int8 quantisation of [`minishlab/potion-multilingual-128M`](https://huggingface.co/minishlab/potion-multilingual-128M), |
| plus precomputed tokenizer artifacts for low-memory loading. |
|
|
| All credit for the model itself goes to [the Minish Lab](https://huggingface.co/minishlab) |
| — this repo adds no training, no distillation and no new data. It only changes |
| how the same weights are stored and loaded. |
|
|
| ## Why |
|
|
| model2vec inference is `tokenize → look up rows → mean-pool`: no context, no |
| forward pass. Loading it should not cost like a transformer, but in practice a |
| stock load decodes the whole weight matrix to `f32` and inflates |
| `tokenizer.json` into an in-memory trie — together about **1 GB of RSS and |
| ~930 ms** before the first vector. That is fine for a long-lived server and |
| prohibitive for a short-lived CLI process. |
|
|
| These artifacts are the same model arranged so it can be memory-mapped instead: |
|
|
| | | stock load | these artifacts | |
| |---|---|---| |
| | RSS | ~1056 MB | **~4.5 MB** | |
| | first embedding | ~930 ms | **~1 ms** | |
| | per embedding | ~0.03 ms | ~0.11 ms | |
|
|
| The per-embedding cost regresses (no pre-decoded matrix, no cache); a process |
| needs roughly 12 700 embeddings in one run before that outweighs the load-time |
| saving. |
|
|
| ## Files |
|
|
| | file | what it is | |
| |---|---| |
| | `model.safetensors` | the embedding matrix, `I8`, `[500353, 256]` | |
| | `vocab.fst` | the Unigram vocab as an FST: token bytes → id | |
| | `vocab.fst.scores` | `f64` unigram log-probs, one per id, in id order | |
| | `vocab.fst.norm` | the normalizer / pre-tokenizer spec and decoding metadata | |
| | `tokenizer.json` | upstream's, unchanged — kept so the FST can be re-derived and verified | |
| | `config.json` | upstream's, unchanged | |
|
|
| ## Quantisation |
|
|
| Symmetric, one global scale (`max|W| / 127`). The scale is deliberately **not** |
| stored: mean-pooling is linear and model2vec L2-normalises its output, so a |
| single global factor cancels exactly. Rows can therefore be read straight from |
| disk as raw `i8`. |
|
|
| This is measured, not assumed. Against `model2vec-rs` reading the f32 original, |
| over 100 133 text segments extracted from real PDF / DOCX / XLSX / EPUB / ODT / |
| ODS / ODP / PPTX files: |
|
|
| * token ids: **0 divergences** |
| * embeddings: worst cosine **0.999994** |
|
|
| Retrieval quality on our internal evaluation is unchanged within noise |
| (0.570/0.413 vs 0.571/0.414 for f32). |
|
|
| ## Caveats |
|
|
| * `[UNK]` handling follows `model2vec-rs` exactly, including that it does *not* |
| drop `[UNK]` rows for a Unigram tokenizer (it looks for a string `unk_token` |
| field, which Unigram does not emit). If you consume these artifacts with your |
| own loader and drop `[UNK]` instead, you will get different vectors on |
| unk-heavy text. |
| * `vocab.fst` maps duplicate tokens to their **last** id, matching how |
| `tokenizers` builds its `token_to_ids` map. |
| * The FST/score/norm layout is ours, not a model2vec standard. Anything reading |
| only `model.safetensors` + `tokenizer.json` (including model2vec itself) works |
| with this repo unmodified — the i8 tensor is a standard safetensors dtype. |
|
|
| ## License |
|
|
| MIT, inherited from the upstream model. |
|
|
