bekko-static-v1-slim-32k

A vocabulary-pruned derivative of hotchpotch/bekko-static-embedding-v1, a static embedding model (token lookup + mean pooling, 1024-d, Matryoshka 32–1024, no transformer). Same architecture, same embedding space, but 32,000 of the original 200,000 WordPiece tokens — the ones most useful for the 14 MNanoBEIR languages:

English, Arabic, German, Spanish, French, Italian, Japanese, Korean, Norwegian, Portuguese, Serbian, Swedish, Thai, Vietnamese.

Text in other languages is not supported: their script-specific tokens were removed and will tokenize to characters or [UNK].

Usage

sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("flower-computer/bekko-static-v1-slim-32k")           # add truncate_dim=256 etc. for Matryoshka sizes
emb = model.encode(["What is the capital of France?", "Paris is the capital of France."])
print(model.similarity(emb[0:1], emb[1:2]))

Queries and documents use no prompts/prefixes, as in the source model.

Rust: encoders

encoders (github.com/flowercomputers/encoders) compiles a static embedding model into your binary at build time (no runtime download, no Python); pick this model with its cargo feature (optional extras: a Matryoshka width via dim-512 … dim-32, an integer table via quant-8 / quant-16, parallel batches via rayon):

$ cargo add encoders --no-default-features --features 'model-bekko-static-v1-slim-32k'

let one   = encoders::encode_single("What is the capital of France?");
let batch = encoders::encode(["Paris is the capital of France.", "Tokyo is the capital of Japan."]);

The table is embedded as 32,000 × 1024 (or the chosen width) values. Output matches sentence-transformers' StaticEmbedding (same tokenization, mean pooling); the crate's tests check its embeddings against the reference implementation.

The family

All members: huggingface.co/collections/flower-computer/bekko-static-v1-slim

model	vocab · fp16 size	langs	NanoBEIR-en	MNanoBEIR	NanoMIRACL
hotchpotch/bekko-static-embedding-v1	200k · 410 MB	40	53.71	43.71	68.03
bekko-static-v1-slim-100k	100k · 205 MB	25	52.91	42.98	66.53
bekko-static-v1-slim-75k	75k · 154 MB	25	52.09	42.15	64.51
bekko-static-v1-slim-50k	50k · 102 MB	25	50.22	40.37	61.35
bekko-static-v1-slim-32k (this model)	32k · 66 MB	14	49.31	39.86	49.25

The 50k+ members keep 25 languages; the 32k member keeps the 14 MNanoBEIR languages (en ar de es fr it ja ko no pt sr sv th vi), because 25 languages do not fit in 32k pieces without the long-word scripts (te/th/bn) shattering into characters. Scores are nDCG@10 × 100 at 1024 dims; NanoMIRACL includes languages the 32k model drops.

How the vocabulary was chosen

The source model's quality lives in its 200k-token table, and in a static model the tokenizer's pieces are exactly the vectors that get averaged, so removing a token is not free: every word that used it is re-tokenized into shorter pieces whose vectors were trained for other contexts. Tokens were therefore ordered by expected damage from removal:

usefulness(t) = corpus frequency of t × ‖ E[t] − Σ E[pieces t falls back to when t is removed] ‖

computed in rounds (drop the least useful 10 %, re-tokenize, re-score the tokens whose fallback changed), then truncated at the target size. Frequencies are a per-language mix of WebFAQ-v2 and Wikipedia text (equal weight per language, English ×3). Special tokens and every single-character piece that occurs in the corpus are always kept, so covered-language text never degrades to [UNK]. After truncation, the surviving rows were gently fine-tuned by self-distillation: cosine loss at 1024/256/64 dims against the full model's sentence embeddings, on WebFAQ-v2 and Wikipedia paragraphs of the covered languages (lr 0.03, 3000 steps, batch 1024, sparse Adam on touched rows only). This is a nudge, not a retrain — the median row moves ~10 % of its norm — and it recovers roughly 1–2 nDCG points at small vocabularies; larger learning rates reduce the distillation loss further but hurt retrieval.

Because the vocabulary is smaller, covered-language text tokenizes into ≈21 % more tokens than with the source tokenizer (measured on Wikipedia paragraphs); embedding cost scales with that.

Evaluation

model	vocab · fp16 size	langs	NanoBEIR-en	MNanoBEIR	NanoMIRACL
hotchpotch/bekko-static-embedding-v1	200k · 410 MB	40	53.71	43.71	68.03
this vocabulary, no recovery	32k · 66 MB	14	48.57	39.33	47.61
bekko-static-v1-slim-32k	32k · 66 MB	14	49.31	39.86	49.25

Measured with a local cosine-retrieval harness on the HAKARI-Bench Nano datasets, which reproduces the source card's NanoBEIR-en 53.71 and NanoMIRACL 68.03 exactly. NanoBEIR-en = 13 tasks; MNanoBEIR = 14 languages × 13 tasks (182); NanoMIRACL = 18 languages. All numbers are nDCG@10 × 100, equal weight per task.

Matryoshka truncation

dims	1024	512	256	128	64
NanoBEIR-en	49.31	48.39	46.16	42.53	36.51
MNanoBEIR	39.86	38.88	36.85	33.12	27.50
NanoMIRACL	49.25	48.60	47.19	44.10	36.47

Matryoshka sizes

dims	1024	512	256	128	64	32
fp16 MB	66	33	16	8	4	2

Caveats

• Losses concentrate in lexically specific tasks (entity names, scientific/biomedical terms) and, at small vocabularies, in languages with long agglutinative words (Telugu, Thai, Bengali), where words fragment into characters.
• Uncased, like the source. Only model.safetensors (fp16) is provided — no ONNX/OpenVINO/int8 exports.
• Pruning code: the ordering, recovery and evaluation scripts live in the bekko-prune repository (link to follow).

License and credit

MIT, same as the source. All credit for the model and its training goes to the source author (hotchpotch); this repository only removes rows from its table.