TwinSunsLLC/multilingual-e5-small-gguf

multilingual-e5-small — GGUF (Q8_0)

GGUF conversion of intfloat/multilingual-e5-small, quantized to Q8_0, for on-device retrieval in Offline AI: Private Chat.

• 384 dimensions, 12 layers, 512 max tokens
• 126 MB (f16 is 236 MB)
• Runs via llama.cpp / llama.swift with embeddings = true and mean pooling

Quantization is free here

Q8_0 matches f16 to three decimal places on cosine similarity, and produces an identical ranking order — which is what retrieval actually depends on.

Pair	Q8_0	f16
EN paraphrase	0.868	0.868
EN → FR translation	0.896	0.896
EN → ES translation	0.843	0.843
query → correct passage	0.848	0.848
EN → unrelated topic	0.737	0.736
query → wrong passage	0.671	0.671

Note the high floor: unrelated text still scores 0.737. Rank by top-k; do not use an absolute similarity threshold.

Usage

E5 is asymmetric — prefix stored text with passage: and queries with query: . Omitting the prefixes degrades retrieval, and mixing them is worse than using neither.

Conversion

convert_hf_to_gguf.py fails on this model with BPE pre-tokenizer was not recognized: it declares architectures: ["BertModel"] but ships an XLM-RoBERTa SentencePiece tokenizer. Rewrite config.json to architectures: ["XLMRobertaModel"] first, then convert and quantize.

License

MIT, inherited from the base model. Copyright belongs to the original authors — see intfloat/multilingual-e5-small.