metadata
pipeline_tag: sentence-similarity
language: hat
license: mit
tags:
- trimmed
library_name: sentence-transformers
base_model: intfloat/multilingual-e5-base
base_model_relation: quantized
datasets:
- lbourdois/fineweb-2-trimming
multilingual-e5-base-hat-16384
This model is a 64.53% smaller version of intfloat/multilingual-e5-base optimized for Haitian Creole language via vocabulary size reduction using the trimming method.
This trimmed model should perform similarly to the original model with only 16,384 tokens and a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.
Model Statistics
| Metric | Original | Trimmed | Reduction |
|---|---|---|---|
| Vocabulary size | 250,037 tokens | 16,384 tokens | 93.44% |
| Model size | 278,043,648 params | 98,625,024 params | 64.53% |
Mining Dataset Statistics
- Number of texts used for mining: 200,000 texts
- Dataset: lbourdois/fineweb-2-trimming
Usage
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("alphaedge-ai/multilingual-e5-base-hat-16384")
# Run inference with queries and documents
query = "My query in Haitian Creole"
documents = [
"Chunk in Haitian Creole",
"Chunk in Haitian Creole",
"Chunk in Haitian Creole",
]
query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# Compute similarities to determine a ranking
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
Citations
Multilingual E5
@article{wang2024multilingual,
title={Multilingual E5 Text Embeddings: A Technical Report},
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
journal={arXiv preprint arXiv:2402.05672},
year={2024}
}
Trimming blog post
@misc{hf_blogpost_trimming,
title={Introduction to Trimming},
author={Loïck BOURDOIS and Tom AARSEN and Bram VANROY and Christopher AKIKI and Woojun JUNG and Manuel ROMERO and Prithiv SAKTHI},
year={2026},
url={https://huggingface.co/blog/lbourdois/introduction-to-trimming},
}