Update model card for Breton

README.md

@@ -1,39 +1,70 @@
-
 ---
-pipeline_tag: fill-mask
+pipeline_tag: sentence-similarity
 language: bre
 license: mit
 tags:
-- trimmed
-library_name: transformers
+  - trimmed
+library_name: sentence-transformers
 base_model: intfloat/multilingual-e5-small
 base_model_relation: quantized
 datasets:
-- Lumberjackk/fineweb-2-trimming
+  - lbourdois/fineweb-2-trimming
 ---
-# multilingual-e5-small-breton-16384
-
-This model is a 76.3% smaller version of [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) 
-for the **bre** language, created using vocabulary pruning.
 
-**Vocabulary size**: 16384 tokens (reduced from 250002)  
-**Tokenizer type**: Unigram  
-**Training samples**: 200000 texts  
-**Dataset**: Lumberjackk/fineweb-2-trimming
+# multilingual-e5-small-bre-16384
+This model is a **76.26% smaller** version of [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) optimized for **Breton** language via vocabulary size reduction using the [trimming](https://huggingface.co/blog/lbourdois/introduction-to-trimming) method.  
+This trimmed model should perform similarly to the original model with only 16,384 tokens and a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.
 
-This pruned model should perform similarly to the original model for bre language tasks with a much smaller 
-memory footprint. However, it may not perform well for other languages present in the original multilingual model as tokens not 
-commonly used in bre were removed from the original multilingual model's vocabulary.
+## Model Statistics
+| Metric | Original | Trimmed | Reduction |
+|--------|----------|---------|-----------|
+| **Vocabulary size** | 250,037 tokens | 16,384 tokens | **93.44%** |
+| **Model size** | 117,653,760 params | 27,931,008 params | **76.26%** |
 
-## Usage
+![image](https://raw.githubusercontent.com/lbourdois/blog/refs/heads/master/assets/images/Trimming/me5-small-16384.png)
 
-You can use this model with the Transformers library:
+## Mining Dataset Statistics
+- **Number of texts used for mining**: 54,409 texts  
+- **Dataset**: [lbourdois/fineweb-2-trimming](https://huggingface.co/datasets/lbourdois/fineweb-2-trimming)
 
+## Usage
 ```python
-from transformers import AutoModel, AutoTokenizer
+from sentence_transformers import SentenceTransformer
+# Download from the 🤗 Hub
+model = SentenceTransformer("alphaedge-ai/multilingual-e5-small-bre-16384")
+# Run inference with queries and documents
+query = "My query in Breton"
+documents = [
+    "Chunk in Breton",
+    "Chunk in Breton",
+    "Chunk in Breton",
+]
+query_embeddings = model.encode_query(query)
+document_embeddings = model.encode_document(documents)
+print(query_embeddings.shape, document_embeddings.shape)
+# Compute similarities to determine a ranking
+similarities = model.similarity(query_embeddings, document_embeddings)
+print(similarities)
+```
+
+## Citations
 
-model_name = "lbourdois/multilingual-e5-small-breton-16384"
-model = AutoModel.from_pretrained(model_name)
-tokenizer = AutoTokenizer.from_pretrained(model_name)
+#### Multilingual E5
+```
+@article{wang2024multilingual,
+  title={Multilingual E5 Text Embeddings: A Technical Report},
+  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
+  journal={arXiv preprint arXiv:2402.05672},
+  year={2024}
+}
 ```
 
+#### Trimming blog post
+```
+@misc{hf_blogpost_trimming,
+      title={Introduction to Trimming}, 
+      author={Loïck BOURDOIS and Tom AARSEN and Bram VANROY and Christopher AKIKI and Woojun JUNG and Manuel ROMERO and Prithiv SAKTHI},
+      year={2026},
+      url={https://huggingface.co/blog/lbourdois/introduction-to-trimming}, 
+}
+```