Feature Extraction
sentence-transformers
Safetensors
xlm-roberta
colbert
late-interaction
multi-vector
information-retrieval
african-languages
multilingual
Generated from Trainer
dataset_size:112864
loss:CachedMultiVectorMultipleNegativesRankingLoss
text-embeddings-inference
Instructions to use nebhailema/afrixir-multivector-e5-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use nebhailema/afrixir-multivector-e5-base with sentence-transformers:
from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("nebhailema/afrixir-multivector-e5-base") queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
docs: link each model in the full-corpus tables
Browse files
README.md
CHANGED
|
@@ -144,7 +144,7 @@ print(similarities)
|
|
| 144 |
```
|
| 145 |
## Evaluation
|
| 146 |
|
| 147 |
-
Full-corpus passage retrieval on [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR) for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. `avg` is the query-weighted average across those languages. The best value in that column is
|
| 148 |
|
| 149 |
Reranker rows rerank the top 50 passages from a frozen dense retriever.
|
| 150 |
|
|
@@ -154,79 +154,79 @@ Reranker rows rerank the top 50 passages from a frozen dense retriever.
|
|
| 154 |
|
| 155 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 156 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 157 |
-
| dense mini | 0.7805 | 0.6940 | 0.5643 | 0.6270 | 0.6637 | 0.7081 | 0.7755 | 0.6024 | 0.7233 |
|
| 158 |
-
| e5-small | 0.8741 | 0.8014 | 0.7418 | 0.8162 | 0.8088 | 0.8451 | 0.8571 | 0.7359 | 0.8379 |
|
| 159 |
-
| e5-base | 0.8886 | 0.8691 | 0.7810 | 0.8000 | 0.8526 | 0.8738 | 0.8571 | 0.7590 | 0.8629 |
|
| 160 |
-
| e5-large | 0.8948 | **0.8809** | 0.7974 | 0.8324 | 0.8841 | **0.8993** | 0.7755 | 0.7550 | 0.8738 |
|
| 161 |
-
| M3 dense | 0.7067 | 0.6264 | 0.4630 | 0.5892 | 0.5894 | 0.6488 | 0.7551 | 0.5231 | 0.6501 |
|
| 162 |
-
| SPLADE mini | 0.7759 | 0.7446 | 0.7603 | 0.7405 | 0.7705 | 0.7546 | 0.8163 | 0.6657 | 0.7599 |
|
| 163 |
-
| M3 sparse | 0.7826 | 0.7356 | 0.7593 | 0.7189 | 0.7627 | 0.7463 | 0.8367 | 0.6536 | 0.7598 |
|
| 164 |
-
| multivector mini | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | 0.9388 | 0.7841 | 0.8471 |
|
| 165 |
-
| multivector e5-small | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
|
| 166 |
-
| multivector e5-base | 0.9034 | 0.8718 | 0.8246 | 0.8432 | 0.8695 | 0.8719 | 0.9388 | 0.7922 | 0.8789 |
|
| 167 |
-
| M3 multi | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
|
| 168 |
-
| reranker mini | 0.8647 | 0.8159 | 0.7462 | 0.7946 | 0.8245 | 0.8305 | 0.9184 | 0.7359 | 0.8336 |
|
| 169 |
-
| reranker e5-small | 0.8907 | 0.8430 | 0.8279 | 0.8541 | 0.8673 | 0.8693 | **0.9592** | 0.7871 | 0.8690 |
|
| 170 |
-
| reranker e5-base | **0.9082** | 0.8727 | **0.8497** | 0.8486 | **0.8943** | 0.8725 | 0.8980 | **0.7932** | **0.8853** |
|
| 171 |
|
| 172 |
#### recall@10
|
| 173 |
|
| 174 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 175 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 176 |
-
| dense mini | 0.8285 | 0.7644 | 0.6362 | 0.6973 | 0.7334 | 0.7827 | 0.8163 | 0.6737 | 0.7820 |
|
| 177 |
-
| e5-small | 0.9072 | 0.8718 | 0.8007 | 0.8811 | 0.8774 | 0.9012 | 0.8980 | 0.7982 | 0.8844 |
|
| 178 |
-
| e5-base | 0.9209 | 0.9070 | 0.8410 | 0.8649 | 0.9224 | 0.9133 | 0.8571 | 0.8233 | 0.9042 |
|
| 179 |
-
| e5-large | 0.9273 | **0.9242** | 0.8551 | 0.8811 | **0.9359** | **0.9344** | 0.8163 | 0.8143 | 0.9131 |
|
| 180 |
-
| M3 dense | 0.7584 | 0.6931 | 0.5381 | 0.6595 | 0.6704 | 0.7208 | 0.7755 | 0.5944 | 0.7112 |
|
| 181 |
-
| SPLADE mini | 0.8293 | 0.7924 | 0.8115 | 0.8162 | 0.8425 | 0.8317 | 0.8776 | 0.7440 | 0.8192 |
|
| 182 |
-
| M3 sparse | 0.8279 | 0.8087 | 0.8137 | 0.8000 | 0.8256 | 0.8107 | 0.8776 | 0.7169 | 0.8138 |
|
| 183 |
-
| multivector mini | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
|
| 184 |
-
| multivector e5-small | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
|
| 185 |
-
| multivector e5-base | 0.9300 | 0.9206 | 0.8758 | 0.8649 | 0.9123 | 0.9152 | 0.9388 | 0.8394 | 0.9140 |
|
| 186 |
-
| M3 multi | 0.8896 | 0.8484 | 0.8529 | 0.8865 | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
|
| 187 |
-
| reranker mini | 0.8872 | 0.8547 | 0.7800 | 0.8324 | 0.8605 | 0.8662 | **0.9592** | 0.7620 | 0.8615 |
|
| 188 |
-
| reranker e5-small | 0.9206 | 0.8917 | 0.8704 | 0.8811 | 0.9168 | 0.9191 | **0.9592** | 0.8414 | 0.9072 |
|
| 189 |
-
| reranker e5-base | **0.9345** | 0.9224 | **0.8987** | **0.8919** | 0.9314 | **0.9344** | **0.9592** | **0.8454** | **0.9229** |
|
| 190 |
|
| 191 |
#### ndcg@10
|
| 192 |
|
| 193 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 194 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 195 |
-
| dense mini | 0.7129 | 0.5933 | 0.4881 | 0.5533 | 0.5709 | 0.6234 | 0.6532 | 0.5160 | 0.6464 |
|
| 196 |
-
| e5-small | 0.8086 | 0.7077 | 0.6464 | 0.7233 | 0.7270 | 0.7653 | 0.7810 | 0.6609 | 0.7635 |
|
| 197 |
-
| e5-base | 0.8264 | 0.7661 | 0.6851 | 0.7362 | 0.7691 | 0.7929 | 0.7583 | 0.6855 | 0.7897 |
|
| 198 |
-
| e5-large | 0.8294 | **0.7962** | 0.7089 | 0.7499 | **0.8061** | **0.8147** | 0.7093 | 0.6677 | 0.7997 |
|
| 199 |
-
| M3 dense | 0.6340 | 0.5278 | 0.4034 | 0.5053 | 0.5045 | 0.5699 | 0.6650 | 0.4406 | 0.5734 |
|
| 200 |
-
| SPLADE mini | 0.6816 | 0.6213 | 0.6629 | 0.6763 | 0.6768 | 0.6579 | 0.7696 | 0.5729 | 0.6633 |
|
| 201 |
-
| M3 sparse | 0.6987 | 0.6469 | 0.6776 | 0.6599 | 0.6835 | 0.6593 | 0.7754 | 0.5562 | 0.6750 |
|
| 202 |
-
| multivector mini | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
|
| 203 |
-
| multivector e5-small | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
|
| 204 |
-
| multivector e5-base | 0.8428 | 0.7853 | 0.7429 | 0.7801 | 0.7867 | 0.7989 | 0.8569 | **0.7137** | 0.8098 |
|
| 205 |
-
| M3 multi | 0.7690 | 0.6967 | 0.7113 | 0.7824 | 0.7315 | 0.7365 | 0.8879 | 0.6265 | 0.7410 |
|
| 206 |
-
| reranker mini | 0.8082 | 0.7483 | 0.6906 | 0.7416 | 0.7613 | 0.7523 | 0.8987 | 0.6671 | 0.7722 |
|
| 207 |
-
| reranker e5-small | 0.8284 | 0.7397 | 0.7435 | 0.7852 | 0.7814 | 0.7815 | **0.9037** | 0.6999 | 0.7946 |
|
| 208 |
-
| reranker e5-base | **0.8449** | 0.7681 | **0.7668** | **0.7903** | 0.7968 | 0.7947 | 0.8769 | 0.7018 | **0.8107** |
|
| 209 |
|
| 210 |
#### mrr@10
|
| 211 |
|
| 212 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 213 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 214 |
-
| dense mini | 0.6759 | 0.5387 | 0.4412 | 0.5077 | 0.5192 | 0.5729 | 0.6004 | 0.4658 | 0.6032 |
|
| 215 |
-
| e5-small | 0.7766 | 0.6548 | 0.5970 | 0.6736 | 0.6790 | 0.7214 | 0.7437 | 0.6169 | 0.7246 |
|
| 216 |
-
| e5-base | 0.7957 | 0.7201 | 0.6350 | 0.6957 | 0.7202 | 0.7534 | 0.7245 | 0.6418 | 0.7527 |
|
| 217 |
-
| e5-large | 0.7976 | **0.7544** | 0.6620 | 0.7077 | **0.7643** | **0.7756** | 0.6741 | 0.6208 | 0.7629 |
|
| 218 |
-
| M3 dense | 0.5941 | 0.4746 | 0.3609 | 0.4563 | 0.4518 | 0.5221 | 0.6291 | 0.3917 | 0.5295 |
|
| 219 |
-
| SPLADE mini | 0.6338 | 0.5656 | 0.6148 | 0.6319 | 0.6237 | 0.6027 | 0.7358 | 0.5185 | 0.6130 |
|
| 220 |
-
| M3 sparse | 0.6568 | 0.5950 | 0.6337 | 0.6157 | 0.6376 | 0.6108 | 0.7417 | 0.5047 | 0.6302 |
|
| 221 |
-
| multivector mini | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
|
| 222 |
-
| multivector e5-small | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
|
| 223 |
-
| multivector e5-base | 0.8145 | 0.7413 | 0.7000 | 0.7524 | 0.7459 | 0.7612 | 0.8296 | **0.6731** | **0.7760** |
|
| 224 |
-
| M3 multi | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | 0.8716 | 0.5789 | 0.6982 |
|
| 225 |
-
| reranker mini | 0.7824 | 0.7139 | 0.6618 | 0.7122 | 0.7296 | 0.7152 | 0.8793 | 0.6361 | 0.7431 |
|
| 226 |
-
| reranker e5-small | 0.7984 | 0.6900 | 0.7023 | 0.7539 | 0.7379 | 0.7369 | **0.8850** | 0.6541 | 0.7580 |
|
| 227 |
-
| reranker e5-base | **0.8156** | 0.7178 | **0.7240** | **0.7572** | 0.7527 | 0.7498 | 0.8509 | 0.6553 | 0.7741 |
|
| 228 |
-
|
| 229 |
-
In Multivector,
|
| 230 |
|
| 231 |
### Multivector
|
| 232 |
|
|
@@ -234,37 +234,37 @@ In Multivector, **bold** is the best multivector model in that column.
|
|
| 234 |
|
| 235 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 236 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 237 |
-
| multivector mini | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | **0.9388** | 0.7841 | 0.8471 |
|
| 238 |
-
| multivector e5-small | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
|
| 239 |
-
| multivector e5-base | **0.9034** | **0.8718** | **0.8246** | 0.8432 | **0.8695** | **0.8719** | **0.9388** | **0.7922** | **0.8789** |
|
| 240 |
-
| M3 multi | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
|
| 241 |
|
| 242 |
#### recall@10
|
| 243 |
|
| 244 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 245 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 246 |
-
| multivector mini | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
|
| 247 |
-
| multivector e5-small | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
|
| 248 |
-
| multivector e5-base | **0.9300** | **0.9206** | **0.8758** | 0.8649 | **0.9123** | **0.9152** | 0.9388 | **0.8394** | **0.9140** |
|
| 249 |
-
| M3 multi | 0.8896 | 0.8484 | 0.8529 | **0.8865** | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
|
| 250 |
|
| 251 |
#### ndcg@10
|
| 252 |
|
| 253 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 254 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 255 |
-
| multivector mini | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
|
| 256 |
-
| multivector e5-small | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
|
| 257 |
-
| multivector e5-base | **0.8428** | **0.7853** | **0.7429** | 0.7801 | **0.7867** | **0.7989** | 0.8569 | **0.7137** | **0.8098** |
|
| 258 |
-
| M3 multi | 0.7690 | 0.6967 | 0.7113 | **0.7824** | 0.7315 | 0.7365 | **0.8879** | 0.6265 | 0.7410 |
|
| 259 |
|
| 260 |
#### mrr@10
|
| 261 |
|
| 262 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 263 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 264 |
-
| multivector mini | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
|
| 265 |
-
| multivector e5-small | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
|
| 266 |
-
| multivector e5-base | **0.8145** | **0.7413** | **0.7000** | **0.7524** | **0.7459** | **0.7612** | 0.8296 | **0.6731** | **0.7760** |
|
| 267 |
-
| M3 multi | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | **0.8716** | 0.5789 | 0.6982 |
|
| 268 |
|
| 269 |
## Training Details
|
| 270 |
|
|
@@ -288,6 +288,19 @@ In Multivector, **bold** is the best multivector model in that column.
|
|
| 288 |
|
| 289 |
### BibTeX
|
| 290 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 291 |
#### CachedMultiVectorMultipleNegativesRankingLoss
|
| 292 |
```bibtex
|
| 293 |
@misc{gao2021scaling,
|
|
|
|
| 144 |
```
|
| 145 |
## Evaluation
|
| 146 |
|
| 147 |
+
Full-corpus passage retrieval on [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR) for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. `avg` is the query-weighted average across those languages. The best value in that column is bold.
|
| 148 |
|
| 149 |
Reranker rows rerank the top 50 passages from a frozen dense retriever.
|
| 150 |
|
|
|
|
| 154 |
|
| 155 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 156 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 157 |
+
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.7805 | 0.6940 | 0.5643 | 0.6270 | 0.6637 | 0.7081 | 0.7755 | 0.6024 | 0.7233 |
|
| 158 |
+
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.8741 | 0.8014 | 0.7418 | 0.8162 | 0.8088 | 0.8451 | 0.8571 | 0.7359 | 0.8379 |
|
| 159 |
+
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.8886 | 0.8691 | 0.7810 | 0.8000 | 0.8526 | 0.8738 | 0.8571 | 0.7590 | 0.8629 |
|
| 160 |
+
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.8948 | **0.8809** | 0.7974 | 0.8324 | 0.8841 | **0.8993** | 0.7755 | 0.7550 | 0.8738 |
|
| 161 |
+
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7067 | 0.6264 | 0.4630 | 0.5892 | 0.5894 | 0.6488 | 0.7551 | 0.5231 | 0.6501 |
|
| 162 |
+
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.7759 | 0.7446 | 0.7603 | 0.7405 | 0.7705 | 0.7546 | 0.8163 | 0.6657 | 0.7599 |
|
| 163 |
+
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7826 | 0.7356 | 0.7593 | 0.7189 | 0.7627 | 0.7463 | 0.8367 | 0.6536 | 0.7598 |
|
| 164 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | 0.9388 | 0.7841 | 0.8471 |
|
| 165 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
|
| 166 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.9034 | 0.8718 | 0.8246 | 0.8432 | 0.8695 | 0.8719 | 0.9388 | 0.7922 | 0.8789 |
|
| 167 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
|
| 168 |
+
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8647 | 0.8159 | 0.7462 | 0.7946 | 0.8245 | 0.8305 | 0.9184 | 0.7359 | 0.8336 |
|
| 169 |
+
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.8907 | 0.8430 | 0.8279 | 0.8541 | 0.8673 | 0.8693 | **0.9592** | 0.7871 | 0.8690 |
|
| 170 |
+
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.9082** | 0.8727 | **0.8497** | 0.8486 | **0.8943** | 0.8725 | 0.8980 | **0.7932** | **0.8853** |
|
| 171 |
|
| 172 |
#### recall@10
|
| 173 |
|
| 174 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 175 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 176 |
+
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.8285 | 0.7644 | 0.6362 | 0.6973 | 0.7334 | 0.7827 | 0.8163 | 0.6737 | 0.7820 |
|
| 177 |
+
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.9072 | 0.8718 | 0.8007 | 0.8811 | 0.8774 | 0.9012 | 0.8980 | 0.7982 | 0.8844 |
|
| 178 |
+
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.9209 | 0.9070 | 0.8410 | 0.8649 | 0.9224 | 0.9133 | 0.8571 | 0.8233 | 0.9042 |
|
| 179 |
+
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.9273 | **0.9242** | 0.8551 | 0.8811 | **0.9359** | **0.9344** | 0.8163 | 0.8143 | 0.9131 |
|
| 180 |
+
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7584 | 0.6931 | 0.5381 | 0.6595 | 0.6704 | 0.7208 | 0.7755 | 0.5944 | 0.7112 |
|
| 181 |
+
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.8293 | 0.7924 | 0.8115 | 0.8162 | 0.8425 | 0.8317 | 0.8776 | 0.7440 | 0.8192 |
|
| 182 |
+
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8279 | 0.8087 | 0.8137 | 0.8000 | 0.8256 | 0.8107 | 0.8776 | 0.7169 | 0.8138 |
|
| 183 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
|
| 184 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
|
| 185 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.9300 | 0.9206 | 0.8758 | 0.8649 | 0.9123 | 0.9152 | 0.9388 | 0.8394 | 0.9140 |
|
| 186 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8896 | 0.8484 | 0.8529 | 0.8865 | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
|
| 187 |
+
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8872 | 0.8547 | 0.7800 | 0.8324 | 0.8605 | 0.8662 | **0.9592** | 0.7620 | 0.8615 |
|
| 188 |
+
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.9206 | 0.8917 | 0.8704 | 0.8811 | 0.9168 | 0.9191 | **0.9592** | 0.8414 | 0.9072 |
|
| 189 |
+
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.9345** | 0.9224 | **0.8987** | **0.8919** | 0.9314 | **0.9344** | **0.9592** | **0.8454** | **0.9229** |
|
| 190 |
|
| 191 |
#### ndcg@10
|
| 192 |
|
| 193 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 194 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 195 |
+
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.7129 | 0.5933 | 0.4881 | 0.5533 | 0.5709 | 0.6234 | 0.6532 | 0.5160 | 0.6464 |
|
| 196 |
+
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.8086 | 0.7077 | 0.6464 | 0.7233 | 0.7270 | 0.7653 | 0.7810 | 0.6609 | 0.7635 |
|
| 197 |
+
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.8264 | 0.7661 | 0.6851 | 0.7362 | 0.7691 | 0.7929 | 0.7583 | 0.6855 | 0.7897 |
|
| 198 |
+
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.8294 | **0.7962** | 0.7089 | 0.7499 | **0.8061** | **0.8147** | 0.7093 | 0.6677 | 0.7997 |
|
| 199 |
+
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6340 | 0.5278 | 0.4034 | 0.5053 | 0.5045 | 0.5699 | 0.6650 | 0.4406 | 0.5734 |
|
| 200 |
+
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.6816 | 0.6213 | 0.6629 | 0.6763 | 0.6768 | 0.6579 | 0.7696 | 0.5729 | 0.6633 |
|
| 201 |
+
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6987 | 0.6469 | 0.6776 | 0.6599 | 0.6835 | 0.6593 | 0.7754 | 0.5562 | 0.6750 |
|
| 202 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
|
| 203 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
|
| 204 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.8428 | 0.7853 | 0.7429 | 0.7801 | 0.7867 | 0.7989 | 0.8569 | **0.7137** | 0.8098 |
|
| 205 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7690 | 0.6967 | 0.7113 | 0.7824 | 0.7315 | 0.7365 | 0.8879 | 0.6265 | 0.7410 |
|
| 206 |
+
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8082 | 0.7483 | 0.6906 | 0.7416 | 0.7613 | 0.7523 | 0.8987 | 0.6671 | 0.7722 |
|
| 207 |
+
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.8284 | 0.7397 | 0.7435 | 0.7852 | 0.7814 | 0.7815 | **0.9037** | 0.6999 | 0.7946 |
|
| 208 |
+
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.8449** | 0.7681 | **0.7668** | **0.7903** | 0.7968 | 0.7947 | 0.8769 | 0.7018 | **0.8107** |
|
| 209 |
|
| 210 |
#### mrr@10
|
| 211 |
|
| 212 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 213 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 214 |
+
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.6759 | 0.5387 | 0.4412 | 0.5077 | 0.5192 | 0.5729 | 0.6004 | 0.4658 | 0.6032 |
|
| 215 |
+
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.7766 | 0.6548 | 0.5970 | 0.6736 | 0.6790 | 0.7214 | 0.7437 | 0.6169 | 0.7246 |
|
| 216 |
+
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.7957 | 0.7201 | 0.6350 | 0.6957 | 0.7202 | 0.7534 | 0.7245 | 0.6418 | 0.7527 |
|
| 217 |
+
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.7976 | **0.7544** | 0.6620 | 0.7077 | **0.7643** | **0.7756** | 0.6741 | 0.6208 | 0.7629 |
|
| 218 |
+
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.5941 | 0.4746 | 0.3609 | 0.4563 | 0.4518 | 0.5221 | 0.6291 | 0.3917 | 0.5295 |
|
| 219 |
+
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.6338 | 0.5656 | 0.6148 | 0.6319 | 0.6237 | 0.6027 | 0.7358 | 0.5185 | 0.6130 |
|
| 220 |
+
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6568 | 0.5950 | 0.6337 | 0.6157 | 0.6376 | 0.6108 | 0.7417 | 0.5047 | 0.6302 |
|
| 221 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
|
| 222 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
|
| 223 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.8145 | 0.7413 | 0.7000 | 0.7524 | 0.7459 | 0.7612 | 0.8296 | **0.6731** | **0.7760** |
|
| 224 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | 0.8716 | 0.5789 | 0.6982 |
|
| 225 |
+
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.7824 | 0.7139 | 0.6618 | 0.7122 | 0.7296 | 0.7152 | 0.8793 | 0.6361 | 0.7431 |
|
| 226 |
+
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.7984 | 0.6900 | 0.7023 | 0.7539 | 0.7379 | 0.7369 | **0.8850** | 0.6541 | 0.7580 |
|
| 227 |
+
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.8156** | 0.7178 | **0.7240** | **0.7572** | 0.7527 | 0.7498 | 0.8509 | 0.6553 | 0.7741 |
|
| 228 |
+
|
| 229 |
+
In Multivector, bold is the best multivector model in that column.
|
| 230 |
|
| 231 |
### Multivector
|
| 232 |
|
|
|
|
| 234 |
|
| 235 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 236 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 237 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | **0.9388** | 0.7841 | 0.8471 |
|
| 238 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
|
| 239 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.9034** | **0.8718** | **0.8246** | 0.8432 | **0.8695** | **0.8719** | **0.9388** | **0.7922** | **0.8789** |
|
| 240 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
|
| 241 |
|
| 242 |
#### recall@10
|
| 243 |
|
| 244 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 245 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 246 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
|
| 247 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
|
| 248 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.9300** | **0.9206** | **0.8758** | 0.8649 | **0.9123** | **0.9152** | 0.9388 | **0.8394** | **0.9140** |
|
| 249 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8896 | 0.8484 | 0.8529 | **0.8865** | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
|
| 250 |
|
| 251 |
#### ndcg@10
|
| 252 |
|
| 253 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 254 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 255 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
|
| 256 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
|
| 257 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.8428** | **0.7853** | **0.7429** | 0.7801 | **0.7867** | **0.7989** | 0.8569 | **0.7137** | **0.8098** |
|
| 258 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7690 | 0.6967 | 0.7113 | **0.7824** | 0.7315 | 0.7365 | **0.8879** | 0.6265 | 0.7410 |
|
| 259 |
|
| 260 |
#### mrr@10
|
| 261 |
|
| 262 |
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
|
| 263 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 264 |
+
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
|
| 265 |
+
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
|
| 266 |
+
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.8145** | **0.7413** | **0.7000** | **0.7524** | **0.7459** | **0.7612** | 0.8296 | **0.6731** | **0.7760** |
|
| 267 |
+
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | **0.8716** | 0.5789 | 0.6982 |
|
| 268 |
|
| 269 |
## Training Details
|
| 270 |
|
|
|
|
| 288 |
|
| 289 |
### BibTeX
|
| 290 |
|
| 291 |
+
#### Sentence Transformers
|
| 292 |
+
```bibtex
|
| 293 |
+
@inproceedings{reimers-2019-sentence-bert,
|
| 294 |
+
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
|
| 295 |
+
author = "Reimers, Nils and Gurevych, Iryna",
|
| 296 |
+
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
|
| 297 |
+
month = "11",
|
| 298 |
+
year = "2019",
|
| 299 |
+
publisher = "Association for Computational Linguistics",
|
| 300 |
+
url = "https://arxiv.org/abs/1908.10084",
|
| 301 |
+
}
|
| 302 |
+
```
|
| 303 |
+
|
| 304 |
#### CachedMultiVectorMultipleNegativesRankingLoss
|
| 305 |
```bibtex
|
| 306 |
@misc{gao2021scaling,
|