AfriXIR dense retriever

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large on the AfriXIR dataset. It maps inputs to a 1024-dimensional dense vector space and can be used for passage retrieval.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: intfloat/multilingual-e5-large
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 1024 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text
  • Training Dataset:
  • Languages: am, ha, om, sn, so, sw, yo, zu
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("nebhailema/afrixir-e5-large")
# Run inference
queries = [
    'query: የኪርጊዝ ሪፐብሊክ እግር ኳስ ፌዴሬሽን',
]
documents = [
    'passage: የኪርጊዝ ሪፐብሊክ እግር ኳስ ፌዴሬሽን የኪርጊዝስታን እግር ኳስ አስተዳዳሪ አካል ነው። ፌዴሬሽኑ የኪርጊዝስታን ብሔራዊ እግር ኳስ ቡድንንና የኪርጊዝስታን ብሔራዊ ፉትሳል ቡድንን ይቆጣጠራል። በተጨማሪም የተለያዩ ብሔራዊ ውድድሮችን ያዘጋጃል።\n\nየውጭ መያያዣዎች\n  ይፋዊ ድረ ገጽ\n  የፊፋ ድረ ገጽ ስለ ኪርጊዝስታን \n\nኪ\nኪርጊዝስታን',
    'passage: ኮንጎ ዴምክራሲያዊት ሪፖብሊክ ምሥራቃዊ አካባቢዎች ውስጥ እሥላማዊ ታጣቂዎች ትናንት ፈፀሙት በተባለ ጥቃት 46 ሲቪሎች መገደላቸውን የክፍለሃገሩ የደህንነት ኃላፊ አድጆ ጊዲ ለሮይተርስ ገልፀዋል።የተባበሩት ዴሞክራሲያዊ ኃይሎች የሚባል ቡድን ከጥቃቱ ጀርባ እንዳለ በመጠቆም የከሰሱት ጊዲ ኢሩሙ በሚባለው አካባቢ ወደምትገኘው ቀበሌ ወታደሮች መላካቸውንና ምርመራ መጀመሩን አመልክተዋል። ምሥራቃዊዎቹ የኮንጎ አካባቢዎች ላይ ባለፉት ሁለት ዓመታት ውስጥ በዚሁ ታጣቂ ቡድን በተጣሉ ጥቂቶች ከአንድ ሺህ በላይ ሲቪሎች መገደላቸውን የተባበሩት መንግሥታት ድርጅት መረጃ ይጠቁማል።በዩሃንዳ፣ በሩዋንዳና በቡሩንዲ አዋሳኝ አካባቢዎች ላይ ከመቶ በላይ ታጣቂ ቡድኖች በየጫካው እንደሚገኙ ተዘግቧል። ቀደም ሲል ለተጣሉት ጥቃቶች እሥላማዊ መንግሥት ቡድን ኃላፊነት ቢወስድም የመንግሥታቱ ድርጅት ግን በታጣቂዎቹና በእሥላማዊ መንግሥት ቡድን መካከል ግንኙነት ሰለመኖሩ ማረጋገጫ አለማግኘቱን ይናገራል።',
    'passage: ስላቫዊ ቋንቋዎች ወይም ስላቪክ ቋንቋዎች አንድ የሕንዳዊ-አውሮፓዊ ቋንቋዎች ቤተሠብ ቅርንጫፍ ናቸው። ከባልታዊ ቋንቋዎች ጋራ አንድላይ የባልቶ-ስላቫዊ ቋንቋዎች ቅርንጫፍ ይሠራሉ።\n\nስላቫዊ ቋንቋዎች ሁሉ ከቅድመ-ስላቭኛ ደረሱ፤ እሱም ከቅድመ-ባልቶ-ስላቭኛ እንደ ደረሰ ይታስባል።\n\nየስላቫዊ ቅርንጫፍ ሦስት ክፍሎች ምሥራቅ ስላቫዊ፣ ምዕራብ ስላቫዊ እና ደቡብ ስላቫዊ ናቸው።\n\nምሥራቅ ስላቫዊ\nሩስኛ\nሩጤንኛ *\nቤላሩስኛ\nዩክሬንኛ\n\nምዕራብ ስላቫዊ\nፖልኛ\nካሹብኛ\nሳይሌሲያን\nቸክኛ\nስሎቫክኛ\nሶርብኛ *\nላይኛ ሶርብኛ\nየታችኛው ሶርብኛ\n\nደቡብ ስላቫዊ\nቡልጋርኛ\nሰርብኛ\nክሮኤሽኛ\nስሎቬንኛ\nመቄዶንኛ\n\nሰው ሠራሽ ቋንቋዎች\nቸርች ስላቪክ\nኢንተርስላቪክ',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.8423, 0.0792, 0.0971]])

Evaluation

Full-corpus passage retrieval on AfriXIR for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. avg is the query-weighted average across those languages. The best value in that column is bold.

Reranker rows rerank the top 50 passages from a frozen dense retriever.

All models

recall@5

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.7805 0.6940 0.5643 0.6270 0.6637 0.7081 0.7755 0.6024 0.7233
e5-small 0.8741 0.8014 0.7418 0.8162 0.8088 0.8451 0.8571 0.7359 0.8379
e5-base 0.8886 0.8691 0.7810 0.8000 0.8526 0.8738 0.8571 0.7590 0.8629
e5-large 0.8948 0.8809 0.7974 0.8324 0.8841 0.8993 0.7755 0.7550 0.8738
M3 dense 0.7067 0.6264 0.4630 0.5892 0.5894 0.6488 0.7551 0.5231 0.6501
SPLADE mini 0.7759 0.7446 0.7603 0.7405 0.7705 0.7546 0.8163 0.6657 0.7599
M3 sparse 0.7826 0.7356 0.7593 0.7189 0.7627 0.7463 0.8367 0.6536 0.7598
multivector mini 0.8710 0.8177 0.8126 0.8162 0.8358 0.8311 0.9388 0.7841 0.8471
multivector e5-small 0.8868 0.8204 0.7832 0.8108 0.8515 0.8662 0.8980 0.7691 0.8578
multivector e5-base 0.9034 0.8718 0.8246 0.8432 0.8695 0.8719 0.9388 0.7922 0.8789
M3 multi 0.8485 0.7960 0.7919 0.8649 0.8211 0.8260 0.9184 0.7108 0.8246
reranker mini 0.8647 0.8159 0.7462 0.7946 0.8245 0.8305 0.9184 0.7359 0.8336
reranker e5-small 0.8907 0.8430 0.8279 0.8541 0.8673 0.8693 0.9592 0.7871 0.8690
reranker e5-base 0.9082 0.8727 0.8497 0.8486 0.8943 0.8725 0.8980 0.7932 0.8853

recall@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.8285 0.7644 0.6362 0.6973 0.7334 0.7827 0.8163 0.6737 0.7820
e5-small 0.9072 0.8718 0.8007 0.8811 0.8774 0.9012 0.8980 0.7982 0.8844
e5-base 0.9209 0.9070 0.8410 0.8649 0.9224 0.9133 0.8571 0.8233 0.9042
e5-large 0.9273 0.9242 0.8551 0.8811 0.9359 0.9344 0.8163 0.8143 0.9131
M3 dense 0.7584 0.6931 0.5381 0.6595 0.6704 0.7208 0.7755 0.5944 0.7112
SPLADE mini 0.8293 0.7924 0.8115 0.8162 0.8425 0.8317 0.8776 0.7440 0.8192
M3 sparse 0.8279 0.8087 0.8137 0.8000 0.8256 0.8107 0.8776 0.7169 0.8138
multivector mini 0.9008 0.8809 0.8725 0.8541 0.8898 0.8802 0.9388 0.8323 0.8877
multivector e5-small 0.9183 0.8755 0.8573 0.8757 0.9033 0.9146 0.9592 0.8102 0.8995
multivector e5-base 0.9300 0.9206 0.8758 0.8649 0.9123 0.9152 0.9388 0.8394 0.9140
M3 multi 0.8896 0.8484 0.8529 0.8865 0.8763 0.8878 0.9388 0.7751 0.8732
reranker mini 0.8872 0.8547 0.7800 0.8324 0.8605 0.8662 0.9592 0.7620 0.8615
reranker e5-small 0.9206 0.8917 0.8704 0.8811 0.9168 0.9191 0.9592 0.8414 0.9072
reranker e5-base 0.9345 0.9224 0.8987 0.8919 0.9314 0.9344 0.9592 0.8454 0.9229

ndcg@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.7129 0.5933 0.4881 0.5533 0.5709 0.6234 0.6532 0.5160 0.6464
e5-small 0.8086 0.7077 0.6464 0.7233 0.7270 0.7653 0.7810 0.6609 0.7635
e5-base 0.8264 0.7661 0.6851 0.7362 0.7691 0.7929 0.7583 0.6855 0.7897
e5-large 0.8294 0.7962 0.7089 0.7499 0.8061 0.8147 0.7093 0.6677 0.7997
M3 dense 0.6340 0.5278 0.4034 0.5053 0.5045 0.5699 0.6650 0.4406 0.5734
SPLADE mini 0.6816 0.6213 0.6629 0.6763 0.6768 0.6579 0.7696 0.5729 0.6633
M3 sparse 0.6987 0.6469 0.6776 0.6599 0.6835 0.6593 0.7754 0.5562 0.6750
multivector mini 0.8028 0.7163 0.7364 0.7599 0.7551 0.7352 0.8785 0.6876 0.7690
multivector e5-small 0.8272 0.7280 0.7167 0.7688 0.7617 0.7801 0.8876 0.6858 0.7880
multivector e5-base 0.8428 0.7853 0.7429 0.7801 0.7867 0.7989 0.8569 0.7137 0.8098
M3 multi 0.7690 0.6967 0.7113 0.7824 0.7315 0.7365 0.8879 0.6265 0.7410
reranker mini 0.8082 0.7483 0.6906 0.7416 0.7613 0.7523 0.8987 0.6671 0.7722
reranker e5-small 0.8284 0.7397 0.7435 0.7852 0.7814 0.7815 0.9037 0.6999 0.7946
reranker e5-base 0.8449 0.7681 0.7668 0.7903 0.7968 0.7947 0.8769 0.7018 0.8107

mrr@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.6759 0.5387 0.4412 0.5077 0.5192 0.5729 0.6004 0.4658 0.6032
e5-small 0.7766 0.6548 0.5970 0.6736 0.6790 0.7214 0.7437 0.6169 0.7246
e5-base 0.7957 0.7201 0.6350 0.6957 0.7202 0.7534 0.7245 0.6418 0.7527
e5-large 0.7976 0.7544 0.6620 0.7077 0.7643 0.7756 0.6741 0.6208 0.7629
M3 dense 0.5941 0.4746 0.3609 0.4563 0.4518 0.5221 0.6291 0.3917 0.5295
SPLADE mini 0.6338 0.5656 0.6148 0.6319 0.6237 0.6027 0.7358 0.5185 0.6130
M3 sparse 0.6568 0.5950 0.6337 0.6157 0.6376 0.6108 0.7417 0.5047 0.6302
multivector mini 0.7709 0.6631 0.6930 0.7291 0.7118 0.6884 0.8571 0.6407 0.7305
multivector e5-small 0.7977 0.6803 0.6723 0.7352 0.7160 0.7366 0.8652 0.6452 0.7520
multivector e5-base 0.8145 0.7413 0.7000 0.7524 0.7459 0.7612 0.8296 0.6731 0.7760
M3 multi 0.7296 0.6476 0.6660 0.7481 0.6850 0.6880 0.8716 0.5789 0.6982
reranker mini 0.7824 0.7139 0.6618 0.7122 0.7296 0.7152 0.8793 0.6361 0.7431
reranker e5-small 0.7984 0.6900 0.7023 0.7539 0.7379 0.7369 0.8850 0.6541 0.7580
reranker e5-base 0.8156 0.7178 0.7240 0.7572 0.7527 0.7498 0.8509 0.6553 0.7741

In Dense, bold is the best dense model in that column.

Dense

recall@5

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.7805 0.6940 0.5643 0.6270 0.6637 0.7081 0.7755 0.6024 0.7233
e5-small 0.8741 0.8014 0.7418 0.8162 0.8088 0.8451 0.8571 0.7359 0.8379
e5-base 0.8886 0.8691 0.7810 0.8000 0.8526 0.8738 0.8571 0.7590 0.8629
e5-large 0.8948 0.8809 0.7974 0.8324 0.8841 0.8993 0.7755 0.7550 0.8738
M3 dense 0.7067 0.6264 0.4630 0.5892 0.5894 0.6488 0.7551 0.5231 0.6501

recall@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.8285 0.7644 0.6362 0.6973 0.7334 0.7827 0.8163 0.6737 0.7820
e5-small 0.9072 0.8718 0.8007 0.8811 0.8774 0.9012 0.8980 0.7982 0.8844
e5-base 0.9209 0.9070 0.8410 0.8649 0.9224 0.9133 0.8571 0.8233 0.9042
e5-large 0.9273 0.9242 0.8551 0.8811 0.9359 0.9344 0.8163 0.8143 0.9131
M3 dense 0.7584 0.6931 0.5381 0.6595 0.6704 0.7208 0.7755 0.5944 0.7112

ndcg@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.7129 0.5933 0.4881 0.5533 0.5709 0.6234 0.6532 0.5160 0.6464
e5-small 0.8086 0.7077 0.6464 0.7233 0.7270 0.7653 0.7810 0.6609 0.7635
e5-base 0.8264 0.7661 0.6851 0.7362 0.7691 0.7929 0.7583 0.6855 0.7897
e5-large 0.8294 0.7962 0.7089 0.7499 0.8061 0.8147 0.7093 0.6677 0.7997
M3 dense 0.6340 0.5278 0.4034 0.5053 0.5045 0.5699 0.6650 0.4406 0.5734

mrr@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.6759 0.5387 0.4412 0.5077 0.5192 0.5729 0.6004 0.4658 0.6032
e5-small 0.7766 0.6548 0.5970 0.6736 0.6790 0.7214 0.7437 0.6169 0.7246
e5-base 0.7957 0.7201 0.6350 0.6957 0.7202 0.7534 0.7245 0.6418 0.7527
e5-large 0.7976 0.7544 0.6620 0.7077 0.7643 0.7756 0.6741 0.6208 0.7629
M3 dense 0.5941 0.4746 0.3609 0.4563 0.4518 0.5221 0.6291 0.3917 0.5295

Training Details

Training Dataset

AfriXIR

  • Dataset: AfriXIR
  • Size: 112,864 training samples
  • Columns: anchor, positive, negative_0, negative_1, negative_2, and negative_3

Evaluation Dataset

AfriXIR

  • Dataset: AfriXIR
  • Size: 12,546 evaluation samples
  • Columns: anchor, positive, negative_0, negative_1, negative_2, and negative_3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
3
Safetensors
Model size
0.6B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nebhailema/afrixir-e5-large

Finetuned
(204)
this model

Dataset used to train nebhailema/afrixir-e5-large

Collections including nebhailema/afrixir-e5-large

Papers for nebhailema/afrixir-e5-large