|
|
--- |
|
|
tags: |
|
|
- sentence-transformers |
|
|
- sentence-similarity |
|
|
- feature-extraction |
|
|
- mteb |
|
|
base_model: sbintuitions/modernbert-ja-130m |
|
|
language: |
|
|
- ja |
|
|
- en |
|
|
model-index: |
|
|
- name: retrieva-jp/amber-base |
|
|
results: |
|
|
- dataset: |
|
|
config: en |
|
|
name: MTEB AmazonCounterfactualClassification (en) |
|
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
|
split: test |
|
|
type: mteb/amazon_counterfactual |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 68.1642 |
|
|
- type: f1 |
|
|
value: 61.9811 |
|
|
- type: f1_weighted |
|
|
value: 71.2157 |
|
|
- type: ap |
|
|
value: 30.6541 |
|
|
- type: ap_weighted |
|
|
value: 30.6541 |
|
|
- type: main_score |
|
|
value: 68.1642 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB ArXivHierarchicalClusteringP2P (default) |
|
|
revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 |
|
|
split: test |
|
|
type: mteb/arxiv-clustering-p2p |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 55.655100000000004 |
|
|
- type: v_measure_std |
|
|
value: 3.2918999999999996 |
|
|
- type: main_score |
|
|
value: 55.655100000000004 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB ArXivHierarchicalClusteringS2S (default) |
|
|
revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 |
|
|
split: test |
|
|
type: mteb/arxiv-clustering-s2s |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 53.6493 |
|
|
- type: v_measure_std |
|
|
value: 3.2359 |
|
|
- type: main_score |
|
|
value: 53.6493 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB ArguAna (default) |
|
|
revision: c22ab2a51041ffd869aaddef7af8d8215647e41a |
|
|
split: test |
|
|
type: mteb/arguana |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 25.249 |
|
|
- type: ndcg_at_3 |
|
|
value: 38.056 |
|
|
- type: ndcg_at_5 |
|
|
value: 43.124 |
|
|
- type: ndcg_at_10 |
|
|
value: 48.068 |
|
|
- type: ndcg_at_20 |
|
|
value: 51.461 |
|
|
- type: ndcg_at_100 |
|
|
value: 53.15800000000001 |
|
|
- type: ndcg_at_1000 |
|
|
value: 53.38 |
|
|
- type: map_at_1 |
|
|
value: 25.249 |
|
|
- type: map_at_3 |
|
|
value: 34.803 |
|
|
- type: map_at_5 |
|
|
value: 37.598 |
|
|
- type: map_at_10 |
|
|
value: 39.611000000000004 |
|
|
- type: map_at_20 |
|
|
value: 40.569 |
|
|
- type: map_at_100 |
|
|
value: 40.821000000000005 |
|
|
- type: map_at_1000 |
|
|
value: 40.83 |
|
|
- type: recall_at_1 |
|
|
value: 25.249 |
|
|
- type: recall_at_3 |
|
|
value: 47.510999999999996 |
|
|
- type: recall_at_5 |
|
|
value: 59.885999999999996 |
|
|
- type: recall_at_10 |
|
|
value: 75.32 |
|
|
- type: recall_at_20 |
|
|
value: 88.549 |
|
|
- type: recall_at_100 |
|
|
value: 97.44 |
|
|
- type: recall_at_1000 |
|
|
value: 99.14699999999999 |
|
|
- type: precision_at_1 |
|
|
value: 25.249 |
|
|
- type: precision_at_3 |
|
|
value: 15.837000000000002 |
|
|
- type: precision_at_5 |
|
|
value: 11.977 |
|
|
- type: precision_at_10 |
|
|
value: 7.532 |
|
|
- type: precision_at_20 |
|
|
value: 4.427 |
|
|
- type: precision_at_100 |
|
|
value: 0.9740000000000001 |
|
|
- type: precision_at_1000 |
|
|
value: 0.099 |
|
|
- type: mrr_at_1 |
|
|
value: 25.817899999999998 |
|
|
- type: mrr_at_3 |
|
|
value: 34.9692 |
|
|
- type: mrr_at_5 |
|
|
value: 37.7928 |
|
|
- type: mrr_at_10 |
|
|
value: 39.8238 |
|
|
- type: mrr_at_20 |
|
|
value: 40.7844 |
|
|
- type: mrr_at_100 |
|
|
value: 41.0403 |
|
|
- type: mrr_at_1000 |
|
|
value: 41.0495 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: -2.6569 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: -2.4726000000000004 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 10.259699999999999 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: -0.8151 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: -3.3642 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 7.884099999999999 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: -0.3906 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: -2.4619 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 7.558 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 1.0935000000000001 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: -1.8624999999999998 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 8.0503 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 1.3164 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: -1.3407 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 7.8992 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 0.8316 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: -0.8725 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 8.5633 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 0.44999999999999996 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: -1.4357 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 8.4438 |
|
|
- type: nauc_map_at_1_max |
|
|
value: -2.6569 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -2.4726000000000004 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 10.259699999999999 |
|
|
- type: nauc_map_at_3_max |
|
|
value: -1.3567 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -3.222 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 8.3557 |
|
|
- type: nauc_map_at_5_max |
|
|
value: -1.162 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -2.7384 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 8.118400000000001 |
|
|
- type: nauc_map_at_10_max |
|
|
value: -0.615 |
|
|
- type: nauc_map_at_10_std |
|
|
value: -2.5394 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 8.283100000000001 |
|
|
- type: nauc_map_at_20_max |
|
|
value: -0.5492 |
|
|
- type: nauc_map_at_20_std |
|
|
value: -2.4076 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 8.280999999999999 |
|
|
- type: nauc_map_at_100_max |
|
|
value: -0.6049 |
|
|
- type: nauc_map_at_100_std |
|
|
value: -2.3560000000000003 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 8.3933 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: -0.6154 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: -2.373 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 8.3902 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: -2.6569 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -2.4726000000000004 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 10.259699999999999 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 0.7234 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -3.7315 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 6.6138 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 2.0847 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: -1.4385000000000001 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 5.9428 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 9.2417 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 1.6372000000000002 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 7.6442 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 17.9819 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 9.3827 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 5.2288 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 46.3576 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 69.5314 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 25.2365 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 47.3173 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 80.3564 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 30.506 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: -2.6569 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: -2.4726000000000004 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 10.259699999999999 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 0.7234 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: -3.7315 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 6.6138 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 2.0847 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: -1.4385000000000001 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 5.9428 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 9.2417 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 1.6372000000000002 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 7.6442 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 17.9819 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 9.3827 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 5.2288 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 46.3576 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 69.5314 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: 25.2365 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 47.3173 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 80.3564 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: 30.506 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: -2.5852 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: -2.7133000000000003 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 8.3902 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: -2.3878 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: -3.1916 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 6.3759999999999994 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: -2.0079 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: -2.9791000000000003 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 6.3531 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: -1.41 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: -2.7921 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 6.514200000000001 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: -1.35 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: -2.6331 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 6.4700999999999995 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: -1.393 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: -2.5819 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 6.5875 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: -1.4037000000000002 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: -2.5989 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 6.583799999999999 |
|
|
- type: main_score |
|
|
value: 48.068 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB AskUbuntuDupQuestions (default) |
|
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
|
split: test |
|
|
type: mteb/askubuntudupquestions-reranking |
|
|
metrics: |
|
|
- type: map |
|
|
value: 56.5225 |
|
|
- type: mrr |
|
|
value: 70.5146 |
|
|
- type: nAUC_map_max |
|
|
value: 18.224 |
|
|
- type: nAUC_map_std |
|
|
value: 12.5352 |
|
|
- type: nAUC_map_diff1 |
|
|
value: 14.0464 |
|
|
- type: nAUC_mrr_max |
|
|
value: 28.619699999999998 |
|
|
- type: nAUC_mrr_std |
|
|
value: 21.69 |
|
|
- type: nAUC_mrr_diff1 |
|
|
value: 15.8021 |
|
|
- type: main_score |
|
|
value: 56.5225 |
|
|
task: |
|
|
type: Reranking |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB BIOSSES (default) |
|
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
|
split: test |
|
|
type: mteb/biosses-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 86.6855 |
|
|
- type: spearman |
|
|
value: 83.17360000000001 |
|
|
- type: cosine_pearson |
|
|
value: 86.6855 |
|
|
- type: cosine_spearman |
|
|
value: 83.17360000000001 |
|
|
- type: manhattan_pearson |
|
|
value: 85.5442 |
|
|
- type: manhattan_spearman |
|
|
value: 83.9501 |
|
|
- type: euclidean_pearson |
|
|
value: 85.0403 |
|
|
- type: euclidean_spearman |
|
|
value: 83.17360000000001 |
|
|
- type: main_score |
|
|
value: 83.17360000000001 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB Banking77Classification (default) |
|
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
|
split: test |
|
|
type: mteb/banking77 |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 76.3312 |
|
|
- type: f1 |
|
|
value: 75.4609 |
|
|
- type: f1_weighted |
|
|
value: 75.4609 |
|
|
- type: main_score |
|
|
value: 76.3312 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB BiorxivClusteringP2P.v2 (default) |
|
|
revision: f5dbc242e11dd8e24def4c4268607a49e02946dc |
|
|
split: test |
|
|
type: mteb/biorxiv-clustering-p2p |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 33.6692 |
|
|
- type: v_measure_std |
|
|
value: 0.769 |
|
|
- type: main_score |
|
|
value: 33.6692 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB CQADupstackGamingRetrieval (default) |
|
|
revision: 4885aa143210c98657558c04aaf3dc47cfb54340 |
|
|
split: test |
|
|
type: mteb/cqadupstack-gaming |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 30.345 |
|
|
- type: ndcg_at_3 |
|
|
value: 37.726 |
|
|
- type: ndcg_at_5 |
|
|
value: 39.999 |
|
|
- type: ndcg_at_10 |
|
|
value: 42.732 |
|
|
- type: ndcg_at_20 |
|
|
value: 44.696000000000005 |
|
|
- type: ndcg_at_100 |
|
|
value: 47.461 |
|
|
- type: ndcg_at_1000 |
|
|
value: 49.341 |
|
|
- type: map_at_1 |
|
|
value: 26.484999999999996 |
|
|
- type: map_at_3 |
|
|
value: 34.474 |
|
|
- type: map_at_5 |
|
|
value: 35.94 |
|
|
- type: map_at_10 |
|
|
value: 37.24 |
|
|
- type: map_at_20 |
|
|
value: 37.852999999999994 |
|
|
- type: map_at_100 |
|
|
value: 38.286 |
|
|
- type: map_at_1000 |
|
|
value: 38.369 |
|
|
- type: recall_at_1 |
|
|
value: 26.484999999999996 |
|
|
- type: recall_at_3 |
|
|
value: 42.857 |
|
|
- type: recall_at_5 |
|
|
value: 48.501 |
|
|
- type: recall_at_10 |
|
|
value: 56.48 |
|
|
- type: recall_at_20 |
|
|
value: 63.81099999999999 |
|
|
- type: recall_at_100 |
|
|
value: 77.518 |
|
|
- type: recall_at_1000 |
|
|
value: 90.89 |
|
|
- type: precision_at_1 |
|
|
value: 30.345 |
|
|
- type: precision_at_3 |
|
|
value: 17.241 |
|
|
- type: precision_at_5 |
|
|
value: 11.962 |
|
|
- type: precision_at_10 |
|
|
value: 7.204000000000001 |
|
|
- type: precision_at_20 |
|
|
value: 4.1290000000000004 |
|
|
- type: precision_at_100 |
|
|
value: 1.0330000000000001 |
|
|
- type: precision_at_1000 |
|
|
value: 0.127 |
|
|
- type: mrr_at_1 |
|
|
value: 30.3448 |
|
|
- type: mrr_at_3 |
|
|
value: 37.5131 |
|
|
- type: mrr_at_5 |
|
|
value: 38.8516 |
|
|
- type: mrr_at_10 |
|
|
value: 39.915299999999995 |
|
|
- type: mrr_at_20 |
|
|
value: 40.428599999999996 |
|
|
- type: mrr_at_100 |
|
|
value: 40.7757 |
|
|
- type: mrr_at_1000 |
|
|
value: 40.8275 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 30.5442 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: -10.3888 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 52.476 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 28.6927 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: -8.8728 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 45.094699999999996 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 29.259600000000002 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: -7.945399999999999 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 44.600699999999996 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 29.9977 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: -6.1746 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 44.2832 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 30.034100000000002 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: -4.8941 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 43.3814 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 30.812800000000003 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: -3.5000999999999998 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 43.345 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 30.9884 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: -3.9316999999999998 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 43.6512 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 27.442800000000002 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -9.8884 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 52.666999999999994 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 27.897100000000002 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -9.777 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 47.013 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 28.3476 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -9.3335 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 46.7246 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 28.921000000000003 |
|
|
- type: nauc_map_at_10_std |
|
|
value: -8.4018 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 46.5358 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 29.033900000000003 |
|
|
- type: nauc_map_at_20_std |
|
|
value: -7.985100000000001 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 46.2362 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 29.2382 |
|
|
- type: nauc_map_at_100_std |
|
|
value: -7.7172 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 46.2663 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 29.263699999999996 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: -7.7108 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 46.2735 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 27.442800000000002 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -9.8884 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 52.666999999999994 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 25.7102 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -8.2064 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 39.145 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 27.244699999999998 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: -5.943 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 38.024 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 29.226000000000003 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: -0.2402 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 36.58 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 29.567500000000003 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 6.2502 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 32.092999999999996 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 33.8086 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 20.092 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 27.5754 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 38.0782 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 34.3309 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 17.712 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 30.5442 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: -10.3888 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 52.476 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 29.0858 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: -5.8233 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 33.480900000000005 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 30.425200000000004 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: -2.0077000000000003 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 29.5631 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 30.8693 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 4.5986 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 23.346600000000002 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 29.6844 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 9.4699 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 15.9193 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 29.7036 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 19.0186 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: 5.9221 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 24.6994 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 18.0033 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -3.2275 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 30.5442 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: -10.3888 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 52.476 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 29.7504 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: -9.5234 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 46.5068 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 30.341099999999997 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: -8.4966 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 46.051199999999994 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 30.6066 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: -7.8854 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 46.035199999999996 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 30.570199999999996 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: -7.614700000000001 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 45.8861 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 30.589100000000002 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: -7.5529 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 45.907 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 30.587799999999998 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: -7.5716 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 45.9244 |
|
|
- type: main_score |
|
|
value: 42.732 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB CQADupstackUnixRetrieval (default) |
|
|
revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 |
|
|
split: test |
|
|
type: mteb/cqadupstack-unix |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 18.843 |
|
|
- type: ndcg_at_3 |
|
|
value: 22.131 |
|
|
- type: ndcg_at_5 |
|
|
value: 23.772 |
|
|
- type: ndcg_at_10 |
|
|
value: 25.661 |
|
|
- type: ndcg_at_20 |
|
|
value: 27.939999999999998 |
|
|
- type: ndcg_at_100 |
|
|
value: 31.645 |
|
|
- type: ndcg_at_1000 |
|
|
value: 34.687 |
|
|
- type: map_at_1 |
|
|
value: 16.194 |
|
|
- type: map_at_3 |
|
|
value: 20.068 |
|
|
- type: map_at_5 |
|
|
value: 21.075 |
|
|
- type: map_at_10 |
|
|
value: 21.913 |
|
|
- type: map_at_20 |
|
|
value: 22.569 |
|
|
- type: map_at_100 |
|
|
value: 23.107 |
|
|
- type: map_at_1000 |
|
|
value: 23.23 |
|
|
- type: recall_at_1 |
|
|
value: 16.194 |
|
|
- type: recall_at_3 |
|
|
value: 24.704 |
|
|
- type: recall_at_5 |
|
|
value: 28.859 |
|
|
- type: recall_at_10 |
|
|
value: 34.402 |
|
|
- type: recall_at_20 |
|
|
value: 42.714 |
|
|
- type: recall_at_100 |
|
|
value: 61.19799999999999 |
|
|
- type: recall_at_1000 |
|
|
value: 82.953 |
|
|
- type: precision_at_1 |
|
|
value: 18.843 |
|
|
- type: precision_at_3 |
|
|
value: 9.919 |
|
|
- type: precision_at_5 |
|
|
value: 7.071 |
|
|
- type: precision_at_10 |
|
|
value: 4.328 |
|
|
- type: precision_at_20 |
|
|
value: 2.752 |
|
|
- type: precision_at_100 |
|
|
value: 0.823 |
|
|
- type: precision_at_1000 |
|
|
value: 0.121 |
|
|
- type: mrr_at_1 |
|
|
value: 18.8433 |
|
|
- type: mrr_at_3 |
|
|
value: 22.776699999999998 |
|
|
- type: mrr_at_5 |
|
|
value: 23.9055 |
|
|
- type: mrr_at_10 |
|
|
value: 24.7244 |
|
|
- type: mrr_at_20 |
|
|
value: 25.3919 |
|
|
- type: mrr_at_100 |
|
|
value: 25.8783 |
|
|
- type: mrr_at_1000 |
|
|
value: 25.957900000000002 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 35.1013 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: 4.116899999999999 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 54.3984 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 35.1035 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: 5.3618 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 47.4455 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 34.3845 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: 5.4364 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 44.8757 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 33.4252 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: 7.100099999999999 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 43.0854 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 33.2135 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 7.750500000000001 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 42.5065 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 34.0845 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 9.0937 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 40.9634 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 34.3716 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 9.8049 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 41.606 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 35.054 |
|
|
- type: nauc_map_at_1_std |
|
|
value: 3.4526000000000003 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 55.69840000000001 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 34.861 |
|
|
- type: nauc_map_at_3_std |
|
|
value: 4.6036 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 49.338 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 34.3213 |
|
|
- type: nauc_map_at_5_std |
|
|
value: 4.7856000000000005 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 47.856 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 33.9813 |
|
|
- type: nauc_map_at_10_std |
|
|
value: 5.649 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 47.0563 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 33.8854 |
|
|
- type: nauc_map_at_20_std |
|
|
value: 5.9026000000000005 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 46.876200000000004 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 33.996500000000005 |
|
|
- type: nauc_map_at_100_std |
|
|
value: 6.094200000000001 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 46.6388 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 34.0082 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 6.1436 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 46.643 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 35.054 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: 3.4526000000000003 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 55.69840000000001 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 34.2271 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: 5.573 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 42.0593 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 32.7785 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: 6.188599999999999 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 36.9345 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 29.7004 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 10.3771 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 31.6352 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 28.474100000000004 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 12.3244 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 29.6458 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 31.2612 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 19.1574 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 19.7616 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 33.2982 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 36.4068 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 15.3188 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 35.1013 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: 4.116899999999999 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 54.3984 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 34.4651 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: 7.8735 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 39.7844 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 32.2792 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: 8.465 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 34.130700000000004 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 28.197699999999998 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 12.1518 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 28.672900000000002 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 27.2073 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 14.113100000000001 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 23.623 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 22.906399999999998 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 16.7201 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: 7.0853 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 10.5167 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 11.5017 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -6.6079 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 35.1013 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: 4.116899999999999 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 54.3984 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 35.489399999999996 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: 5.097700000000001 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 48.8783 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 35.2093 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: 5.2317 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 47.3602 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 34.731 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: 5.7762 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 46.495999999999995 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 34.6509 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: 5.8511 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 46.386500000000005 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 34.7761 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: 6.0355 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 46.2476 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 34.792699999999996 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: 6.0607 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 46.281800000000004 |
|
|
- type: main_score |
|
|
value: 25.661 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB ClimateFEVERHardNegatives (default) |
|
|
revision: 3a309e201f3c2c4b13bd4a367a8f37eee2ec1d21 |
|
|
split: test |
|
|
type: mteb/ClimateFEVER_test_top_250_only_w_correct-v2 |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 16.8 |
|
|
- type: ndcg_at_3 |
|
|
value: 15.503 |
|
|
- type: ndcg_at_5 |
|
|
value: 17.5 |
|
|
- type: ndcg_at_10 |
|
|
value: 20.642 |
|
|
- type: ndcg_at_20 |
|
|
value: 23.07 |
|
|
- type: ndcg_at_100 |
|
|
value: 27.639000000000003 |
|
|
- type: ndcg_at_1000 |
|
|
value: 32.041 |
|
|
- type: map_at_1 |
|
|
value: 7.885000000000001 |
|
|
- type: map_at_3 |
|
|
value: 11.128 |
|
|
- type: map_at_5 |
|
|
value: 12.565999999999999 |
|
|
- type: map_at_10 |
|
|
value: 13.876 |
|
|
- type: map_at_20 |
|
|
value: 14.66 |
|
|
- type: map_at_100 |
|
|
value: 15.432000000000002 |
|
|
- type: map_at_1000 |
|
|
value: 15.655 |
|
|
- type: recall_at_1 |
|
|
value: 7.885000000000001 |
|
|
- type: recall_at_3 |
|
|
value: 14.957 |
|
|
- type: recall_at_5 |
|
|
value: 19.675 |
|
|
- type: recall_at_10 |
|
|
value: 26.868 |
|
|
- type: recall_at_20 |
|
|
value: 33.94 |
|
|
- type: recall_at_100 |
|
|
value: 51.833 |
|
|
- type: recall_at_1000 |
|
|
value: 76.822 |
|
|
- type: precision_at_1 |
|
|
value: 16.8 |
|
|
- type: precision_at_3 |
|
|
value: 11.533 |
|
|
- type: precision_at_5 |
|
|
value: 9.56 |
|
|
- type: precision_at_10 |
|
|
value: 6.83 |
|
|
- type: precision_at_20 |
|
|
value: 4.41 |
|
|
- type: precision_at_100 |
|
|
value: 1.432 |
|
|
- type: precision_at_1000 |
|
|
value: 0.22499999999999998 |
|
|
- type: mrr_at_1 |
|
|
value: 16.8 |
|
|
- type: mrr_at_3 |
|
|
value: 23.2333 |
|
|
- type: mrr_at_5 |
|
|
value: 25.2183 |
|
|
- type: mrr_at_10 |
|
|
value: 26.775 |
|
|
- type: mrr_at_20 |
|
|
value: 27.4121 |
|
|
- type: mrr_at_100 |
|
|
value: 27.882299999999997 |
|
|
- type: mrr_at_1000 |
|
|
value: 27.9472 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 28.3609 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: 10.5951 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 16.566 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 33.3794 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: 14.645900000000001 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 15.4617 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 33.5092 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: 16.209699999999998 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 16.7386 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 37.101299999999995 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: 20.939 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 15.1232 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 38.3563 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 22.3038 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 14.613100000000001 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 39.5793 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 23.3348 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 13.6571 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 39.2582 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 22.5989 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 12.6784 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 36.9819 |
|
|
- type: nauc_map_at_1_std |
|
|
value: 11.5065 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 22.4791 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 35.324299999999994 |
|
|
- type: nauc_map_at_3_std |
|
|
value: 13.572000000000001 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 19.3415 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 35.0138 |
|
|
- type: nauc_map_at_5_std |
|
|
value: 14.857600000000001 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 19.5352 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 36.8267 |
|
|
- type: nauc_map_at_10_std |
|
|
value: 17.6287 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 18.2802 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 37.5214 |
|
|
- type: nauc_map_at_20_std |
|
|
value: 18.319399999999998 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 18.0343 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 37.933499999999995 |
|
|
- type: nauc_map_at_100_std |
|
|
value: 18.6864 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 17.7119 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 37.9509 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 18.6975 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 17.5997 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 36.9819 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: 11.5065 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 22.4791 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 33.0875 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: 16.3976 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 15.6164 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 30.604799999999997 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: 17.1699 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 15.639800000000001 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 35.342400000000005 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 24.665599999999998 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 11.9499 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 35.956700000000005 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 26.556800000000003 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 10.0239 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 36.1012 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 27.8055 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 6.3591 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 34.7202 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 26.378 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: -0.7171000000000001 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 28.3609 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: 10.5951 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 16.566 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 30.490000000000002 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: 16.270899999999997 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 9.7026 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 29.3491 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: 19.084699999999998 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 10.7809 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 34.753699999999995 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 28.155 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 5.6554 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 33.3812 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 27.122400000000003 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 3.6636 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 28.7799 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 23.9905 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: -0.5301 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 13.068399999999999 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 12.9133 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -8.8717 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 28.3609 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: 10.5951 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 16.566 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 30.9311 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: 13.9549 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 12.851399999999998 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 30.893700000000003 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: 14.464599999999999 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 13.2001 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 32.277499999999996 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: 15.9378 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 12.9887 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 32.3817 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: 16.0469 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 13.039200000000001 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 32.386900000000004 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: 15.966800000000001 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 12.982 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 32.347300000000004 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: 15.9096 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 12.9742 |
|
|
- type: main_score |
|
|
value: 20.642 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB FEVERHardNegatives (default) |
|
|
revision: 080c9ed6267b65029207906e815d44a9240bafca |
|
|
split: test |
|
|
type: mteb/FEVER_test_top_250_only_w_correct-v2 |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 46.9 |
|
|
- type: ndcg_at_3 |
|
|
value: 57.825 |
|
|
- type: ndcg_at_5 |
|
|
value: 61.245000000000005 |
|
|
- type: ndcg_at_10 |
|
|
value: 63.836000000000006 |
|
|
- type: ndcg_at_20 |
|
|
value: 65.408 |
|
|
- type: ndcg_at_100 |
|
|
value: 66.796 |
|
|
- type: ndcg_at_1000 |
|
|
value: 67.216 |
|
|
- type: map_at_1 |
|
|
value: 43.999 |
|
|
- type: map_at_3 |
|
|
value: 53.813 |
|
|
- type: map_at_5 |
|
|
value: 55.741 |
|
|
- type: map_at_10 |
|
|
value: 56.852999999999994 |
|
|
- type: map_at_20 |
|
|
value: 57.30800000000001 |
|
|
- type: map_at_100 |
|
|
value: 57.54 |
|
|
- type: map_at_1000 |
|
|
value: 57.56099999999999 |
|
|
- type: recall_at_1 |
|
|
value: 43.999 |
|
|
- type: recall_at_3 |
|
|
value: 66.184 |
|
|
- type: recall_at_5 |
|
|
value: 74.557 |
|
|
- type: recall_at_10 |
|
|
value: 82.394 |
|
|
- type: recall_at_20 |
|
|
value: 88.51 |
|
|
- type: recall_at_100 |
|
|
value: 95.253 |
|
|
- type: recall_at_1000 |
|
|
value: 98.031 |
|
|
- type: precision_at_1 |
|
|
value: 46.9 |
|
|
- type: precision_at_3 |
|
|
value: 23.599999999999998 |
|
|
- type: precision_at_5 |
|
|
value: 15.98 |
|
|
- type: precision_at_10 |
|
|
value: 8.85 |
|
|
- type: precision_at_20 |
|
|
value: 4.760000000000001 |
|
|
- type: precision_at_100 |
|
|
value: 1.045 |
|
|
- type: precision_at_1000 |
|
|
value: 0.11 |
|
|
- type: mrr_at_1 |
|
|
value: 46.9 |
|
|
- type: mrr_at_3 |
|
|
value: 57.0167 |
|
|
- type: mrr_at_5 |
|
|
value: 59.046699999999994 |
|
|
- type: mrr_at_10 |
|
|
value: 60.1422 |
|
|
- type: mrr_at_20 |
|
|
value: 60.535799999999995 |
|
|
- type: mrr_at_100 |
|
|
value: 60.716 |
|
|
- type: mrr_at_1000 |
|
|
value: 60.7232 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 12.741900000000001 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: -20.011000000000003 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 51.02100000000001 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 17.416400000000003 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: -20.9336 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 46.3134 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 18.2369 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: -21.5645 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 46.261799999999994 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 18.8528 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: -20.6893 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 46.5862 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 18.0211 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: -19.652 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 46.5482 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 17.766000000000002 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: -18.7245 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 47.0345 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 17.596500000000002 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: -19.0628 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 47.12 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 13.017599999999998 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -18.8296 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 49.8762 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 16.2438 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -20.1711 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 47.2236 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 16.541 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -20.4952 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 47.1971 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 16.7266 |
|
|
- type: nauc_map_at_10_std |
|
|
value: -20.1189 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 47.2762 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 16.5198 |
|
|
- type: nauc_map_at_20_std |
|
|
value: -19.8167 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 47.266799999999996 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 16.467200000000002 |
|
|
- type: nauc_map_at_100_std |
|
|
value: -19.7016 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 47.3389 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 16.466900000000003 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: -19.704 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 47.341 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 13.017599999999998 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -18.8296 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 49.8762 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 20.579700000000003 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -21.263399999999997 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 40.7412 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 23.308799999999998 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: -23.0915 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 38.2001 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 27.296 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: -19.2697 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 35.9711 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 23.9957 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: -10.1564 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 30.5332 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 27.0148 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 25.655299999999997 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 23.1136 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 28.9392 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 47.491 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 15.6225 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 12.741900000000001 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: -20.011000000000003 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 51.02100000000001 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 20.477999999999998 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: -24.4646 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 41.1551 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 24.364 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: -27.1997 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 38.9501 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 30.684299999999997 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: -23.1531 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 34.6829 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 24.1828 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: -10.783800000000001 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 22.662399999999998 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 12.189 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 10.600999999999999 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: -0.2197 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 1.1533 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 6.2423 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -10.4662 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 12.741900000000001 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: -20.011000000000003 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 51.02100000000001 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 16.4501 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: -21.337500000000002 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 48.4594 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 16.8928 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: -21.7254 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 48.619299999999996 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 17.0057 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: -21.465899999999998 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 48.848200000000006 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 16.745099999999997 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: -21.2914 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 48.861900000000006 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 16.653399999999998 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: -21.1954 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 48.9097 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 16.650000000000002 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: -21.2048 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 48.911500000000004 |
|
|
- type: main_score |
|
|
value: 63.836000000000006 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB FiQA2018 (default) |
|
|
revision: 27a168819829fe9bcd655c2df245fb19452e8e06 |
|
|
split: test |
|
|
type: mteb/fiqa |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 25.154 |
|
|
- type: ndcg_at_3 |
|
|
value: 22.85 |
|
|
- type: ndcg_at_5 |
|
|
value: 23.788999999999998 |
|
|
- type: ndcg_at_10 |
|
|
value: 25.657000000000004 |
|
|
- type: ndcg_at_20 |
|
|
value: 28.058 |
|
|
- type: ndcg_at_100 |
|
|
value: 32.019999999999996 |
|
|
- type: ndcg_at_1000 |
|
|
value: 36.124 |
|
|
- type: map_at_1 |
|
|
value: 12.594 |
|
|
- type: map_at_3 |
|
|
value: 17.345 |
|
|
- type: map_at_5 |
|
|
value: 18.740000000000002 |
|
|
- type: map_at_10 |
|
|
value: 19.871 |
|
|
- type: map_at_20 |
|
|
value: 20.71 |
|
|
- type: map_at_100 |
|
|
value: 21.404 |
|
|
- type: map_at_1000 |
|
|
value: 21.616 |
|
|
- type: recall_at_1 |
|
|
value: 12.594 |
|
|
- type: recall_at_3 |
|
|
value: 20.682000000000002 |
|
|
- type: recall_at_5 |
|
|
value: 24.735 |
|
|
- type: recall_at_10 |
|
|
value: 30.217 |
|
|
- type: recall_at_20 |
|
|
value: 37.714999999999996 |
|
|
- type: recall_at_100 |
|
|
value: 54.364000000000004 |
|
|
- type: recall_at_1000 |
|
|
value: 79.487 |
|
|
- type: precision_at_1 |
|
|
value: 25.154 |
|
|
- type: precision_at_3 |
|
|
value: 15.174999999999999 |
|
|
- type: precision_at_5 |
|
|
value: 11.235000000000001 |
|
|
- type: precision_at_10 |
|
|
value: 7.13 |
|
|
- type: precision_at_20 |
|
|
value: 4.522 |
|
|
- type: precision_at_100 |
|
|
value: 1.341 |
|
|
- type: precision_at_1000 |
|
|
value: 0.20500000000000002 |
|
|
- type: mrr_at_1 |
|
|
value: 25.154300000000003 |
|
|
- type: mrr_at_3 |
|
|
value: 30.324099999999998 |
|
|
- type: mrr_at_5 |
|
|
value: 31.581799999999998 |
|
|
- type: mrr_at_10 |
|
|
value: 32.5208 |
|
|
- type: mrr_at_20 |
|
|
value: 33.055 |
|
|
- type: mrr_at_100 |
|
|
value: 33.4738 |
|
|
- type: mrr_at_1000 |
|
|
value: 33.5533 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 20.836199999999998 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: -2.4346 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 41.3264 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 21.4673 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: -0.35760000000000003 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 36.5457 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 21.0022 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: 0.30079999999999996 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 35.1377 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 21.4511 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: 1.9931 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 35.367599999999996 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 21.9794 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 3.2666 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 33.9954 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 22.666900000000002 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 6.1648000000000005 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 32.5715 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 23.9645 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 7.031 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 32.6535 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 13.436699999999998 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -6.1377 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 46.1518 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 17.6491 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -3.3383000000000003 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 39.909800000000004 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 18.4969 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -1.8129 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 38.4072 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 19.4823 |
|
|
- type: nauc_map_at_10_std |
|
|
value: -0.2211 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 38.1346 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 19.9898 |
|
|
- type: nauc_map_at_20_std |
|
|
value: 0.6002000000000001 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 37.755100000000006 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 20.2321 |
|
|
- type: nauc_map_at_100_std |
|
|
value: 1.2189999999999999 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 37.379 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 20.3676 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 1.3561999999999999 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 37.3216 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 13.436699999999998 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -6.1377 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 46.1518 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 17.4283 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -2.0456 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 34.5422 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 18.2169 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: 0.7002 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 29.7798 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 19.6832 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 4.6769 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 27.8829 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 20.095 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 6.884899999999999 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 22.7741 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 20.5351 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 19.2636 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 16.2238 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 27.9838 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 33.3099 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 12.701699999999999 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 20.836199999999998 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: -2.4346 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 41.3264 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 26.558500000000002 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: 3.6578 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 27.0323 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 28.794199999999996 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: 8.6533 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 21.9488 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 29.7713 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 13.645399999999999 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 20.1386 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 28.0465 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 16.3569 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 14.969299999999999 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 26.7123 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 19.1407 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: 5.7822 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 23.6681 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 16.3438 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -3.3699 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 20.836199999999998 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: -2.4346 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 41.3264 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 22.4267 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: -0.1948 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 36.9255 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 22.6662 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: 0.4444 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 35.957 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 22.5111 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: 0.7020000000000001 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 35.6976 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 22.4416 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: 0.8706999999999999 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 35.2034 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 22.4571 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: 1.0563 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 35.177 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 22.4743 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: 1.0505 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 35.2186 |
|
|
- type: main_score |
|
|
value: 25.657000000000004 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB HotpotQAHardNegatives (default) |
|
|
revision: 617612fa63afcb60e3b134bed8b7216a99707c37 |
|
|
split: test |
|
|
type: mteb/HotpotQA_test_top_250_only_w_correct-v2 |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 58.9 |
|
|
- type: ndcg_at_3 |
|
|
value: 45.092999999999996 |
|
|
- type: ndcg_at_5 |
|
|
value: 47.806 |
|
|
- type: ndcg_at_10 |
|
|
value: 50.666 |
|
|
- type: ndcg_at_20 |
|
|
value: 52.644000000000005 |
|
|
- type: ndcg_at_100 |
|
|
value: 56.071000000000005 |
|
|
- type: ndcg_at_1000 |
|
|
value: 58.262 |
|
|
- type: map_at_1 |
|
|
value: 29.45 |
|
|
- type: map_at_3 |
|
|
value: 37.675 |
|
|
- type: map_at_5 |
|
|
value: 39.562999999999995 |
|
|
- type: map_at_10 |
|
|
value: 41.056 |
|
|
- type: map_at_20 |
|
|
value: 41.765 |
|
|
- type: map_at_100 |
|
|
value: 42.425000000000004 |
|
|
- type: map_at_1000 |
|
|
value: 42.54 |
|
|
- type: recall_at_1 |
|
|
value: 29.45 |
|
|
- type: recall_at_3 |
|
|
value: 41.75 |
|
|
- type: recall_at_5 |
|
|
value: 47.099999999999994 |
|
|
- type: recall_at_10 |
|
|
value: 54.300000000000004 |
|
|
- type: recall_at_20 |
|
|
value: 60.699999999999996 |
|
|
- type: recall_at_100 |
|
|
value: 75.9 |
|
|
- type: recall_at_1000 |
|
|
value: 90.3 |
|
|
- type: precision_at_1 |
|
|
value: 58.9 |
|
|
- type: precision_at_3 |
|
|
value: 27.833000000000002 |
|
|
- type: precision_at_5 |
|
|
value: 18.84 |
|
|
- type: precision_at_10 |
|
|
value: 10.86 |
|
|
- type: precision_at_20 |
|
|
value: 6.069999999999999 |
|
|
- type: precision_at_100 |
|
|
value: 1.518 |
|
|
- type: precision_at_1000 |
|
|
value: 0.181 |
|
|
- type: mrr_at_1 |
|
|
value: 58.9 |
|
|
- type: mrr_at_3 |
|
|
value: 64.81670000000001 |
|
|
- type: mrr_at_5 |
|
|
value: 65.9717 |
|
|
- type: mrr_at_10 |
|
|
value: 66.84750000000001 |
|
|
- type: mrr_at_20 |
|
|
value: 67.1864 |
|
|
- type: mrr_at_100 |
|
|
value: 67.3796 |
|
|
- type: mrr_at_1000 |
|
|
value: 67.3962 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 40.6699 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: -6.4051 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 61.4074 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 36.086200000000005 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: -3.8372 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 44.0991 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 35.1661 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: -3.4778000000000002 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 41.2298 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 34.5689 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: -0.7254 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 38.9824 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 35.4153 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 0.9502999999999999 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 38.5558 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 36.187799999999996 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 3.3059 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 37.775 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 36.9076 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 3.2030000000000003 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 39.6691 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 40.6699 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -6.4051 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 61.4074 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 34.8654 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -1.9401000000000002 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 40.4559 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 34.0362 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -1.677 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 38.384 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 33.8136 |
|
|
- type: nauc_map_at_10_std |
|
|
value: -0.2753 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 37.1326 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 34.1981 |
|
|
- type: nauc_map_at_20_std |
|
|
value: 0.2882 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 36.996 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 34.2694 |
|
|
- type: nauc_map_at_100_std |
|
|
value: 0.596 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 36.858200000000004 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 34.3301 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 0.6459 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 36.9437 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 40.6699 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -6.4051 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 61.4074 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 33.4227 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -2.6978 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 35.5329 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 29.759900000000002 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: -1.7928 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 27.8553 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 27.2765 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 5.0284 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 21.5188 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 27.456500000000002 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 10.4452 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 17.377100000000002 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 27.960400000000003 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 26.0653 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 5.9226 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 33.996700000000004 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 44.291199999999996 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 7.6986 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 40.6699 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: -6.4051 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 61.4074 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 33.4227 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: -2.6978 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 35.5329 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 29.759900000000002 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: -1.7928 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 27.8553 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 27.2765 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 5.0284 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 21.5188 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 27.456500000000002 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 10.4452 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 17.377100000000002 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 27.960400000000003 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 26.0653 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: 5.9226 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 33.996700000000004 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 44.291199999999996 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: 7.6986 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 40.6699 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: -6.4051 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 61.4074 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 40.4193 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: -8.072899999999999 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 58.589400000000005 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 40.6559 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: -8.1937 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 58.30650000000001 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 40.515699999999995 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: -7.4325 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 58.1284 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 40.63 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: -7.1578 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 58.215799999999994 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 40.693 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: -7.0889 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 58.22389999999999 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 40.700900000000004 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: -7.098400000000001 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 58.2458 |
|
|
- type: main_score |
|
|
value: 50.666 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB ImdbClassification (default) |
|
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
|
split: test |
|
|
type: mteb/imdb |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 68.1712 |
|
|
- type: f1 |
|
|
value: 67.982 |
|
|
- type: f1_weighted |
|
|
value: 67.982 |
|
|
- type: ap |
|
|
value: 62.572799999999994 |
|
|
- type: ap_weighted |
|
|
value: 62.572799999999994 |
|
|
- type: main_score |
|
|
value: 68.1712 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: en |
|
|
name: MTEB MTOPDomainClassification (en) |
|
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
|
split: test |
|
|
type: mteb/mtop_domain |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 90.4423 |
|
|
- type: f1 |
|
|
value: 90.08840000000001 |
|
|
- type: f1_weighted |
|
|
value: 90.44919999999999 |
|
|
- type: main_score |
|
|
value: 90.4423 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: en |
|
|
name: MTEB MassiveIntentClassification (en) |
|
|
revision: 4672e20407010da34463acc759c162ca9734bca6 |
|
|
split: test |
|
|
type: mteb/amazon_massive_intent |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 65.4371 |
|
|
- type: f1 |
|
|
value: 62.8737 |
|
|
- type: f1_weighted |
|
|
value: 64.2218 |
|
|
- type: main_score |
|
|
value: 65.4371 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: en |
|
|
name: MTEB MassiveScenarioClassification (en) |
|
|
revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 |
|
|
split: test |
|
|
type: mteb/amazon_massive_scenario |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 70.4371 |
|
|
- type: f1 |
|
|
value: 69.75200000000001 |
|
|
- type: f1_weighted |
|
|
value: 69.7839 |
|
|
- type: main_score |
|
|
value: 70.4371 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB MedrxivClusteringP2P.v2 (default) |
|
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
|
split: test |
|
|
type: mteb/medrxiv-clustering-p2p |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 35.1864 |
|
|
- type: v_measure_std |
|
|
value: 0.7835 |
|
|
- type: main_score |
|
|
value: 35.1864 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB MedrxivClusteringS2S.v2 (default) |
|
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
|
split: test |
|
|
type: mteb/medrxiv-clustering-s2s |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 31.8693 |
|
|
- type: v_measure_std |
|
|
value: 0.662 |
|
|
- type: main_score |
|
|
value: 31.8693 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB MindSmallReranking (default) |
|
|
revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 |
|
|
split: test |
|
|
type: mteb/mind_small |
|
|
metrics: |
|
|
- type: map |
|
|
value: 29.4367 |
|
|
- type: mrr |
|
|
value: 30.318299999999997 |
|
|
- type: nAUC_map_max |
|
|
value: -21.5343 |
|
|
- type: nAUC_map_std |
|
|
value: -6.4848 |
|
|
- type: nAUC_map_diff1 |
|
|
value: 12.8559 |
|
|
- type: nAUC_mrr_max |
|
|
value: -15.981200000000001 |
|
|
- type: nAUC_mrr_std |
|
|
value: -4.2437000000000005 |
|
|
- type: nAUC_mrr_diff1 |
|
|
value: 12.4087 |
|
|
- type: main_score |
|
|
value: 29.4367 |
|
|
task: |
|
|
type: Reranking |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB SCIDOCS (default) |
|
|
revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 |
|
|
split: test |
|
|
type: mteb/scidocs |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 19.5 |
|
|
- type: ndcg_at_3 |
|
|
value: 15.673 |
|
|
- type: ndcg_at_5 |
|
|
value: 13.389000000000001 |
|
|
- type: ndcg_at_10 |
|
|
value: 16.179 |
|
|
- type: ndcg_at_20 |
|
|
value: 18.88 |
|
|
- type: ndcg_at_100 |
|
|
value: 23.812 |
|
|
- type: ndcg_at_1000 |
|
|
value: 29.833 |
|
|
- type: map_at_1 |
|
|
value: 3.963 |
|
|
- type: map_at_3 |
|
|
value: 6.93 |
|
|
- type: map_at_5 |
|
|
value: 8.062 |
|
|
- type: map_at_10 |
|
|
value: 9.328 |
|
|
- type: map_at_20 |
|
|
value: 10.283000000000001 |
|
|
- type: map_at_100 |
|
|
value: 11.197 |
|
|
- type: map_at_1000 |
|
|
value: 11.522 |
|
|
- type: recall_at_1 |
|
|
value: 3.963 |
|
|
- type: recall_at_3 |
|
|
value: 8.813 |
|
|
- type: recall_at_5 |
|
|
value: 11.658 |
|
|
- type: recall_at_10 |
|
|
value: 16.803 |
|
|
- type: recall_at_20 |
|
|
value: 23.169999999999998 |
|
|
- type: recall_at_100 |
|
|
value: 39.163 |
|
|
- type: recall_at_1000 |
|
|
value: 68.572 |
|
|
- type: precision_at_1 |
|
|
value: 19.5 |
|
|
- type: precision_at_3 |
|
|
value: 14.499999999999998 |
|
|
- type: precision_at_5 |
|
|
value: 11.5 |
|
|
- type: precision_at_10 |
|
|
value: 8.3 |
|
|
- type: precision_at_20 |
|
|
value: 5.71 |
|
|
- type: precision_at_100 |
|
|
value: 1.9300000000000002 |
|
|
- type: precision_at_1000 |
|
|
value: 0.338 |
|
|
- type: mrr_at_1 |
|
|
value: 19.5 |
|
|
- type: mrr_at_3 |
|
|
value: 26.016699999999997 |
|
|
- type: mrr_at_5 |
|
|
value: 27.526699999999998 |
|
|
- type: mrr_at_10 |
|
|
value: 28.9305 |
|
|
- type: mrr_at_20 |
|
|
value: 29.628100000000003 |
|
|
- type: mrr_at_100 |
|
|
value: 30.131400000000003 |
|
|
- type: mrr_at_1000 |
|
|
value: 30.201800000000002 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 25.1197 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: 4.7176 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 24.2336 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 30.050900000000002 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: 11.4719 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 20.4572 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 32.224399999999996 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: 15.0585 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 19.991600000000002 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 33.7156 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: 19.2797 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 20.3735 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 34.7518 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 23.227600000000002 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 19.2851 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 36.6006 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 28.511599999999998 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 18.0315 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 36.3651 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 29.7201 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 16.5988 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 24.954 |
|
|
- type: nauc_map_at_1_std |
|
|
value: 4.7878 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 24.7611 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 30.0634 |
|
|
- type: nauc_map_at_3_std |
|
|
value: 9.9217 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 21.9063 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 32.1685 |
|
|
- type: nauc_map_at_5_std |
|
|
value: 12.8527 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 21.033099999999997 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 33.840199999999996 |
|
|
- type: nauc_map_at_10_std |
|
|
value: 16.304299999999998 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 21.9142 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 34.2084 |
|
|
- type: nauc_map_at_20_std |
|
|
value: 18.709799999999998 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 21.2113 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 35.1304 |
|
|
- type: nauc_map_at_100_std |
|
|
value: 20.8559 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 20.8642 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 35.1972 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 21.2306 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 20.7425 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 24.954 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: 4.7878 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 24.7611 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: 31.1016 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: 14.1642 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 18.676000000000002 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: 33.8509 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: 19.503899999999998 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 17.1764 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 34.085300000000004 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 25.536199999999997 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 16.8913 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 34.1879 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 31.5486 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 13.852300000000001 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 34.313700000000004 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 40.6137 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 9.043800000000001 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 27.090500000000002 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 42.398799999999994 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: -0.9452999999999999 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 25.1197 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: 4.7176 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 24.2336 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 31.4429 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: 14.1941 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 18.4824 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 34.2219 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: 19.703699999999998 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 17.0964 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 34.380300000000005 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 25.6554 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 16.8487 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 34.462199999999996 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 31.465500000000002 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 13.9038 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 34.7074 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 40.3278 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: 9.2637 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 27.213900000000002 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 40.8382 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -0.5306 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 25.1197 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: 4.7176 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 24.2336 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 27.9362 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: 9.9578 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 20.809 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 29.0381 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: 11.7807 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 20.8787 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 28.860799999999998 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: 12.269 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 20.7762 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 29.2051 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: 12.7588 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 20.9176 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 29.2288 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: 12.7523 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 20.9235 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 29.2119 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: 12.697600000000001 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 20.9131 |
|
|
- type: main_score |
|
|
value: 16.179 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB SICK-R (default) |
|
|
revision: 20a6d6f312dd54037fe07a32d58e5e168867909d |
|
|
split: test |
|
|
type: mteb/sickr-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 84.5347 |
|
|
- type: spearman |
|
|
value: 79.80850000000001 |
|
|
- type: cosine_pearson |
|
|
value: 84.5347 |
|
|
- type: cosine_spearman |
|
|
value: 79.80850000000001 |
|
|
- type: manhattan_pearson |
|
|
value: 81.0701 |
|
|
- type: manhattan_spearman |
|
|
value: 79.6721 |
|
|
- type: euclidean_pearson |
|
|
value: 81.20349999999999 |
|
|
- type: euclidean_spearman |
|
|
value: 79.80850000000001 |
|
|
- type: main_score |
|
|
value: 79.80850000000001 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB STS12 (default) |
|
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
|
split: test |
|
|
type: mteb/sts12-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 86.88 |
|
|
- type: spearman |
|
|
value: 78.1076 |
|
|
- type: cosine_pearson |
|
|
value: 86.88 |
|
|
- type: cosine_spearman |
|
|
value: 78.1052 |
|
|
- type: manhattan_pearson |
|
|
value: 83.3712 |
|
|
- type: manhattan_spearman |
|
|
value: 78.0898 |
|
|
- type: euclidean_pearson |
|
|
value: 83.3731 |
|
|
- type: euclidean_spearman |
|
|
value: 78.1052 |
|
|
- type: main_score |
|
|
value: 78.1052 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB STS13 (default) |
|
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
|
split: test |
|
|
type: mteb/sts13-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 83.5938 |
|
|
- type: spearman |
|
|
value: 84.2951 |
|
|
- type: cosine_pearson |
|
|
value: 83.5938 |
|
|
- type: cosine_spearman |
|
|
value: 84.2951 |
|
|
- type: manhattan_pearson |
|
|
value: 83.2541 |
|
|
- type: manhattan_spearman |
|
|
value: 83.8292 |
|
|
- type: euclidean_pearson |
|
|
value: 83.69640000000001 |
|
|
- type: euclidean_spearman |
|
|
value: 84.2951 |
|
|
- type: main_score |
|
|
value: 84.2951 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB STS14 (default) |
|
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
|
split: test |
|
|
type: mteb/sts14-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 82.6003 |
|
|
- type: spearman |
|
|
value: 81.3569 |
|
|
- type: cosine_pearson |
|
|
value: 82.6003 |
|
|
- type: cosine_spearman |
|
|
value: 81.357 |
|
|
- type: manhattan_pearson |
|
|
value: 81.5087 |
|
|
- type: manhattan_spearman |
|
|
value: 81.17229999999999 |
|
|
- type: euclidean_pearson |
|
|
value: 81.7147 |
|
|
- type: euclidean_spearman |
|
|
value: 81.3569 |
|
|
- type: main_score |
|
|
value: 81.357 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB STS15 (default) |
|
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
|
split: test |
|
|
type: mteb/sts15-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 86.4161 |
|
|
- type: spearman |
|
|
value: 87.0039 |
|
|
- type: cosine_pearson |
|
|
value: 86.4161 |
|
|
- type: cosine_spearman |
|
|
value: 87.0039 |
|
|
- type: manhattan_pearson |
|
|
value: 86.2482 |
|
|
- type: manhattan_spearman |
|
|
value: 86.934 |
|
|
- type: euclidean_pearson |
|
|
value: 86.3344 |
|
|
- type: euclidean_spearman |
|
|
value: 87.0039 |
|
|
- type: main_score |
|
|
value: 87.0039 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: en-en |
|
|
name: MTEB STS17 (en-en) |
|
|
revision: faeb762787bd10488a50c8b5be4a3b82e411949c |
|
|
split: test |
|
|
type: mteb/sts17-crosslingual-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 88.6011 |
|
|
- type: spearman |
|
|
value: 88.1023 |
|
|
- type: cosine_pearson |
|
|
value: 88.6011 |
|
|
- type: cosine_spearman |
|
|
value: 88.1023 |
|
|
- type: manhattan_pearson |
|
|
value: 88.18639999999999 |
|
|
- type: manhattan_spearman |
|
|
value: 88.55380000000001 |
|
|
- type: euclidean_pearson |
|
|
value: 88.011 |
|
|
- type: euclidean_spearman |
|
|
value: 88.1023 |
|
|
- type: main_score |
|
|
value: 88.1023 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: en |
|
|
name: MTEB STS22.v2 (en) |
|
|
revision: d31f33a128469b20e357535c39b82fb3c3f6f2bd |
|
|
split: test |
|
|
type: mteb/sts22-crosslingual-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 65.7746 |
|
|
- type: spearman |
|
|
value: 64.7997 |
|
|
- type: cosine_pearson |
|
|
value: 65.7746 |
|
|
- type: cosine_spearman |
|
|
value: 64.7997 |
|
|
- type: manhattan_pearson |
|
|
value: 67.5417 |
|
|
- type: manhattan_spearman |
|
|
value: 65.27629999999999 |
|
|
- type: euclidean_pearson |
|
|
value: 67.2574 |
|
|
- type: euclidean_spearman |
|
|
value: 64.7997 |
|
|
- type: main_score |
|
|
value: 64.7997 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB STSBenchmark (default) |
|
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
|
split: test |
|
|
type: mteb/stsbenchmark-sts |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 84.4276 |
|
|
- type: spearman |
|
|
value: 84.9631 |
|
|
- type: cosine_pearson |
|
|
value: 84.4276 |
|
|
- type: cosine_spearman |
|
|
value: 84.9631 |
|
|
- type: manhattan_pearson |
|
|
value: 84.4743 |
|
|
- type: manhattan_spearman |
|
|
value: 84.7686 |
|
|
- type: euclidean_pearson |
|
|
value: 84.6058 |
|
|
- type: euclidean_spearman |
|
|
value: 84.9631 |
|
|
- type: main_score |
|
|
value: 84.9631 |
|
|
task: |
|
|
type: STS |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB SprintDuplicateQuestions (default) |
|
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
|
split: test |
|
|
type: mteb/sprintduplicatequestions-pairclassification |
|
|
metrics: |
|
|
- type: similarity_accuracy |
|
|
value: 99.7931 |
|
|
- type: similarity_accuracy_threshold |
|
|
value: 69.6798 |
|
|
- type: similarity_f1 |
|
|
value: 89.4293 |
|
|
- type: similarity_f1_threshold |
|
|
value: 68.3132 |
|
|
- type: similarity_precision |
|
|
value: 88.76849999999999 |
|
|
- type: similarity_recall |
|
|
value: 90.10000000000001 |
|
|
- type: similarity_ap |
|
|
value: 94.3099 |
|
|
- type: cosine_accuracy |
|
|
value: 99.7931 |
|
|
- type: cosine_accuracy_threshold |
|
|
value: 69.6798 |
|
|
- type: cosine_f1 |
|
|
value: 89.4293 |
|
|
- type: cosine_f1_threshold |
|
|
value: 68.3132 |
|
|
- type: cosine_precision |
|
|
value: 88.76849999999999 |
|
|
- type: cosine_recall |
|
|
value: 90.10000000000001 |
|
|
- type: cosine_ap |
|
|
value: 94.3099 |
|
|
- type: manhattan_accuracy |
|
|
value: 99.7792 |
|
|
- type: manhattan_accuracy_threshold |
|
|
value: 1354.3922 |
|
|
- type: manhattan_f1 |
|
|
value: 88.71289999999999 |
|
|
- type: manhattan_f1_threshold |
|
|
value: 1389.3319999999999 |
|
|
- type: manhattan_precision |
|
|
value: 87.84309999999999 |
|
|
- type: manhattan_recall |
|
|
value: 89.60000000000001 |
|
|
- type: manhattan_ap |
|
|
value: 93.8459 |
|
|
- type: euclidean_accuracy |
|
|
value: 99.7931 |
|
|
- type: euclidean_accuracy_threshold |
|
|
value: 77.872 |
|
|
- type: euclidean_f1 |
|
|
value: 89.4293 |
|
|
- type: euclidean_f1_threshold |
|
|
value: 79.6075 |
|
|
- type: euclidean_precision |
|
|
value: 88.76849999999999 |
|
|
- type: euclidean_recall |
|
|
value: 90.10000000000001 |
|
|
- type: euclidean_ap |
|
|
value: 94.3099 |
|
|
- type: dot_accuracy |
|
|
value: 99.7931 |
|
|
- type: dot_accuracy_threshold |
|
|
value: 69.6798 |
|
|
- type: dot_f1 |
|
|
value: 89.4293 |
|
|
- type: dot_f1_threshold |
|
|
value: 68.3132 |
|
|
- type: dot_precision |
|
|
value: 88.76849999999999 |
|
|
- type: dot_recall |
|
|
value: 90.10000000000001 |
|
|
- type: dot_ap |
|
|
value: 94.3099 |
|
|
- type: max_accuracy |
|
|
value: 99.7931 |
|
|
- type: max_f1 |
|
|
value: 89.4293 |
|
|
- type: max_precision |
|
|
value: 88.76849999999999 |
|
|
- type: max_recall |
|
|
value: 90.10000000000001 |
|
|
- type: max_ap |
|
|
value: 94.3099 |
|
|
- type: main_score |
|
|
value: 94.3099 |
|
|
task: |
|
|
type: PairClassification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB StackExchangeClustering.v2 (default) |
|
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
|
split: test |
|
|
type: mteb/stackexchange-clustering |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 53.9397 |
|
|
- type: v_measure_std |
|
|
value: 0.7764 |
|
|
- type: main_score |
|
|
value: 53.9397 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB StackExchangeClusteringP2P.v2 (default) |
|
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
|
split: test |
|
|
type: mteb/stackexchange-clustering-p2p |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 40.6498 |
|
|
- type: v_measure_std |
|
|
value: 0.439 |
|
|
- type: main_score |
|
|
value: 40.6498 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB SummEvalSummarization.v2 (default) |
|
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
|
split: test |
|
|
type: mteb/summeval |
|
|
metrics: |
|
|
- type: pearson |
|
|
value: 28.6283 |
|
|
- type: spearman |
|
|
value: 26.0828 |
|
|
- type: cosine_spearman |
|
|
value: 26.0828 |
|
|
- type: cosine_pearson |
|
|
value: 28.6283 |
|
|
- type: dot_spearman |
|
|
value: 26.0828 |
|
|
- type: dot_pearson |
|
|
value: 28.6283 |
|
|
- type: main_score |
|
|
value: 26.0828 |
|
|
task: |
|
|
type: Summarization |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB TRECCOVID (default) |
|
|
revision: bb9466bac8153a0349341eb1b22e06409e78ef4e |
|
|
split: test |
|
|
type: mteb/trec-covid |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 66 |
|
|
- type: ndcg_at_3 |
|
|
value: 64.592 |
|
|
- type: ndcg_at_5 |
|
|
value: 63.405 |
|
|
- type: ndcg_at_10 |
|
|
value: 60.077999999999996 |
|
|
- type: ndcg_at_20 |
|
|
value: 57.202 |
|
|
- type: ndcg_at_100 |
|
|
value: 44.643 |
|
|
- type: ndcg_at_1000 |
|
|
value: 42.104 |
|
|
- type: map_at_1 |
|
|
value: 0.193 |
|
|
- type: map_at_3 |
|
|
value: 0.514 |
|
|
- type: map_at_5 |
|
|
value: 0.783 |
|
|
- type: map_at_10 |
|
|
value: 1.3719999999999999 |
|
|
- type: map_at_20 |
|
|
value: 2.371 |
|
|
- type: map_at_100 |
|
|
value: 7.353 |
|
|
- type: map_at_1000 |
|
|
value: 17.855999999999998 |
|
|
- type: recall_at_1 |
|
|
value: 0.193 |
|
|
- type: recall_at_3 |
|
|
value: 0.563 |
|
|
- type: recall_at_5 |
|
|
value: 0.907 |
|
|
- type: recall_at_10 |
|
|
value: 1.683 |
|
|
- type: recall_at_20 |
|
|
value: 3.118 |
|
|
- type: recall_at_100 |
|
|
value: 11.051 |
|
|
- type: recall_at_1000 |
|
|
value: 39.973 |
|
|
- type: precision_at_1 |
|
|
value: 74 |
|
|
- type: precision_at_3 |
|
|
value: 71.333 |
|
|
- type: precision_at_5 |
|
|
value: 68.8 |
|
|
- type: precision_at_10 |
|
|
value: 63.800000000000004 |
|
|
- type: precision_at_20 |
|
|
value: 60.5 |
|
|
- type: precision_at_100 |
|
|
value: 45.519999999999996 |
|
|
- type: precision_at_1000 |
|
|
value: 18.451999999999998 |
|
|
- type: mrr_at_1 |
|
|
value: 74 |
|
|
- type: mrr_at_3 |
|
|
value: 83.3333 |
|
|
- type: mrr_at_5 |
|
|
value: 83.7333 |
|
|
- type: mrr_at_10 |
|
|
value: 84.3524 |
|
|
- type: mrr_at_20 |
|
|
value: 84.3524 |
|
|
- type: mrr_at_100 |
|
|
value: 84.3524 |
|
|
- type: mrr_at_1000 |
|
|
value: 84.3524 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: 11.527800000000001 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: 17.1352 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 24.955199999999998 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: 11.7829 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: 23.1421 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 20.884 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: 8.8058 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: 27.9156 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 7.002 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: 16.561 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: 40.528999999999996 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: -6.1467 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: 25.0792 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 54.0689 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: -9.6224 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: 43.2818 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 75.4432 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: -11.4618 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: 50.360099999999996 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 76.03999999999999 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: -12.5796 |
|
|
- type: nauc_map_at_1_max |
|
|
value: 4.3809000000000005 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -17.5338 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 24.837 |
|
|
- type: nauc_map_at_3_max |
|
|
value: 4.7842 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -8.9273 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 19.7729 |
|
|
- type: nauc_map_at_5_max |
|
|
value: 3.6865 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -1.1584 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 7.3548 |
|
|
- type: nauc_map_at_10_max |
|
|
value: 7.556400000000001 |
|
|
- type: nauc_map_at_10_std |
|
|
value: 11.2599 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: -3.4863999999999997 |
|
|
- type: nauc_map_at_20_max |
|
|
value: 12.6951 |
|
|
- type: nauc_map_at_20_std |
|
|
value: 27.3531 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: -11.968 |
|
|
- type: nauc_map_at_100_max |
|
|
value: 41.625099999999996 |
|
|
- type: nauc_map_at_100_std |
|
|
value: 66.5204 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: -12.020999999999999 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: 56.6014 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 80.6523 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: -11.9876 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: 4.3809000000000005 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -17.5338 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 24.837 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: -0.8904000000000001 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -11.2455 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 17.6352 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: -4.6216 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: -3.5367999999999995 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 3.3192 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: 1.8993 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: 6.844600000000001 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: -6.0693 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: 5.733 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: 20.6114 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: -11.631 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: 32.7146 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 55.6053 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: -10.7219 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 50.7544 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 68.4639 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: -10.431600000000001 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: 13.8681 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: -3.4711 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 36.945 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: 11.6309 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: 5.0299000000000005 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 28.5186 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: 10.1297 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: 19.049599999999998 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 7.918500000000001 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: 21.3492 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 39.6679 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: -10.7691 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 32.4627 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 57.2564 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: -12.0336 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 47.7277 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 77.0329 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: -9.2173 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 47.6622 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 62.8329 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -5.9713 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: 13.8681 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: -3.4711 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 36.945 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: 9.6673 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: -4.3877 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 39.2075 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: 7.9742999999999995 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: -4.8388 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 38.314 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: 11.6962 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: -2.7085000000000004 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 37.695 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: 11.6962 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: -2.7085000000000004 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 37.695 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: 11.6962 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: -2.7085000000000004 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 37.695 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: 11.6962 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: -2.7085000000000004 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 37.695 |
|
|
- type: main_score |
|
|
value: 60.077999999999996 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB Touche2020Retrieval.v3 (default) |
|
|
revision: 431886eaecc48f067a3975b70d0949ea2862463c |
|
|
split: test |
|
|
type: mteb/webis-touche2020-v3 |
|
|
metrics: |
|
|
- type: ndcg_at_1 |
|
|
value: 58.163 |
|
|
- type: ndcg_at_3 |
|
|
value: 58.884 |
|
|
- type: ndcg_at_5 |
|
|
value: 53.062 |
|
|
- type: ndcg_at_10 |
|
|
value: 47.571999999999996 |
|
|
- type: ndcg_at_20 |
|
|
value: 43.984 |
|
|
- type: ndcg_at_100 |
|
|
value: 51.559999999999995 |
|
|
- type: ndcg_at_1000 |
|
|
value: 64.25800000000001 |
|
|
- type: map_at_1 |
|
|
value: 2.759 |
|
|
- type: map_at_3 |
|
|
value: 7.310999999999999 |
|
|
- type: map_at_5 |
|
|
value: 10.077 |
|
|
- type: map_at_10 |
|
|
value: 15.722 |
|
|
- type: map_at_20 |
|
|
value: 21.917 |
|
|
- type: map_at_100 |
|
|
value: 29.582000000000004 |
|
|
- type: map_at_1000 |
|
|
value: 32.608 |
|
|
- type: recall_at_1 |
|
|
value: 2.759 |
|
|
- type: recall_at_3 |
|
|
value: 7.870000000000001 |
|
|
- type: recall_at_5 |
|
|
value: 11.26 |
|
|
- type: recall_at_10 |
|
|
value: 19.211 |
|
|
- type: recall_at_20 |
|
|
value: 30.134 |
|
|
- type: recall_at_100 |
|
|
value: 54.96 |
|
|
- type: recall_at_1000 |
|
|
value: 85.78099999999999 |
|
|
- type: precision_at_1 |
|
|
value: 67.34700000000001 |
|
|
- type: precision_at_3 |
|
|
value: 68.027 |
|
|
- type: precision_at_5 |
|
|
value: 59.184000000000005 |
|
|
- type: precision_at_10 |
|
|
value: 50.815999999999995 |
|
|
- type: precision_at_20 |
|
|
value: 41.939 |
|
|
- type: precision_at_100 |
|
|
value: 17.041 |
|
|
- type: precision_at_1000 |
|
|
value: 2.963 |
|
|
- type: mrr_at_1 |
|
|
value: 67.3469 |
|
|
- type: mrr_at_3 |
|
|
value: 80.6122 |
|
|
- type: mrr_at_5 |
|
|
value: 80.6122 |
|
|
- type: mrr_at_10 |
|
|
value: 80.9524 |
|
|
- type: mrr_at_20 |
|
|
value: 80.9524 |
|
|
- type: mrr_at_100 |
|
|
value: 80.9524 |
|
|
- type: mrr_at_1000 |
|
|
value: 80.9524 |
|
|
- type: nauc_ndcg_at_1_max |
|
|
value: -18.7982 |
|
|
- type: nauc_ndcg_at_1_std |
|
|
value: 13.605500000000001 |
|
|
- type: nauc_ndcg_at_1_diff1 |
|
|
value: 21.2588 |
|
|
- type: nauc_ndcg_at_3_max |
|
|
value: -9.0937 |
|
|
- type: nauc_ndcg_at_3_std |
|
|
value: 23.259900000000002 |
|
|
- type: nauc_ndcg_at_3_diff1 |
|
|
value: 24.2989 |
|
|
- type: nauc_ndcg_at_5_max |
|
|
value: -13.242300000000002 |
|
|
- type: nauc_ndcg_at_5_std |
|
|
value: 9.7464 |
|
|
- type: nauc_ndcg_at_5_diff1 |
|
|
value: 18.601799999999997 |
|
|
- type: nauc_ndcg_at_10_max |
|
|
value: -12.045599999999999 |
|
|
- type: nauc_ndcg_at_10_std |
|
|
value: 7.5604000000000005 |
|
|
- type: nauc_ndcg_at_10_diff1 |
|
|
value: 20.1203 |
|
|
- type: nauc_ndcg_at_20_max |
|
|
value: -13.2776 |
|
|
- type: nauc_ndcg_at_20_std |
|
|
value: 8.2692 |
|
|
- type: nauc_ndcg_at_20_diff1 |
|
|
value: 21.38 |
|
|
- type: nauc_ndcg_at_100_max |
|
|
value: -21.1315 |
|
|
- type: nauc_ndcg_at_100_std |
|
|
value: 8.4079 |
|
|
- type: nauc_ndcg_at_100_diff1 |
|
|
value: 29.3124 |
|
|
- type: nauc_ndcg_at_1000_max |
|
|
value: -3.7026999999999997 |
|
|
- type: nauc_ndcg_at_1000_std |
|
|
value: 34.970600000000005 |
|
|
- type: nauc_ndcg_at_1000_diff1 |
|
|
value: 22.3636 |
|
|
- type: nauc_map_at_1_max |
|
|
value: -36.432500000000005 |
|
|
- type: nauc_map_at_1_std |
|
|
value: -23.9669 |
|
|
- type: nauc_map_at_1_diff1 |
|
|
value: 37.2073 |
|
|
- type: nauc_map_at_3_max |
|
|
value: -32.8613 |
|
|
- type: nauc_map_at_3_std |
|
|
value: -18.0951 |
|
|
- type: nauc_map_at_3_diff1 |
|
|
value: 36.3228 |
|
|
- type: nauc_map_at_5_max |
|
|
value: -31.355 |
|
|
- type: nauc_map_at_5_std |
|
|
value: -21.148500000000002 |
|
|
- type: nauc_map_at_5_diff1 |
|
|
value: 27.999200000000002 |
|
|
- type: nauc_map_at_10_max |
|
|
value: -25.3787 |
|
|
- type: nauc_map_at_10_std |
|
|
value: -18.564700000000002 |
|
|
- type: nauc_map_at_10_diff1 |
|
|
value: 24.076800000000002 |
|
|
- type: nauc_map_at_20_max |
|
|
value: -20.954 |
|
|
- type: nauc_map_at_20_std |
|
|
value: -12.6847 |
|
|
- type: nauc_map_at_20_diff1 |
|
|
value: 24.3842 |
|
|
- type: nauc_map_at_100_max |
|
|
value: -15.7801 |
|
|
- type: nauc_map_at_100_std |
|
|
value: -2.823 |
|
|
- type: nauc_map_at_100_diff1 |
|
|
value: 24.8472 |
|
|
- type: nauc_map_at_1000_max |
|
|
value: -11.8023 |
|
|
- type: nauc_map_at_1000_std |
|
|
value: 3.9041 |
|
|
- type: nauc_map_at_1000_diff1 |
|
|
value: 23.3312 |
|
|
- type: nauc_recall_at_1_max |
|
|
value: -36.432500000000005 |
|
|
- type: nauc_recall_at_1_std |
|
|
value: -23.9669 |
|
|
- type: nauc_recall_at_1_diff1 |
|
|
value: 37.2073 |
|
|
- type: nauc_recall_at_3_max |
|
|
value: -36.3448 |
|
|
- type: nauc_recall_at_3_std |
|
|
value: -18.4742 |
|
|
- type: nauc_recall_at_3_diff1 |
|
|
value: 38.4857 |
|
|
- type: nauc_recall_at_5_max |
|
|
value: -35.4207 |
|
|
- type: nauc_recall_at_5_std |
|
|
value: -23.7906 |
|
|
- type: nauc_recall_at_5_diff1 |
|
|
value: 28.3854 |
|
|
- type: nauc_recall_at_10_max |
|
|
value: -28.4266 |
|
|
- type: nauc_recall_at_10_std |
|
|
value: -21.3224 |
|
|
- type: nauc_recall_at_10_diff1 |
|
|
value: 27.0746 |
|
|
- type: nauc_recall_at_20_max |
|
|
value: -23.1205 |
|
|
- type: nauc_recall_at_20_std |
|
|
value: -12.3539 |
|
|
- type: nauc_recall_at_20_diff1 |
|
|
value: 27.127499999999998 |
|
|
- type: nauc_recall_at_100_max |
|
|
value: -22.0703 |
|
|
- type: nauc_recall_at_100_std |
|
|
value: 10.1339 |
|
|
- type: nauc_recall_at_100_diff1 |
|
|
value: 29.759900000000002 |
|
|
- type: nauc_recall_at_1000_max |
|
|
value: 13.5147 |
|
|
- type: nauc_recall_at_1000_std |
|
|
value: 78.4907 |
|
|
- type: nauc_recall_at_1000_diff1 |
|
|
value: 12.151 |
|
|
- type: nauc_precision_at_1_max |
|
|
value: -20.1082 |
|
|
- type: nauc_precision_at_1_std |
|
|
value: 13.5123 |
|
|
- type: nauc_precision_at_1_diff1 |
|
|
value: 16.7562 |
|
|
- type: nauc_precision_at_3_max |
|
|
value: -11.2979 |
|
|
- type: nauc_precision_at_3_std |
|
|
value: 23.0876 |
|
|
- type: nauc_precision_at_3_diff1 |
|
|
value: 20.738 |
|
|
- type: nauc_precision_at_5_max |
|
|
value: -18.1198 |
|
|
- type: nauc_precision_at_5_std |
|
|
value: -2.4168 |
|
|
- type: nauc_precision_at_5_diff1 |
|
|
value: 5.1223 |
|
|
- type: nauc_precision_at_10_max |
|
|
value: -4.7656 |
|
|
- type: nauc_precision_at_10_std |
|
|
value: 1.5377 |
|
|
- type: nauc_precision_at_10_diff1 |
|
|
value: 8.2175 |
|
|
- type: nauc_precision_at_20_max |
|
|
value: 7.571999999999999 |
|
|
- type: nauc_precision_at_20_std |
|
|
value: 17.309 |
|
|
- type: nauc_precision_at_20_diff1 |
|
|
value: 5.2156 |
|
|
- type: nauc_precision_at_100_max |
|
|
value: 35.02 |
|
|
- type: nauc_precision_at_100_std |
|
|
value: 57.2867 |
|
|
- type: nauc_precision_at_100_diff1 |
|
|
value: -12.814200000000001 |
|
|
- type: nauc_precision_at_1000_max |
|
|
value: 54.8988 |
|
|
- type: nauc_precision_at_1000_std |
|
|
value: 55.970699999999994 |
|
|
- type: nauc_precision_at_1000_diff1 |
|
|
value: -36.8074 |
|
|
- type: nauc_mrr_at_1_max |
|
|
value: -20.1082 |
|
|
- type: nauc_mrr_at_1_std |
|
|
value: 13.5123 |
|
|
- type: nauc_mrr_at_1_diff1 |
|
|
value: 16.7562 |
|
|
- type: nauc_mrr_at_3_max |
|
|
value: -23.668300000000002 |
|
|
- type: nauc_mrr_at_3_std |
|
|
value: 16.883699999999997 |
|
|
- type: nauc_mrr_at_3_diff1 |
|
|
value: 20.6687 |
|
|
- type: nauc_mrr_at_5_max |
|
|
value: -23.668300000000002 |
|
|
- type: nauc_mrr_at_5_std |
|
|
value: 16.883699999999997 |
|
|
- type: nauc_mrr_at_5_diff1 |
|
|
value: 20.6687 |
|
|
- type: nauc_mrr_at_10_max |
|
|
value: -21.8234 |
|
|
- type: nauc_mrr_at_10_std |
|
|
value: 15.1609 |
|
|
- type: nauc_mrr_at_10_diff1 |
|
|
value: 19.6023 |
|
|
- type: nauc_mrr_at_20_max |
|
|
value: -21.8234 |
|
|
- type: nauc_mrr_at_20_std |
|
|
value: 15.1609 |
|
|
- type: nauc_mrr_at_20_diff1 |
|
|
value: 19.6023 |
|
|
- type: nauc_mrr_at_100_max |
|
|
value: -21.8234 |
|
|
- type: nauc_mrr_at_100_std |
|
|
value: 15.1609 |
|
|
- type: nauc_mrr_at_100_diff1 |
|
|
value: 19.6023 |
|
|
- type: nauc_mrr_at_1000_max |
|
|
value: -21.8234 |
|
|
- type: nauc_mrr_at_1000_std |
|
|
value: 15.1609 |
|
|
- type: nauc_mrr_at_1000_diff1 |
|
|
value: 19.6023 |
|
|
- type: main_score |
|
|
value: 47.571999999999996 |
|
|
task: |
|
|
type: Retrieval |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB ToxicConversationsClassification (default) |
|
|
revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de |
|
|
split: test |
|
|
type: mteb/toxic_conversations_50k |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 63.608399999999996 |
|
|
- type: f1 |
|
|
value: 48.6248 |
|
|
- type: f1_weighted |
|
|
value: 71.6158 |
|
|
- type: ap |
|
|
value: 10.9541 |
|
|
- type: ap_weighted |
|
|
value: 10.9541 |
|
|
- type: main_score |
|
|
value: 63.608399999999996 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB TweetSentimentExtractionClassification (default) |
|
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
|
split: test |
|
|
type: mteb/tweet_sentiment_extraction |
|
|
metrics: |
|
|
- type: accuracy |
|
|
value: 60.506499999999996 |
|
|
- type: f1 |
|
|
value: 60.711499999999994 |
|
|
- type: f1_weighted |
|
|
value: 59.695699999999995 |
|
|
- type: main_score |
|
|
value: 60.506499999999996 |
|
|
task: |
|
|
type: Classification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB TwentyNewsgroupsClustering.v2 (default) |
|
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
|
split: test |
|
|
type: mteb/twentynewsgroups-clustering |
|
|
metrics: |
|
|
- type: v_measure |
|
|
value: 33.5462 |
|
|
- type: v_measure_std |
|
|
value: 1.3361 |
|
|
- type: main_score |
|
|
value: 33.5462 |
|
|
task: |
|
|
type: Clustering |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB TwitterSemEval2015 (default) |
|
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
|
split: test |
|
|
type: mteb/twittersemeval2015-pairclassification |
|
|
metrics: |
|
|
- type: similarity_accuracy |
|
|
value: 82.51180000000001 |
|
|
- type: similarity_accuracy_threshold |
|
|
value: 69.4516 |
|
|
- type: similarity_f1 |
|
|
value: 58.483399999999996 |
|
|
- type: similarity_f1_threshold |
|
|
value: 61.3852 |
|
|
- type: similarity_precision |
|
|
value: 56.29880000000001 |
|
|
- type: similarity_recall |
|
|
value: 60.8443 |
|
|
- type: similarity_ap |
|
|
value: 61.8784 |
|
|
- type: cosine_accuracy |
|
|
value: 82.51180000000001 |
|
|
- type: cosine_accuracy_threshold |
|
|
value: 69.4516 |
|
|
- type: cosine_f1 |
|
|
value: 58.483399999999996 |
|
|
- type: cosine_f1_threshold |
|
|
value: 61.3852 |
|
|
- type: cosine_precision |
|
|
value: 56.29880000000001 |
|
|
- type: cosine_recall |
|
|
value: 60.8443 |
|
|
- type: cosine_ap |
|
|
value: 61.8784 |
|
|
- type: manhattan_accuracy |
|
|
value: 82.60119999999999 |
|
|
- type: manhattan_accuracy_threshold |
|
|
value: 1395.2354 |
|
|
- type: manhattan_f1 |
|
|
value: 59.3387 |
|
|
- type: manhattan_f1_threshold |
|
|
value: 1544.4108 |
|
|
- type: manhattan_precision |
|
|
value: 56.284 |
|
|
- type: manhattan_recall |
|
|
value: 62.7441 |
|
|
- type: manhattan_ap |
|
|
value: 62.407999999999994 |
|
|
- type: euclidean_accuracy |
|
|
value: 82.51180000000001 |
|
|
- type: euclidean_accuracy_threshold |
|
|
value: 78.1645 |
|
|
- type: euclidean_f1 |
|
|
value: 58.483399999999996 |
|
|
- type: euclidean_f1_threshold |
|
|
value: 87.88040000000001 |
|
|
- type: euclidean_precision |
|
|
value: 56.29880000000001 |
|
|
- type: euclidean_recall |
|
|
value: 60.8443 |
|
|
- type: euclidean_ap |
|
|
value: 61.8784 |
|
|
- type: dot_accuracy |
|
|
value: 82.51180000000001 |
|
|
- type: dot_accuracy_threshold |
|
|
value: 69.4516 |
|
|
- type: dot_f1 |
|
|
value: 58.483399999999996 |
|
|
- type: dot_f1_threshold |
|
|
value: 61.3852 |
|
|
- type: dot_precision |
|
|
value: 56.29880000000001 |
|
|
- type: dot_recall |
|
|
value: 60.8443 |
|
|
- type: dot_ap |
|
|
value: 61.8784 |
|
|
- type: max_accuracy |
|
|
value: 82.60119999999999 |
|
|
- type: max_f1 |
|
|
value: 59.3387 |
|
|
- type: max_precision |
|
|
value: 56.29880000000001 |
|
|
- type: max_recall |
|
|
value: 62.7441 |
|
|
- type: max_ap |
|
|
value: 62.407999999999994 |
|
|
- type: main_score |
|
|
value: 62.407999999999994 |
|
|
task: |
|
|
type: PairClassification |
|
|
- dataset: |
|
|
config: default |
|
|
name: MTEB TwitterURLCorpus (default) |
|
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
|
split: test |
|
|
type: mteb/twitterurlcorpus-pairclassification |
|
|
metrics: |
|
|
- type: similarity_accuracy |
|
|
value: 87.84880000000001 |
|
|
- type: similarity_accuracy_threshold |
|
|
value: 62.77890000000001 |
|
|
- type: similarity_f1 |
|
|
value: 75.968 |
|
|
- type: similarity_f1_threshold |
|
|
value: 57.5925 |
|
|
- type: similarity_precision |
|
|
value: 71.909 |
|
|
- type: similarity_recall |
|
|
value: 80.5128 |
|
|
- type: similarity_ap |
|
|
value: 83.6557 |
|
|
- type: cosine_accuracy |
|
|
value: 87.84880000000001 |
|
|
- type: cosine_accuracy_threshold |
|
|
value: 62.77890000000001 |
|
|
- type: cosine_f1 |
|
|
value: 75.968 |
|
|
- type: cosine_f1_threshold |
|
|
value: 57.5925 |
|
|
- type: cosine_precision |
|
|
value: 71.909 |
|
|
- type: cosine_recall |
|
|
value: 80.5128 |
|
|
- type: cosine_ap |
|
|
value: 83.6557 |
|
|
- type: manhattan_accuracy |
|
|
value: 87.69940000000001 |
|
|
- type: manhattan_accuracy_threshold |
|
|
value: 1524.1733 |
|
|
- type: manhattan_f1 |
|
|
value: 76.01830000000001 |
|
|
- type: manhattan_f1_threshold |
|
|
value: 1597.1845 |
|
|
- type: manhattan_precision |
|
|
value: 72.981 |
|
|
- type: manhattan_recall |
|
|
value: 79.3194 |
|
|
- type: manhattan_ap |
|
|
value: 83.63629999999999 |
|
|
- type: euclidean_accuracy |
|
|
value: 87.84880000000001 |
|
|
- type: euclidean_accuracy_threshold |
|
|
value: 86.2799 |
|
|
- type: euclidean_f1 |
|
|
value: 75.968 |
|
|
- type: euclidean_f1_threshold |
|
|
value: 92.0951 |
|
|
- type: euclidean_precision |
|
|
value: 71.909 |
|
|
- type: euclidean_recall |
|
|
value: 80.5128 |
|
|
- type: euclidean_ap |
|
|
value: 83.6557 |
|
|
- type: dot_accuracy |
|
|
value: 87.84880000000001 |
|
|
- type: dot_accuracy_threshold |
|
|
value: 62.77890000000001 |
|
|
- type: dot_f1 |
|
|
value: 75.968 |
|
|
- type: dot_f1_threshold |
|
|
value: 57.5925 |
|
|
- type: dot_precision |
|
|
value: 71.909 |
|
|
- type: dot_recall |
|
|
value: 80.5128 |
|
|
- type: dot_ap |
|
|
value: 83.6557 |
|
|
- type: max_accuracy |
|
|
value: 87.84880000000001 |
|
|
- type: max_f1 |
|
|
value: 76.01830000000001 |
|
|
- type: max_precision |
|
|
value: 72.981 |
|
|
- type: max_recall |
|
|
value: 80.5128 |
|
|
- type: max_ap |
|
|
value: 83.6557 |
|
|
- type: main_score |
|
|
value: 83.6557 |
|
|
task: |
|
|
type: PairClassification |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# RetrievaEmbedding-01: AMBER |
|
|
|
|
|
The **AMBER (Adaptive Multitask Bilingual Embedding Representations)** is a text embedding model trained by Retrieva, Inc. |
|
|
This model is primarily designed for Japanese, but it also supports English. |
|
|
We trained this model on various datasets related to Japanese and English. |
|
|
|
|
|
This model size is 132M parameters (base size). |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
The AMBER model is a text embedding model based on the [sbintuitions/modernbert-ja-130m](https://huggingface.co/sbintuitions/modernbert-ja-130m) architecture, designed for Japanese text. |
|
|
This model was trained on a variety of datasets related to Japanese, and also includes English datasets. |
|
|
The model can be used for English text as well. |
|
|
During training, prompts (instructions) in natural language were included, allowing the model to generate embeddings tailored to specific tasks. |
|
|
|
|
|
- **Developed by:** Retrieva, Inc. |
|
|
- **Model type:** Based on the [ModernBERT](https://arxiv.org/abs/2412.13663) Architecture. |
|
|
- **Language(s) (NLP):** Primarily Japanese (optional support for English). |
|
|
- **License:** Apache 2.0 |
|
|
- **Finetuned from model:** `sbintuitions/modernbert-ja-130m` |
|
|
- **Model Type:** Sentence Transformer |
|
|
- **Maximum Sequence Length:** 512 tokens |
|
|
- **Output Dimensionality:** 512 dimensions |
|
|
- **Similarity Function:** Cosine Similarity |
|
|
|
|
|
## Uses |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
### Install Library |
|
|
|
|
|
First install the python library using pip: |
|
|
|
|
|
```bash |
|
|
pip install sentence-transformers sentencepiece |
|
|
``` |
|
|
|
|
|
### Run Inference |
|
|
|
|
|
Then you can load this model and run inference. |
|
|
|
|
|
You can specify the prompt at inference time by adding an argument called `prompt` to `model.encode`. |
|
|
The prompts used in the Japanese benchmark are described in `jmteb/tasks`, and the prompts used in the English benchmark are described in `mteb/models/retrieva_en.py`. |
|
|
|
|
|
```python |
|
|
from sentence_transformers import SentenceTransformer |
|
|
|
|
|
# Download from the 🤗 Hub |
|
|
model = SentenceTransformer("retrieva-jp/amber-base") |
|
|
# Run inference |
|
|
queries = [ |
|
|
"自然言語処理とはなんですか？", |
|
|
"株式会社レトリバについて教えて", |
|
|
] |
|
|
documents = [ |
|
|
"自然言語処理（しぜんげんごしょり、英語: Natural language processing、略称：NLP）は、人間が日常的に使っている自然言語をコンピュータに処理させる一連の技術であり、人工知能と言語学の一分野である。", |
|
|
"株式会社レトリバは、自然言語処理と機械学習を核としたAI技術で組織の課題解決を支援するテクノロジー企業である。", |
|
|
] |
|
|
|
|
|
queries_embeddings = model.encode(queries, prompt_name="Retrieval-query") |
|
|
documents_embeddings = model.encode(documents, prompt_name="Retrieval-passage") |
|
|
|
|
|
similarities = model.similarity(queries_embeddings, documents_embeddings) |
|
|
print(similarities.shape) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
We used multiple datasets to train this model. |
|
|
We selected datasets from [llm-jp-eval](https://github.com/llm-jp/llm-jp-eval), [llm-japanese-dataset](https://github.com/masanorihirano/llm-japanese-dataset), and [hpprc/emb](https://huggingface.co/datasets/hpprc/emb) for Japanese datasets. |
|
|
For English datasets, we mainly used some of the datasets utilized in [Asai et al. (2023)](https://arxiv.org/abs/2211.09260). |
|
|
Additionally, we partially used the English datasets at [the sentence-transformers repository](https://huggingface.co/sentence-transformers) and [kilt-tasks](https://huggingface.co/datasets/facebook/kilt_tasks). |
|
|
To consider cross-lingual between Japanese and English, we also used translation datasets between Japanese and English. |
|
|
|
|
|
For Japanese, we used synthetic data created by LLM to prepare a sufficient amount of training data. |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
We evaluated the model on the following benchmarks: |
|
|
- Japanese Benchmark: [JMTEB](https://github.com/sbintuitions/JMTEB) |
|
|
- Japanese Retrieval Tasks: [JQaRA](https://github.com/hotchpotch/JQaRA/), [JaCWIR](https://github.com/hotchpotch/JaCWIR/), [MLDR Japanese Subset](https://huggingface.co/datasets/Shitao/MLDR) |
|
|
- English Benchmark: [MTEB(eng, v2)](https://github.com/embeddings-benchmark/mteb). |
|
|
|
|
|
The scores in the table are all calculated by us unless otherwise noted. |
|
|
|
|
|
### Japanese Benchmark: JMTEB |
|
|
|
|
|
Note that the `Mean (TaskType)` in the following leaderboard is the same as the `Avg.` in the original JMTEB leaderboard. |
|
|
|
|
|
The files used for evaluation are stored in the `jmteb` directory. |
|
|
|
|
|
| Model | # Parameters | Mean (TaskType) | Mean (Task) | Retrieval | STS | Classification | Reranking | Clustering | PairClassification | |
|
|
| :--- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
|
|
| base models | < 300M | | | | | | | | | |
|
|
| [cl-nagoya/ruri-base](https://huggingface.co/cl-nagoya/ruri-base) | 111M | 72.60 | 71.56 | 69.53 | 82.87 | 75.49 | 92.91 | 52.40 | 62.38 | |
|
|
| AMBER-base <br> (this model) | 130M | 72.12 | 72.12 | **73.40** | 77.81 | **76.14** | **93.27** | 48.05 | **64.03** | |
|
|
| [pkshatech/GLuCoSE-base-ja-v2](https://huggingface.co/pkshatech/GLuCoSE-base-ja-v2) | 133M | **72.89** | **72.47** | 73.03 | **82.96** | 74.02 | 93.01 | 51.96 | 62.37 | |
|
|
| [pkshatech/RoSEtta-base-ja](https://huggingface.co/pkshatech/RoSEtta-base-ja) | 190M | 72.49 | 72.05 | 73.14 | 81.39 | 72.37 | 92.69 | **53.60** | 61.74 | |
|
|
| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 278M | 71.11 | 69.72 | 69.45 | 80.45 | 69.86 | 92.90 | 51.62 | 62.35 | |
|
|
| large models | 300M < | | | | | | | | | |
|
|
| [AMBER-large](https://huggingface.co/retrieva-jp/amber-large) | 315M | 72.52 | **73.22** | **75.40** | 79.32 | 77.14 | **93.54** | 48.73 | 60.97 | |
|
|
| [cl-nagoya/ruri-large](https://huggingface.co/cl-nagoya/ruri-large) | 337M | **73.20** | 73.06 | 72.86 | **83.14** | **77.15** | 93.00 | 50.78 | 62.29 | |
|
|
| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 560M | 72.06 | 71.29 | 71.71 | 80.87 | 72.45 | 93.29 | **51.59** | **62.42** | |
|
|
|
|
|
|
|
|
### Japanese Retrieval Tasks: JQaRA, JaCWIR, MLDR Japanese Subset |
|
|
|
|
|
The files used for MLDR are stored in the `mldr` directory. |
|
|
|
|
|
The prompts used in JQaRA and JaCWIR are `Retrieval-query` and `Retrieval-passage` described in `config_sentence_transformers.json`. |
|
|
|
|
|
| Model | # Parameters | JQaRA (nDCG@10) | JaCWIR (MAP@10) | MLDR Japanese Subset (nDCG@10) | |
|
|
| :--- | --- | ---: | ---: | ---: | |
|
|
| base models | < 300M | | | | |
|
|
| [cl-nagoya/ruri-base](https://huggingface.co/cl-nagoya/ruri-base) | 111M | 58.4 | 83.3 | 32.77 | |
|
|
| AMBER-base <br> (this model) | 130M | 57.1 | 81.6 | **35.69** | |
|
|
| [pkshatech/GLuCoSE-base-ja-v2](https://huggingface.co/pkshatech/GLuCoSE-base-ja-v2) | 133M | **60.6** | **85.3** | 33.99 | |
|
|
| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 278M | 47.1 | **85.3** | 25.46 | |
|
|
| large models | 300M < | | | | |
|
|
| [AMBER-large](https://huggingface.co/retrieva-jp/amber-large) | 315M | 62.5 | 82.4 | 34.57 | |
|
|
| [cl-nagoya/ruri-large](https://huggingface.co/cl-nagoya/ruri-large) | 337M | **62.8** | 82.5 | **34.78** | |
|
|
| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 560M | 55.4| **87.3** | 29.95 | |
|
|
|
|
|
### English Benchmark: MTEB(eng, v2) |
|
|
|
|
|
The files used for evaluation are stored in the `mteb` directory. |
|
|
|
|
|
| Model | # Parameters | Mean (TaskType) | Mean (Task) | Retrieval | STS | Classification | Reranking | Clustering | PairClassification | Summarization | |
|
|
| :--- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
|
|
| base models | < 300M | | | | | | | | | | |
|
|
| AMBER-base <br> (this model) | 130M | 54.75 | 58.20 | 40.11 | **81.29** | 70.39 | 42.98 | **42.27** | 80.12 | 26.08 | |
|
|
| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 278M | **56.21** | **59.75** | **43.22** | 80.50 | **73.84** | **43.87** | 42.19 | **83.74** | **26.10** | |
|
|
| large models | 300M < | | | | | | | | | | |
|
|
| [AMBER-large](https://huggingface.co/retrieva-jp/amber-large) | 315M | 56.08 | 59.13 | 41.04 | **81.52** | 72.23 | 43.83 | **42.71** | 81.00 | **30.21** | |
|
|
| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 560M | **57.06** | **60.84** | **46.17** | 81.11 | **74.88** | **44.31** | 41.91 | **84.33** | 26.67 | |
|
|
|
|
|
## Citation |
|
|
|
|
|
**BibTeX:** |
|
|
|
|
|
```bibtex |
|
|
@inproceedings{amber2025, |
|
|
title = {インストラクションと複数タスクを利用した日本語向け分散表現モデルの構築}, |
|
|
author = {勝又智 and 木村大翼 and 西鳥羽二郎}, |
|
|
booktitle = {言語処理学会第31回年次大会発表論文集}, |
|
|
year = {2025}, |
|
|
} |
|
|
``` |
|
|
|
|
|
## More Information |
|
|
|
|
|
https://note.com/retrieva/n/n4ee9d304f44d |
|
|
(in Japanese) |
|
|
|
|
|
## Model Card Authors |
|
|
|
|
|
Satoru Katsumata, Daisuke Kimura, Jiro Nishitoba |
|
|
|
|
|
## Model Card Contact |
|
|
|
|
|
pr[at]retrieva.jp |
