Severian commited on
Commit
0b64fb9
·
1 Parent(s): c1824c4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -2748
README.md CHANGED
@@ -1,2748 +0,0 @@
1
- ---
2
- tags:
3
- - finetuner
4
- - mteb
5
- - sentence-transformers
6
- - feature-extraction
7
- - sentence-similarity
8
- - alibi
9
- datasets:
10
- - allenai/c4
11
- language: en
12
- license: apache-2.0
13
- model-index:
14
- - name: jina-embedding-b-en-v2
15
- results:
16
- - task:
17
- type: Classification
18
- dataset:
19
- type: mteb/amazon_counterfactual
20
- name: MTEB AmazonCounterfactualClassification (en)
21
- config: en
22
- split: test
23
- revision: e8379541af4e31359cca9fbcf4b00f2671dba205
24
- metrics:
25
- - type: accuracy
26
- value: 74.73134328358209
27
- - type: ap
28
- value: 37.765427081831035
29
- - type: f1
30
- value: 68.79367444339518
31
- - task:
32
- type: Classification
33
- dataset:
34
- type: mteb/amazon_polarity
35
- name: MTEB AmazonPolarityClassification
36
- config: default
37
- split: test
38
- revision: e2d317d38cd51312af73b3d32a06d1a08b442046
39
- metrics:
40
- - type: accuracy
41
- value: 88.544275
42
- - type: ap
43
- value: 84.61328675662887
44
- - type: f1
45
- value: 88.51879035862375
46
- - task:
47
- type: Classification
48
- dataset:
49
- type: mteb/amazon_reviews_multi
50
- name: MTEB AmazonReviewsClassification (en)
51
- config: en
52
- split: test
53
- revision: 1399c76144fd37290681b995c656ef9b2e06e26d
54
- metrics:
55
- - type: accuracy
56
- value: 45.263999999999996
57
- - type: f1
58
- value: 43.778759656699435
59
- - task:
60
- type: Retrieval
61
- dataset:
62
- type: arguana
63
- name: MTEB ArguAna
64
- config: default
65
- split: test
66
- revision: None
67
- metrics:
68
- - type: map_at_1
69
- value: 21.693
70
- - type: map_at_10
71
- value: 35.487
72
- - type: map_at_100
73
- value: 36.862
74
- - type: map_at_1000
75
- value: 36.872
76
- - type: map_at_3
77
- value: 30.049999999999997
78
- - type: map_at_5
79
- value: 32.966
80
- - type: mrr_at_1
81
- value: 21.977
82
- - type: mrr_at_10
83
- value: 35.565999999999995
84
- - type: mrr_at_100
85
- value: 36.948
86
- - type: mrr_at_1000
87
- value: 36.958
88
- - type: mrr_at_3
89
- value: 30.121
90
- - type: mrr_at_5
91
- value: 33.051
92
- - type: ndcg_at_1
93
- value: 21.693
94
- - type: ndcg_at_10
95
- value: 44.181
96
- - type: ndcg_at_100
97
- value: 49.982
98
- - type: ndcg_at_1000
99
- value: 50.233000000000004
100
- - type: ndcg_at_3
101
- value: 32.830999999999996
102
- - type: ndcg_at_5
103
- value: 38.080000000000005
104
- - type: precision_at_1
105
- value: 21.693
106
- - type: precision_at_10
107
- value: 7.248
108
- - type: precision_at_100
109
- value: 0.9769999999999999
110
- - type: precision_at_1000
111
- value: 0.1
112
- - type: precision_at_3
113
- value: 13.632
114
- - type: precision_at_5
115
- value: 10.725
116
- - type: recall_at_1
117
- value: 21.693
118
- - type: recall_at_10
119
- value: 72.475
120
- - type: recall_at_100
121
- value: 97.653
122
- - type: recall_at_1000
123
- value: 99.57300000000001
124
- - type: recall_at_3
125
- value: 40.896
126
- - type: recall_at_5
127
- value: 53.627
128
- - task:
129
- type: Clustering
130
- dataset:
131
- type: mteb/arxiv-clustering-p2p
132
- name: MTEB ArxivClusteringP2P
133
- config: default
134
- split: test
135
- revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
136
- metrics:
137
- - type: v_measure
138
- value: 45.39242428696777
139
- - task:
140
- type: Clustering
141
- dataset:
142
- type: mteb/arxiv-clustering-s2s
143
- name: MTEB ArxivClusteringS2S
144
- config: default
145
- split: test
146
- revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53
147
- metrics:
148
- - type: v_measure
149
- value: 36.675626784714
150
- - task:
151
- type: Reranking
152
- dataset:
153
- type: mteb/askubuntudupquestions-reranking
154
- name: MTEB AskUbuntuDupQuestions
155
- config: default
156
- split: test
157
- revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
158
- metrics:
159
- - type: map
160
- value: 62.247725694904034
161
- - type: mrr
162
- value: 74.91359978894604
163
- - task:
164
- type: STS
165
- dataset:
166
- type: mteb/biosses-sts
167
- name: MTEB BIOSSES
168
- config: default
169
- split: test
170
- revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
171
- metrics:
172
- - type: cos_sim_pearson
173
- value: 82.68003802970496
174
- - type: cos_sim_spearman
175
- value: 81.23438110096286
176
- - type: euclidean_pearson
177
- value: 81.87462986142582
178
- - type: euclidean_spearman
179
- value: 81.23438110096286
180
- - type: manhattan_pearson
181
- value: 81.61162566600755
182
- - type: manhattan_spearman
183
- value: 81.11329400456184
184
- - task:
185
- type: Classification
186
- dataset:
187
- type: mteb/banking77
188
- name: MTEB Banking77Classification
189
- config: default
190
- split: test
191
- revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
192
- metrics:
193
- - type: accuracy
194
- value: 84.01298701298701
195
- - type: f1
196
- value: 83.31690714969382
197
- - task:
198
- type: Clustering
199
- dataset:
200
- type: mteb/biorxiv-clustering-p2p
201
- name: MTEB BiorxivClusteringP2P
202
- config: default
203
- split: test
204
- revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
205
- metrics:
206
- - type: v_measure
207
- value: 37.050108150972086
208
- - task:
209
- type: Clustering
210
- dataset:
211
- type: mteb/biorxiv-clustering-s2s
212
- name: MTEB BiorxivClusteringS2S
213
- config: default
214
- split: test
215
- revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
216
- metrics:
217
- - type: v_measure
218
- value: 30.15731442819715
219
- - task:
220
- type: Retrieval
221
- dataset:
222
- type: BeIR/cqadupstack
223
- name: MTEB CQADupstackAndroidRetrieval
224
- config: default
225
- split: test
226
- revision: None
227
- metrics:
228
- - type: map_at_1
229
- value: 31.391999999999996
230
- - type: map_at_10
231
- value: 42.597
232
- - type: map_at_100
233
- value: 44.07
234
- - type: map_at_1000
235
- value: 44.198
236
- - type: map_at_3
237
- value: 38.957
238
- - type: map_at_5
239
- value: 40.961
240
- - type: mrr_at_1
241
- value: 37.196
242
- - type: mrr_at_10
243
- value: 48.152
244
- - type: mrr_at_100
245
- value: 48.928
246
- - type: mrr_at_1000
247
- value: 48.964999999999996
248
- - type: mrr_at_3
249
- value: 45.446
250
- - type: mrr_at_5
251
- value: 47.205999999999996
252
- - type: ndcg_at_1
253
- value: 37.196
254
- - type: ndcg_at_10
255
- value: 49.089
256
- - type: ndcg_at_100
257
- value: 54.471000000000004
258
- - type: ndcg_at_1000
259
- value: 56.385
260
- - type: ndcg_at_3
261
- value: 43.699
262
- - type: ndcg_at_5
263
- value: 46.22
264
- - type: precision_at_1
265
- value: 37.196
266
- - type: precision_at_10
267
- value: 9.313
268
- - type: precision_at_100
269
- value: 1.478
270
- - type: precision_at_1000
271
- value: 0.198
272
- - type: precision_at_3
273
- value: 20.839
274
- - type: precision_at_5
275
- value: 14.936
276
- - type: recall_at_1
277
- value: 31.391999999999996
278
- - type: recall_at_10
279
- value: 61.876
280
- - type: recall_at_100
281
- value: 84.214
282
- - type: recall_at_1000
283
- value: 95.985
284
- - type: recall_at_3
285
- value: 46.6
286
- - type: recall_at_5
287
- value: 53.588
288
- - task:
289
- type: Retrieval
290
- dataset:
291
- type: BeIR/cqadupstack
292
- name: MTEB CQADupstackEnglishRetrieval
293
- config: default
294
- split: test
295
- revision: None
296
- metrics:
297
- - type: map_at_1
298
- value: 29.083
299
- - type: map_at_10
300
- value: 38.812999999999995
301
- - type: map_at_100
302
- value: 40.053
303
- - type: map_at_1000
304
- value: 40.188
305
- - type: map_at_3
306
- value: 36.111
307
- - type: map_at_5
308
- value: 37.519000000000005
309
- - type: mrr_at_1
310
- value: 36.497
311
- - type: mrr_at_10
312
- value: 44.85
313
- - type: mrr_at_100
314
- value: 45.546
315
- - type: mrr_at_1000
316
- value: 45.593
317
- - type: mrr_at_3
318
- value: 42.686
319
- - type: mrr_at_5
320
- value: 43.909
321
- - type: ndcg_at_1
322
- value: 36.497
323
- - type: ndcg_at_10
324
- value: 44.443
325
- - type: ndcg_at_100
326
- value: 48.979
327
- - type: ndcg_at_1000
328
- value: 51.154999999999994
329
- - type: ndcg_at_3
330
- value: 40.660000000000004
331
- - type: ndcg_at_5
332
- value: 42.193000000000005
333
- - type: precision_at_1
334
- value: 36.497
335
- - type: precision_at_10
336
- value: 8.433
337
- - type: precision_at_100
338
- value: 1.369
339
- - type: precision_at_1000
340
- value: 0.185
341
- - type: precision_at_3
342
- value: 19.894000000000002
343
- - type: precision_at_5
344
- value: 13.873
345
- - type: recall_at_1
346
- value: 29.083
347
- - type: recall_at_10
348
- value: 54.313
349
- - type: recall_at_100
350
- value: 73.792
351
- - type: recall_at_1000
352
- value: 87.629
353
- - type: recall_at_3
354
- value: 42.257
355
- - type: recall_at_5
356
- value: 47.066
357
- - task:
358
- type: Retrieval
359
- dataset:
360
- type: BeIR/cqadupstack
361
- name: MTEB CQADupstackGamingRetrieval
362
- config: default
363
- split: test
364
- revision: None
365
- metrics:
366
- - type: map_at_1
367
- value: 38.556000000000004
368
- - type: map_at_10
369
- value: 50.698
370
- - type: map_at_100
371
- value: 51.705
372
- - type: map_at_1000
373
- value: 51.768
374
- - type: map_at_3
375
- value: 47.848
376
- - type: map_at_5
377
- value: 49.358000000000004
378
- - type: mrr_at_1
379
- value: 43.95
380
- - type: mrr_at_10
381
- value: 54.191
382
- - type: mrr_at_100
383
- value: 54.852999999999994
384
- - type: mrr_at_1000
385
- value: 54.885
386
- - type: mrr_at_3
387
- value: 51.954
388
- - type: mrr_at_5
389
- value: 53.13
390
- - type: ndcg_at_1
391
- value: 43.95
392
- - type: ndcg_at_10
393
- value: 56.516
394
- - type: ndcg_at_100
395
- value: 60.477000000000004
396
- - type: ndcg_at_1000
397
- value: 61.746
398
- - type: ndcg_at_3
399
- value: 51.601
400
- - type: ndcg_at_5
401
- value: 53.795
402
- - type: precision_at_1
403
- value: 43.95
404
- - type: precision_at_10
405
- value: 9.009
406
- - type: precision_at_100
407
- value: 1.189
408
- - type: precision_at_1000
409
- value: 0.135
410
- - type: precision_at_3
411
- value: 22.989
412
- - type: precision_at_5
413
- value: 15.473
414
- - type: recall_at_1
415
- value: 38.556000000000004
416
- - type: recall_at_10
417
- value: 70.159
418
- - type: recall_at_100
419
- value: 87.132
420
- - type: recall_at_1000
421
- value: 96.16
422
- - type: recall_at_3
423
- value: 56.906
424
- - type: recall_at_5
425
- value: 62.332
426
- - task:
427
- type: Retrieval
428
- dataset:
429
- type: BeIR/cqadupstack
430
- name: MTEB CQADupstackGisRetrieval
431
- config: default
432
- split: test
433
- revision: None
434
- metrics:
435
- - type: map_at_1
436
- value: 24.238
437
- - type: map_at_10
438
- value: 32.5
439
- - type: map_at_100
440
- value: 33.637
441
- - type: map_at_1000
442
- value: 33.719
443
- - type: map_at_3
444
- value: 30.026999999999997
445
- - type: map_at_5
446
- value: 31.555
447
- - type: mrr_at_1
448
- value: 26.328000000000003
449
- - type: mrr_at_10
450
- value: 34.44
451
- - type: mrr_at_100
452
- value: 35.455999999999996
453
- - type: mrr_at_1000
454
- value: 35.521
455
- - type: mrr_at_3
456
- value: 32.034
457
- - type: mrr_at_5
458
- value: 33.565
459
- - type: ndcg_at_1
460
- value: 26.328000000000003
461
- - type: ndcg_at_10
462
- value: 37.202
463
- - type: ndcg_at_100
464
- value: 42.728
465
- - type: ndcg_at_1000
466
- value: 44.792
467
- - type: ndcg_at_3
468
- value: 32.368
469
- - type: ndcg_at_5
470
- value: 35.008
471
- - type: precision_at_1
472
- value: 26.328000000000003
473
- - type: precision_at_10
474
- value: 5.7059999999999995
475
- - type: precision_at_100
476
- value: 0.8880000000000001
477
- - type: precision_at_1000
478
- value: 0.11100000000000002
479
- - type: precision_at_3
480
- value: 13.672
481
- - type: precision_at_5
482
- value: 9.74
483
- - type: recall_at_1
484
- value: 24.238
485
- - type: recall_at_10
486
- value: 49.829
487
- - type: recall_at_100
488
- value: 75.21
489
- - type: recall_at_1000
490
- value: 90.521
491
- - type: recall_at_3
492
- value: 36.867
493
- - type: recall_at_5
494
- value: 43.241
495
- - task:
496
- type: Retrieval
497
- dataset:
498
- type: BeIR/cqadupstack
499
- name: MTEB CQADupstackMathematicaRetrieval
500
- config: default
501
- split: test
502
- revision: None
503
- metrics:
504
- - type: map_at_1
505
- value: 15.378
506
- - type: map_at_10
507
- value: 22.817999999999998
508
- - type: map_at_100
509
- value: 23.977999999999998
510
- - type: map_at_1000
511
- value: 24.108
512
- - type: map_at_3
513
- value: 20.719
514
- - type: map_at_5
515
- value: 21.889
516
- - type: mrr_at_1
517
- value: 19.03
518
- - type: mrr_at_10
519
- value: 27.022000000000002
520
- - type: mrr_at_100
521
- value: 28.011999999999997
522
- - type: mrr_at_1000
523
- value: 28.096
524
- - type: mrr_at_3
525
- value: 24.855
526
- - type: mrr_at_5
527
- value: 26.029999999999998
528
- - type: ndcg_at_1
529
- value: 19.03
530
- - type: ndcg_at_10
531
- value: 27.526
532
- - type: ndcg_at_100
533
- value: 33.040000000000006
534
- - type: ndcg_at_1000
535
- value: 36.187000000000005
536
- - type: ndcg_at_3
537
- value: 23.497
538
- - type: ndcg_at_5
539
- value: 25.334
540
- - type: precision_at_1
541
- value: 19.03
542
- - type: precision_at_10
543
- value: 4.963
544
- - type: precision_at_100
545
- value: 0.893
546
- - type: precision_at_1000
547
- value: 0.13
548
- - type: precision_at_3
549
- value: 11.360000000000001
550
- - type: precision_at_5
551
- value: 8.134
552
- - type: recall_at_1
553
- value: 15.378
554
- - type: recall_at_10
555
- value: 38.061
556
- - type: recall_at_100
557
- value: 61.754
558
- - type: recall_at_1000
559
- value: 84.259
560
- - type: recall_at_3
561
- value: 26.788
562
- - type: recall_at_5
563
- value: 31.326999999999998
564
- - task:
565
- type: Retrieval
566
- dataset:
567
- type: BeIR/cqadupstack
568
- name: MTEB CQADupstackPhysicsRetrieval
569
- config: default
570
- split: test
571
- revision: None
572
- metrics:
573
- - type: map_at_1
574
- value: 27.511999999999997
575
- - type: map_at_10
576
- value: 37.429
577
- - type: map_at_100
578
- value: 38.818000000000005
579
- - type: map_at_1000
580
- value: 38.924
581
- - type: map_at_3
582
- value: 34.625
583
- - type: map_at_5
584
- value: 36.064
585
- - type: mrr_at_1
586
- value: 33.300999999999995
587
- - type: mrr_at_10
588
- value: 43.036
589
- - type: mrr_at_100
590
- value: 43.894
591
- - type: mrr_at_1000
592
- value: 43.936
593
- - type: mrr_at_3
594
- value: 40.825
595
- - type: mrr_at_5
596
- value: 42.028
597
- - type: ndcg_at_1
598
- value: 33.300999999999995
599
- - type: ndcg_at_10
600
- value: 43.229
601
- - type: ndcg_at_100
602
- value: 48.992000000000004
603
- - type: ndcg_at_1000
604
- value: 51.02100000000001
605
- - type: ndcg_at_3
606
- value: 38.794000000000004
607
- - type: ndcg_at_5
608
- value: 40.65
609
- - type: precision_at_1
610
- value: 33.300999999999995
611
- - type: precision_at_10
612
- value: 7.777000000000001
613
- - type: precision_at_100
614
- value: 1.269
615
- - type: precision_at_1000
616
- value: 0.163
617
- - type: precision_at_3
618
- value: 18.351
619
- - type: precision_at_5
620
- value: 12.762
621
- - type: recall_at_1
622
- value: 27.511999999999997
623
- - type: recall_at_10
624
- value: 54.788000000000004
625
- - type: recall_at_100
626
- value: 79.105
627
- - type: recall_at_1000
628
- value: 92.49199999999999
629
- - type: recall_at_3
630
- value: 41.924
631
- - type: recall_at_5
632
- value: 47.026
633
- - task:
634
- type: Retrieval
635
- dataset:
636
- type: BeIR/cqadupstack
637
- name: MTEB CQADupstackProgrammersRetrieval
638
- config: default
639
- split: test
640
- revision: None
641
- metrics:
642
- - type: map_at_1
643
- value: 24.117
644
- - type: map_at_10
645
- value: 33.32
646
- - type: map_at_100
647
- value: 34.677
648
- - type: map_at_1000
649
- value: 34.78
650
- - type: map_at_3
651
- value: 30.233999999999998
652
- - type: map_at_5
653
- value: 31.668000000000003
654
- - type: mrr_at_1
655
- value: 29.566
656
- - type: mrr_at_10
657
- value: 38.244
658
- - type: mrr_at_100
659
- value: 39.245000000000005
660
- - type: mrr_at_1000
661
- value: 39.296
662
- - type: mrr_at_3
663
- value: 35.864000000000004
664
- - type: mrr_at_5
665
- value: 36.919999999999995
666
- - type: ndcg_at_1
667
- value: 29.566
668
- - type: ndcg_at_10
669
- value: 39.127
670
- - type: ndcg_at_100
671
- value: 44.989000000000004
672
- - type: ndcg_at_1000
673
- value: 47.189
674
- - type: ndcg_at_3
675
- value: 34.039
676
- - type: ndcg_at_5
677
- value: 35.744
678
- - type: precision_at_1
679
- value: 29.566
680
- - type: precision_at_10
681
- value: 7.385999999999999
682
- - type: precision_at_100
683
- value: 1.204
684
- - type: precision_at_1000
685
- value: 0.158
686
- - type: precision_at_3
687
- value: 16.286
688
- - type: precision_at_5
689
- value: 11.484
690
- - type: recall_at_1
691
- value: 24.117
692
- - type: recall_at_10
693
- value: 51.559999999999995
694
- - type: recall_at_100
695
- value: 77.104
696
- - type: recall_at_1000
697
- value: 91.79899999999999
698
- - type: recall_at_3
699
- value: 36.82
700
- - type: recall_at_5
701
- value: 41.453
702
- - task:
703
- type: Retrieval
704
- dataset:
705
- type: BeIR/cqadupstack
706
- name: MTEB CQADupstackRetrieval
707
- config: default
708
- split: test
709
- revision: None
710
- metrics:
711
- - type: map_at_1
712
- value: 25.17625
713
- - type: map_at_10
714
- value: 34.063916666666664
715
- - type: map_at_100
716
- value: 35.255500000000005
717
- - type: map_at_1000
718
- value: 35.37275
719
- - type: map_at_3
720
- value: 31.351666666666667
721
- - type: map_at_5
722
- value: 32.80608333333333
723
- - type: mrr_at_1
724
- value: 29.59783333333333
725
- - type: mrr_at_10
726
- value: 38.0925
727
- - type: mrr_at_100
728
- value: 38.957249999999995
729
- - type: mrr_at_1000
730
- value: 39.01608333333333
731
- - type: mrr_at_3
732
- value: 35.77625
733
- - type: mrr_at_5
734
- value: 37.04991666666667
735
- - type: ndcg_at_1
736
- value: 29.59783333333333
737
- - type: ndcg_at_10
738
- value: 39.343666666666664
739
- - type: ndcg_at_100
740
- value: 44.488249999999994
741
- - type: ndcg_at_1000
742
- value: 46.83358333333334
743
- - type: ndcg_at_3
744
- value: 34.69708333333333
745
- - type: ndcg_at_5
746
- value: 36.75075
747
- - type: precision_at_1
748
- value: 29.59783333333333
749
- - type: precision_at_10
750
- value: 6.884083333333332
751
- - type: precision_at_100
752
- value: 1.114
753
- - type: precision_at_1000
754
- value: 0.15108333333333332
755
- - type: precision_at_3
756
- value: 15.965250000000003
757
- - type: precision_at_5
758
- value: 11.246500000000001
759
- - type: recall_at_1
760
- value: 25.17625
761
- - type: recall_at_10
762
- value: 51.015999999999984
763
- - type: recall_at_100
764
- value: 73.60174999999998
765
- - type: recall_at_1000
766
- value: 89.849
767
- - type: recall_at_3
768
- value: 37.88399999999999
769
- - type: recall_at_5
770
- value: 43.24541666666666
771
- - task:
772
- type: Retrieval
773
- dataset:
774
- type: BeIR/cqadupstack
775
- name: MTEB CQADupstackStatsRetrieval
776
- config: default
777
- split: test
778
- revision: None
779
- metrics:
780
- - type: map_at_1
781
- value: 24.537
782
- - type: map_at_10
783
- value: 31.081999999999997
784
- - type: map_at_100
785
- value: 32.042
786
- - type: map_at_1000
787
- value: 32.141
788
- - type: map_at_3
789
- value: 29.137
790
- - type: map_at_5
791
- value: 30.079
792
- - type: mrr_at_1
793
- value: 27.454
794
- - type: mrr_at_10
795
- value: 33.694
796
- - type: mrr_at_100
797
- value: 34.579
798
- - type: mrr_at_1000
799
- value: 34.649
800
- - type: mrr_at_3
801
- value: 32.004
802
- - type: mrr_at_5
803
- value: 32.794000000000004
804
- - type: ndcg_at_1
805
- value: 27.454
806
- - type: ndcg_at_10
807
- value: 34.915
808
- - type: ndcg_at_100
809
- value: 39.641
810
- - type: ndcg_at_1000
811
- value: 42.105
812
- - type: ndcg_at_3
813
- value: 31.276
814
- - type: ndcg_at_5
815
- value: 32.65
816
- - type: precision_at_1
817
- value: 27.454
818
- - type: precision_at_10
819
- value: 5.337
820
- - type: precision_at_100
821
- value: 0.8250000000000001
822
- - type: precision_at_1000
823
- value: 0.11199999999999999
824
- - type: precision_at_3
825
- value: 13.241
826
- - type: precision_at_5
827
- value: 8.895999999999999
828
- - type: recall_at_1
829
- value: 24.537
830
- - type: recall_at_10
831
- value: 44.324999999999996
832
- - type: recall_at_100
833
- value: 65.949
834
- - type: recall_at_1000
835
- value: 84.017
836
- - type: recall_at_3
837
- value: 33.857
838
- - type: recall_at_5
839
- value: 37.316
840
- - task:
841
- type: Retrieval
842
- dataset:
843
- type: BeIR/cqadupstack
844
- name: MTEB CQADupstackTexRetrieval
845
- config: default
846
- split: test
847
- revision: None
848
- metrics:
849
- - type: map_at_1
850
- value: 17.122
851
- - type: map_at_10
852
- value: 24.32
853
- - type: map_at_100
854
- value: 25.338
855
- - type: map_at_1000
856
- value: 25.462
857
- - type: map_at_3
858
- value: 22.064
859
- - type: map_at_5
860
- value: 23.322000000000003
861
- - type: mrr_at_1
862
- value: 20.647
863
- - type: mrr_at_10
864
- value: 27.858
865
- - type: mrr_at_100
866
- value: 28.743999999999996
867
- - type: mrr_at_1000
868
- value: 28.819
869
- - type: mrr_at_3
870
- value: 25.769
871
- - type: mrr_at_5
872
- value: 26.964
873
- - type: ndcg_at_1
874
- value: 20.647
875
- - type: ndcg_at_10
876
- value: 28.849999999999998
877
- - type: ndcg_at_100
878
- value: 33.849000000000004
879
- - type: ndcg_at_1000
880
- value: 36.802
881
- - type: ndcg_at_3
882
- value: 24.799
883
- - type: ndcg_at_5
884
- value: 26.682
885
- - type: precision_at_1
886
- value: 20.647
887
- - type: precision_at_10
888
- value: 5.2170000000000005
889
- - type: precision_at_100
890
- value: 0.906
891
- - type: precision_at_1000
892
- value: 0.134
893
- - type: precision_at_3
894
- value: 11.769
895
- - type: precision_at_5
896
- value: 8.486
897
- - type: recall_at_1
898
- value: 17.122
899
- - type: recall_at_10
900
- value: 38.999
901
- - type: recall_at_100
902
- value: 61.467000000000006
903
- - type: recall_at_1000
904
- value: 82.716
905
- - type: recall_at_3
906
- value: 27.601
907
- - type: recall_at_5
908
- value: 32.471
909
- - task:
910
- type: Retrieval
911
- dataset:
912
- type: BeIR/cqadupstack
913
- name: MTEB CQADupstackUnixRetrieval
914
- config: default
915
- split: test
916
- revision: None
917
- metrics:
918
- - type: map_at_1
919
- value: 24.396
920
- - type: map_at_10
921
- value: 33.415
922
- - type: map_at_100
923
- value: 34.521
924
- - type: map_at_1000
925
- value: 34.631
926
- - type: map_at_3
927
- value: 30.703999999999997
928
- - type: map_at_5
929
- value: 32.166
930
- - type: mrr_at_1
931
- value: 28.825
932
- - type: mrr_at_10
933
- value: 37.397000000000006
934
- - type: mrr_at_100
935
- value: 38.286
936
- - type: mrr_at_1000
937
- value: 38.346000000000004
938
- - type: mrr_at_3
939
- value: 35.028
940
- - type: mrr_at_5
941
- value: 36.32
942
- - type: ndcg_at_1
943
- value: 28.825
944
- - type: ndcg_at_10
945
- value: 38.656
946
- - type: ndcg_at_100
947
- value: 43.856
948
- - type: ndcg_at_1000
949
- value: 46.31
950
- - type: ndcg_at_3
951
- value: 33.793
952
- - type: ndcg_at_5
953
- value: 35.909
954
- - type: precision_at_1
955
- value: 28.825
956
- - type: precision_at_10
957
- value: 6.567
958
- - type: precision_at_100
959
- value: 1.0330000000000001
960
- - type: precision_at_1000
961
- value: 0.135
962
- - type: precision_at_3
963
- value: 15.516
964
- - type: precision_at_5
965
- value: 10.914
966
- - type: recall_at_1
967
- value: 24.396
968
- - type: recall_at_10
969
- value: 50.747
970
- - type: recall_at_100
971
- value: 73.477
972
- - type: recall_at_1000
973
- value: 90.801
974
- - type: recall_at_3
975
- value: 37.1
976
- - type: recall_at_5
977
- value: 42.589
978
- - task:
979
- type: Retrieval
980
- dataset:
981
- type: BeIR/cqadupstack
982
- name: MTEB CQADupstackWebmastersRetrieval
983
- config: default
984
- split: test
985
- revision: None
986
- metrics:
987
- - type: map_at_1
988
- value: 25.072
989
- - type: map_at_10
990
- value: 34.307
991
- - type: map_at_100
992
- value: 35.725
993
- - type: map_at_1000
994
- value: 35.943999999999996
995
- - type: map_at_3
996
- value: 30.906
997
- - type: map_at_5
998
- value: 32.818000000000005
999
- - type: mrr_at_1
1000
- value: 29.644
1001
- - type: mrr_at_10
1002
- value: 38.673
1003
- - type: mrr_at_100
1004
- value: 39.459
1005
- - type: mrr_at_1000
1006
- value: 39.527
1007
- - type: mrr_at_3
1008
- value: 35.771
1009
- - type: mrr_at_5
1010
- value: 37.332
1011
- - type: ndcg_at_1
1012
- value: 29.644
1013
- - type: ndcg_at_10
1014
- value: 40.548
1015
- - type: ndcg_at_100
1016
- value: 45.678999999999995
1017
- - type: ndcg_at_1000
1018
- value: 48.488
1019
- - type: ndcg_at_3
1020
- value: 34.887
1021
- - type: ndcg_at_5
1022
- value: 37.543
1023
- - type: precision_at_1
1024
- value: 29.644
1025
- - type: precision_at_10
1026
- value: 7.688000000000001
1027
- - type: precision_at_100
1028
- value: 1.482
1029
- - type: precision_at_1000
1030
- value: 0.23600000000000002
1031
- - type: precision_at_3
1032
- value: 16.206
1033
- - type: precision_at_5
1034
- value: 12.016
1035
- - type: recall_at_1
1036
- value: 25.072
1037
- - type: recall_at_10
1038
- value: 53.478
1039
- - type: recall_at_100
1040
- value: 76.07300000000001
1041
- - type: recall_at_1000
1042
- value: 93.884
1043
- - type: recall_at_3
1044
- value: 37.583
1045
- - type: recall_at_5
1046
- value: 44.464
1047
- - task:
1048
- type: Retrieval
1049
- dataset:
1050
- type: BeIR/cqadupstack
1051
- name: MTEB CQADupstackWordpressRetrieval
1052
- config: default
1053
- split: test
1054
- revision: None
1055
- metrics:
1056
- - type: map_at_1
1057
- value: 20.712
1058
- - type: map_at_10
1059
- value: 27.467999999999996
1060
- - type: map_at_100
1061
- value: 28.502
1062
- - type: map_at_1000
1063
- value: 28.610000000000003
1064
- - type: map_at_3
1065
- value: 24.887999999999998
1066
- - type: map_at_5
1067
- value: 26.273999999999997
1068
- - type: mrr_at_1
1069
- value: 22.736
1070
- - type: mrr_at_10
1071
- value: 29.553
1072
- - type: mrr_at_100
1073
- value: 30.485
1074
- - type: mrr_at_1000
1075
- value: 30.56
1076
- - type: mrr_at_3
1077
- value: 27.078999999999997
1078
- - type: mrr_at_5
1079
- value: 28.401
1080
- - type: ndcg_at_1
1081
- value: 22.736
1082
- - type: ndcg_at_10
1083
- value: 32.023
1084
- - type: ndcg_at_100
1085
- value: 37.158
1086
- - type: ndcg_at_1000
1087
- value: 39.823
1088
- - type: ndcg_at_3
1089
- value: 26.951999999999998
1090
- - type: ndcg_at_5
1091
- value: 29.281000000000002
1092
- - type: precision_at_1
1093
- value: 22.736
1094
- - type: precision_at_10
1095
- value: 5.213
1096
- - type: precision_at_100
1097
- value: 0.832
1098
- - type: precision_at_1000
1099
- value: 0.116
1100
- - type: precision_at_3
1101
- value: 11.459999999999999
1102
- - type: precision_at_5
1103
- value: 8.244
1104
- - type: recall_at_1
1105
- value: 20.712
1106
- - type: recall_at_10
1107
- value: 44.057
1108
- - type: recall_at_100
1109
- value: 67.944
1110
- - type: recall_at_1000
1111
- value: 87.925
1112
- - type: recall_at_3
1113
- value: 30.305
1114
- - type: recall_at_5
1115
- value: 36.071999999999996
1116
- - task:
1117
- type: Retrieval
1118
- dataset:
1119
- type: climate-fever
1120
- name: MTEB ClimateFEVER
1121
- config: default
1122
- split: test
1123
- revision: None
1124
- metrics:
1125
- - type: map_at_1
1126
- value: 10.181999999999999
1127
- - type: map_at_10
1128
- value: 16.66
1129
- - type: map_at_100
1130
- value: 18.273
1131
- - type: map_at_1000
1132
- value: 18.45
1133
- - type: map_at_3
1134
- value: 14.141
1135
- - type: map_at_5
1136
- value: 15.455
1137
- - type: mrr_at_1
1138
- value: 22.15
1139
- - type: mrr_at_10
1140
- value: 32.062000000000005
1141
- - type: mrr_at_100
1142
- value: 33.116
1143
- - type: mrr_at_1000
1144
- value: 33.168
1145
- - type: mrr_at_3
1146
- value: 28.827
1147
- - type: mrr_at_5
1148
- value: 30.892999999999997
1149
- - type: ndcg_at_1
1150
- value: 22.15
1151
- - type: ndcg_at_10
1152
- value: 23.532
1153
- - type: ndcg_at_100
1154
- value: 30.358
1155
- - type: ndcg_at_1000
1156
- value: 33.783
1157
- - type: ndcg_at_3
1158
- value: 19.222
1159
- - type: ndcg_at_5
1160
- value: 20.919999999999998
1161
- - type: precision_at_1
1162
- value: 22.15
1163
- - type: precision_at_10
1164
- value: 7.185999999999999
1165
- - type: precision_at_100
1166
- value: 1.433
1167
- - type: precision_at_1000
1168
- value: 0.207
1169
- - type: precision_at_3
1170
- value: 13.941
1171
- - type: precision_at_5
1172
- value: 10.906
1173
- - type: recall_at_1
1174
- value: 10.181999999999999
1175
- - type: recall_at_10
1176
- value: 28.104000000000003
1177
- - type: recall_at_100
1178
- value: 51.998999999999995
1179
- - type: recall_at_1000
1180
- value: 71.311
1181
- - type: recall_at_3
1182
- value: 17.698
1183
- - type: recall_at_5
1184
- value: 22.262999999999998
1185
- - task:
1186
- type: Retrieval
1187
- dataset:
1188
- type: dbpedia-entity
1189
- name: MTEB DBPedia
1190
- config: default
1191
- split: test
1192
- revision: None
1193
- metrics:
1194
- - type: map_at_1
1195
- value: 6.669
1196
- - type: map_at_10
1197
- value: 15.552
1198
- - type: map_at_100
1199
- value: 21.865000000000002
1200
- - type: map_at_1000
1201
- value: 23.268
1202
- - type: map_at_3
1203
- value: 11.309
1204
- - type: map_at_5
1205
- value: 13.084000000000001
1206
- - type: mrr_at_1
1207
- value: 55.50000000000001
1208
- - type: mrr_at_10
1209
- value: 66.46600000000001
1210
- - type: mrr_at_100
1211
- value: 66.944
1212
- - type: mrr_at_1000
1213
- value: 66.956
1214
- - type: mrr_at_3
1215
- value: 64.542
1216
- - type: mrr_at_5
1217
- value: 65.717
1218
- - type: ndcg_at_1
1219
- value: 44.75
1220
- - type: ndcg_at_10
1221
- value: 35.049
1222
- - type: ndcg_at_100
1223
- value: 39.073
1224
- - type: ndcg_at_1000
1225
- value: 46.208
1226
- - type: ndcg_at_3
1227
- value: 39.525
1228
- - type: ndcg_at_5
1229
- value: 37.156
1230
- - type: precision_at_1
1231
- value: 55.50000000000001
1232
- - type: precision_at_10
1233
- value: 27.800000000000004
1234
- - type: precision_at_100
1235
- value: 9.013
1236
- - type: precision_at_1000
1237
- value: 1.8800000000000001
1238
- - type: precision_at_3
1239
- value: 42.667
1240
- - type: precision_at_5
1241
- value: 36.0
1242
- - type: recall_at_1
1243
- value: 6.669
1244
- - type: recall_at_10
1245
- value: 21.811
1246
- - type: recall_at_100
1247
- value: 45.112
1248
- - type: recall_at_1000
1249
- value: 67.806
1250
- - type: recall_at_3
1251
- value: 13.373
1252
- - type: recall_at_5
1253
- value: 16.615
1254
- - task:
1255
- type: Classification
1256
- dataset:
1257
- type: mteb/emotion
1258
- name: MTEB EmotionClassification
1259
- config: default
1260
- split: test
1261
- revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
1262
- metrics:
1263
- - type: accuracy
1264
- value: 48.769999999999996
1265
- - type: f1
1266
- value: 42.91448356376592
1267
- - task:
1268
- type: Retrieval
1269
- dataset:
1270
- type: fever
1271
- name: MTEB FEVER
1272
- config: default
1273
- split: test
1274
- revision: None
1275
- metrics:
1276
- - type: map_at_1
1277
- value: 54.013
1278
- - type: map_at_10
1279
- value: 66.239
1280
- - type: map_at_100
1281
- value: 66.62599999999999
1282
- - type: map_at_1000
1283
- value: 66.644
1284
- - type: map_at_3
1285
- value: 63.965
1286
- - type: map_at_5
1287
- value: 65.45400000000001
1288
- - type: mrr_at_1
1289
- value: 58.221000000000004
1290
- - type: mrr_at_10
1291
- value: 70.43700000000001
1292
- - type: mrr_at_100
1293
- value: 70.744
1294
- - type: mrr_at_1000
1295
- value: 70.75099999999999
1296
- - type: mrr_at_3
1297
- value: 68.284
1298
- - type: mrr_at_5
1299
- value: 69.721
1300
- - type: ndcg_at_1
1301
- value: 58.221000000000004
1302
- - type: ndcg_at_10
1303
- value: 72.327
1304
- - type: ndcg_at_100
1305
- value: 73.953
1306
- - type: ndcg_at_1000
1307
- value: 74.312
1308
- - type: ndcg_at_3
1309
- value: 68.062
1310
- - type: ndcg_at_5
1311
- value: 70.56400000000001
1312
- - type: precision_at_1
1313
- value: 58.221000000000004
1314
- - type: precision_at_10
1315
- value: 9.521
1316
- - type: precision_at_100
1317
- value: 1.045
1318
- - type: precision_at_1000
1319
- value: 0.109
1320
- - type: precision_at_3
1321
- value: 27.348
1322
- - type: precision_at_5
1323
- value: 17.794999999999998
1324
- - type: recall_at_1
1325
- value: 54.013
1326
- - type: recall_at_10
1327
- value: 86.957
1328
- - type: recall_at_100
1329
- value: 93.911
1330
- - type: recall_at_1000
1331
- value: 96.38
1332
- - type: recall_at_3
1333
- value: 75.555
1334
- - type: recall_at_5
1335
- value: 81.671
1336
- - task:
1337
- type: Retrieval
1338
- dataset:
1339
- type: fiqa
1340
- name: MTEB FiQA2018
1341
- config: default
1342
- split: test
1343
- revision: None
1344
- metrics:
1345
- - type: map_at_1
1346
- value: 21.254
1347
- - type: map_at_10
1348
- value: 33.723
1349
- - type: map_at_100
1350
- value: 35.574
1351
- - type: map_at_1000
1352
- value: 35.730000000000004
1353
- - type: map_at_3
1354
- value: 29.473
1355
- - type: map_at_5
1356
- value: 31.543
1357
- - type: mrr_at_1
1358
- value: 41.358
1359
- - type: mrr_at_10
1360
- value: 49.498
1361
- - type: mrr_at_100
1362
- value: 50.275999999999996
1363
- - type: mrr_at_1000
1364
- value: 50.308
1365
- - type: mrr_at_3
1366
- value: 47.016000000000005
1367
- - type: mrr_at_5
1368
- value: 48.336
1369
- - type: ndcg_at_1
1370
- value: 41.358
1371
- - type: ndcg_at_10
1372
- value: 41.579
1373
- - type: ndcg_at_100
1374
- value: 48.455
1375
- - type: ndcg_at_1000
1376
- value: 51.165000000000006
1377
- - type: ndcg_at_3
1378
- value: 37.681
1379
- - type: ndcg_at_5
1380
- value: 38.49
1381
- - type: precision_at_1
1382
- value: 41.358
1383
- - type: precision_at_10
1384
- value: 11.543000000000001
1385
- - type: precision_at_100
1386
- value: 1.87
1387
- - type: precision_at_1000
1388
- value: 0.23600000000000002
1389
- - type: precision_at_3
1390
- value: 24.743000000000002
1391
- - type: precision_at_5
1392
- value: 17.994
1393
- - type: recall_at_1
1394
- value: 21.254
1395
- - type: recall_at_10
1396
- value: 48.698
1397
- - type: recall_at_100
1398
- value: 74.588
1399
- - type: recall_at_1000
1400
- value: 91.00200000000001
1401
- - type: recall_at_3
1402
- value: 33.939
1403
- - type: recall_at_5
1404
- value: 39.367000000000004
1405
- - task:
1406
- type: Retrieval
1407
- dataset:
1408
- type: hotpotqa
1409
- name: MTEB HotpotQA
1410
- config: default
1411
- split: test
1412
- revision: None
1413
- metrics:
1414
- - type: map_at_1
1415
- value: 35.922
1416
- - type: map_at_10
1417
- value: 52.32599999999999
1418
- - type: map_at_100
1419
- value: 53.18000000000001
1420
- - type: map_at_1000
1421
- value: 53.245
1422
- - type: map_at_3
1423
- value: 49.294
1424
- - type: map_at_5
1425
- value: 51.202999999999996
1426
- - type: mrr_at_1
1427
- value: 71.843
1428
- - type: mrr_at_10
1429
- value: 78.24600000000001
1430
- - type: mrr_at_100
1431
- value: 78.515
1432
- - type: mrr_at_1000
1433
- value: 78.527
1434
- - type: mrr_at_3
1435
- value: 77.17500000000001
1436
- - type: mrr_at_5
1437
- value: 77.852
1438
- - type: ndcg_at_1
1439
- value: 71.843
1440
- - type: ndcg_at_10
1441
- value: 61.379
1442
- - type: ndcg_at_100
1443
- value: 64.535
1444
- - type: ndcg_at_1000
1445
- value: 65.888
1446
- - type: ndcg_at_3
1447
- value: 56.958
1448
- - type: ndcg_at_5
1449
- value: 59.434
1450
- - type: precision_at_1
1451
- value: 71.843
1452
- - type: precision_at_10
1453
- value: 12.686
1454
- - type: precision_at_100
1455
- value: 1.517
1456
- - type: precision_at_1000
1457
- value: 0.16999999999999998
1458
- - type: precision_at_3
1459
- value: 35.778
1460
- - type: precision_at_5
1461
- value: 23.422
1462
- - type: recall_at_1
1463
- value: 35.922
1464
- - type: recall_at_10
1465
- value: 63.43
1466
- - type: recall_at_100
1467
- value: 75.868
1468
- - type: recall_at_1000
1469
- value: 84.88900000000001
1470
- - type: recall_at_3
1471
- value: 53.666000000000004
1472
- - type: recall_at_5
1473
- value: 58.555
1474
- - task:
1475
- type: Classification
1476
- dataset:
1477
- type: mteb/imdb
1478
- name: MTEB ImdbClassification
1479
- config: default
1480
- split: test
1481
- revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
1482
- metrics:
1483
- - type: accuracy
1484
- value: 79.4408
1485
- - type: ap
1486
- value: 73.52820871620366
1487
- - type: f1
1488
- value: 79.36240238685001
1489
- - task:
1490
- type: Retrieval
1491
- dataset:
1492
- type: msmarco
1493
- name: MTEB MSMARCO
1494
- config: default
1495
- split: dev
1496
- revision: None
1497
- metrics:
1498
- - type: map_at_1
1499
- value: 21.826999999999998
1500
- - type: map_at_10
1501
- value: 34.04
1502
- - type: map_at_100
1503
- value: 35.226
1504
- - type: map_at_1000
1505
- value: 35.275
1506
- - type: map_at_3
1507
- value: 30.165999999999997
1508
- - type: map_at_5
1509
- value: 32.318000000000005
1510
- - type: mrr_at_1
1511
- value: 22.464000000000002
1512
- - type: mrr_at_10
1513
- value: 34.631
1514
- - type: mrr_at_100
1515
- value: 35.752
1516
- - type: mrr_at_1000
1517
- value: 35.795
1518
- - type: mrr_at_3
1519
- value: 30.798
1520
- - type: mrr_at_5
1521
- value: 32.946999999999996
1522
- - type: ndcg_at_1
1523
- value: 22.464000000000002
1524
- - type: ndcg_at_10
1525
- value: 40.919
1526
- - type: ndcg_at_100
1527
- value: 46.632
1528
- - type: ndcg_at_1000
1529
- value: 47.833
1530
- - type: ndcg_at_3
1531
- value: 32.992
1532
- - type: ndcg_at_5
1533
- value: 36.834
1534
- - type: precision_at_1
1535
- value: 22.464000000000002
1536
- - type: precision_at_10
1537
- value: 6.494
1538
- - type: precision_at_100
1539
- value: 0.9369999999999999
1540
- - type: precision_at_1000
1541
- value: 0.104
1542
- - type: precision_at_3
1543
- value: 14.021
1544
- - type: precision_at_5
1545
- value: 10.347000000000001
1546
- - type: recall_at_1
1547
- value: 21.826999999999998
1548
- - type: recall_at_10
1549
- value: 62.132
1550
- - type: recall_at_100
1551
- value: 88.55199999999999
1552
- - type: recall_at_1000
1553
- value: 97.707
1554
- - type: recall_at_3
1555
- value: 40.541
1556
- - type: recall_at_5
1557
- value: 49.739
1558
- - task:
1559
- type: Classification
1560
- dataset:
1561
- type: mteb/mtop_domain
1562
- name: MTEB MTOPDomainClassification (en)
1563
- config: en
1564
- split: test
1565
- revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
1566
- metrics:
1567
- - type: accuracy
1568
- value: 95.68399452804377
1569
- - type: f1
1570
- value: 95.25490609832268
1571
- - task:
1572
- type: Classification
1573
- dataset:
1574
- type: mteb/mtop_intent
1575
- name: MTEB MTOPIntentClassification (en)
1576
- config: en
1577
- split: test
1578
- revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
1579
- metrics:
1580
- - type: accuracy
1581
- value: 83.15321477428182
1582
- - type: f1
1583
- value: 60.35476439087966
1584
- - task:
1585
- type: Classification
1586
- dataset:
1587
- type: mteb/amazon_massive_intent
1588
- name: MTEB MassiveIntentClassification (en)
1589
- config: en
1590
- split: test
1591
- revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
1592
- metrics:
1593
- - type: accuracy
1594
- value: 71.92669804976462
1595
- - type: f1
1596
- value: 69.22815107207565
1597
- - task:
1598
- type: Classification
1599
- dataset:
1600
- type: mteb/amazon_massive_scenario
1601
- name: MTEB MassiveScenarioClassification (en)
1602
- config: en
1603
- split: test
1604
- revision: 7d571f92784cd94a019292a1f45445077d0ef634
1605
- metrics:
1606
- - type: accuracy
1607
- value: 74.4855413584398
1608
- - type: f1
1609
- value: 72.92107516103387
1610
- - task:
1611
- type: Clustering
1612
- dataset:
1613
- type: mteb/medrxiv-clustering-p2p
1614
- name: MTEB MedrxivClusteringP2P
1615
- config: default
1616
- split: test
1617
- revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
1618
- metrics:
1619
- - type: v_measure
1620
- value: 32.412679360205544
1621
- - task:
1622
- type: Clustering
1623
- dataset:
1624
- type: mteb/medrxiv-clustering-s2s
1625
- name: MTEB MedrxivClusteringS2S
1626
- config: default
1627
- split: test
1628
- revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
1629
- metrics:
1630
- - type: v_measure
1631
- value: 28.09211869875204
1632
- - task:
1633
- type: Reranking
1634
- dataset:
1635
- type: mteb/mind_small
1636
- name: MTEB MindSmallReranking
1637
- config: default
1638
- split: test
1639
- revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
1640
- metrics:
1641
- - type: map
1642
- value: 30.540919056982545
1643
- - type: mrr
1644
- value: 31.529904607063536
1645
- - task:
1646
- type: Retrieval
1647
- dataset:
1648
- type: nfcorpus
1649
- name: MTEB NFCorpus
1650
- config: default
1651
- split: test
1652
- revision: None
1653
- metrics:
1654
- - type: map_at_1
1655
- value: 5.745
1656
- - type: map_at_10
1657
- value: 12.013
1658
- - type: map_at_100
1659
- value: 15.040000000000001
1660
- - type: map_at_1000
1661
- value: 16.427
1662
- - type: map_at_3
1663
- value: 8.841000000000001
1664
- - type: map_at_5
1665
- value: 10.289
1666
- - type: mrr_at_1
1667
- value: 45.201
1668
- - type: mrr_at_10
1669
- value: 53.483999999999995
1670
- - type: mrr_at_100
1671
- value: 54.20700000000001
1672
- - type: mrr_at_1000
1673
- value: 54.252
1674
- - type: mrr_at_3
1675
- value: 51.29
1676
- - type: mrr_at_5
1677
- value: 52.73
1678
- - type: ndcg_at_1
1679
- value: 43.808
1680
- - type: ndcg_at_10
1681
- value: 32.445
1682
- - type: ndcg_at_100
1683
- value: 30.031000000000002
1684
- - type: ndcg_at_1000
1685
- value: 39.007
1686
- - type: ndcg_at_3
1687
- value: 37.204
1688
- - type: ndcg_at_5
1689
- value: 35.07
1690
- - type: precision_at_1
1691
- value: 45.201
1692
- - type: precision_at_10
1693
- value: 23.684
1694
- - type: precision_at_100
1695
- value: 7.600999999999999
1696
- - type: precision_at_1000
1697
- value: 2.043
1698
- - type: precision_at_3
1699
- value: 33.953
1700
- - type: precision_at_5
1701
- value: 29.412
1702
- - type: recall_at_1
1703
- value: 5.745
1704
- - type: recall_at_10
1705
- value: 16.168
1706
- - type: recall_at_100
1707
- value: 30.875999999999998
1708
- - type: recall_at_1000
1709
- value: 62.686
1710
- - type: recall_at_3
1711
- value: 9.75
1712
- - type: recall_at_5
1713
- value: 12.413
1714
- - task:
1715
- type: Retrieval
1716
- dataset:
1717
- type: nq
1718
- name: MTEB NQ
1719
- config: default
1720
- split: test
1721
- revision: None
1722
- metrics:
1723
- - type: map_at_1
1724
- value: 37.828
1725
- - type: map_at_10
1726
- value: 53.239000000000004
1727
- - type: map_at_100
1728
- value: 54.035999999999994
1729
- - type: map_at_1000
1730
- value: 54.067
1731
- - type: map_at_3
1732
- value: 49.289
1733
- - type: map_at_5
1734
- value: 51.784
1735
- - type: mrr_at_1
1736
- value: 42.497
1737
- - type: mrr_at_10
1738
- value: 55.916999999999994
1739
- - type: mrr_at_100
1740
- value: 56.495
1741
- - type: mrr_at_1000
1742
- value: 56.516999999999996
1743
- - type: mrr_at_3
1744
- value: 52.800000000000004
1745
- - type: mrr_at_5
1746
- value: 54.722
1747
- - type: ndcg_at_1
1748
- value: 42.468
1749
- - type: ndcg_at_10
1750
- value: 60.437
1751
- - type: ndcg_at_100
1752
- value: 63.731
1753
- - type: ndcg_at_1000
1754
- value: 64.41799999999999
1755
- - type: ndcg_at_3
1756
- value: 53.230999999999995
1757
- - type: ndcg_at_5
1758
- value: 57.26
1759
- - type: precision_at_1
1760
- value: 42.468
1761
- - type: precision_at_10
1762
- value: 9.47
1763
- - type: precision_at_100
1764
- value: 1.1360000000000001
1765
- - type: precision_at_1000
1766
- value: 0.12
1767
- - type: precision_at_3
1768
- value: 23.724999999999998
1769
- - type: precision_at_5
1770
- value: 16.593
1771
- - type: recall_at_1
1772
- value: 37.828
1773
- - type: recall_at_10
1774
- value: 79.538
1775
- - type: recall_at_100
1776
- value: 93.646
1777
- - type: recall_at_1000
1778
- value: 98.72999999999999
1779
- - type: recall_at_3
1780
- value: 61.134
1781
- - type: recall_at_5
1782
- value: 70.377
1783
- - task:
1784
- type: Retrieval
1785
- dataset:
1786
- type: quora
1787
- name: MTEB QuoraRetrieval
1788
- config: default
1789
- split: test
1790
- revision: None
1791
- metrics:
1792
- - type: map_at_1
1793
- value: 70.548
1794
- - type: map_at_10
1795
- value: 84.466
1796
- - type: map_at_100
1797
- value: 85.10600000000001
1798
- - type: map_at_1000
1799
- value: 85.123
1800
- - type: map_at_3
1801
- value: 81.57600000000001
1802
- - type: map_at_5
1803
- value: 83.399
1804
- - type: mrr_at_1
1805
- value: 81.24
1806
- - type: mrr_at_10
1807
- value: 87.457
1808
- - type: mrr_at_100
1809
- value: 87.574
1810
- - type: mrr_at_1000
1811
- value: 87.575
1812
- - type: mrr_at_3
1813
- value: 86.507
1814
- - type: mrr_at_5
1815
- value: 87.205
1816
- - type: ndcg_at_1
1817
- value: 81.25
1818
- - type: ndcg_at_10
1819
- value: 88.203
1820
- - type: ndcg_at_100
1821
- value: 89.457
1822
- - type: ndcg_at_1000
1823
- value: 89.563
1824
- - type: ndcg_at_3
1825
- value: 85.465
1826
- - type: ndcg_at_5
1827
- value: 87.007
1828
- - type: precision_at_1
1829
- value: 81.25
1830
- - type: precision_at_10
1831
- value: 13.373
1832
- - type: precision_at_100
1833
- value: 1.5270000000000001
1834
- - type: precision_at_1000
1835
- value: 0.157
1836
- - type: precision_at_3
1837
- value: 37.417
1838
- - type: precision_at_5
1839
- value: 24.556
1840
- - type: recall_at_1
1841
- value: 70.548
1842
- - type: recall_at_10
1843
- value: 95.208
1844
- - type: recall_at_100
1845
- value: 99.514
1846
- - type: recall_at_1000
1847
- value: 99.988
1848
- - type: recall_at_3
1849
- value: 87.214
1850
- - type: recall_at_5
1851
- value: 91.696
1852
- - task:
1853
- type: Clustering
1854
- dataset:
1855
- type: mteb/reddit-clustering
1856
- name: MTEB RedditClustering
1857
- config: default
1858
- split: test
1859
- revision: 24640382cdbf8abc73003fb0fa6d111a705499eb
1860
- metrics:
1861
- - type: v_measure
1862
- value: 53.04822095496839
1863
- - task:
1864
- type: Clustering
1865
- dataset:
1866
- type: mteb/reddit-clustering-p2p
1867
- name: MTEB RedditClusteringP2P
1868
- config: default
1869
- split: test
1870
- revision: 282350215ef01743dc01b456c7f5241fa8937f16
1871
- metrics:
1872
- - type: v_measure
1873
- value: 60.30778476474675
1874
- - task:
1875
- type: Retrieval
1876
- dataset:
1877
- type: scidocs
1878
- name: MTEB SCIDOCS
1879
- config: default
1880
- split: test
1881
- revision: None
1882
- metrics:
1883
- - type: map_at_1
1884
- value: 4.692
1885
- - type: map_at_10
1886
- value: 11.766
1887
- - type: map_at_100
1888
- value: 13.904
1889
- - type: map_at_1000
1890
- value: 14.216999999999999
1891
- - type: map_at_3
1892
- value: 8.245
1893
- - type: map_at_5
1894
- value: 9.92
1895
- - type: mrr_at_1
1896
- value: 23.0
1897
- - type: mrr_at_10
1898
- value: 33.78
1899
- - type: mrr_at_100
1900
- value: 34.922
1901
- - type: mrr_at_1000
1902
- value: 34.973
1903
- - type: mrr_at_3
1904
- value: 30.2
1905
- - type: mrr_at_5
1906
- value: 32.565
1907
- - type: ndcg_at_1
1908
- value: 23.0
1909
- - type: ndcg_at_10
1910
- value: 19.863
1911
- - type: ndcg_at_100
1912
- value: 28.141
1913
- - type: ndcg_at_1000
1914
- value: 33.549
1915
- - type: ndcg_at_3
1916
- value: 18.434
1917
- - type: ndcg_at_5
1918
- value: 16.384
1919
- - type: precision_at_1
1920
- value: 23.0
1921
- - type: precision_at_10
1922
- value: 10.39
1923
- - type: precision_at_100
1924
- value: 2.235
1925
- - type: precision_at_1000
1926
- value: 0.35300000000000004
1927
- - type: precision_at_3
1928
- value: 17.133000000000003
1929
- - type: precision_at_5
1930
- value: 14.44
1931
- - type: recall_at_1
1932
- value: 4.692
1933
- - type: recall_at_10
1934
- value: 21.025
1935
- - type: recall_at_100
1936
- value: 45.324999999999996
1937
- - type: recall_at_1000
1938
- value: 71.675
1939
- - type: recall_at_3
1940
- value: 10.440000000000001
1941
- - type: recall_at_5
1942
- value: 14.64
1943
- - task:
1944
- type: STS
1945
- dataset:
1946
- type: mteb/sickr-sts
1947
- name: MTEB SICK-R
1948
- config: default
1949
- split: test
1950
- revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
1951
- metrics:
1952
- - type: cos_sim_pearson
1953
- value: 84.96178184892842
1954
- - type: cos_sim_spearman
1955
- value: 79.6487740813199
1956
- - type: euclidean_pearson
1957
- value: 82.06661161625023
1958
- - type: euclidean_spearman
1959
- value: 79.64876769031183
1960
- - type: manhattan_pearson
1961
- value: 82.07061164575131
1962
- - type: manhattan_spearman
1963
- value: 79.65197039464537
1964
- - task:
1965
- type: STS
1966
- dataset:
1967
- type: mteb/sts12-sts
1968
- name: MTEB STS12
1969
- config: default
1970
- split: test
1971
- revision: a0d554a64d88156834ff5ae9920b964011b16384
1972
- metrics:
1973
- - type: cos_sim_pearson
1974
- value: 84.15305604100027
1975
- - type: cos_sim_spearman
1976
- value: 74.27447427941591
1977
- - type: euclidean_pearson
1978
- value: 80.52737337565307
1979
- - type: euclidean_spearman
1980
- value: 74.27416077132192
1981
- - type: manhattan_pearson
1982
- value: 80.53728571140387
1983
- - type: manhattan_spearman
1984
- value: 74.28853605753457
1985
- - task:
1986
- type: STS
1987
- dataset:
1988
- type: mteb/sts13-sts
1989
- name: MTEB STS13
1990
- config: default
1991
- split: test
1992
- revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
1993
- metrics:
1994
- - type: cos_sim_pearson
1995
- value: 83.44386080639279
1996
- - type: cos_sim_spearman
1997
- value: 84.17947648159536
1998
- - type: euclidean_pearson
1999
- value: 83.34145388129387
2000
- - type: euclidean_spearman
2001
- value: 84.17947648159536
2002
- - type: manhattan_pearson
2003
- value: 83.30699061927966
2004
- - type: manhattan_spearman
2005
- value: 84.18125737380451
2006
- - task:
2007
- type: STS
2008
- dataset:
2009
- type: mteb/sts14-sts
2010
- name: MTEB STS14
2011
- config: default
2012
- split: test
2013
- revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
2014
- metrics:
2015
- - type: cos_sim_pearson
2016
- value: 81.57392220985612
2017
- - type: cos_sim_spearman
2018
- value: 78.80745014464101
2019
- - type: euclidean_pearson
2020
- value: 80.01660371487199
2021
- - type: euclidean_spearman
2022
- value: 78.80741240102256
2023
- - type: manhattan_pearson
2024
- value: 79.96810779507953
2025
- - type: manhattan_spearman
2026
- value: 78.75600400119448
2027
- - task:
2028
- type: STS
2029
- dataset:
2030
- type: mteb/sts15-sts
2031
- name: MTEB STS15
2032
- config: default
2033
- split: test
2034
- revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
2035
- metrics:
2036
- - type: cos_sim_pearson
2037
- value: 86.85421063026625
2038
- - type: cos_sim_spearman
2039
- value: 87.55320285299192
2040
- - type: euclidean_pearson
2041
- value: 86.69750143323517
2042
- - type: euclidean_spearman
2043
- value: 87.55320284326378
2044
- - type: manhattan_pearson
2045
- value: 86.63379169960379
2046
- - type: manhattan_spearman
2047
- value: 87.4815029877984
2048
- - task:
2049
- type: STS
2050
- dataset:
2051
- type: mteb/sts16-sts
2052
- name: MTEB STS16
2053
- config: default
2054
- split: test
2055
- revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
2056
- metrics:
2057
- - type: cos_sim_pearson
2058
- value: 84.31314130411842
2059
- - type: cos_sim_spearman
2060
- value: 85.3489588181433
2061
- - type: euclidean_pearson
2062
- value: 84.13240933463535
2063
- - type: euclidean_spearman
2064
- value: 85.34902871403281
2065
- - type: manhattan_pearson
2066
- value: 84.01183086503559
2067
- - type: manhattan_spearman
2068
- value: 85.19316703166102
2069
- - task:
2070
- type: STS
2071
- dataset:
2072
- type: mteb/sts17-crosslingual-sts
2073
- name: MTEB STS17 (en-en)
2074
- config: en-en
2075
- split: test
2076
- revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
2077
- metrics:
2078
- - type: cos_sim_pearson
2079
- value: 89.09979781689536
2080
- - type: cos_sim_spearman
2081
- value: 88.87813323759015
2082
- - type: euclidean_pearson
2083
- value: 88.65413031123792
2084
- - type: euclidean_spearman
2085
- value: 88.87813323759015
2086
- - type: manhattan_pearson
2087
- value: 88.61818758256024
2088
- - type: manhattan_spearman
2089
- value: 88.81044100494604
2090
- - task:
2091
- type: STS
2092
- dataset:
2093
- type: mteb/sts22-crosslingual-sts
2094
- name: MTEB STS22 (en)
2095
- config: en
2096
- split: test
2097
- revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
2098
- metrics:
2099
- - type: cos_sim_pearson
2100
- value: 62.30693258111531
2101
- - type: cos_sim_spearman
2102
- value: 62.195516523251946
2103
- - type: euclidean_pearson
2104
- value: 62.951283701049476
2105
- - type: euclidean_spearman
2106
- value: 62.195516523251946
2107
- - type: manhattan_pearson
2108
- value: 63.068322281439535
2109
- - type: manhattan_spearman
2110
- value: 62.10621171028406
2111
- - task:
2112
- type: STS
2113
- dataset:
2114
- type: mteb/stsbenchmark-sts
2115
- name: MTEB STSBenchmark
2116
- config: default
2117
- split: test
2118
- revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
2119
- metrics:
2120
- - type: cos_sim_pearson
2121
- value: 84.27092833763909
2122
- - type: cos_sim_spearman
2123
- value: 84.84429717949759
2124
- - type: euclidean_pearson
2125
- value: 84.8516966060792
2126
- - type: euclidean_spearman
2127
- value: 84.84429717949759
2128
- - type: manhattan_pearson
2129
- value: 84.82203139242881
2130
- - type: manhattan_spearman
2131
- value: 84.8358503952945
2132
- - task:
2133
- type: Reranking
2134
- dataset:
2135
- type: mteb/scidocs-reranking
2136
- name: MTEB SciDocsRR
2137
- config: default
2138
- split: test
2139
- revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
2140
- metrics:
2141
- - type: map
2142
- value: 83.10290863981409
2143
- - type: mrr
2144
- value: 95.31168450286097
2145
- - task:
2146
- type: Retrieval
2147
- dataset:
2148
- type: scifact
2149
- name: MTEB SciFact
2150
- config: default
2151
- split: test
2152
- revision: None
2153
- metrics:
2154
- - type: map_at_1
2155
- value: 52.161
2156
- - type: map_at_10
2157
- value: 62.138000000000005
2158
- - type: map_at_100
2159
- value: 62.769
2160
- - type: map_at_1000
2161
- value: 62.812
2162
- - type: map_at_3
2163
- value: 59.111000000000004
2164
- - type: map_at_5
2165
- value: 60.995999999999995
2166
- - type: mrr_at_1
2167
- value: 55.333
2168
- - type: mrr_at_10
2169
- value: 63.504000000000005
2170
- - type: mrr_at_100
2171
- value: 64.036
2172
- - type: mrr_at_1000
2173
- value: 64.08
2174
- - type: mrr_at_3
2175
- value: 61.278
2176
- - type: mrr_at_5
2177
- value: 62.778
2178
- - type: ndcg_at_1
2179
- value: 55.333
2180
- - type: ndcg_at_10
2181
- value: 66.678
2182
- - type: ndcg_at_100
2183
- value: 69.415
2184
- - type: ndcg_at_1000
2185
- value: 70.453
2186
- - type: ndcg_at_3
2187
- value: 61.755
2188
- - type: ndcg_at_5
2189
- value: 64.546
2190
- - type: precision_at_1
2191
- value: 55.333
2192
- - type: precision_at_10
2193
- value: 9.033
2194
- - type: precision_at_100
2195
- value: 1.043
2196
- - type: precision_at_1000
2197
- value: 0.11199999999999999
2198
- - type: precision_at_3
2199
- value: 24.221999999999998
2200
- - type: precision_at_5
2201
- value: 16.333000000000002
2202
- - type: recall_at_1
2203
- value: 52.161
2204
- - type: recall_at_10
2205
- value: 79.156
2206
- - type: recall_at_100
2207
- value: 91.333
2208
- - type: recall_at_1000
2209
- value: 99.333
2210
- - type: recall_at_3
2211
- value: 66.43299999999999
2212
- - type: recall_at_5
2213
- value: 73.272
2214
- - task:
2215
- type: PairClassification
2216
- dataset:
2217
- type: mteb/sprintduplicatequestions-pairclassification
2218
- name: MTEB SprintDuplicateQuestions
2219
- config: default
2220
- split: test
2221
- revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
2222
- metrics:
2223
- - type: cos_sim_accuracy
2224
- value: 99.81287128712871
2225
- - type: cos_sim_ap
2226
- value: 95.30034785910676
2227
- - type: cos_sim_f1
2228
- value: 90.28629856850716
2229
- - type: cos_sim_precision
2230
- value: 92.36401673640168
2231
- - type: cos_sim_recall
2232
- value: 88.3
2233
- - type: dot_accuracy
2234
- value: 99.81287128712871
2235
- - type: dot_ap
2236
- value: 95.30034785910676
2237
- - type: dot_f1
2238
- value: 90.28629856850716
2239
- - type: dot_precision
2240
- value: 92.36401673640168
2241
- - type: dot_recall
2242
- value: 88.3
2243
- - type: euclidean_accuracy
2244
- value: 99.81287128712871
2245
- - type: euclidean_ap
2246
- value: 95.30034785910676
2247
- - type: euclidean_f1
2248
- value: 90.28629856850716
2249
- - type: euclidean_precision
2250
- value: 92.36401673640168
2251
- - type: euclidean_recall
2252
- value: 88.3
2253
- - type: manhattan_accuracy
2254
- value: 99.80990099009901
2255
- - type: manhattan_ap
2256
- value: 95.26880751950654
2257
- - type: manhattan_f1
2258
- value: 90.22177419354838
2259
- - type: manhattan_precision
2260
- value: 90.95528455284553
2261
- - type: manhattan_recall
2262
- value: 89.5
2263
- - type: max_accuracy
2264
- value: 99.81287128712871
2265
- - type: max_ap
2266
- value: 95.30034785910676
2267
- - type: max_f1
2268
- value: 90.28629856850716
2269
- - task:
2270
- type: Clustering
2271
- dataset:
2272
- type: mteb/stackexchange-clustering
2273
- name: MTEB StackExchangeClustering
2274
- config: default
2275
- split: test
2276
- revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
2277
- metrics:
2278
- - type: v_measure
2279
- value: 58.518662504351184
2280
- - task:
2281
- type: Clustering
2282
- dataset:
2283
- type: mteb/stackexchange-clustering-p2p
2284
- name: MTEB StackExchangeClusteringP2P
2285
- config: default
2286
- split: test
2287
- revision: 815ca46b2622cec33ccafc3735d572c266efdb44
2288
- metrics:
2289
- - type: v_measure
2290
- value: 34.96168178378587
2291
- - task:
2292
- type: Reranking
2293
- dataset:
2294
- type: mteb/stackoverflowdupquestions-reranking
2295
- name: MTEB StackOverflowDupQuestions
2296
- config: default
2297
- split: test
2298
- revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
2299
- metrics:
2300
- - type: map
2301
- value: 52.04862593471896
2302
- - type: mrr
2303
- value: 52.97238402936932
2304
- - task:
2305
- type: Summarization
2306
- dataset:
2307
- type: mteb/summeval
2308
- name: MTEB SummEval
2309
- config: default
2310
- split: test
2311
- revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
2312
- metrics:
2313
- - type: cos_sim_pearson
2314
- value: 30.092545236479946
2315
- - type: cos_sim_spearman
2316
- value: 31.599851000175498
2317
- - type: dot_pearson
2318
- value: 30.092542723901676
2319
- - type: dot_spearman
2320
- value: 31.599851000175498
2321
- - task:
2322
- type: Retrieval
2323
- dataset:
2324
- type: trec-covid
2325
- name: MTEB TRECCOVID
2326
- config: default
2327
- split: test
2328
- revision: None
2329
- metrics:
2330
- - type: map_at_1
2331
- value: 0.189
2332
- - type: map_at_10
2333
- value: 1.662
2334
- - type: map_at_100
2335
- value: 9.384
2336
- - type: map_at_1000
2337
- value: 22.669
2338
- - type: map_at_3
2339
- value: 0.5559999999999999
2340
- - type: map_at_5
2341
- value: 0.9039999999999999
2342
- - type: mrr_at_1
2343
- value: 68.0
2344
- - type: mrr_at_10
2345
- value: 81.01899999999999
2346
- - type: mrr_at_100
2347
- value: 81.01899999999999
2348
- - type: mrr_at_1000
2349
- value: 81.01899999999999
2350
- - type: mrr_at_3
2351
- value: 79.333
2352
- - type: mrr_at_5
2353
- value: 80.733
2354
- - type: ndcg_at_1
2355
- value: 63.0
2356
- - type: ndcg_at_10
2357
- value: 65.913
2358
- - type: ndcg_at_100
2359
- value: 51.895
2360
- - type: ndcg_at_1000
2361
- value: 46.967
2362
- - type: ndcg_at_3
2363
- value: 65.49199999999999
2364
- - type: ndcg_at_5
2365
- value: 66.69699999999999
2366
- - type: precision_at_1
2367
- value: 68.0
2368
- - type: precision_at_10
2369
- value: 71.6
2370
- - type: precision_at_100
2371
- value: 53.66
2372
- - type: precision_at_1000
2373
- value: 21.124000000000002
2374
- - type: precision_at_3
2375
- value: 72.667
2376
- - type: precision_at_5
2377
- value: 74.0
2378
- - type: recall_at_1
2379
- value: 0.189
2380
- - type: recall_at_10
2381
- value: 1.913
2382
- - type: recall_at_100
2383
- value: 12.601999999999999
2384
- - type: recall_at_1000
2385
- value: 44.296
2386
- - type: recall_at_3
2387
- value: 0.605
2388
- - type: recall_at_5
2389
- value: 1.018
2390
- - task:
2391
- type: Retrieval
2392
- dataset:
2393
- type: webis-touche2020
2394
- name: MTEB Touche2020
2395
- config: default
2396
- split: test
2397
- revision: None
2398
- metrics:
2399
- - type: map_at_1
2400
- value: 2.701
2401
- - type: map_at_10
2402
- value: 10.445
2403
- - type: map_at_100
2404
- value: 17.324
2405
- - type: map_at_1000
2406
- value: 19.161
2407
- - type: map_at_3
2408
- value: 5.497
2409
- - type: map_at_5
2410
- value: 7.278
2411
- - type: mrr_at_1
2412
- value: 30.612000000000002
2413
- - type: mrr_at_10
2414
- value: 45.534
2415
- - type: mrr_at_100
2416
- value: 45.792
2417
- - type: mrr_at_1000
2418
- value: 45.806999999999995
2419
- - type: mrr_at_3
2420
- value: 37.755
2421
- - type: mrr_at_5
2422
- value: 43.469
2423
- - type: ndcg_at_1
2424
- value: 26.531
2425
- - type: ndcg_at_10
2426
- value: 26.235000000000003
2427
- - type: ndcg_at_100
2428
- value: 39.17
2429
- - type: ndcg_at_1000
2430
- value: 51.038
2431
- - type: ndcg_at_3
2432
- value: 23.625
2433
- - type: ndcg_at_5
2434
- value: 24.338
2435
- - type: precision_at_1
2436
- value: 30.612000000000002
2437
- - type: precision_at_10
2438
- value: 24.285999999999998
2439
- - type: precision_at_100
2440
- value: 8.224
2441
- - type: precision_at_1000
2442
- value: 1.6179999999999999
2443
- - type: precision_at_3
2444
- value: 24.490000000000002
2445
- - type: precision_at_5
2446
- value: 24.898
2447
- - type: recall_at_1
2448
- value: 2.701
2449
- - type: recall_at_10
2450
- value: 17.997
2451
- - type: recall_at_100
2452
- value: 51.766999999999996
2453
- - type: recall_at_1000
2454
- value: 87.863
2455
- - type: recall_at_3
2456
- value: 6.295000000000001
2457
- - type: recall_at_5
2458
- value: 9.993
2459
- - task:
2460
- type: Classification
2461
- dataset:
2462
- type: mteb/toxic_conversations_50k
2463
- name: MTEB ToxicConversationsClassification
2464
- config: default
2465
- split: test
2466
- revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
2467
- metrics:
2468
- - type: accuracy
2469
- value: 73.3474
2470
- - type: ap
2471
- value: 15.393431414459924
2472
- - type: f1
2473
- value: 56.466681887882416
2474
- - task:
2475
- type: Classification
2476
- dataset:
2477
- type: mteb/tweet_sentiment_extraction
2478
- name: MTEB TweetSentimentExtractionClassification
2479
- config: default
2480
- split: test
2481
- revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
2482
- metrics:
2483
- - type: accuracy
2484
- value: 62.062818336163
2485
- - type: f1
2486
- value: 62.11230840463252
2487
- - task:
2488
- type: Clustering
2489
- dataset:
2490
- type: mteb/twentynewsgroups-clustering
2491
- name: MTEB TwentyNewsgroupsClustering
2492
- config: default
2493
- split: test
2494
- revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
2495
- metrics:
2496
- - type: v_measure
2497
- value: 42.464892820845115
2498
- - task:
2499
- type: PairClassification
2500
- dataset:
2501
- type: mteb/twittersemeval2015-pairclassification
2502
- name: MTEB TwitterSemEval2015
2503
- config: default
2504
- split: test
2505
- revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
2506
- metrics:
2507
- - type: cos_sim_accuracy
2508
- value: 86.15962329379508
2509
- - type: cos_sim_ap
2510
- value: 74.73674057919256
2511
- - type: cos_sim_f1
2512
- value: 68.81245642574947
2513
- - type: cos_sim_precision
2514
- value: 61.48255813953488
2515
- - type: cos_sim_recall
2516
- value: 78.12664907651715
2517
- - type: dot_accuracy
2518
- value: 86.15962329379508
2519
- - type: dot_ap
2520
- value: 74.7367634988281
2521
- - type: dot_f1
2522
- value: 68.81245642574947
2523
- - type: dot_precision
2524
- value: 61.48255813953488
2525
- - type: dot_recall
2526
- value: 78.12664907651715
2527
- - type: euclidean_accuracy
2528
- value: 86.15962329379508
2529
- - type: euclidean_ap
2530
- value: 74.7367761466634
2531
- - type: euclidean_f1
2532
- value: 68.81245642574947
2533
- - type: euclidean_precision
2534
- value: 61.48255813953488
2535
- - type: euclidean_recall
2536
- value: 78.12664907651715
2537
- - type: manhattan_accuracy
2538
- value: 86.21326816474935
2539
- - type: manhattan_ap
2540
- value: 74.64416473733951
2541
- - type: manhattan_f1
2542
- value: 68.80924855491331
2543
- - type: manhattan_precision
2544
- value: 61.23456790123457
2545
- - type: manhattan_recall
2546
- value: 78.52242744063325
2547
- - type: max_accuracy
2548
- value: 86.21326816474935
2549
- - type: max_ap
2550
- value: 74.7367761466634
2551
- - type: max_f1
2552
- value: 68.81245642574947
2553
- - task:
2554
- type: PairClassification
2555
- dataset:
2556
- type: mteb/twitterurlcorpus-pairclassification
2557
- name: MTEB TwitterURLCorpus
2558
- config: default
2559
- split: test
2560
- revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
2561
- metrics:
2562
- - type: cos_sim_accuracy
2563
- value: 88.97620988085536
2564
- - type: cos_sim_ap
2565
- value: 86.08680845745758
2566
- - type: cos_sim_f1
2567
- value: 78.02793637114438
2568
- - type: cos_sim_precision
2569
- value: 73.11082699683736
2570
- - type: cos_sim_recall
2571
- value: 83.65414228518632
2572
- - type: dot_accuracy
2573
- value: 88.97620988085536
2574
- - type: dot_ap
2575
- value: 86.08681149437946
2576
- - type: dot_f1
2577
- value: 78.02793637114438
2578
- - type: dot_precision
2579
- value: 73.11082699683736
2580
- - type: dot_recall
2581
- value: 83.65414228518632
2582
- - type: euclidean_accuracy
2583
- value: 88.97620988085536
2584
- - type: euclidean_ap
2585
- value: 86.08681215460771
2586
- - type: euclidean_f1
2587
- value: 78.02793637114438
2588
- - type: euclidean_precision
2589
- value: 73.11082699683736
2590
- - type: euclidean_recall
2591
- value: 83.65414228518632
2592
- - type: manhattan_accuracy
2593
- value: 88.88888888888889
2594
- - type: manhattan_ap
2595
- value: 86.02916327562438
2596
- - type: manhattan_f1
2597
- value: 78.02063045516843
2598
- - type: manhattan_precision
2599
- value: 73.38851947346994
2600
- - type: manhattan_recall
2601
- value: 83.2768709578072
2602
- - type: max_accuracy
2603
- value: 88.97620988085536
2604
- - type: max_ap
2605
- value: 86.08681215460771
2606
- - type: max_f1
2607
- value: 78.02793637114438
2608
- ---
2609
- <!-- TODO: add evaluation results here -->
2610
- <br><br>
2611
-
2612
- <p align="center">
2613
- <img src="https://github.com/jina-ai/finetuner/blob/main/docs/_static/finetuner-logo-ani.svg?raw=true" alt="Finetuner logo: Finetuner helps you to create experiments in order to improve embeddings on search tasks. It accompanies you to deliver the last mile of performance-tuning for neural search applications." width="150px">
2614
- </p>
2615
-
2616
-
2617
- <p align="center">
2618
- <b>The text embedding set trained by <a href="https://jina.ai/"><b>Jina AI</b></a>, <a href="https://github.com/jina-ai/finetuner"><b>Finetuner</b></a> team.</b>
2619
- </p>
2620
-
2621
-
2622
- ## Intended Usage & Model Info
2623
-
2624
- `jina-embeddings-v2-base-en` is an English, monolingual **embedding model** supporting **8192 sequence length**.
2625
- It is based on a Bert architecture (JinaBert) that supports the symmetric bidirectional variant of [ALiBi](https://arxiv.org/abs/2108.12409) to allow longer sequence length.
2626
- The backbone `jina-bert-v2-base-en` is pretrained on the C4 dataset.
2627
- The model is further trained on Jina AI's collection of more than 400 millions of sentence pairs and hard negatives.
2628
- These pairs were obtained from various domains and were carefully selected through a thorough cleaning process.
2629
-
2630
- The embedding model was trained using 512 sequence length, but extrapolates to 8k sequence length (or even longer) thanks to ALiBi.
2631
- This makes our model useful for a range of use cases, especially when processing long documents is needed, including long document retrieval, semantic textual similarity, text reranking, recommendation, RAG and LLM-based generative search, etc.
2632
-
2633
- With a standard size of 137 million parameters, the model enables fast inference while delivering better performance than our small model. It is recommended to use a single GPU for inference.
2634
- Additionally, we provide the following embedding models:
2635
-
2636
- **V1 (Based on T5, 512 Seq)**
2637
-
2638
- - [`jina-embeddings-v1-small-en`](https://huggingface.co/jinaai/jina-embedding-s-en-v1): 35 million parameters.
2639
- - [`jina-embeddings-v1-base-en`](https://huggingface.co/jinaai/jina-embedding-b-en-v1): 110 million parameters.
2640
- - [`jina-embeddings-v1-large-en`](https://huggingface.co/jinaai/jina-embedding-l-en-v1): 330 million parameters.
2641
-
2642
- **V2 (Based on JinaBert, 8k Seq)**
2643
-
2644
- - [`jina-embeddings-v2-small-en`](https://huggingface.co/jinaai/jina-embeddings-v2-small-en): 33 million parameters.
2645
- - [`jina-embeddings-v2-base-en`](https://huggingface.co/jinaai/jina-embeddings-v2-base-en): 137 million parameters **(you are here)**.
2646
- - [`jina-embeddings-v2-large-en`](): 435 million parameters (releasing soon).
2647
-
2648
- ## Data & Parameters
2649
-
2650
- Jina Embeddings V2 [technical report](https://arxiv.org/abs/2310.19923)
2651
-
2652
- ## Usage
2653
-
2654
- **<details><summary>Please apply mean pooling when integrating the model.</summary>**
2655
- <p>
2656
-
2657
- ### Why mean pooling?
2658
-
2659
- `mean poooling` takes all token embeddings from model output and averaging them at sentence/paragraph level.
2660
- It has been proved to be the most effective way to produce high-quality sentence embeddings.
2661
- We offer an `encode` function to deal with this.
2662
-
2663
- However, if you would like to do it without using the default `encode` function:
2664
-
2665
- ```python
2666
- import torch
2667
- import torch.nn.functional as F
2668
- from transformers import AutoTokenizer, AutoModel
2669
-
2670
- def mean_pooling(model_output, attention_mask):
2671
- token_embeddings = model_output[0]
2672
- input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
2673
- return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
2674
-
2675
- sentences = ['How is the weather today?', 'What is the current weather like today?']
2676
-
2677
- tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v2-small-en')
2678
- model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-small-en', trust_remote_code=True)
2679
-
2680
- encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
2681
-
2682
- with torch.no_grad():
2683
- model_output = model(**encoded_input)
2684
-
2685
- embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
2686
- embeddings = F.normalize(embeddings, p=2, dim=1)
2687
- ```
2688
-
2689
- </p>
2690
- </details>
2691
-
2692
- You can use Jina Embedding models directly from transformers package:
2693
- ```python
2694
- !pip install transformers
2695
- from transformers import AutoModel
2696
- from numpy.linalg import norm
2697
-
2698
- cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
2699
- model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True) # trust_remote_code is needed to use the encode method
2700
- embeddings = model.encode(['How is the weather today?', 'What is the current weather like today?'])
2701
- print(cos_sim(embeddings[0], embeddings[1]))
2702
- ```
2703
-
2704
- If you only want to handle shorter sequence, such as 2k, pass the `max_length` parameter to the `encode` function:
2705
-
2706
- ```python
2707
- embeddings = model.encode(
2708
- ['Very long ... document'],
2709
- max_length=2048
2710
- )
2711
- ```
2712
-
2713
- ## Fully-managed Embeddings Service
2714
-
2715
- Alternatively, you can use Jina AI's [Embedding platform](https://jina.ai/embeddings/) for fully-managed access to Jina Embeddings models.
2716
-
2717
- ## Use Jina Embeddings for RAG
2718
-
2719
- According to the latest blog post from [LLamaIndex](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83),
2720
-
2721
- > In summary, to achieve the peak performance in both hit rate and MRR, the combination of OpenAI or JinaAI-Base embeddings with the CohereRerank/bge-reranker-large reranker stands out.
2722
-
2723
- <img src="https://miro.medium.com/v2/resize:fit:4800/format:webp/1*ZP2RVejCZovF3FDCg-Bx3A.png" width="780px">
2724
-
2725
-
2726
- ## Plans
2727
-
2728
- The development of new bilingual models is currently underway. We will be targeting mainly the German and Spanish languages.
2729
- The upcoming models will be called `jina-embeddings-v2-base-de/es`.
2730
-
2731
- ## Contact
2732
-
2733
- Join our [Discord community](https://discord.jina.ai) and chat with other community members about ideas.
2734
-
2735
- ## Citation
2736
-
2737
- If you find Jina Embeddings useful in your research, please cite the following paper:
2738
-
2739
- ```
2740
- @misc{günther2023jina,
2741
- title={Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents},
2742
- author={Michael Günther and Jackmin Ong and Isabelle Mohr and Alaeddine Abdessalem and Tanguy Abel and Mohammad Kalim Akram and Susana Guzman and Georgios Mastrapas and Saba Sturua and Bo Wang and Maximilian Werk and Nan Wang and Han Xiao},
2743
- year={2023},
2744
- eprint={2310.19923},
2745
- archivePrefix={arXiv},
2746
- primaryClass={cs.CL}
2747
- }
2748
- ```