Hugging Face's logo

zeroshot
/
gte-small-quant

Feature Extraction
Transformers
ONNX
English
bert
sparse sparsity quantized onnx embeddings int8
mteb
Eval Results (legacy)
Model card Files
xet
Community
zeroshot commited on
Commit
9397de3
·
1 Parent(s): b0a1ca0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +581 -0
README.md CHANGED
@@ -1,6 +1,587 @@
1
  ---
2
  tags:
3
  - sparse sparsity quantized onnx embeddings int8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  license: mit
5
  language:
6
  - en
 
1
  ---
2
  tags:
3
  - sparse sparsity quantized onnx embeddings int8
4
+ model-index:
5
+ - name: gte-small-quant
6
+ results:
7
+ - task:
8
+ type: Classification
9
+ dataset:
10
+ type: mteb/amazon_counterfactual
11
+ name: MTEB AmazonCounterfactualClassification (en)
12
+ config: en
13
+ split: test
14
+ revision: e8379541af4e31359cca9fbcf4b00f2671dba205
15
+ metrics:
16
+ - type: accuracy
17
+ value: 72.88059701492537
18
+ - type: ap
19
+ value: 35.74239003564444
20
+ - type: f1
21
+ value: 66.98065758287116
22
+ - task:
23
+ type: Classification
24
+ dataset:
25
+ type: mteb/amazon_polarity
26
+ name: MTEB AmazonPolarityClassification
27
+ config: default
28
+ split: test
29
+ revision: e2d317d38cd51312af73b3d32a06d1a08b442046
30
+ metrics:
31
+ - type: accuracy
32
+ value: 91.031575
33
+ - type: ap
34
+ value: 87.60741691468986
35
+ - type: f1
36
+ value: 91.00983458583187
37
+ - task:
38
+ type: Classification
39
+ dataset:
40
+ type: mteb/amazon_reviews_multi
41
+ name: MTEB AmazonReviewsClassification (en)
42
+ config: en
43
+ split: test
44
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
45
+ metrics:
46
+ - type: accuracy
47
+ value: 46.943999999999996
48
+ - type: f1
49
+ value: 46.33280307575562
50
+ - task:
51
+ type: Reranking
52
+ dataset:
53
+ type: mteb/askubuntudupquestions-reranking
54
+ name: MTEB AskUbuntuDupQuestions
55
+ config: default
56
+ split: test
57
+ revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
58
+ metrics:
59
+ - type: map
60
+ value: 60.75683986813218
61
+ - type: mrr
62
+ value: 73.51624675724399
63
+ - task:
64
+ type: STS
65
+ dataset:
66
+ type: mteb/biosses-sts
67
+ name: MTEB BIOSSES
68
+ config: default
69
+ split: test
70
+ revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
71
+ metrics:
72
+ - type: cos_sim_pearson
73
+ value: 89.07092347634877
74
+ - type: cos_sim_spearman
75
+ value: 87.80621759170344
76
+ - type: euclidean_pearson
77
+ value: 87.29751551472525
78
+ - type: euclidean_spearman
79
+ value: 87.5634409755362
80
+ - type: manhattan_pearson
81
+ value: 87.56100206227441
82
+ - type: manhattan_spearman
83
+ value: 87.45982415672536
84
+ - task:
85
+ type: Classification
86
+ dataset:
87
+ type: mteb/banking77
88
+ name: MTEB Banking77Classification
89
+ config: default
90
+ split: test
91
+ revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
92
+ metrics:
93
+ - type: accuracy
94
+ value: 83.46753246753246
95
+ - type: f1
96
+ value: 83.39526091362032
97
+ - task:
98
+ type: Classification
99
+ dataset:
100
+ type: mteb/emotion
101
+ name: MTEB EmotionClassification
102
+ config: default
103
+ split: test
104
+ revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
105
+ metrics:
106
+ - type: accuracy
107
+ value: 45.800000000000004
108
+ - type: f1
109
+ value: 40.76055487612189
110
+ - task:
111
+ type: Classification
112
+ dataset:
113
+ type: mteb/imdb
114
+ name: MTEB ImdbClassification
115
+ config: default
116
+ split: test
117
+ revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
118
+ metrics:
119
+ - type: accuracy
120
+ value: 85.0096
121
+ - type: ap
122
+ value: 79.91059611360778
123
+ - type: f1
124
+ value: 84.9738791599706
125
+ - task:
126
+ type: Classification
127
+ dataset:
128
+ type: mteb/mtop_domain
129
+ name: MTEB MTOPDomainClassification (en)
130
+ config: en
131
+ split: test
132
+ revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
133
+ metrics:
134
+ - type: accuracy
135
+ value: 92.51025991792065
136
+ - type: f1
137
+ value: 92.2852224639839
138
+ - task:
139
+ type: Classification
140
+ dataset:
141
+ type: mteb/mtop_intent
142
+ name: MTEB MTOPIntentClassification (en)
143
+ config: en
144
+ split: test
145
+ revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
146
+ metrics:
147
+ - type: accuracy
148
+ value: 69.61924304605563
149
+ - type: f1
150
+ value: 51.832892524807505
151
+ - task:
152
+ type: Classification
153
+ dataset:
154
+ type: mteb/amazon_massive_intent
155
+ name: MTEB MassiveIntentClassification (en)
156
+ config: en
157
+ split: test
158
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
159
+ metrics:
160
+ - type: accuracy
161
+ value: 70.2320107599193
162
+ - type: f1
163
+ value: 68.03367707473218
164
+ - task:
165
+ type: Classification
166
+ dataset:
167
+ type: mteb/amazon_massive_scenario
168
+ name: MTEB MassiveScenarioClassification (en)
169
+ config: en
170
+ split: test
171
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
172
+ metrics:
173
+ - type: accuracy
174
+ value: 75.28581035642232
175
+ - type: f1
176
+ value: 75.43554941058956
177
+ - task:
178
+ type: STS
179
+ dataset:
180
+ type: mteb/sickr-sts
181
+ name: MTEB SICK-R
182
+ config: default
183
+ split: test
184
+ revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
185
+ metrics:
186
+ - type: cos_sim_pearson
187
+ value: 83.58628262329275
188
+ - type: cos_sim_spearman
189
+ value: 77.30534089053104
190
+ - type: euclidean_pearson
191
+ value: 80.86400799226335
192
+ - type: euclidean_spearman
193
+ value: 77.26947744139412
194
+ - type: manhattan_pearson
195
+ value: 80.79442484789072
196
+ - type: manhattan_spearman
197
+ value: 77.18043722794019
198
+ - task:
199
+ type: STS
200
+ dataset:
201
+ type: mteb/sts12-sts
202
+ name: MTEB STS12
203
+ config: default
204
+ split: test
205
+ revision: a0d554a64d88156834ff5ae9920b964011b16384
206
+ metrics:
207
+ - type: cos_sim_pearson
208
+ value: 82.77293561742106
209
+ - type: cos_sim_spearman
210
+ value: 73.98616407095425
211
+ - type: euclidean_pearson
212
+ value: 78.7096804108132
213
+ - type: euclidean_spearman
214
+ value: 73.52379687387366
215
+ - type: manhattan_pearson
216
+ value: 78.80694876432868
217
+ - type: manhattan_spearman
218
+ value: 73.64907838788528
219
+ - task:
220
+ type: STS
221
+ dataset:
222
+ type: mteb/sts13-sts
223
+ name: MTEB STS13
224
+ config: default
225
+ split: test
226
+ revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
227
+ metrics:
228
+ - type: cos_sim_pearson
229
+ value: 82.12995363427328
230
+ - type: cos_sim_spearman
231
+ value: 84.23345798311749
232
+ - type: euclidean_pearson
233
+ value: 83.94003648503143
234
+ - type: euclidean_spearman
235
+ value: 84.74522675669463
236
+ - type: manhattan_pearson
237
+ value: 83.82868963165394
238
+ - type: manhattan_spearman
239
+ value: 84.61059125620956
240
+ - task:
241
+ type: STS
242
+ dataset:
243
+ type: mteb/sts14-sts
244
+ name: MTEB STS14
245
+ config: default
246
+ split: test
247
+ revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
248
+ metrics:
249
+ - type: cos_sim_pearson
250
+ value: 81.88504872832357
251
+ - type: cos_sim_spearman
252
+ value: 80.09345991196561
253
+ - type: euclidean_pearson
254
+ value: 81.99899431994811
255
+ - type: euclidean_spearman
256
+ value: 80.25520445997002
257
+ - type: manhattan_pearson
258
+ value: 81.9635758954928
259
+ - type: manhattan_spearman
260
+ value: 80.24335353637277
261
+ - task:
262
+ type: STS
263
+ dataset:
264
+ type: mteb/sts15-sts
265
+ name: MTEB STS15
266
+ config: default
267
+ split: test
268
+ revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
269
+ metrics:
270
+ - type: cos_sim_pearson
271
+ value: 86.55052353126385
272
+ - type: cos_sim_spearman
273
+ value: 88.1950992730786
274
+ - type: euclidean_pearson
275
+ value: 87.83472249083056
276
+ - type: euclidean_spearman
277
+ value: 88.43301043636015
278
+ - type: manhattan_pearson
279
+ value: 87.75102815516877
280
+ - type: manhattan_spearman
281
+ value: 88.34719608377306
282
+ - task:
283
+ type: STS
284
+ dataset:
285
+ type: mteb/sts16-sts
286
+ name: MTEB STS16
287
+ config: default
288
+ split: test
289
+ revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
290
+ metrics:
291
+ - type: cos_sim_pearson
292
+ value: 81.58832350766542
293
+ - type: cos_sim_spearman
294
+ value: 83.60857270697358
295
+ - type: euclidean_pearson
296
+ value: 82.9059299279255
297
+ - type: euclidean_spearman
298
+ value: 83.87380773329784
299
+ - type: manhattan_pearson
300
+ value: 82.76009241925925
301
+ - type: manhattan_spearman
302
+ value: 83.72876466499108
303
+ - task:
304
+ type: STS
305
+ dataset:
306
+ type: mteb/sts17-crosslingual-sts
307
+ name: MTEB STS17 (en-en)
308
+ config: en-en
309
+ split: test
310
+ revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
311
+ metrics:
312
+ - type: cos_sim_pearson
313
+ value: 87.96440735880392
314
+ - type: cos_sim_spearman
315
+ value: 87.79655666183349
316
+ - type: euclidean_pearson
317
+ value: 88.47129589774806
318
+ - type: euclidean_spearman
319
+ value: 87.95235258398374
320
+ - type: manhattan_pearson
321
+ value: 88.37144209103296
322
+ - type: manhattan_spearman
323
+ value: 87.81869790317533
324
+ - task:
325
+ type: STS
326
+ dataset:
327
+ type: mteb/sts22-crosslingual-sts
328
+ name: MTEB STS22 (en)
329
+ config: en
330
+ split: test
331
+ revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
332
+ metrics:
333
+ - type: cos_sim_pearson
334
+ value: 66.66468384683428
335
+ - type: cos_sim_spearman
336
+ value: 66.84275911821702
337
+ - type: euclidean_pearson
338
+ value: 67.73972664535547
339
+ - type: euclidean_spearman
340
+ value: 66.57863145583491
341
+ - type: manhattan_pearson
342
+ value: 67.91309920462287
343
+ - type: manhattan_spearman
344
+ value: 66.67487869242575
345
+ - task:
346
+ type: STS
347
+ dataset:
348
+ type: mteb/stsbenchmark-sts
349
+ name: MTEB STSBenchmark
350
+ config: default
351
+ split: test
352
+ revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
353
+ metrics:
354
+ - type: cos_sim_pearson
355
+ value: 84.07668437020894
356
+ - type: cos_sim_spearman
357
+ value: 85.13186558138277
358
+ - type: euclidean_pearson
359
+ value: 85.28607166042313
360
+ - type: euclidean_spearman
361
+ value: 85.25082312265897
362
+ - type: manhattan_pearson
363
+ value: 85.0870328315141
364
+ - type: manhattan_spearman
365
+ value: 85.10612962221282
366
+ - task:
367
+ type: Reranking
368
+ dataset:
369
+ type: mteb/scidocs-reranking
370
+ name: MTEB SciDocsRR
371
+ config: default
372
+ split: test
373
+ revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
374
+ metrics:
375
+ - type: map
376
+ value: 84.33835340608282
377
+ - type: mrr
378
+ value: 95.54063220729888
379
+ - task:
380
+ type: PairClassification
381
+ dataset:
382
+ type: mteb/sprintduplicatequestions-pairclassification
383
+ name: MTEB SprintDuplicateQuestions
384
+ config: default
385
+ split: test
386
+ revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
387
+ metrics:
388
+ - type: cos_sim_accuracy
389
+ value: 99.81386138613861
390
+ - type: cos_sim_ap
391
+ value: 95.49398397880566
392
+ - type: cos_sim_f1
393
+ value: 90.5050505050505
394
+ - type: cos_sim_precision
395
+ value: 91.42857142857143
396
+ - type: cos_sim_recall
397
+ value: 89.60000000000001
398
+ - type: dot_accuracy
399
+ value: 99.75742574257426
400
+ - type: dot_ap
401
+ value: 93.40675781804289
402
+ - type: dot_f1
403
+ value: 87.45519713261648
404
+ - type: dot_precision
405
+ value: 89.61175236096537
406
+ - type: dot_recall
407
+ value: 85.39999999999999
408
+ - type: euclidean_accuracy
409
+ value: 99.81485148514851
410
+ - type: euclidean_ap
411
+ value: 95.39724876386569
412
+ - type: euclidean_f1
413
+ value: 90.5793450881612
414
+ - type: euclidean_precision
415
+ value: 91.26903553299492
416
+ - type: euclidean_recall
417
+ value: 89.9
418
+ - type: manhattan_accuracy
419
+ value: 99.81485148514851
420
+ - type: manhattan_ap
421
+ value: 95.46515830873487
422
+ - type: manhattan_f1
423
+ value: 90.56974459724951
424
+ - type: manhattan_precision
425
+ value: 88.996138996139
426
+ - type: manhattan_recall
427
+ value: 92.2
428
+ - type: max_accuracy
429
+ value: 99.81485148514851
430
+ - type: max_ap
431
+ value: 95.49398397880566
432
+ - type: max_f1
433
+ value: 90.5793450881612
434
+ - task:
435
+ type: Reranking
436
+ dataset:
437
+ type: mteb/stackoverflowdupquestions-reranking
438
+ name: MTEB StackOverflowDupQuestions
439
+ config: default
440
+ split: test
441
+ revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
442
+ metrics:
443
+ - type: map
444
+ value: 51.68384236354744
445
+ - type: mrr
446
+ value: 52.52933749257278
447
+ - task:
448
+ type: Classification
449
+ dataset:
450
+ type: mteb/toxic_conversations_50k
451
+ name: MTEB ToxicConversationsClassification
452
+ config: default
453
+ split: test
454
+ revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
455
+ metrics:
456
+ - type: accuracy
457
+ value: 69.7972
458
+ - type: ap
459
+ value: 13.790209566654962
460
+ - type: f1
461
+ value: 53.73625700975159
462
+ - task:
463
+ type: Classification
464
+ dataset:
465
+ type: mteb/tweet_sentiment_extraction
466
+ name: MTEB TweetSentimentExtractionClassification
467
+ config: default
468
+ split: test
469
+ revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
470
+ metrics:
471
+ - type: accuracy
472
+ value: 57.81550650820599
473
+ - type: f1
474
+ value: 58.22494506904567
475
+ - task:
476
+ type: PairClassification
477
+ dataset:
478
+ type: mteb/twittersemeval2015-pairclassification
479
+ name: MTEB TwitterSemEval2015
480
+ config: default
481
+ split: test
482
+ revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
483
+ metrics:
484
+ - type: cos_sim_accuracy
485
+ value: 84.30589497526375
486
+ - type: cos_sim_ap
487
+ value: 68.60854966172107
488
+ - type: cos_sim_f1
489
+ value: 65.06926244852113
490
+ - type: cos_sim_precision
491
+ value: 61.733364906464594
492
+ - type: cos_sim_recall
493
+ value: 68.7862796833773
494
+ - type: dot_accuracy
495
+ value: 81.63557250998392
496
+ - type: dot_ap
497
+ value: 58.80135920860792
498
+ - type: dot_f1
499
+ value: 57.39889705882353
500
+ - type: dot_precision
501
+ value: 50.834350834350836
502
+ - type: dot_recall
503
+ value: 65.91029023746702
504
+ - type: euclidean_accuracy
505
+ value: 84.37742146986946
506
+ - type: euclidean_ap
507
+ value: 68.88494996210581
508
+ - type: euclidean_f1
509
+ value: 65.23647001462702
510
+ - type: euclidean_precision
511
+ value: 60.62528318985048
512
+ - type: euclidean_recall
513
+ value: 70.60686015831135
514
+ - type: manhattan_accuracy
515
+ value: 84.21648685700661
516
+ - type: manhattan_ap
517
+ value: 68.54917405273397
518
+ - type: manhattan_f1
519
+ value: 64.97045701193778
520
+ - type: manhattan_precision
521
+ value: 59.826782145236514
522
+ - type: manhattan_recall
523
+ value: 71.08179419525065
524
+ - type: max_accuracy
525
+ value: 84.37742146986946
526
+ - type: max_ap
527
+ value: 68.88494996210581
528
+ - type: max_f1
529
+ value: 65.23647001462702
530
+ - task:
531
+ type: PairClassification
532
+ dataset:
533
+ type: mteb/twitterurlcorpus-pairclassification
534
+ name: MTEB TwitterURLCorpus
535
+ config: default
536
+ split: test
537
+ revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
538
+ metrics:
539
+ - type: cos_sim_accuracy
540
+ value: 88.60752124810804
541
+ - type: cos_sim_ap
542
+ value: 85.16030341274225
543
+ - type: cos_sim_f1
544
+ value: 77.50186985789081
545
+ - type: cos_sim_precision
546
+ value: 75.34904013961605
547
+ - type: cos_sim_recall
548
+ value: 79.781336618417
549
+ - type: dot_accuracy
550
+ value: 86.00147475453099
551
+ - type: dot_ap
552
+ value: 79.24446611557556
553
+ - type: dot_f1
554
+ value: 72.34317740892433
555
+ - type: dot_precision
556
+ value: 67.81624680048498
557
+ - type: dot_recall
558
+ value: 77.51770865414228
559
+ - type: euclidean_accuracy
560
+ value: 88.7026041060271
561
+ - type: euclidean_ap
562
+ value: 85.30879801684605
563
+ - type: euclidean_f1
564
+ value: 77.60992108229988
565
+ - type: euclidean_precision
566
+ value: 75.80384671854354
567
+ - type: euclidean_recall
568
+ value: 79.50415768401602
569
+ - type: manhattan_accuracy
570
+ value: 88.75305623471883
571
+ - type: manhattan_ap
572
+ value: 85.24656615741652
573
+ - type: manhattan_f1
574
+ value: 77.5542141739325
575
+ - type: manhattan_precision
576
+ value: 75.14079422382672
577
+ - type: manhattan_recall
578
+ value: 80.12781028641824
579
+ - type: max_accuracy
580
+ value: 88.75305623471883
581
+ - type: max_ap
582
+ value: 85.30879801684605
583
+ - type: max_f1
584
+ value: 77.60992108229988
585
  license: mit
586
  language:
587
  - en