metadata
base_model: almanach/camembert-base
datasets:
- CATIE-AQ/frenchSTS
- CATIE-AQ/frenchNLI
language: fr
library_name: sentence-transformers
tags:
- sentence-transformers
- sentence-similarity
pipeline_tag: sentence-similarity
model-index:
- name: CATIE-AQ/camembert-base-embedding
results:
- dataset:
config: default
name: MTEB AlloProfClusteringP2P (default)
revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b
split: test
type: lyon-nlp/alloprof
metrics:
- type: main_score
value: 62.81514840418616
- type: v_measure
value: 62.81514840418616
- type: v_measure_std
value: 3.132725810965268
task:
type: Clustering
- dataset:
config: default
name: MTEB AlloProfClusteringS2S (default)
revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b
split: test
type: lyon-nlp/alloprof
metrics:
- type: main_score
value: 45.87590134482225
- type: v_measure
value: 45.87590134482225
- type: v_measure_std
value: 3.1798952505373275
task:
type: Clustering
- dataset:
config: default
name: MTEB AlloprofReranking (default)
revision: 65393d0d7a08a10b4e348135e824f385d420b0fd
split: test
type: lyon-nlp/mteb-fr-reranking-alloprof-s2p
metrics:
- type: main_score
value: 64.6956883099902
- type: map
value: 64.6956883099902
- type: mrr
value: 65.92878731803602
- type: nAUC_map_diff1
value: 43.11417842279244
- type: nAUC_map_max
value: 11.830624365241297
- type: nAUC_map_std
value: 27.41656377559847
- type: nAUC_mrr_diff1
value: 42.33557798726354
- type: nAUC_mrr_max
value: 11.698885231653506
- type: nAUC_mrr_std
value: 27.15603079445034
task:
type: Reranking
- dataset:
config: default
name: MTEB AlloprofRetrieval (default)
revision: fcf295ea64c750f41fadbaa37b9b861558e1bfbd
split: test
type: lyon-nlp/alloprof
metrics:
- type: main_score
value: 32.765
- type: map_at_1
value: 19.473
- type: map_at_10
value: 27.994999999999997
- type: map_at_100
value: 29.104999999999997
- type: map_at_1000
value: 29.199
- type: map_at_20
value: 28.63
- type: map_at_3
value: 25.396
- type: map_at_5
value: 26.745
- type: mrr_at_1
value: 19.473229706390327
- type: mrr_at_10
value: 27.995140773638184
- type: mrr_at_100
value: 29.10470977277557
- type: mrr_at_1000
value: 29.19859919011471
- type: mrr_at_20
value: 28.629888260231024
- type: mrr_at_3
value: 25.3957973517559
- type: mrr_at_5
value: 26.745106505469195
- type: nauc_map_at_1000_diff1
value: 30.241033919960458
- type: nauc_map_at_1000_max
value: 27.690496841937534
- type: nauc_map_at_1000_std
value: 2.646985164121684
- type: nauc_map_at_100_diff1
value: 30.220842681045223
- type: nauc_map_at_100_max
value: 27.711541510509484
- type: nauc_map_at_100_std
value: 2.664964721880645
- type: nauc_map_at_10_diff1
value: 30.110964192209465
- type: nauc_map_at_10_max
value: 27.36351483446033
- type: nauc_map_at_10_std
value: 2.2690551732906017
- type: nauc_map_at_1_diff1
value: 36.57849195648472
- type: nauc_map_at_1_max
value: 24.41828027807538
- type: nauc_map_at_1_std
value: -1.4959086055716733
- type: nauc_map_at_20_diff1
value: 30.264433733622155
- type: nauc_map_at_20_max
value: 27.609024309353856
- type: nauc_map_at_20_std
value: 2.4900033832350257
- type: nauc_map_at_3_diff1
value: 30.977925617910252
- type: nauc_map_at_3_max
value: 26.503174401427348
- type: nauc_map_at_3_std
value: 1.1409136908689934
- type: nauc_map_at_5_diff1
value: 30.40381270754491
- type: nauc_map_at_5_max
value: 27.02396755802422
- type: nauc_map_at_5_std
value: 1.8889735596815345
- type: nauc_mrr_at_1000_diff1
value: 30.241033919960458
- type: nauc_mrr_at_1000_max
value: 27.690496841937534
- type: nauc_mrr_at_1000_std
value: 2.646985164121684
- type: nauc_mrr_at_100_diff1
value: 30.220842681045223
- type: nauc_mrr_at_100_max
value: 27.711541510509484
- type: nauc_mrr_at_100_std
value: 2.664964721880645
- type: nauc_mrr_at_10_diff1
value: 30.110964192209465
- type: nauc_mrr_at_10_max
value: 27.36351483446033
- type: nauc_mrr_at_10_std
value: 2.2690551732906017
- type: nauc_mrr_at_1_diff1
value: 36.57849195648472
- type: nauc_mrr_at_1_max
value: 24.41828027807538
- type: nauc_mrr_at_1_std
value: -1.4959086055716733
- type: nauc_mrr_at_20_diff1
value: 30.264433733622155
- type: nauc_mrr_at_20_max
value: 27.609024309353856
- type: nauc_mrr_at_20_std
value: 2.4900033832350257
- type: nauc_mrr_at_3_diff1
value: 30.977925617910252
- type: nauc_mrr_at_3_max
value: 26.503174401427348
- type: nauc_mrr_at_3_std
value: 1.1409136908689934
- type: nauc_mrr_at_5_diff1
value: 30.40381270754491
- type: nauc_mrr_at_5_max
value: 27.02396755802422
- type: nauc_mrr_at_5_std
value: 1.8889735596815345
- type: nauc_ndcg_at_1000_diff1
value: 28.781294362482978
- type: nauc_ndcg_at_1000_max
value: 29.586129771015806
- type: nauc_ndcg_at_1000_std
value: 5.201207556039119
- type: nauc_ndcg_at_100_diff1
value: 28.271541462228537
- type: nauc_ndcg_at_100_max
value: 30.73005674016489
- type: nauc_ndcg_at_100_std
value: 6.428375999185625
- type: nauc_ndcg_at_10_diff1
value: 27.758657548840677
- type: nauc_ndcg_at_10_max
value: 28.904484139374652
- type: nauc_ndcg_at_10_std
value: 4.18074095938288
- type: nauc_ndcg_at_1_diff1
value: 36.57849195648472
- type: nauc_ndcg_at_1_max
value: 24.41828027807538
- type: nauc_ndcg_at_1_std
value: -1.4959086055716733
- type: nauc_ndcg_at_20_diff1
value: 28.325927584145415
- type: nauc_ndcg_at_20_max
value: 29.781188360596445
- type: nauc_ndcg_at_20_std
value: 5.018358478792948
- type: nauc_ndcg_at_3_diff1
value: 29.453679222478712
- type: nauc_ndcg_at_3_max
value: 27.169325446315863
- type: nauc_ndcg_at_3_std
value: 1.959274197035818
- type: nauc_ndcg_at_5_diff1
value: 28.4305777218966
- type: nauc_ndcg_at_5_max
value: 28.039534691743157
- type: nauc_ndcg_at_5_std
value: 3.2216617298195387
- type: nauc_precision_at_1000_diff1
value: 13.929965402261416
- type: nauc_precision_at_1000_max
value: 62.06764851885686
- type: nauc_precision_at_1000_std
value: 60.38250744674588
- type: nauc_precision_at_100_diff1
value: 21.05204958969327
- type: nauc_precision_at_100_max
value: 48.28535293362649
- type: nauc_precision_at_100_std
value: 28.391655966980906
- type: nauc_precision_at_10_diff1
value: 21.314944216545907
- type: nauc_precision_at_10_max
value: 33.38194006739203
- type: nauc_precision_at_10_std
value: 9.688442640271612
- type: nauc_precision_at_1_diff1
value: 36.57849195648472
- type: nauc_precision_at_1_max
value: 24.41828027807538
- type: nauc_precision_at_1_std
value: -1.4959086055716733
- type: nauc_precision_at_20_diff1
value: 23.08727857679785
- type: nauc_precision_at_20_max
value: 37.08955712841807
- type: nauc_precision_at_20_std
value: 13.417543826400546
- type: nauc_precision_at_3_diff1
value: 25.520470746119884
- type: nauc_precision_at_3_max
value: 28.917460502087827
- type: nauc_precision_at_3_std
value: 4.100765101479203
- type: nauc_precision_at_5_diff1
value: 23.283276148921086
- type: nauc_precision_at_5_max
value: 30.730432699902373
- type: nauc_precision_at_5_std
value: 6.764994172330351
- type: nauc_recall_at_1000_diff1
value: 13.929965402260464
- type: nauc_recall_at_1000_max
value: 62.06764851885672
- type: nauc_recall_at_1000_std
value: 60.38250744674567
- type: nauc_recall_at_100_diff1
value: 21.052049589693247
- type: nauc_recall_at_100_max
value: 48.28535293362645
- type: nauc_recall_at_100_std
value: 28.39165596698092
- type: nauc_recall_at_10_diff1
value: 21.314944216545946
- type: nauc_recall_at_10_max
value: 33.38194006739202
- type: nauc_recall_at_10_std
value: 9.68844264027159
- type: nauc_recall_at_1_diff1
value: 36.57849195648472
- type: nauc_recall_at_1_max
value: 24.41828027807538
- type: nauc_recall_at_1_std
value: -1.4959086055716733
- type: nauc_recall_at_20_diff1
value: 23.08727857679781
- type: nauc_recall_at_20_max
value: 37.08955712841808
- type: nauc_recall_at_20_std
value: 13.417543826400491
- type: nauc_recall_at_3_diff1
value: 25.520470746119877
- type: nauc_recall_at_3_max
value: 28.91746050208786
- type: nauc_recall_at_3_std
value: 4.100765101479203
- type: nauc_recall_at_5_diff1
value: 23.28327614892109
- type: nauc_recall_at_5_max
value: 30.730432699902366
- type: nauc_recall_at_5_std
value: 6.764994172330383
- type: ndcg_at_1
value: 19.473
- type: ndcg_at_10
value: 32.765
- type: ndcg_at_100
value: 38.619
- type: ndcg_at_1000
value: 41.23
- type: ndcg_at_20
value: 35.071999999999996
- type: ndcg_at_3
value: 27.334000000000003
- type: ndcg_at_5
value: 29.768
- type: precision_at_1
value: 19.473
- type: precision_at_10
value: 4.806
- type: precision_at_100
value: 0.765
- type: precision_at_1000
value: 0.097
- type: precision_at_20
value: 2.858
- type: precision_at_3
value: 10.982
- type: precision_at_5
value: 7.771999999999999
- type: recall_at_1
value: 19.473
- type: recall_at_10
value: 48.057
- type: recall_at_100
value: 76.511
- type: recall_at_1000
value: 97.28
- type: recall_at_20
value: 57.168
- type: recall_at_3
value: 32.945
- type: recall_at_5
value: 38.86
task:
type: Retrieval
- dataset:
config: fr
name: MTEB AmazonReviewsClassification (fr)
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
split: test
type: mteb/amazon_reviews_multi
metrics:
- type: accuracy
value: 38.647999999999996
- type: f1
value: 38.208525191489606
- type: f1_weighted
value: 38.2085251914896
- type: main_score
value: 38.647999999999996
task:
type: Classification
- dataset:
config: default
name: MTEB BSARDRetrieval (default)
revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59
split: test
type: maastrichtlawtech/bsard
metrics:
- type: main_score
value: 51.80200000000001
- type: map_at_1
value: 5.856
- type: map_at_10
value: 10.335999999999999
- type: map_at_100
value: 11.371
- type: map_at_1000
value: 11.484
- type: map_at_20
value: 10.905
- type: map_at_3
value: 8.483
- type: map_at_5
value: 9.272
- type: mrr_at_1
value: 5.8558558558558556
- type: mrr_at_10
value: 10.336407836407837
- type: mrr_at_100
value: 11.371270723447942
- type: mrr_at_1000
value: 11.48380324519674
- type: mrr_at_20
value: 10.90535085891123
- type: mrr_at_3
value: 8.483483483483482
- type: mrr_at_5
value: 9.271771771771771
- type: nauc_map_at_1000_diff1
value: 23.998881641118615
- type: nauc_map_at_1000_max
value: 11.451148757019597
- type: nauc_map_at_1000_std
value: -6.095276818382939
- type: nauc_map_at_100_diff1
value: 24.016804645947758
- type: nauc_map_at_100_max
value: 11.463837274810707
- type: nauc_map_at_100_std
value: -6.143667423142305
- type: nauc_map_at_10_diff1
value: 25.285515257226137
- type: nauc_map_at_10_max
value: 11.439661938133314
- type: nauc_map_at_10_std
value: -7.172016283125754
- type: nauc_map_at_1_diff1
value: 33.126157053422425
- type: nauc_map_at_1_max
value: 7.841998655018221
- type: nauc_map_at_1_std
value: -1.974070057397683
- type: nauc_map_at_20_diff1
value: 24.008416731789556
- type: nauc_map_at_20_max
value: 11.171880025274657
- type: nauc_map_at_20_std
value: -6.812377451564844
- type: nauc_map_at_3_diff1
value: 25.56328629217215
- type: nauc_map_at_3_max
value: 11.314106166746091
- type: nauc_map_at_3_std
value: -4.81459592066856
- type: nauc_map_at_5_diff1
value: 22.918643874830273
- type: nauc_map_at_5_max
value: 10.752530109615206
- type: nauc_map_at_5_std
value: -5.449673066849446
- type: nauc_mrr_at_1000_diff1
value: 23.998881641118615
- type: nauc_mrr_at_1000_max
value: 11.451148757019597
- type: nauc_mrr_at_1000_std
value: -6.095276818382939
- type: nauc_mrr_at_100_diff1
value: 24.016804645947758
- type: nauc_mrr_at_100_max
value: 11.463837274810707
- type: nauc_mrr_at_100_std
value: -6.143667423142305
- type: nauc_mrr_at_10_diff1
value: 25.285515257226137
- type: nauc_mrr_at_10_max
value: 11.439661938133314
- type: nauc_mrr_at_10_std
value: -7.172016283125754
- type: nauc_mrr_at_1_diff1
value: 33.126157053422425
- type: nauc_mrr_at_1_max
value: 7.841998655018221
- type: nauc_mrr_at_1_std
value: -1.974070057397683
- type: nauc_mrr_at_20_diff1
value: 24.008416731789556
- type: nauc_mrr_at_20_max
value: 11.171880025274657
- type: nauc_mrr_at_20_std
value: -6.812377451564844
- type: nauc_mrr_at_3_diff1
value: 25.56328629217215
- type: nauc_mrr_at_3_max
value: 11.314106166746091
- type: nauc_mrr_at_3_std
value: -4.81459592066856
- type: nauc_mrr_at_5_diff1
value: 22.918643874830273
- type: nauc_mrr_at_5_max
value: 10.752530109615206
- type: nauc_mrr_at_5_std
value: -5.449673066849446
- type: nauc_ndcg_at_1000_diff1
value: 21.11846424892469
- type: nauc_ndcg_at_1000_max
value: 13.762965589541624
- type: nauc_ndcg_at_1000_std
value: -0.9049620104888738
- type: nauc_ndcg_at_100_diff1
value: 21.04760573646968
- type: nauc_ndcg_at_100_max
value: 13.566320323833287
- type: nauc_ndcg_at_100_std
value: -2.890976829670434
- type: nauc_ndcg_at_10_diff1
value: 24.272788631698305
- type: nauc_ndcg_at_10_max
value: 11.978861552095738
- type: nauc_ndcg_at_10_std
value: -9.76679807599373
- type: nauc_ndcg_at_1_diff1
value: 33.126157053422425
- type: nauc_ndcg_at_1_max
value: 7.841998655018221
- type: nauc_ndcg_at_1_std
value: -1.974070057397683
- type: nauc_ndcg_at_20_diff1
value: 20.856189995510878
- type: nauc_ndcg_at_20_max
value: 11.375601238002009
- type: nauc_ndcg_at_20_std
value: -8.265683285756289
- type: nauc_ndcg_at_3_diff1
value: 23.996883168471285
- type: nauc_ndcg_at_3_max
value: 11.83103272225721
- type: nauc_ndcg_at_3_std
value: -5.6278054989183675
- type: nauc_ndcg_at_5_diff1
value: 19.74654034525485
- type: nauc_ndcg_at_5_max
value: 10.82478247800965
- type: nauc_ndcg_at_5_std
value: -6.483124115754927
- type: nauc_precision_at_1000_diff1
value: 12.438592768281895
- type: nauc_precision_at_1000_max
value: 27.28154735140418
- type: nauc_precision_at_1000_std
value: 39.53231591230107
- type: nauc_precision_at_100_diff1
value: 15.98575745798744
- type: nauc_precision_at_100_max
value: 19.26116052862835
- type: nauc_precision_at_100_std
value: 9.34249516386986
- type: nauc_precision_at_10_diff1
value: 22.95439189046463
- type: nauc_precision_at_10_max
value: 12.790416123390521
- type: nauc_precision_at_10_std
value: -14.454663912966312
- type: nauc_precision_at_1_diff1
value: 33.126157053422425
- type: nauc_precision_at_1_max
value: 7.841998655018221
- type: nauc_precision_at_1_std
value: -1.974070057397683
- type: nauc_precision_at_20_diff1
value: 15.361565724277282
- type: nauc_precision_at_20_max
value: 11.502927678745186
- type: nauc_precision_at_20_std
value: -10.009814161653562
- type: nauc_precision_at_3_diff1
value: 20.622257983195322
- type: nauc_precision_at_3_max
value: 12.893388052538466
- type: nauc_precision_at_3_std
value: -7.436820520489828
- type: nauc_precision_at_5_diff1
value: 13.263055121165085
- type: nauc_precision_at_5_max
value: 10.790024461151273
- type: nauc_precision_at_5_std
value: -8.519678351816069
- type: nauc_recall_at_1000_diff1
value: 12.438592768281932
- type: nauc_recall_at_1000_max
value: 27.28154735140412
- type: nauc_recall_at_1000_std
value: 39.53231591230112
- type: nauc_recall_at_100_diff1
value: 15.98575745798741
- type: nauc_recall_at_100_max
value: 19.261160528628295
- type: nauc_recall_at_100_std
value: 9.342495163869868
- type: nauc_recall_at_10_diff1
value: 22.954391890464585
- type: nauc_recall_at_10_max
value: 12.790416123390461
- type: nauc_recall_at_10_std
value: -14.454663912966359
- type: nauc_recall_at_1_diff1
value: 33.126157053422425
- type: nauc_recall_at_1_max
value: 7.841998655018221
- type: nauc_recall_at_1_std
value: -1.974070057397683
- type: nauc_recall_at_20_diff1
value: 15.361565724277243
- type: nauc_recall_at_20_max
value: 11.502927678745152
- type: nauc_recall_at_20_std
value: -10.00981416165357
- type: nauc_recall_at_3_diff1
value: 20.622257983195322
- type: nauc_recall_at_3_max
value: 12.893388052538457
- type: nauc_recall_at_3_std
value: -7.436820520489838
- type: nauc_recall_at_5_diff1
value: 13.263055121165095
- type: nauc_recall_at_5_max
value: 10.790024461151285
- type: nauc_recall_at_5_std
value: -8.519678351816067
- type: ndcg_at_1
value: 5.856
- type: ndcg_at_10
value: 13.458
- type: ndcg_at_100
value: 19.103
- type: ndcg_at_1000
value: 22.706
- type: ndcg_at_20
value: 15.516
- type: ndcg_at_3
value: 9.315
- type: ndcg_at_5
value: 10.768
- type: precision_at_1
value: 5.856
- type: precision_at_10
value: 2.387
- type: precision_at_100
value: 0.518
- type: precision_at_1000
value: 0.082
- type: precision_at_20
value: 1.599
- type: precision_at_3
value: 3.904
- type: precision_at_5
value: 3.063
- type: recall_at_1
value: 5.856
- type: recall_at_10
value: 23.874000000000002
- type: recall_at_100
value: 51.80200000000001
- type: recall_at_1000
value: 81.53200000000001
- type: recall_at_20
value: 31.982
- type: recall_at_3
value: 11.712
- type: recall_at_5
value: 15.315000000000001
task:
type: Retrieval
- dataset:
config: default
name: MTEB HALClusteringS2S (default)
revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915
split: test
type: lyon-nlp/clustering-hal-s2s
metrics:
- type: main_score
value: 22.977568920606508
- type: v_measure
value: 22.977568920606508
- type: v_measure_std
value: 2.955082488913792
task:
type: Clustering
- dataset:
config: fr
name: MTEB MLSUMClusteringP2P (fr)
revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7
split: test
type: reciTAL/mlsum
metrics:
- type: main_score
value: 45.14143832379743
- type: v_measure
value: 45.14143832379743
- type: v_measure_std
value: 1.839999362285958
task:
type: Clustering
- dataset:
config: fr
name: MTEB MLSUMClusteringS2S (fr)
revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7
split: test
type: reciTAL/mlsum
metrics:
- type: main_score
value: 44.33871248859759
- type: v_measure
value: 44.33871248859759
- type: v_measure_std
value: 2.3753641237309555
task:
type: Clustering
- dataset:
config: fr
name: MTEB MTOPDomainClassification (fr)
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
split: test
type: mteb/mtop_domain
metrics:
- type: accuracy
value: 83.13185092389602
- type: f1
value: 83.26194336267176
- type: f1_weighted
value: 83.20435337584968
- type: main_score
value: 83.13185092389602
task:
type: Classification
- dataset:
config: fr
name: MTEB MTOPIntentClassification (fr)
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
split: test
type: mteb/mtop_intent
metrics:
- type: accuracy
value: 64.4127779517695
- type: f1
value: 46.121296881617425
- type: f1_weighted
value: 67.79937396761608
- type: main_score
value: 64.4127779517695
task:
type: Classification
- dataset:
config: fra
name: MTEB MasakhaNEWSClassification (fra)
revision: 18193f187b92da67168c655c9973a165ed9593dd
split: test
type: mteb/masakhanews
metrics:
- type: accuracy
value: 75.87677725118482
- type: f1
value: 72.33385300453897
- type: f1_weighted
value: 76.33530272965092
- type: main_score
value: 75.87677725118482
task:
type: Classification
- dataset:
config: fra
name: MTEB MasakhaNEWSClusteringP2P (fra)
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
split: test
type: masakhane/masakhanews
metrics:
- type: main_score
value: 63.530462963627464
- type: v_measure
value: 63.530462963627464
- type: v_measure_std
value: 35.259382473374814
task:
type: Clustering
- dataset:
config: fra
name: MTEB MasakhaNEWSClusteringS2S (fra)
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
split: test
type: masakhane/masakhanews
metrics:
- type: main_score
value: 33.19255819863289
- type: v_measure
value: 33.19255819863289
- type: v_measure_std
value: 38.312807275467
task:
type: Clustering
- dataset:
config: fr
name: MTEB MassiveIntentClassification (fr)
revision: 4672e20407010da34463acc759c162ca9734bca6
split: test
type: mteb/amazon_massive_intent
metrics:
- type: accuracy
value: 64.35776731674513
- type: f1
value: 62.09518087404688
- type: f1_weighted
value: 64.82712755229456
- type: main_score
value: 64.35776731674513
task:
type: Classification
- dataset:
config: fr
name: MTEB MassiveScenarioClassification (fr)
revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
split: test
type: mteb/amazon_massive_scenario
metrics:
- type: accuracy
value: 70.27236045729657
- type: f1
value: 70.42233098566706
- type: f1_weighted
value: 70.47699232511306
- type: main_score
value: 70.27236045729657
task:
type: Classification
- dataset:
config: fr
name: MTEB MintakaRetrieval (fr)
revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e
split: test
type: jinaai/mintakaqa
metrics:
- type: main_score
value: 19.725
- type: map_at_1
value: 10.401
- type: map_at_10
value: 16.332
- type: map_at_100
value: 17.253
- type: map_at_1000
value: 17.383000000000003
- type: map_at_20
value: 16.834
- type: map_at_3
value: 14.353
- type: map_at_5
value: 15.547
- type: mrr_at_1
value: 10.4013104013104
- type: mrr_at_10
value: 16.331965081965084
- type: mrr_at_100
value: 17.252844638794183
- type: mrr_at_1000
value: 17.382941774504403
- type: mrr_at_20
value: 16.833716866611603
- type: mrr_at_3
value: 14.352989352989354
- type: mrr_at_5
value: 15.546683046683047
- type: nauc_map_at_1000_diff1
value: 13.274461283694494
- type: nauc_map_at_1000_max
value: 21.400004682214327
- type: nauc_map_at_1000_std
value: 11.814086789693494
- type: nauc_map_at_100_diff1
value: 13.227192364413309
- type: nauc_map_at_100_max
value: 21.40460916292035
- type: nauc_map_at_100_std
value: 11.841814584576975
- type: nauc_map_at_10_diff1
value: 13.5469197720684
- type: nauc_map_at_10_max
value: 21.60503097576083
- type: nauc_map_at_10_std
value: 11.451034692800333
- type: nauc_map_at_1_diff1
value: 23.295783817688516
- type: nauc_map_at_1_max
value: 19.241866825929467
- type: nauc_map_at_1_std
value: 5.632198991737862
- type: nauc_map_at_20_diff1
value: 13.410950031613398
- type: nauc_map_at_20_max
value: 21.56234915384663
- type: nauc_map_at_20_std
value: 11.733058466598747
- type: nauc_map_at_3_diff1
value: 16.332989633746436
- type: nauc_map_at_3_max
value: 21.07236981362303
- type: nauc_map_at_3_std
value: 8.860732553996627
- type: nauc_map_at_5_diff1
value: 14.35059347765584
- type: nauc_map_at_5_max
value: 21.565556594465317
- type: nauc_map_at_5_std
value: 10.77858599641232
- type: nauc_mrr_at_1000_diff1
value: 13.274461283694494
- type: nauc_mrr_at_1000_max
value: 21.400004682214327
- type: nauc_mrr_at_1000_std
value: 11.814086789693494
- type: nauc_mrr_at_100_diff1
value: 13.227192364413309
- type: nauc_mrr_at_100_max
value: 21.40460916292035
- type: nauc_mrr_at_100_std
value: 11.841814584576975
- type: nauc_mrr_at_10_diff1
value: 13.5469197720684
- type: nauc_mrr_at_10_max
value: 21.60503097576083
- type: nauc_mrr_at_10_std
value: 11.451034692800333
- type: nauc_mrr_at_1_diff1
value: 23.295783817688516
- type: nauc_mrr_at_1_max
value: 19.241866825929467
- type: nauc_mrr_at_1_std
value: 5.632198991737862
- type: nauc_mrr_at_20_diff1
value: 13.410950031613398
- type: nauc_mrr_at_20_max
value: 21.56234915384663
- type: nauc_mrr_at_20_std
value: 11.733058466598747
- type: nauc_mrr_at_3_diff1
value: 16.332989633746436
- type: nauc_mrr_at_3_max
value: 21.07236981362303
- type: nauc_mrr_at_3_std
value: 8.860732553996627
- type: nauc_mrr_at_5_diff1
value: 14.35059347765584
- type: nauc_mrr_at_5_max
value: 21.565556594465317
- type: nauc_mrr_at_5_std
value: 10.77858599641232
- type: nauc_ndcg_at_1000_diff1
value: 9.624519184835286
- type: nauc_ndcg_at_1000_max
value: 21.112312234571796
- type: nauc_ndcg_at_1000_std
value: 14.61165208239704
- type: nauc_ndcg_at_100_diff1
value: 8.467300115659219
- type: nauc_ndcg_at_100_max
value: 20.998551549861922
- type: nauc_ndcg_at_100_std
value: 15.617444412498482
- type: nauc_ndcg_at_10_diff1
value: 9.794329961155967
- type: nauc_ndcg_at_10_max
value: 22.21307447982454
- type: nauc_ndcg_at_10_std
value: 14.140470010292761
- type: nauc_ndcg_at_1_diff1
value: 23.295783817688516
- type: nauc_ndcg_at_1_max
value: 19.241866825929467
- type: nauc_ndcg_at_1_std
value: 5.632198991737862
- type: nauc_ndcg_at_20_diff1
value: 9.471394907455924
- type: nauc_ndcg_at_20_max
value: 22.017754976774302
- type: nauc_ndcg_at_20_std
value: 14.925299910164098
- type: nauc_ndcg_at_3_diff1
value: 14.628461006556087
- type: nauc_ndcg_at_3_max
value: 21.327667951962866
- type: nauc_ndcg_at_3_std
value: 9.584594104651016
- type: nauc_ndcg_at_5_diff1
value: 11.491269733236242
- type: nauc_ndcg_at_5_max
value: 22.119385774444975
- type: nauc_ndcg_at_5_std
value: 12.66004487696584
- type: nauc_precision_at_1000_diff1
value: -10.6610303615593
- type: nauc_precision_at_1000_max
value: 16.293618723581197
- type: nauc_precision_at_1000_std
value: 29.115987282239725
- type: nauc_precision_at_100_diff1
value: -3.54934760447827
- type: nauc_precision_at_100_max
value: 18.550090074498577
- type: nauc_precision_at_100_std
value: 25.594349823082478
- type: nauc_precision_at_10_diff1
value: 1.5964851660196469
- type: nauc_precision_at_10_max
value: 23.491559294184867
- type: nauc_precision_at_10_std
value: 20.18314104053199
- type: nauc_precision_at_1_diff1
value: 23.295783817688516
- type: nauc_precision_at_1_max
value: 19.241866825929467
- type: nauc_precision_at_1_std
value: 5.632198991737862
- type: nauc_precision_at_20_diff1
value: 1.0490694050962388
- type: nauc_precision_at_20_max
value: 22.804832311240293
- type: nauc_precision_at_20_std
value: 22.12336463692607
- type: nauc_precision_at_3_diff1
value: 10.71150665032358
- type: nauc_precision_at_3_max
value: 21.864712928603865
- type: nauc_precision_at_3_std
value: 11.230891649165665
- type: nauc_precision_at_5_diff1
value: 5.0972783500259435
- type: nauc_precision_at_5_max
value: 23.332591677345178
- type: nauc_precision_at_5_std
value: 16.945600967239475
- type: nauc_recall_at_1000_diff1
value: -10.661030361559408
- type: nauc_recall_at_1000_max
value: 16.293618723581222
- type: nauc_recall_at_1000_std
value: 29.115987282239836
- type: nauc_recall_at_100_diff1
value: -3.549347604478282
- type: nauc_recall_at_100_max
value: 18.550090074498538
- type: nauc_recall_at_100_std
value: 25.59434982308244
- type: nauc_recall_at_10_diff1
value: 1.5964851660196395
- type: nauc_recall_at_10_max
value: 23.49155929418485
- type: nauc_recall_at_10_std
value: 20.18314104053199
- type: nauc_recall_at_1_diff1
value: 23.295783817688516
- type: nauc_recall_at_1_max
value: 19.241866825929467
- type: nauc_recall_at_1_std
value: 5.632198991737862
- type: nauc_recall_at_20_diff1
value: 1.0490694050962253
- type: nauc_recall_at_20_max
value: 22.804832311240265
- type: nauc_recall_at_20_std
value: 22.12336463692609
- type: nauc_recall_at_3_diff1
value: 10.711506650323582
- type: nauc_recall_at_3_max
value: 21.864712928603858
- type: nauc_recall_at_3_std
value: 11.23089164916565
- type: nauc_recall_at_5_diff1
value: 5.097278350025921
- type: nauc_recall_at_5_max
value: 23.332591677345178
- type: nauc_recall_at_5_std
value: 16.945600967239464
- type: ndcg_at_1
value: 10.401
- type: ndcg_at_10
value: 19.725
- type: ndcg_at_100
value: 24.773999999999997
- type: ndcg_at_1000
value: 29.073999999999998
- type: ndcg_at_20
value: 21.57
- type: ndcg_at_3
value: 15.651000000000002
- type: ndcg_at_5
value: 17.798
- type: precision_at_1
value: 10.401
- type: precision_at_10
value: 3.063
- type: precision_at_100
value: 0.5559999999999999
- type: precision_at_1000
value: 0.091
- type: precision_at_20
value: 1.8980000000000001
- type: precision_at_3
value: 6.47
- type: precision_at_5
value: 4.922
- type: recall_at_1
value: 10.401
- type: recall_at_10
value: 30.631000000000004
- type: recall_at_100
value: 55.569
- type: recall_at_1000
value: 91.31899999999999
- type: recall_at_20
value: 37.961
- type: recall_at_3
value: 19.41
- type: recall_at_5
value: 24.611
task:
type: Retrieval
- dataset:
config: fr
name: MTEB OpusparcusPC (fr)
revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a
split: test
type: GEM/opusparcus
metrics:
- type: main_score
value: 100
task:
type: PairClassification
- dataset:
config: fr
name: MTEB PawsX (fr)
revision: 8a04d940a42cd40658986fdd8e3da561533a3646
split: test
type: google-research-datasets/paws-x
metrics:
- type: main_score
value: 59.350898072045524
task:
type: PairClassification
- dataset:
config: default
name: MTEB SICKFr (default)
revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a
split: test
type: Lajavaness/SICK-fr
metrics:
- type: cosine_pearson
value: 87.27626503447769
- type: cosine_spearman
value: 82.97811185613928
- type: euclidean_pearson
value: 84.23492888124656
- type: euclidean_spearman
value: 82.70148623477802
- type: main_score
value: 82.97811185613928
- type: manhattan_pearson
value: 84.297157107182
- type: manhattan_spearman
value: 82.74716982383642
task:
type: STS
- dataset:
config: fr
name: MTEB STS22 (fr)
revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3
split: test
type: mteb/sts22-crosslingual-sts
metrics:
- type: cosine_pearson
value: 79.31660828924933
- type: cosine_spearman
value: 79.81279059580181
- type: euclidean_pearson
value: 76.97214428475466
- type: euclidean_spearman
value: 78.71684748795504
- type: main_score
value: 79.81279059580181
- type: manhattan_pearson
value: 77.31973152122153
- type: manhattan_spearman
value: 79.39135921375463
task:
type: STS
- dataset:
config: fr
name: MTEB STSBenchmarkMultilingualSTS (fr)
revision: 29afa2569dcedaaa2fe6a3dcfebab33d28b82e8c
split: test
type: mteb/stsb_multi_mt
metrics:
- type: cosine_pearson
value: 83.25110537943343
- type: cosine_spearman
value: 83.3549848907322
- type: euclidean_pearson
value: 83.76643622531708
- type: euclidean_spearman
value: 84.26736484765411
- type: main_score
value: 83.3549848907322
- type: manhattan_pearson
value: 83.85819348663743
- type: manhattan_spearman
value: 84.3238855655977
task:
type: STS
- dataset:
config: default
name: MTEB SummEvalFr (default)
revision: b385812de6a9577b6f4d0f88c6a6e35395a94054
split: test
type: lyon-nlp/summarization-summeval-fr-p2p
metrics:
- type: cosine_pearson
value: 30.041871461290647
- type: cosine_spearman
value: 30.07396842547381
- type: dot_pearson
value: 27.801917921027048
- type: dot_spearman
value: 29.45369912070655
- type: main_score
value: 30.07396842547381
- type: pearson
value: 30.04186698260713
- type: spearman
value: 30.07396842547381
task:
type: Summarization
- dataset:
config: default
name: MTEB SyntecReranking (default)
revision: daf0863838cd9e3ba50544cdce3ac2b338a1b0ad
split: test
type: lyon-nlp/mteb-fr-reranking-syntec-s2p
metrics:
- type: main_score
value: 77.9095238095238
- type: map
value: 77.9095238095238
- type: mrr
value: 77.9095238095238
- type: nAUC_map_diff1
value: 66.27107953653064
- type: nAUC_map_max
value: 0.2695668659856748
- type: nAUC_map_std
value: 26.93797112914763
- type: nAUC_mrr_diff1
value: 66.27107953653064
- type: nAUC_mrr_max
value: 0.2695668659856748
- type: nAUC_mrr_std
value: 26.93797112914763
task:
type: Reranking
- dataset:
config: default
name: MTEB SyntecRetrieval (default)
revision: 19661ccdca4dfc2d15122d776b61685f48c68ca9
split: test
type: lyon-nlp/mteb-fr-retrieval-syntec-s2p
metrics:
- type: main_score
value: 70.816
- type: map_at_1
value: 56.00000000000001
- type: map_at_10
value: 65.24
- type: map_at_100
value: 65.861
- type: map_at_1000
value: 65.861
- type: map_at_20
value: 65.694
- type: map_at_3
value: 61.667
- type: map_at_5
value: 63.517
- type: mrr_at_1
value: 56.00000000000001
- type: mrr_at_10
value: 65.23968253968253
- type: mrr_at_100
value: 65.86075928457507
- type: mrr_at_1000
value: 65.86075928457507
- type: mrr_at_20
value: 65.6941359079517
- type: mrr_at_3
value: 61.66666666666667
- type: mrr_at_5
value: 63.516666666666666
- type: nauc_map_at_1000_diff1
value: 57.80107526241379
- type: nauc_map_at_1000_max
value: 17.788895135066365
- type: nauc_map_at_1000_std
value: -16.080088232653964
- type: nauc_map_at_100_diff1
value: 57.80107526241379
- type: nauc_map_at_100_max
value: 17.788895135066365
- type: nauc_map_at_100_std
value: -16.080088232653964
- type: nauc_map_at_10_diff1
value: 58.15441410208685
- type: nauc_map_at_10_max
value: 17.62427688711543
- type: nauc_map_at_10_std
value: -16.06858334210365
- type: nauc_map_at_1_diff1
value: 57.191119691119674
- type: nauc_map_at_1_max
value: 10.798035430388317
- type: nauc_map_at_1_std
value: -21.245457642516502
- type: nauc_map_at_20_diff1
value: 57.63714861477662
- type: nauc_map_at_20_max
value: 17.872891516043435
- type: nauc_map_at_20_std
value: -15.851696265890538
- type: nauc_map_at_3_diff1
value: 58.30962645532695
- type: nauc_map_at_3_max
value: 19.41580401166845
- type: nauc_map_at_3_std
value: -16.08746160002065
- type: nauc_map_at_5_diff1
value: 58.54473584909214
- type: nauc_map_at_5_max
value: 18.564995584175005
- type: nauc_map_at_5_std
value: -16.479444125875393
- type: nauc_mrr_at_1000_diff1
value: 57.80107526241379
- type: nauc_mrr_at_1000_max
value: 17.788895135066365
- type: nauc_mrr_at_1000_std
value: -16.080088232653964
- type: nauc_mrr_at_100_diff1
value: 57.80107526241379
- type: nauc_mrr_at_100_max
value: 17.788895135066365
- type: nauc_mrr_at_100_std
value: -16.080088232653964
- type: nauc_mrr_at_10_diff1
value: 58.15441410208685
- type: nauc_mrr_at_10_max
value: 17.62427688711543
- type: nauc_mrr_at_10_std
value: -16.06858334210365
- type: nauc_mrr_at_1_diff1
value: 57.191119691119674
- type: nauc_mrr_at_1_max
value: 10.798035430388317
- type: nauc_mrr_at_1_std
value: -21.245457642516502
- type: nauc_mrr_at_20_diff1
value: 57.63714861477662
- type: nauc_mrr_at_20_max
value: 17.872891516043435
- type: nauc_mrr_at_20_std
value: -15.851696265890538
- type: nauc_mrr_at_3_diff1
value: 58.30962645532695
- type: nauc_mrr_at_3_max
value: 19.41580401166845
- type: nauc_mrr_at_3_std
value: -16.08746160002065
- type: nauc_mrr_at_5_diff1
value: 58.54473584909214
- type: nauc_mrr_at_5_max
value: 18.564995584175005
- type: nauc_mrr_at_5_std
value: -16.479444125875393
- type: nauc_ndcg_at_1000_diff1
value: 57.94873661161248
- type: nauc_ndcg_at_1000_max
value: 18.303142573985006
- type: nauc_ndcg_at_1000_std
value: -15.471339542689467
- type: nauc_ndcg_at_100_diff1
value: 57.94873661161248
- type: nauc_ndcg_at_100_max
value: 18.303142573985006
- type: nauc_ndcg_at_100_std
value: -15.471339542689467
- type: nauc_ndcg_at_10_diff1
value: 59.07524779658788
- type: nauc_ndcg_at_10_max
value: 17.58898442202306
- type: nauc_ndcg_at_10_std
value: -14.786464420733475
- type: nauc_ndcg_at_1_diff1
value: 57.191119691119674
- type: nauc_ndcg_at_1_max
value: 10.798035430388317
- type: nauc_ndcg_at_1_std
value: -21.245457642516502
- type: nauc_ndcg_at_20_diff1
value: 56.833941496402886
- type: nauc_ndcg_at_20_max
value: 18.89867354518282
- type: nauc_ndcg_at_20_std
value: -13.879721060843933
- type: nauc_ndcg_at_3_diff1
value: 59.33013207022601
- type: nauc_ndcg_at_3_max
value: 21.96126648737589
- type: nauc_ndcg_at_3_std
value: -15.01220810337584
- type: nauc_ndcg_at_5_diff1
value: 59.96524000642845
- type: nauc_ndcg_at_5_max
value: 20.396683050862023
- type: nauc_ndcg_at_5_std
value: -15.745995152140422
- type: nauc_precision_at_1000_diff1
value: .nan
- type: nauc_precision_at_1000_max
value: .nan
- type: nauc_precision_at_1000_std
value: .nan
- type: nauc_precision_at_100_diff1
value: .nan
- type: nauc_precision_at_100_max
value: .nan
- type: nauc_precision_at_100_std
value: .nan
- type: nauc_precision_at_10_diff1
value: 65.79331672536726
- type: nauc_precision_at_10_max
value: 12.314234172321926
- type: nauc_precision_at_10_std
value: -6.086246886006163
- type: nauc_precision_at_1_diff1
value: 57.191119691119674
- type: nauc_precision_at_1_max
value: 10.798035430388317
- type: nauc_precision_at_1_std
value: -21.245457642516502
- type: nauc_precision_at_20_diff1
value: 28.81652661064429
- type: nauc_precision_at_20_max
value: 34.37208216620027
- type: nauc_precision_at_20_std
value: 26.890756302521208
- type: nauc_precision_at_3_diff1
value: 62.825651302605166
- type: nauc_precision_at_3_max
value: 29.97374058461758
- type: nauc_precision_at_3_std
value: -11.811554142768218
- type: nauc_precision_at_5_diff1
value: 65.92541316950764
- type: nauc_precision_at_5_max
value: 26.81924374837766
- type: nauc_precision_at_5_std
value: -13.44206974128227
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_100_diff1
value: .nan
- type: nauc_recall_at_100_max
value: .nan
- type: nauc_recall_at_100_std
value: .nan
- type: nauc_recall_at_10_diff1
value: 65.79331672536732
- type: nauc_recall_at_10_max
value: 12.314234172321907
- type: nauc_recall_at_10_std
value: -6.086246886006318
- type: nauc_recall_at_1_diff1
value: 57.191119691119674
- type: nauc_recall_at_1_max
value: 10.798035430388317
- type: nauc_recall_at_1_std
value: -21.245457642516502
- type: nauc_recall_at_20_diff1
value: 28.816526610644416
- type: nauc_recall_at_20_max
value: 34.372082166200116
- type: nauc_recall_at_20_std
value: 26.89075630252124
- type: nauc_recall_at_3_diff1
value: 62.825651302605166
- type: nauc_recall_at_3_max
value: 29.97374058461751
- type: nauc_recall_at_3_std
value: -11.811554142768365
- type: nauc_recall_at_5_diff1
value: 65.92541316950765
- type: nauc_recall_at_5_max
value: 26.819243748377637
- type: nauc_recall_at_5_std
value: -13.442069741282372
- type: ndcg_at_1
value: 56.00000000000001
- type: ndcg_at_10
value: 70.816
- type: ndcg_at_100
value: 73.407
- type: ndcg_at_1000
value: 73.407
- type: ndcg_at_20
value: 72.54599999999999
- type: ndcg_at_3
value: 63.309000000000005
- type: ndcg_at_5
value: 66.623
- type: precision_at_1
value: 56.00000000000001
- type: precision_at_10
value: 8.9
- type: precision_at_100
value: 1
- type: precision_at_1000
value: 0.1
- type: precision_at_20
value: 4.8
- type: precision_at_3
value: 22.667
- type: precision_at_5
value: 15.2
- type: recall_at_1
value: 56.00000000000001
- type: recall_at_10
value: 89
- type: recall_at_100
value: 100
- type: recall_at_1000
value: 100
- type: recall_at_20
value: 96
- type: recall_at_3
value: 68
- type: recall_at_5
value: 76
task:
type: Retrieval
- dataset:
config: fra-fra
name: MTEB XPQARetrieval (fra-fra)
revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f
split: test
type: jinaai/xpqa
metrics:
- type: main_score
value: 53.736
- type: map_at_1
value: 30.509999999999998
- type: map_at_10
value: 47.132000000000005
- type: map_at_100
value: 48.821999999999996
- type: map_at_1000
value: 48.931000000000004
- type: map_at_20
value: 48.124
- type: map_at_3
value: 41.427
- type: map_at_5
value: 45.109
- type: mrr_at_1
value: 48.73164218958612
- type: mrr_at_10
value: 56.385657066564946
- type: mrr_at_100
value: 57.11851611532053
- type: mrr_at_1000
value: 57.15740285337062
- type: mrr_at_20
value: 56.865661966890855
- type: mrr_at_3
value: 53.91633288829551
- type: mrr_at_5
value: 55.51846906987093
- type: nauc_map_at_1000_diff1
value: 45.903881003685385
- type: nauc_map_at_1000_max
value: 40.24033911245095
- type: nauc_map_at_1000_std
value: -0.29774228919788903
- type: nauc_map_at_100_diff1
value: 45.8951282902959
- type: nauc_map_at_100_max
value: 40.24612049826258
- type: nauc_map_at_100_std
value: -0.30037049930255116
- type: nauc_map_at_10_diff1
value: 45.6564521063323
- type: nauc_map_at_10_max
value: 38.76075848701305
- type: nauc_map_at_10_std
value: -1.4252064121260086
- type: nauc_map_at_1_diff1
value: 54.11412008072518
- type: nauc_map_at_1_max
value: 28.5583920066647
- type: nauc_map_at_1_std
value: -2.032440075433671
- type: nauc_map_at_20_diff1
value: 45.78769301971449
- type: nauc_map_at_20_max
value: 39.62447659942229
- type: nauc_map_at_20_std
value: -0.8680223144310233
- type: nauc_map_at_3_diff1
value: 47.79253805734147
- type: nauc_map_at_3_max
value: 35.90915728121236
- type: nauc_map_at_3_std
value: -2.7837914848287544
- type: nauc_map_at_5_diff1
value: 46.533221489021315
- type: nauc_map_at_5_max
value: 37.614720807495914
- type: nauc_map_at_5_std
value: -2.8457694672333393
- type: nauc_mrr_at_1000_diff1
value: 49.113863291952455
- type: nauc_mrr_at_1000_max
value: 45.27137646954461
- type: nauc_mrr_at_1000_std
value: 2.3936124099071066
- type: nauc_mrr_at_100_diff1
value: 49.10443817738001
- type: nauc_mrr_at_100_max
value: 45.28526599942805
- type: nauc_mrr_at_100_std
value: 2.4293661343031006
- type: nauc_mrr_at_10_diff1
value: 48.841023586768166
- type: nauc_mrr_at_10_max
value: 44.81183781341252
- type: nauc_mrr_at_10_std
value: 1.9312130690544813
- type: nauc_mrr_at_1_diff1
value: 52.36433700317802
- type: nauc_mrr_at_1_max
value: 46.10446435296976
- type: nauc_mrr_at_1_std
value: 0.6338946522527386
- type: nauc_mrr_at_20_diff1
value: 49.10392312642867
- type: nauc_mrr_at_20_max
value: 45.22847441282849
- type: nauc_mrr_at_20_std
value: 2.3542725547873755
- type: nauc_mrr_at_3_diff1
value: 49.71231586771101
- type: nauc_mrr_at_3_max
value: 44.86214301866909
- type: nauc_mrr_at_3_std
value: 1.415227182519296
- type: nauc_mrr_at_5_diff1
value: 48.97212979755398
- type: nauc_mrr_at_5_max
value: 44.627002551409376
- type: nauc_mrr_at_5_std
value: 1.372422201307736
- type: nauc_ndcg_at_1000_diff1
value: 46.026334178565584
- type: nauc_ndcg_at_1000_max
value: 43.87335105323267
- type: nauc_ndcg_at_1000_std
value: 3.2650114134706114
- type: nauc_ndcg_at_100_diff1
value: 45.745131108447154
- type: nauc_ndcg_at_100_max
value: 44.52713580213882
- type: nauc_ndcg_at_100_std
value: 4.245662387188146
- type: nauc_ndcg_at_10_diff1
value: 44.397290711991424
- type: nauc_ndcg_at_10_max
value: 40.091783546193
- type: nauc_ndcg_at_10_std
value: 0.20786173246599443
- type: nauc_ndcg_at_1_diff1
value: 52.36433700317802
- type: nauc_ndcg_at_1_max
value: 46.10446435296976
- type: nauc_ndcg_at_1_std
value: 0.6338946522527386
- type: nauc_ndcg_at_20_diff1
value: 45.11466246736486
- type: nauc_ndcg_at_20_max
value: 42.071765598990694
- type: nauc_ndcg_at_20_std
value: 1.8857293316791688
- type: nauc_ndcg_at_3_diff1
value: 45.549078702113526
- type: nauc_ndcg_at_3_max
value: 40.37376933329564
- type: nauc_ndcg_at_3_std
value: -1.2092966231656004
- type: nauc_ndcg_at_5_diff1
value: 45.542796430967975
- type: nauc_ndcg_at_5_max
value: 38.51415439452183
- type: nauc_ndcg_at_5_std
value: -2.358084841952356
- type: nauc_precision_at_1000_diff1
value: -12.950780679125476
- type: nauc_precision_at_1000_max
value: 21.412917471461256
- type: nauc_precision_at_1000_std
value: 13.825405119536201
- type: nauc_precision_at_100_diff1
value: -5.272242499865375
- type: nauc_precision_at_100_max
value: 31.146070261389795
- type: nauc_precision_at_100_std
value: 16.76037098641707
- type: nauc_precision_at_10_diff1
value: 5.022078446921512
- type: nauc_precision_at_10_max
value: 32.40126524945171
- type: nauc_precision_at_10_std
value: 6.8199402218300165
- type: nauc_precision_at_1_diff1
value: 52.36433700317802
- type: nauc_precision_at_1_max
value: 46.10446435296976
- type: nauc_precision_at_1_std
value: 0.6338946522527386
- type: nauc_precision_at_20_diff1
value: 1.8507191498204303
- type: nauc_precision_at_20_max
value: 32.79240428757858
- type: nauc_precision_at_20_std
value: 10.475564903497089
- type: nauc_precision_at_3_diff1
value: 18.485676250880545
- type: nauc_precision_at_3_max
value: 37.83868390515382
- type: nauc_precision_at_3_std
value: 0.8764756728424954
- type: nauc_precision_at_5_diff1
value: 11.175975339969739
- type: nauc_precision_at_5_max
value: 34.3923017648371
- type: nauc_precision_at_5_std
value: 1.970006195503709
- type: nauc_recall_at_1000_diff1
value: 50.00849402651506
- type: nauc_recall_at_1000_max
value: 76.53252732514734
- type: nauc_recall_at_1000_std
value: 77.98271681562578
- type: nauc_recall_at_100_diff1
value: 37.40486283884057
- type: nauc_recall_at_100_max
value: 53.937687925250145
- type: nauc_recall_at_100_std
value: 27.02303686880432
- type: nauc_recall_at_10_diff1
value: 34.16588036777468
- type: nauc_recall_at_10_max
value: 33.462391339457454
- type: nauc_recall_at_10_std
value: 0.8457483878677011
- type: nauc_recall_at_1_diff1
value: 54.11412008072518
- type: nauc_recall_at_1_max
value: 28.5583920066647
- type: nauc_recall_at_1_std
value: -2.032440075433671
- type: nauc_recall_at_20_diff1
value: 35.648128649471246
- type: nauc_recall_at_20_max
value: 38.25932742254366
- type: nauc_recall_at_20_std
value: 7.011710372084609
- type: nauc_recall_at_3_diff1
value: 41.91414480084279
- type: nauc_recall_at_3_max
value: 31.453812281128368
- type: nauc_recall_at_3_std
value: -3.674860959584926
- type: nauc_recall_at_5_diff1
value: 37.937536670026674
- type: nauc_recall_at_5_max
value: 30.827227889874187
- type: nauc_recall_at_5_std
value: -5.600317070261898
- type: ndcg_at_1
value: 48.732
- type: ndcg_at_10
value: 53.736
- type: ndcg_at_100
value: 59.65200000000001
- type: ndcg_at_1000
value: 61.633
- type: ndcg_at_20
value: 56.313
- type: ndcg_at_3
value: 48.02
- type: ndcg_at_5
value: 50.368
- type: precision_at_1
value: 48.732
- type: precision_at_10
value: 12.737000000000002
- type: precision_at_100
value: 1.79
- type: precision_at_1000
value: 0.20500000000000002
- type: precision_at_20
value: 7.303
- type: precision_at_3
value: 28.838
- type: precision_at_5
value: 21.602
- type: recall_at_1
value: 30.509999999999998
- type: recall_at_10
value: 63.23800000000001
- type: recall_at_100
value: 85.739
- type: recall_at_1000
value: 98.923
- type: recall_at_20
value: 71.41999999999999
- type: recall_at_3
value: 46.208
- type: recall_at_5
value: 54.913000000000004
task:
type: Retrieval
CATIE-AQ/camembert-base-embedding
Description
This is a sentence-transformers model finetuned from almanach/camembert-base (111M parameters). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Score on the MTEB leaderboard:
| Model | Average | Classification | Clustering | PairClassification | Reranking | Retrieval | STS | Summarization |
|---|---|---|---|---|---|---|---|---|
| CATIE-AQ/camembert-base-embedding (111M) | 60,057 | 66,117 | 45,41 | 79,675 | 71,303 | 45,769 | 82,049 | 30,074 |
| CATIE-AQ/distilcamembert-base-embedding (68M) | 58,297 | 63,904 | 44,549 | 79,102 | 67,961 | 42,222 | 80,204 | 30,138 |
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: almanach/camembert-base
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: CamembertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("CATIE-AQ/camembert-base-embedding")
# Run inference
sentences = [
"Tenet est sous surveillance depuis novembre, lorsque l'ancien directeur général Jeffrey Barbakow a déclaré que la société a utilisé des prix agressifs pour déclencher des paiements plus élevés pour les patients les plus malades de l'assurance maladie.",
"En novembre, Jeffrey Brabakow, le directeur général de l'époque, a déclaré que la société utilisait des prix agressifs pour obtenir des paiements plus élevés pour les patients les plus malades de l'assurance maladie.",
'La femme est en route pour un rendez-vous.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Citation
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}