diff --git "a/README.md" "b/README.md" new file mode 100644--- /dev/null +++ "b/README.md" @@ -0,0 +1,7610 @@ +--- +model-index: +- name: voyage-code-3 + results: + - dataset: + config: default + name: MTEB AppsRetrieval (default) + revision: f22508f96b7a36c2415181ed8bb76f76e04ae2d5 + split: test + type: CoIR-Retrieval/apps + metrics: + - type: main_score + value: 93.621 + - type: map_at_1 + value: 87.968 + - type: map_at_10 + value: 92.047 + - type: map_at_100 + value: 92.11500000000001 + - type: map_at_1000 + value: 92.11500000000001 + - type: map_at_20 + value: 92.091 + - type: map_at_3 + value: 91.532 + - type: map_at_5 + value: 91.874 + - type: mrr_at_1 + value: 87.99468791500664 + - type: mrr_at_10 + value: 92.0623432197138 + - type: mrr_at_100 + value: 92.13012638701028 + - type: mrr_at_1000 + value: 92.1308801595124 + - type: mrr_at_20 + value: 92.10639248723756 + - type: mrr_at_3 + value: 91.54050464807443 + - type: mrr_at_5 + value: 91.88977423638782 + - type: nauc_map_at_1000_diff1 + value: 90.15490174664647 + - type: nauc_map_at_1000_max + value: 67.02866848649028 + - type: nauc_map_at_1000_std + value: -43.43778333375324 + - type: nauc_map_at_100_diff1 + value: 90.15545196165746 + - type: nauc_map_at_100_max + value: 67.03097567361978 + - type: nauc_map_at_100_std + value: -43.43758797549862 + - type: nauc_map_at_10_diff1 + value: 90.14099743587612 + - type: nauc_map_at_10_max + value: 67.16096459752251 + - type: nauc_map_at_10_std + value: -43.44370410695505 + - type: nauc_map_at_1_diff1 + value: 90.99585708349436 + - type: nauc_map_at_1_max + value: 63.57782533912141 + - type: nauc_map_at_1_std + value: -42.361341394612126 + - type: nauc_map_at_20_diff1 + value: 90.1551408410763 + - type: nauc_map_at_20_max + value: 67.07765073421193 + - type: nauc_map_at_20_std + value: -43.41448959858248 + - type: nauc_map_at_3_diff1 + value: 89.8995753062064 + - type: nauc_map_at_3_max + value: 67.61366242819365 + - type: nauc_map_at_3_std + value: -44.226914049035386 + - type: nauc_map_at_5_diff1 + value: 90.09003396740809 + - type: nauc_map_at_5_max + value: 67.4059817584288 + - type: nauc_map_at_5_std + value: -43.62598115209535 + - type: nauc_mrr_at_1000_diff1 + value: 90.1224401379764 + - type: nauc_mrr_at_1000_max + value: 67.0844465768374 + - type: nauc_mrr_at_1000_std + value: -43.634995805553366 + - type: nauc_mrr_at_100_diff1 + value: 90.12299449341405 + - type: nauc_mrr_at_100_max + value: 67.08675243260438 + - type: nauc_mrr_at_100_std + value: -43.634780889033514 + - type: nauc_mrr_at_10_diff1 + value: 90.10879634384949 + - type: nauc_mrr_at_10_max + value: 67.21648611418671 + - type: nauc_mrr_at_10_std + value: -43.63919542502753 + - type: nauc_mrr_at_1_diff1 + value: 90.94858533614968 + - type: nauc_mrr_at_1_max + value: 63.65698161247239 + - type: nauc_mrr_at_1_std + value: -42.61438951155136 + - type: nauc_mrr_at_20_diff1 + value: 90.1227820706247 + - type: nauc_mrr_at_20_max + value: 67.1333396707349 + - type: nauc_mrr_at_20_std + value: -43.61103858801164 + - type: nauc_mrr_at_3_diff1 + value: 89.87704691693779 + - type: nauc_mrr_at_3_max + value: 67.69227631885192 + - type: nauc_mrr_at_3_std + value: -44.3319988628551 + - type: nauc_mrr_at_5_diff1 + value: 90.05844149558918 + - type: nauc_mrr_at_5_max + value: 67.46072486731265 + - type: nauc_mrr_at_5_std + value: -43.81755835911823 + - type: nauc_ndcg_at_1000_diff1 + value: 90.04496527123014 + - type: nauc_ndcg_at_1000_max + value: 67.60247091688248 + - type: nauc_ndcg_at_1000_std + value: -43.03623963675031 + - type: nauc_ndcg_at_100_diff1 + value: 90.070471931644 + - type: nauc_ndcg_at_100_max + value: 67.68039037445462 + - type: nauc_ndcg_at_100_std + value: -43.036110157737035 + - type: nauc_ndcg_at_10_diff1 + value: 90.0028263117178 + - type: nauc_ndcg_at_10_max + value: 68.38084649569117 + - type: nauc_ndcg_at_10_std + value: -42.96245497355038 + - type: nauc_ndcg_at_1_diff1 + value: 90.99585708349436 + - type: nauc_ndcg_at_1_max + value: 63.57782533912141 + - type: nauc_ndcg_at_1_std + value: -42.361341394612126 + - type: nauc_ndcg_at_20_diff1 + value: 90.06473833364464 + - type: nauc_ndcg_at_20_max + value: 68.04393781250494 + - type: nauc_ndcg_at_20_std + value: -42.754643190221756 + - type: nauc_ndcg_at_3_diff1 + value: 89.47150366489366 + - type: nauc_ndcg_at_3_max + value: 69.28073683657388 + - type: nauc_ndcg_at_3_std + value: -44.80650989406889 + - type: nauc_ndcg_at_5_diff1 + value: 89.85493663258895 + - type: nauc_ndcg_at_5_max + value: 68.97238224506673 + - type: nauc_ndcg_at_5_std + value: -43.59588098814929 + - type: nauc_precision_at_1000_diff1 + value: 71.5228076434295 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 95.64334438407597 + - type: nauc_precision_at_100_diff1 + value: 89.05538400916919 + - type: nauc_precision_at_100_max + value: 93.59380801456066 + - type: nauc_precision_at_100_std + value: 3.4218682638907 + - type: nauc_precision_at_10_diff1 + value: 88.82693413470759 + - type: nauc_precision_at_10_max + value: 83.71281148941699 + - type: nauc_precision_at_10_std + value: -33.65732721164693 + - type: nauc_precision_at_1_diff1 + value: 90.99585708349436 + - type: nauc_precision_at_1_max + value: 63.57782533912141 + - type: nauc_precision_at_1_std + value: -42.361341394612126 + - type: nauc_precision_at_20_diff1 + value: 89.60083706628117 + - type: nauc_precision_at_20_max + value: 85.08806968533703 + - type: nauc_precision_at_20_std + value: -22.269540063924854 + - type: nauc_precision_at_3_diff1 + value: 87.16551109941072 + - type: nauc_precision_at_3_max + value: 78.32435110823177 + - type: nauc_precision_at_3_std + value: -47.825568017450934 + - type: nauc_precision_at_5_diff1 + value: 88.21645334780426 + - type: nauc_precision_at_5_max + value: 81.21614910200276 + - type: nauc_precision_at_5_std + value: -42.6577409583649 + - type: nauc_recall_at_1000_diff1 + value: 71.52280764343946 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 95.64334438409236 + - type: nauc_recall_at_100_diff1 + value: 89.05538400917727 + - type: nauc_recall_at_100_max + value: 93.59380801455806 + - type: nauc_recall_at_100_std + value: 3.4218682638950693 + - type: nauc_recall_at_10_diff1 + value: 88.82693413470616 + - type: nauc_recall_at_10_max + value: 83.71281148941586 + - type: nauc_recall_at_10_std + value: -33.65732721164498 + - type: nauc_recall_at_1_diff1 + value: 90.99585708349436 + - type: nauc_recall_at_1_max + value: 63.57782533912141 + - type: nauc_recall_at_1_std + value: -42.361341394612126 + - type: nauc_recall_at_20_diff1 + value: 89.60083706628052 + - type: nauc_recall_at_20_max + value: 85.08806968533526 + - type: nauc_recall_at_20_std + value: -22.269540063922193 + - type: nauc_recall_at_3_diff1 + value: 87.16551109941042 + - type: nauc_recall_at_3_max + value: 78.32435110823131 + - type: nauc_recall_at_3_std + value: -47.825568017451694 + - type: nauc_recall_at_5_diff1 + value: 88.21645334780379 + - type: nauc_recall_at_5_max + value: 81.21614910200307 + - type: nauc_recall_at_5_std + value: -42.65774095836507 + - type: ndcg_at_1 + value: 87.968 + - type: ndcg_at_10 + value: 93.621 + - type: ndcg_at_100 + value: 93.929 + - type: ndcg_at_1000 + value: 93.949 + - type: ndcg_at_20 + value: 93.77799999999999 + - type: ndcg_at_3 + value: 92.60199999999999 + - type: ndcg_at_5 + value: 93.217 + - type: precision_at_1 + value: 87.968 + - type: precision_at_10 + value: 9.838 + - type: precision_at_100 + value: 0.9979999999999999 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.95 + - type: precision_at_3 + value: 31.89 + - type: precision_at_5 + value: 19.432 + - type: recall_at_1 + value: 87.968 + - type: recall_at_10 + value: 98.38 + - type: recall_at_100 + value: 99.761 + - type: recall_at_1000 + value: 99.92 + - type: recall_at_20 + value: 98.991 + - type: recall_at_3 + value: 95.67099999999999 + - type: recall_at_5 + value: 97.158 + task: + type: Retrieval + - dataset: + config: python + name: MTEB COIRCodeSearchNetRetrieval (python) + revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 + split: test + type: CoIR-Retrieval/CodeSearchNet + metrics: + - type: main_score + value: 98.367 + - type: map_at_1 + value: 96.494 + - type: map_at_10 + value: 97.881 + - type: map_at_100 + value: 97.89099999999999 + - type: map_at_1000 + value: 97.89099999999999 + - type: map_at_20 + value: 97.88799999999999 + - type: map_at_3 + value: 97.761 + - type: map_at_5 + value: 97.849 + - type: mrr_at_1 + value: 96.49416811905081 + - type: mrr_at_10 + value: 97.88111783570305 + - type: mrr_at_100 + value: 97.8906549372564 + - type: mrr_at_1000 + value: 97.89071223051654 + - type: mrr_at_20 + value: 97.88807761475906 + - type: mrr_at_3 + value: 97.76109398042645 + - type: mrr_at_5 + value: 97.84924252580788 + - type: nauc_map_at_1000_diff1 + value: 98.15528913593722 + - type: nauc_map_at_1000_max + value: 85.10151231357158 + - type: nauc_map_at_1000_std + value: -42.70593910947264 + - type: nauc_map_at_100_diff1 + value: 98.15533924129993 + - type: nauc_map_at_100_max + value: 85.10116247232071 + - type: nauc_map_at_100_std + value: -42.70950485228204 + - type: nauc_map_at_10_diff1 + value: 98.15404628849377 + - type: nauc_map_at_10_max + value: 85.129420000526 + - type: nauc_map_at_10_std + value: -43.05569868322259 + - type: nauc_map_at_1_diff1 + value: 98.05311494766504 + - type: nauc_map_at_1_max + value: 83.37279874457242 + - type: nauc_map_at_1_std + value: -36.78783832567891 + - type: nauc_map_at_20_diff1 + value: 98.15680755256112 + - type: nauc_map_at_20_max + value: 85.1144859632837 + - type: nauc_map_at_20_std + value: -42.76588304453336 + - type: nauc_map_at_3_diff1 + value: 98.19103616317174 + - type: nauc_map_at_3_max + value: 85.64151924417325 + - type: nauc_map_at_3_std + value: -43.86751765575472 + - type: nauc_map_at_5_diff1 + value: 98.16782356611095 + - type: nauc_map_at_5_max + value: 85.28003038549159 + - type: nauc_map_at_5_std + value: -43.74493490831982 + - type: nauc_mrr_at_1000_diff1 + value: 98.15528913593722 + - type: nauc_mrr_at_1000_max + value: 85.10151231357158 + - type: nauc_mrr_at_1000_std + value: -42.70593910947264 + - type: nauc_mrr_at_100_diff1 + value: 98.15533924129993 + - type: nauc_mrr_at_100_max + value: 85.10116247232071 + - type: nauc_mrr_at_100_std + value: -42.70950485228204 + - type: nauc_mrr_at_10_diff1 + value: 98.15404628849377 + - type: nauc_mrr_at_10_max + value: 85.129420000526 + - type: nauc_mrr_at_10_std + value: -43.05569868322259 + - type: nauc_mrr_at_1_diff1 + value: 98.05311494766504 + - type: nauc_mrr_at_1_max + value: 83.37279874457242 + - type: nauc_mrr_at_1_std + value: -36.78783832567891 + - type: nauc_mrr_at_20_diff1 + value: 98.15680755256112 + - type: nauc_mrr_at_20_max + value: 85.1144859632837 + - type: nauc_mrr_at_20_std + value: -42.76588304453336 + - type: nauc_mrr_at_3_diff1 + value: 98.19103616317174 + - type: nauc_mrr_at_3_max + value: 85.64151924417325 + - type: nauc_mrr_at_3_std + value: -43.86751765575472 + - type: nauc_mrr_at_5_diff1 + value: 98.16782356611095 + - type: nauc_mrr_at_5_max + value: 85.28003038549159 + - type: nauc_mrr_at_5_std + value: -43.74493490831982 + - type: nauc_ndcg_at_1000_diff1 + value: 98.16554701617589 + - type: nauc_ndcg_at_1000_max + value: 85.2748828477838 + - type: nauc_ndcg_at_1000_std + value: -43.53566701835024 + - type: nauc_ndcg_at_100_diff1 + value: 98.16666937775803 + - type: nauc_ndcg_at_100_max + value: 85.26689650159106 + - type: nauc_ndcg_at_100_std + value: -43.61547874575031 + - type: nauc_ndcg_at_10_diff1 + value: 98.15895834975365 + - type: nauc_ndcg_at_10_max + value: 85.43419484878363 + - type: nauc_ndcg_at_10_std + value: -45.349653395420944 + - type: nauc_ndcg_at_1_diff1 + value: 98.05311494766504 + - type: nauc_ndcg_at_1_max + value: 83.37279874457242 + - type: nauc_ndcg_at_1_std + value: -36.78783832567891 + - type: nauc_ndcg_at_20_diff1 + value: 98.17535928747724 + - type: nauc_ndcg_at_20_max + value: 85.37092010493639 + - type: nauc_ndcg_at_20_std + value: -43.98828652303495 + - type: nauc_ndcg_at_3_diff1 + value: 98.25821737148956 + - type: nauc_ndcg_at_3_max + value: 86.61782450729814 + - type: nauc_ndcg_at_3_std + value: -47.32915673717569 + - type: nauc_ndcg_at_5_diff1 + value: 98.2061154768608 + - type: nauc_ndcg_at_5_max + value: 85.87935683953549 + - type: nauc_ndcg_at_5_std + value: -47.302585930633896 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 100.0 + - type: nauc_precision_at_100_max + value: 72.22153459022752 + - type: nauc_precision_at_100_std + value: -173.98459168247265 + - type: nauc_precision_at_10_diff1 + value: 97.94083448220364 + - type: nauc_precision_at_10_max + value: 91.42141149896693 + - type: nauc_precision_at_10_std + value: -110.27705518270434 + - type: nauc_precision_at_1_diff1 + value: 98.05311494766504 + - type: nauc_precision_at_1_max + value: 83.37279874457242 + - type: nauc_precision_at_1_std + value: -36.78783832567891 + - type: nauc_precision_at_20_diff1 + value: 98.99445460386596 + - type: nauc_precision_at_20_max + value: 93.841013383867 + - type: nauc_precision_at_20_std + value: -85.33131689356598 + - type: nauc_precision_at_3_diff1 + value: 98.77736667583922 + - type: nauc_precision_at_3_max + value: 94.04942862307165 + - type: nauc_precision_at_3_std + value: -74.09082528149943 + - type: nauc_precision_at_5_diff1 + value: 98.69061226878082 + - type: nauc_precision_at_5_max + value: 93.54052780940046 + - type: nauc_precision_at_5_std + value: -97.44201029419808 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 100.0 + - type: nauc_recall_at_100_max + value: 72.22153459013214 + - type: nauc_recall_at_100_std + value: -173.98459168226447 + - type: nauc_recall_at_10_diff1 + value: 97.94083448220702 + - type: nauc_recall_at_10_max + value: 91.42141149896996 + - type: nauc_recall_at_10_std + value: -110.27705518269966 + - type: nauc_recall_at_1_diff1 + value: 98.05311494766504 + - type: nauc_recall_at_1_max + value: 83.37279874457242 + - type: nauc_recall_at_1_std + value: -36.78783832567891 + - type: nauc_recall_at_20_diff1 + value: 98.99445460384722 + - type: nauc_recall_at_20_max + value: 93.84101338387075 + - type: nauc_recall_at_20_std + value: -85.33131689356598 + - type: nauc_recall_at_3_diff1 + value: 98.77736667583844 + - type: nauc_recall_at_3_max + value: 94.0494286230709 + - type: nauc_recall_at_3_std + value: -74.09082528149965 + - type: nauc_recall_at_5_diff1 + value: 98.69061226878156 + - type: nauc_recall_at_5_max + value: 93.5405278094034 + - type: nauc_recall_at_5_std + value: -97.44201029420337 + - type: ndcg_at_1 + value: 96.494 + - type: ndcg_at_10 + value: 98.367 + - type: ndcg_at_100 + value: 98.408 + - type: ndcg_at_1000 + value: 98.409 + - type: ndcg_at_20 + value: 98.392 + - type: ndcg_at_3 + value: 98.13 + - type: ndcg_at_5 + value: 98.289 + - type: precision_at_1 + value: 96.494 + - type: precision_at_10 + value: 9.981 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.9959999999999996 + - type: precision_at_3 + value: 33.061 + - type: precision_at_5 + value: 19.914 + - type: recall_at_1 + value: 96.494 + - type: recall_at_10 + value: 99.812 + - type: recall_at_100 + value: 99.993 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.913 + - type: recall_at_3 + value: 99.182 + - type: recall_at_5 + value: 99.571 + task: + type: Retrieval + - dataset: + config: javascript + name: MTEB COIRCodeSearchNetRetrieval (javascript) + revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 + split: test + type: CoIR-Retrieval/CodeSearchNet + metrics: + - type: main_score + value: 83.479 + - type: map_at_1 + value: 74.354 + - type: map_at_10 + value: 80.71300000000001 + - type: map_at_100 + value: 80.95700000000001 + - type: map_at_1000 + value: 80.963 + - type: map_at_20 + value: 80.87100000000001 + - type: map_at_3 + value: 79.712 + - type: map_at_5 + value: 80.28999999999999 + - type: mrr_at_1 + value: 74.35429960498328 + - type: mrr_at_10 + value: 80.71294728769666 + - type: mrr_at_100 + value: 80.95719685924738 + - type: mrr_at_1000 + value: 80.9626426876391 + - type: mrr_at_20 + value: 80.87121221349531 + - type: mrr_at_3 + value: 79.71234680441609 + - type: mrr_at_5 + value: 80.28967892231339 + - type: nauc_map_at_1000_diff1 + value: 82.88577068822386 + - type: nauc_map_at_1000_max + value: 67.67862670522469 + - type: nauc_map_at_1000_std + value: -3.522371246361021 + - type: nauc_map_at_100_diff1 + value: 82.8829326501484 + - type: nauc_map_at_100_max + value: 67.68203242273655 + - type: nauc_map_at_100_std + value: -3.522160367469023 + - type: nauc_map_at_10_diff1 + value: 82.8083126411219 + - type: nauc_map_at_10_max + value: 67.76494610226807 + - type: nauc_map_at_10_std + value: -3.3188673363705137 + - type: nauc_map_at_1_diff1 + value: 85.06863588556931 + - type: nauc_map_at_1_max + value: 64.5628789135785 + - type: nauc_map_at_1_std + value: -6.191657522827031 + - type: nauc_map_at_20_diff1 + value: 82.85819113195612 + - type: nauc_map_at_20_max + value: 67.75920634659379 + - type: nauc_map_at_20_std + value: -3.3782092153909744 + - type: nauc_map_at_3_diff1 + value: 82.93140971183675 + - type: nauc_map_at_3_max + value: 67.88657739487108 + - type: nauc_map_at_3_std + value: -3.9042280696240383 + - type: nauc_map_at_5_diff1 + value: 82.78238115943125 + - type: nauc_map_at_5_max + value: 67.78270133394841 + - type: nauc_map_at_5_std + value: -3.6293201723127693 + - type: nauc_mrr_at_1000_diff1 + value: 82.88577068822386 + - type: nauc_mrr_at_1000_max + value: 67.67862670522469 + - type: nauc_mrr_at_1000_std + value: -3.522371246361021 + - type: nauc_mrr_at_100_diff1 + value: 82.8829326501484 + - type: nauc_mrr_at_100_max + value: 67.68203242273655 + - type: nauc_mrr_at_100_std + value: -3.522160367469023 + - type: nauc_mrr_at_10_diff1 + value: 82.8083126411219 + - type: nauc_mrr_at_10_max + value: 67.76494610226807 + - type: nauc_mrr_at_10_std + value: -3.3188673363705137 + - type: nauc_mrr_at_1_diff1 + value: 85.06863588556931 + - type: nauc_mrr_at_1_max + value: 64.5628789135785 + - type: nauc_mrr_at_1_std + value: -6.191657522827031 + - type: nauc_mrr_at_20_diff1 + value: 82.85819113195612 + - type: nauc_mrr_at_20_max + value: 67.75920634659379 + - type: nauc_mrr_at_20_std + value: -3.3782092153909744 + - type: nauc_mrr_at_3_diff1 + value: 82.93140971183675 + - type: nauc_mrr_at_3_max + value: 67.88657739487108 + - type: nauc_mrr_at_3_std + value: -3.9042280696240383 + - type: nauc_mrr_at_5_diff1 + value: 82.78238115943125 + - type: nauc_mrr_at_5_max + value: 67.78270133394841 + - type: nauc_mrr_at_5_std + value: -3.6293201723127693 + - type: nauc_ndcg_at_1000_diff1 + value: 82.3199098700142 + - type: nauc_ndcg_at_1000_max + value: 68.120342472938 + - type: nauc_ndcg_at_1000_std + value: -2.8508509319395734 + - type: nauc_ndcg_at_100_diff1 + value: 82.1633159731435 + - type: nauc_ndcg_at_100_max + value: 68.26748139014519 + - type: nauc_ndcg_at_100_std + value: -2.714633476731751 + - type: nauc_ndcg_at_10_diff1 + value: 81.82903994006058 + - type: nauc_ndcg_at_10_max + value: 68.89584312261682 + - type: nauc_ndcg_at_10_std + value: -1.4252080306379409 + - type: nauc_ndcg_at_1_diff1 + value: 85.06863588556931 + - type: nauc_ndcg_at_1_max + value: 64.5628789135785 + - type: nauc_ndcg_at_1_std + value: -6.191657522827031 + - type: nauc_ndcg_at_20_diff1 + value: 81.98789282274657 + - type: nauc_ndcg_at_20_max + value: 68.8959043317764 + - type: nauc_ndcg_at_20_std + value: -1.6185846906781556 + - type: nauc_ndcg_at_3_diff1 + value: 82.10544479985309 + - type: nauc_ndcg_at_3_max + value: 68.98284750281354 + - type: nauc_ndcg_at_3_std + value: -3.02774137516056 + - type: nauc_ndcg_at_5_diff1 + value: 81.77317252542665 + - type: nauc_ndcg_at_5_max + value: 68.8430670861106 + - type: nauc_ndcg_at_5_std + value: -2.3676013953659583 + - type: nauc_precision_at_1000_diff1 + value: 71.2357013205843 + - type: nauc_precision_at_1000_max + value: 59.54259563274979 + - type: nauc_precision_at_1000_std + value: -6.453769813732959 + - type: nauc_precision_at_100_diff1 + value: 70.6646143169478 + - type: nauc_precision_at_100_max + value: 72.21908402194197 + - type: nauc_precision_at_100_std + value: 3.6390309142884423 + - type: nauc_precision_at_10_diff1 + value: 75.69911710504576 + - type: nauc_precision_at_10_max + value: 75.9705779221398 + - type: nauc_precision_at_10_std + value: 11.632934387302264 + - type: nauc_precision_at_1_diff1 + value: 85.06863588556931 + - type: nauc_precision_at_1_max + value: 64.5628789135785 + - type: nauc_precision_at_1_std + value: -6.191657522827031 + - type: nauc_precision_at_20_diff1 + value: 74.91685968954766 + - type: nauc_precision_at_20_max + value: 78.48940148220586 + - type: nauc_precision_at_20_std + value: 14.477347988107262 + - type: nauc_precision_at_3_diff1 + value: 78.78186792721571 + - type: nauc_precision_at_3_max + value: 73.31020934353046 + - type: nauc_precision_at_3_std + value: 0.4957741917380896 + - type: nauc_precision_at_5_diff1 + value: 76.96340948709872 + - type: nauc_precision_at_5_max + value: 73.6957286199771 + - type: nauc_precision_at_5_std + value: 3.7822933962717555 + - type: nauc_recall_at_1000_diff1 + value: 71.23570132058367 + - type: nauc_recall_at_1000_max + value: 59.54259563275034 + - type: nauc_recall_at_1000_std + value: -6.453769813732673 + - type: nauc_recall_at_100_diff1 + value: 70.66461431694707 + - type: nauc_recall_at_100_max + value: 72.21908402194222 + - type: nauc_recall_at_100_std + value: 3.639030914287266 + - type: nauc_recall_at_10_diff1 + value: 75.69911710504581 + - type: nauc_recall_at_10_max + value: 75.97057792213957 + - type: nauc_recall_at_10_std + value: 11.632934387302335 + - type: nauc_recall_at_1_diff1 + value: 85.06863588556931 + - type: nauc_recall_at_1_max + value: 64.5628789135785 + - type: nauc_recall_at_1_std + value: -6.191657522827031 + - type: nauc_recall_at_20_diff1 + value: 74.91685968954776 + - type: nauc_recall_at_20_max + value: 78.4894014822059 + - type: nauc_recall_at_20_std + value: 14.477347988107574 + - type: nauc_recall_at_3_diff1 + value: 78.78186792721584 + - type: nauc_recall_at_3_max + value: 73.31020934353042 + - type: nauc_recall_at_3_std + value: 0.49577419173804144 + - type: nauc_recall_at_5_diff1 + value: 76.96340948709855 + - type: nauc_recall_at_5_max + value: 73.695728619977 + - type: nauc_recall_at_5_std + value: 3.782293396271726 + - type: ndcg_at_1 + value: 74.354 + - type: ndcg_at_10 + value: 83.479 + - type: ndcg_at_100 + value: 84.65599999999999 + - type: ndcg_at_1000 + value: 84.851 + - type: ndcg_at_20 + value: 84.03500000000001 + - type: ndcg_at_3 + value: 81.42200000000001 + - type: ndcg_at_5 + value: 82.461 + - type: precision_at_1 + value: 74.354 + - type: precision_at_10 + value: 9.200999999999999 + - type: precision_at_100 + value: 0.975 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 4.707999999999999 + - type: precision_at_3 + value: 28.786 + - type: precision_at_5 + value: 17.776 + - type: recall_at_1 + value: 74.354 + - type: recall_at_10 + value: 92.009 + - type: recall_at_100 + value: 97.508 + - type: recall_at_1000 + value: 99.149 + - type: recall_at_20 + value: 94.16600000000001 + - type: recall_at_3 + value: 86.357 + - type: recall_at_5 + value: 88.87899999999999 + task: + type: Retrieval + - dataset: + config: go + name: MTEB COIRCodeSearchNetRetrieval (go) + revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 + split: test + type: CoIR-Retrieval/CodeSearchNet + metrics: + - type: main_score + value: 91.093 + - type: map_at_1 + value: 83.95700000000001 + - type: map_at_10 + value: 89.095 + - type: map_at_100 + value: 89.21900000000001 + - type: map_at_1000 + value: 89.22099999999999 + - type: map_at_20 + value: 89.181 + - type: map_at_3 + value: 88.472 + - type: map_at_5 + value: 88.866 + - type: mrr_at_1 + value: 83.95715341049002 + - type: mrr_at_10 + value: 89.0947231309824 + - type: mrr_at_100 + value: 89.21908275588152 + - type: mrr_at_1000 + value: 89.22056604897018 + - type: mrr_at_20 + value: 89.18054785129605 + - type: mrr_at_3 + value: 88.47164081096584 + - type: mrr_at_5 + value: 88.86624805056191 + - type: nauc_map_at_1000_diff1 + value: 86.24973579936615 + - type: nauc_map_at_1000_max + value: 43.76713186644932 + - type: nauc_map_at_1000_std + value: -47.20109532688898 + - type: nauc_map_at_100_diff1 + value: 86.24862049034168 + - type: nauc_map_at_100_max + value: 43.76881022585186 + - type: nauc_map_at_100_std + value: -47.20426521434973 + - type: nauc_map_at_10_diff1 + value: 86.25350003349301 + - type: nauc_map_at_10_max + value: 43.771945888748306 + - type: nauc_map_at_10_std + value: -47.62265974147625 + - type: nauc_map_at_1_diff1 + value: 86.98798340919 + - type: nauc_map_at_1_max + value: 43.024043355768995 + - type: nauc_map_at_1_std + value: -41.469271817341955 + - type: nauc_map_at_20_diff1 + value: 86.25325068130977 + - type: nauc_map_at_20_max + value: 43.80727376815637 + - type: nauc_map_at_20_std + value: -47.31520043996017 + - type: nauc_map_at_3_diff1 + value: 86.10288537323795 + - type: nauc_map_at_3_max + value: 43.79879378738717 + - type: nauc_map_at_3_std + value: -48.546995258168884 + - type: nauc_map_at_5_diff1 + value: 86.18610042665851 + - type: nauc_map_at_5_max + value: 43.4814674201994 + - type: nauc_map_at_5_std + value: -48.409761968549084 + - type: nauc_mrr_at_1000_diff1 + value: 86.24973579936615 + - type: nauc_mrr_at_1000_max + value: 43.76713186644932 + - type: nauc_mrr_at_1000_std + value: -47.20109532688898 + - type: nauc_mrr_at_100_diff1 + value: 86.24862049034168 + - type: nauc_mrr_at_100_max + value: 43.76881022585186 + - type: nauc_mrr_at_100_std + value: -47.20426521434973 + - type: nauc_mrr_at_10_diff1 + value: 86.25350003349301 + - type: nauc_mrr_at_10_max + value: 43.771945888748306 + - type: nauc_mrr_at_10_std + value: -47.62265974147625 + - type: nauc_mrr_at_1_diff1 + value: 86.98798340919 + - type: nauc_mrr_at_1_max + value: 43.024043355768995 + - type: nauc_mrr_at_1_std + value: -41.469271817341955 + - type: nauc_mrr_at_20_diff1 + value: 86.25325068130977 + - type: nauc_mrr_at_20_max + value: 43.80727376815637 + - type: nauc_mrr_at_20_std + value: -47.31520043996017 + - type: nauc_mrr_at_3_diff1 + value: 86.10288537323795 + - type: nauc_mrr_at_3_max + value: 43.79879378738717 + - type: nauc_mrr_at_3_std + value: -48.546995258168884 + - type: nauc_mrr_at_5_diff1 + value: 86.18610042665851 + - type: nauc_mrr_at_5_max + value: 43.4814674201994 + - type: nauc_mrr_at_5_std + value: -48.409761968549084 + - type: nauc_ndcg_at_1000_diff1 + value: 86.1495231100838 + - type: nauc_ndcg_at_1000_max + value: 44.07212654503798 + - type: nauc_ndcg_at_1000_std + value: -47.62163979276151 + - type: nauc_ndcg_at_100_diff1 + value: 86.0998353963385 + - type: nauc_ndcg_at_100_max + value: 44.12175911440964 + - type: nauc_ndcg_at_100_std + value: -47.60497182589526 + - type: nauc_ndcg_at_10_diff1 + value: 86.15470113116336 + - type: nauc_ndcg_at_10_max + value: 44.22094281301947 + - type: nauc_ndcg_at_10_std + value: -49.68035065822525 + - type: nauc_ndcg_at_1_diff1 + value: 86.98798340919 + - type: nauc_ndcg_at_1_max + value: 43.024043355768995 + - type: nauc_ndcg_at_1_std + value: -41.469271817341955 + - type: nauc_ndcg_at_20_diff1 + value: 86.153687239383 + - type: nauc_ndcg_at_20_max + value: 44.413683910958994 + - type: nauc_ndcg_at_20_std + value: -48.390979780431394 + - type: nauc_ndcg_at_3_diff1 + value: 85.81806042008513 + - type: nauc_ndcg_at_3_max + value: 44.05606453052025 + - type: nauc_ndcg_at_3_std + value: -51.77857306868798 + - type: nauc_ndcg_at_5_diff1 + value: 85.97978206190466 + - type: nauc_ndcg_at_5_max + value: 43.41162519222535 + - type: nauc_ndcg_at_5_std + value: -51.75858549128195 + - type: nauc_precision_at_1000_diff1 + value: 80.08890508034311 + - type: nauc_precision_at_1000_max + value: 94.23440421215071 + - type: nauc_precision_at_1000_std + value: 55.93973044712788 + - type: nauc_precision_at_100_diff1 + value: 75.93187211974605 + - type: nauc_precision_at_100_max + value: 67.07999533740335 + - type: nauc_precision_at_100_std + value: -8.479533178162365 + - type: nauc_precision_at_10_diff1 + value: 85.56280199155816 + - type: nauc_precision_at_10_max + value: 49.46503821359178 + - type: nauc_precision_at_10_std + value: -68.7017504439551 + - type: nauc_precision_at_1_diff1 + value: 86.98798340919 + - type: nauc_precision_at_1_max + value: 43.024043355768995 + - type: nauc_precision_at_1_std + value: -41.469271817341955 + - type: nauc_precision_at_20_diff1 + value: 85.16456431737308 + - type: nauc_precision_at_20_max + value: 57.105231459014604 + - type: nauc_precision_at_20_std + value: -57.20616563582712 + - type: nauc_precision_at_3_diff1 + value: 84.42151445378559 + - type: nauc_precision_at_3_max + value: 45.32354792405939 + - type: nauc_precision_at_3_std + value: -68.34786794101257 + - type: nauc_precision_at_5_diff1 + value: 84.72626746318639 + - type: nauc_precision_at_5_max + value: 42.65450965313161 + - type: nauc_precision_at_5_std + value: -74.63108495140406 + - type: nauc_recall_at_1000_diff1 + value: 80.08890508033205 + - type: nauc_recall_at_1000_max + value: 94.23440421215781 + - type: nauc_recall_at_1000_std + value: 55.93973044711981 + - type: nauc_recall_at_100_diff1 + value: 75.93187211975196 + - type: nauc_recall_at_100_max + value: 67.07999533740575 + - type: nauc_recall_at_100_std + value: -8.479533178153222 + - type: nauc_recall_at_10_diff1 + value: 85.56280199155815 + - type: nauc_recall_at_10_max + value: 49.46503821359128 + - type: nauc_recall_at_10_std + value: -68.70175044395225 + - type: nauc_recall_at_1_diff1 + value: 86.98798340919 + - type: nauc_recall_at_1_max + value: 43.024043355768995 + - type: nauc_recall_at_1_std + value: -41.469271817341955 + - type: nauc_recall_at_20_diff1 + value: 85.16456431737305 + - type: nauc_recall_at_20_max + value: 57.10523145901444 + - type: nauc_recall_at_20_std + value: -57.20616563582539 + - type: nauc_recall_at_3_diff1 + value: 84.42151445378548 + - type: nauc_recall_at_3_max + value: 45.32354792405929 + - type: nauc_recall_at_3_std + value: -68.34786794101288 + - type: nauc_recall_at_5_diff1 + value: 84.72626746318646 + - type: nauc_recall_at_5_max + value: 42.65450965313161 + - type: nauc_recall_at_5_std + value: -74.63108495140318 + - type: ndcg_at_1 + value: 83.95700000000001 + - type: ndcg_at_10 + value: 91.093 + - type: ndcg_at_100 + value: 91.644 + - type: ndcg_at_1000 + value: 91.67999999999999 + - type: ndcg_at_20 + value: 91.39699999999999 + - type: ndcg_at_3 + value: 89.84299999999999 + - type: ndcg_at_5 + value: 90.551 + - type: precision_at_1 + value: 83.95700000000001 + - type: precision_at_10 + value: 9.714 + - type: precision_at_100 + value: 0.996 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.916 + - type: precision_at_3 + value: 31.261 + - type: precision_at_5 + value: 19.099 + - type: recall_at_1 + value: 83.95700000000001 + - type: recall_at_10 + value: 97.14399999999999 + - type: recall_at_100 + value: 99.59400000000001 + - type: recall_at_1000 + value: 99.87700000000001 + - type: recall_at_20 + value: 98.32600000000001 + - type: recall_at_3 + value: 93.782 + - type: recall_at_5 + value: 95.494 + task: + type: Retrieval + - dataset: + config: ruby + name: MTEB COIRCodeSearchNetRetrieval (ruby) + revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 + split: test + type: CoIR-Retrieval/CodeSearchNet + metrics: + - type: main_score + value: 86.521 + - type: map_at_1 + value: 77.082 + - type: map_at_10 + value: 83.552 + - type: map_at_100 + value: 83.687 + - type: map_at_1000 + value: 83.69 + - type: map_at_20 + value: 83.64200000000001 + - type: map_at_3 + value: 82.11699999999999 + - type: map_at_5 + value: 83.164 + - type: mrr_at_1 + value: 77.08168120539256 + - type: mrr_at_10 + value: 83.55194919627917 + - type: mrr_at_100 + value: 83.68701866558312 + - type: mrr_at_1000 + value: 83.6903032381438 + - type: mrr_at_20 + value: 83.64219478925247 + - type: mrr_at_3 + value: 82.11736716891363 + - type: mrr_at_5 + value: 83.16415543219676 + - type: nauc_map_at_1000_diff1 + value: 79.64679377977846 + - type: nauc_map_at_1000_max + value: 70.74499431961593 + - type: nauc_map_at_1000_std + value: -14.853409296691982 + - type: nauc_map_at_100_diff1 + value: 79.64499589721636 + - type: nauc_map_at_100_max + value: 70.74759843854231 + - type: nauc_map_at_100_std + value: -14.844258734825894 + - type: nauc_map_at_10_diff1 + value: 79.53290732095073 + - type: nauc_map_at_10_max + value: 70.76525352436302 + - type: nauc_map_at_10_std + value: -14.744825626716187 + - type: nauc_map_at_1_diff1 + value: 82.27728044611361 + - type: nauc_map_at_1_max + value: 69.29382107125889 + - type: nauc_map_at_1_std + value: -16.02966665531533 + - type: nauc_map_at_20_diff1 + value: 79.57275942354842 + - type: nauc_map_at_20_max + value: 70.73838311952618 + - type: nauc_map_at_20_std + value: -14.86925109860625 + - type: nauc_map_at_3_diff1 + value: 79.47572156907859 + - type: nauc_map_at_3_max + value: 70.85276903670264 + - type: nauc_map_at_3_std + value: -15.755792005612602 + - type: nauc_map_at_5_diff1 + value: 79.42518354242955 + - type: nauc_map_at_5_max + value: 70.69226154953026 + - type: nauc_map_at_5_std + value: -15.473047773228993 + - type: nauc_mrr_at_1000_diff1 + value: 79.64679377977846 + - type: nauc_mrr_at_1000_max + value: 70.74499431961593 + - type: nauc_mrr_at_1000_std + value: -14.853409296691982 + - type: nauc_mrr_at_100_diff1 + value: 79.64499589721636 + - type: nauc_mrr_at_100_max + value: 70.74759843854231 + - type: nauc_mrr_at_100_std + value: -14.844258734825894 + - type: nauc_mrr_at_10_diff1 + value: 79.53290732095073 + - type: nauc_mrr_at_10_max + value: 70.76525352436302 + - type: nauc_mrr_at_10_std + value: -14.744825626716187 + - type: nauc_mrr_at_1_diff1 + value: 82.27728044611361 + - type: nauc_mrr_at_1_max + value: 69.29382107125889 + - type: nauc_mrr_at_1_std + value: -16.02966665531533 + - type: nauc_mrr_at_20_diff1 + value: 79.57275942354842 + - type: nauc_mrr_at_20_max + value: 70.73838311952618 + - type: nauc_mrr_at_20_std + value: -14.86925109860625 + - type: nauc_mrr_at_3_diff1 + value: 79.47572156907859 + - type: nauc_mrr_at_3_max + value: 70.85276903670264 + - type: nauc_mrr_at_3_std + value: -15.755792005612602 + - type: nauc_mrr_at_5_diff1 + value: 79.42518354242955 + - type: nauc_mrr_at_5_max + value: 70.69226154953026 + - type: nauc_mrr_at_5_std + value: -15.473047773228993 + - type: nauc_ndcg_at_1000_diff1 + value: 79.15083604230807 + - type: nauc_ndcg_at_1000_max + value: 71.1178743998364 + - type: nauc_ndcg_at_1000_std + value: -14.28015599243118 + - type: nauc_ndcg_at_100_diff1 + value: 79.03711984795275 + - type: nauc_ndcg_at_100_max + value: 71.20364489826714 + - type: nauc_ndcg_at_100_std + value: -13.915390197654142 + - type: nauc_ndcg_at_10_diff1 + value: 78.38949093885134 + - type: nauc_ndcg_at_10_max + value: 71.26426621928836 + - type: nauc_ndcg_at_10_std + value: -13.497441432215215 + - type: nauc_ndcg_at_1_diff1 + value: 82.27728044611361 + - type: nauc_ndcg_at_1_max + value: 69.29382107125889 + - type: nauc_ndcg_at_1_std + value: -16.02966665531533 + - type: nauc_ndcg_at_20_diff1 + value: 78.51349613885007 + - type: nauc_ndcg_at_20_max + value: 71.13292440385801 + - type: nauc_ndcg_at_20_std + value: -14.035580557966087 + - type: nauc_ndcg_at_3_diff1 + value: 78.3820255722769 + - type: nauc_ndcg_at_3_max + value: 71.32668614112852 + - type: nauc_ndcg_at_3_std + value: -16.04098840431829 + - type: nauc_ndcg_at_5_diff1 + value: 78.14070256474213 + - type: nauc_ndcg_at_5_max + value: 71.04026107145508 + - type: nauc_ndcg_at_5_std + value: -15.592128711464067 + - type: nauc_precision_at_1000_diff1 + value: 77.2067992384371 + - type: nauc_precision_at_1000_max + value: 85.55623308018635 + - type: nauc_precision_at_1000_std + value: -3.0163245425889005 + - type: nauc_precision_at_100_diff1 + value: 71.01994754486307 + - type: nauc_precision_at_100_max + value: 82.30803790261785 + - type: nauc_precision_at_100_std + value: 12.633506693103701 + - type: nauc_precision_at_10_diff1 + value: 67.04993913892164 + - type: nauc_precision_at_10_max + value: 76.23886937696508 + - type: nauc_precision_at_10_std + value: 1.5250700897226326 + - type: nauc_precision_at_1_diff1 + value: 82.27728044611361 + - type: nauc_precision_at_1_max + value: 69.29382107125889 + - type: nauc_precision_at_1_std + value: -16.02966665531533 + - type: nauc_precision_at_20_diff1 + value: 63.666986838468034 + - type: nauc_precision_at_20_max + value: 76.14090190912508 + - type: nauc_precision_at_20_std + value: -0.5871594787522816 + - type: nauc_precision_at_3_diff1 + value: 73.86799722582569 + - type: nauc_precision_at_3_max + value: 73.220598048437 + - type: nauc_precision_at_3_std + value: -17.398868434567206 + - type: nauc_precision_at_5_diff1 + value: 70.02914493269556 + - type: nauc_precision_at_5_max + value: 73.00069716131422 + - type: nauc_precision_at_5_std + value: -16.614393845477327 + - type: nauc_recall_at_1000_diff1 + value: 77.20679923843755 + - type: nauc_recall_at_1000_max + value: 85.55623308018376 + - type: nauc_recall_at_1000_std + value: -3.016324542591852 + - type: nauc_recall_at_100_diff1 + value: 71.01994754486401 + - type: nauc_recall_at_100_max + value: 82.30803790261768 + - type: nauc_recall_at_100_std + value: 12.633506693104875 + - type: nauc_recall_at_10_diff1 + value: 67.04993913892174 + - type: nauc_recall_at_10_max + value: 76.23886937696521 + - type: nauc_recall_at_10_std + value: 1.525070089722257 + - type: nauc_recall_at_1_diff1 + value: 82.27728044611361 + - type: nauc_recall_at_1_max + value: 69.29382107125889 + - type: nauc_recall_at_1_std + value: -16.02966665531533 + - type: nauc_recall_at_20_diff1 + value: 63.66698683846833 + - type: nauc_recall_at_20_max + value: 76.1409019091247 + - type: nauc_recall_at_20_std + value: -0.5871594787518352 + - type: nauc_recall_at_3_diff1 + value: 73.86799722582569 + - type: nauc_recall_at_3_max + value: 73.22059804843683 + - type: nauc_recall_at_3_std + value: -17.398868434566936 + - type: nauc_recall_at_5_diff1 + value: 70.02914493269549 + - type: nauc_recall_at_5_max + value: 73.00069716131398 + - type: nauc_recall_at_5_std + value: -16.614393845477526 + - type: ndcg_at_1 + value: 77.082 + - type: ndcg_at_10 + value: 86.521 + - type: ndcg_at_100 + value: 87.117 + - type: ndcg_at_1000 + value: 87.23100000000001 + - type: ndcg_at_20 + value: 86.858 + - type: ndcg_at_3 + value: 83.7 + - type: ndcg_at_5 + value: 85.591 + - type: precision_at_1 + value: 77.082 + - type: precision_at_10 + value: 9.572 + - type: precision_at_100 + value: 0.983 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 4.853 + - type: precision_at_3 + value: 29.421000000000003 + - type: precision_at_5 + value: 18.573 + - type: recall_at_1 + value: 77.082 + - type: recall_at_10 + value: 95.718 + - type: recall_at_100 + value: 98.335 + - type: recall_at_1000 + value: 99.286 + - type: recall_at_20 + value: 97.066 + - type: recall_at_3 + value: 88.263 + - type: recall_at_5 + value: 92.863 + task: + type: Retrieval + - dataset: + config: java + name: MTEB COIRCodeSearchNetRetrieval (java) + revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 + split: test + type: CoIR-Retrieval/CodeSearchNet + metrics: + - type: main_score + value: 87.26899999999999 + - type: map_at_1 + value: 78.84100000000001 + - type: map_at_10 + value: 84.821 + - type: map_at_100 + value: 84.966 + - type: map_at_1000 + value: 84.97200000000001 + - type: map_at_20 + value: 84.91199999999999 + - type: map_at_3 + value: 83.949 + - type: map_at_5 + value: 84.514 + - type: mrr_at_1 + value: 78.8407120036513 + - type: mrr_at_10 + value: 84.82137604775663 + - type: mrr_at_100 + value: 84.96600236904305 + - type: mrr_at_1000 + value: 84.97188010768014 + - type: mrr_at_20 + value: 84.91215021114384 + - type: mrr_at_3 + value: 83.94949033926656 + - type: mrr_at_5 + value: 84.51361630914327 + - type: nauc_map_at_1000_diff1 + value: 81.29745727169212 + - type: nauc_map_at_1000_max + value: 61.3738482977934 + - type: nauc_map_at_1000_std + value: -14.611667076737653 + - type: nauc_map_at_100_diff1 + value: 81.29611575498951 + - type: nauc_map_at_100_max + value: 61.38415642876098 + - type: nauc_map_at_100_std + value: -14.60020471629818 + - type: nauc_map_at_10_diff1 + value: 81.24492887826995 + - type: nauc_map_at_10_max + value: 61.476804728960545 + - type: nauc_map_at_10_std + value: -14.7216428942423 + - type: nauc_map_at_1_diff1 + value: 83.07021879054804 + - type: nauc_map_at_1_max + value: 57.45256621613931 + - type: nauc_map_at_1_std + value: -16.94155323782564 + - type: nauc_map_at_20_diff1 + value: 81.28505757844012 + - type: nauc_map_at_20_max + value: 61.43691967628063 + - type: nauc_map_at_20_std + value: -14.583239037636378 + - type: nauc_map_at_3_diff1 + value: 81.0101519179173 + - type: nauc_map_at_3_max + value: 61.45133191553263 + - type: nauc_map_at_3_std + value: -15.098090101575302 + - type: nauc_map_at_5_diff1 + value: 81.16287036289346 + - type: nauc_map_at_5_max + value: 61.59538703827228 + - type: nauc_map_at_5_std + value: -14.843988367319087 + - type: nauc_mrr_at_1000_diff1 + value: 81.29745727169212 + - type: nauc_mrr_at_1000_max + value: 61.3738482977934 + - type: nauc_mrr_at_1000_std + value: -14.611667076737653 + - type: nauc_mrr_at_100_diff1 + value: 81.29611575498951 + - type: nauc_mrr_at_100_max + value: 61.38415642876098 + - type: nauc_mrr_at_100_std + value: -14.60020471629818 + - type: nauc_mrr_at_10_diff1 + value: 81.24492887826995 + - type: nauc_mrr_at_10_max + value: 61.476804728960545 + - type: nauc_mrr_at_10_std + value: -14.7216428942423 + - type: nauc_mrr_at_1_diff1 + value: 83.07021879054804 + - type: nauc_mrr_at_1_max + value: 57.45256621613931 + - type: nauc_mrr_at_1_std + value: -16.94155323782564 + - type: nauc_mrr_at_20_diff1 + value: 81.28505757844012 + - type: nauc_mrr_at_20_max + value: 61.43691967628063 + - type: nauc_mrr_at_20_std + value: -14.583239037636378 + - type: nauc_mrr_at_3_diff1 + value: 81.0101519179173 + - type: nauc_mrr_at_3_max + value: 61.45133191553263 + - type: nauc_mrr_at_3_std + value: -15.098090101575302 + - type: nauc_mrr_at_5_diff1 + value: 81.16287036289346 + - type: nauc_mrr_at_5_max + value: 61.59538703827228 + - type: nauc_mrr_at_5_std + value: -14.843988367319087 + - type: nauc_ndcg_at_1000_diff1 + value: 80.93566924723379 + - type: nauc_ndcg_at_1000_max + value: 62.39699094971305 + - type: nauc_ndcg_at_1000_std + value: -13.127328464119836 + - type: nauc_ndcg_at_100_diff1 + value: 80.87553513050011 + - type: nauc_ndcg_at_100_max + value: 62.72180161053774 + - type: nauc_ndcg_at_100_std + value: -12.649445624854962 + - type: nauc_ndcg_at_10_diff1 + value: 80.608466242348 + - type: nauc_ndcg_at_10_max + value: 63.24038696080356 + - type: nauc_ndcg_at_10_std + value: -13.10528733578738 + - type: nauc_ndcg_at_1_diff1 + value: 83.07021879054804 + - type: nauc_ndcg_at_1_max + value: 57.45256621613931 + - type: nauc_ndcg_at_1_std + value: -16.94155323782564 + - type: nauc_ndcg_at_20_diff1 + value: 80.77174027485287 + - type: nauc_ndcg_at_20_max + value: 63.1297458573586 + - type: nauc_ndcg_at_20_std + value: -12.43612330644508 + - type: nauc_ndcg_at_3_diff1 + value: 80.1778653998912 + - type: nauc_ndcg_at_3_max + value: 63.029112839728796 + - type: nauc_ndcg_at_3_std + value: -14.201414014064817 + - type: nauc_ndcg_at_5_diff1 + value: 80.41507067529044 + - type: nauc_ndcg_at_5_max + value: 63.456703437014475 + - type: nauc_ndcg_at_5_std + value: -13.566365227692753 + - type: nauc_precision_at_1000_diff1 + value: 73.89525267370539 + - type: nauc_precision_at_1000_max + value: 83.46793489687887 + - type: nauc_precision_at_1000_std + value: 43.36957143495421 + - type: nauc_precision_at_100_diff1 + value: 75.33082548811534 + - type: nauc_precision_at_100_max + value: 85.08356966408422 + - type: nauc_precision_at_100_std + value: 30.91532174961418 + - type: nauc_precision_at_10_diff1 + value: 75.96120115397336 + - type: nauc_precision_at_10_max + value: 76.87957072290979 + - type: nauc_precision_at_10_std + value: 0.48296998647622513 + - type: nauc_precision_at_1_diff1 + value: 83.07021879054804 + - type: nauc_precision_at_1_max + value: 57.45256621613931 + - type: nauc_precision_at_1_std + value: -16.94155323782564 + - type: nauc_precision_at_20_diff1 + value: 76.37431221666498 + - type: nauc_precision_at_20_max + value: 80.17866322470198 + - type: nauc_precision_at_20_std + value: 12.914278225658554 + - type: nauc_precision_at_3_diff1 + value: 76.55984367763438 + - type: nauc_precision_at_3_max + value: 69.86960822047166 + - type: nauc_precision_at_3_std + value: -10.230045456623477 + - type: nauc_precision_at_5_diff1 + value: 76.32796311102129 + - type: nauc_precision_at_5_max + value: 73.88101356144463 + - type: nauc_precision_at_5_std + value: -6.1510867874405095 + - type: nauc_recall_at_1000_diff1 + value: 73.89525267370503 + - type: nauc_recall_at_1000_max + value: 83.46793489687492 + - type: nauc_recall_at_1000_std + value: 43.36957143495507 + - type: nauc_recall_at_100_diff1 + value: 75.33082548811574 + - type: nauc_recall_at_100_max + value: 85.08356966408451 + - type: nauc_recall_at_100_std + value: 30.91532174961395 + - type: nauc_recall_at_10_diff1 + value: 75.96120115397329 + - type: nauc_recall_at_10_max + value: 76.87957072290959 + - type: nauc_recall_at_10_std + value: 0.4829699864760997 + - type: nauc_recall_at_1_diff1 + value: 83.07021879054804 + - type: nauc_recall_at_1_max + value: 57.45256621613931 + - type: nauc_recall_at_1_std + value: -16.94155323782564 + - type: nauc_recall_at_20_diff1 + value: 76.37431221666506 + - type: nauc_recall_at_20_max + value: 80.17866322470138 + - type: nauc_recall_at_20_std + value: 12.914278225659201 + - type: nauc_recall_at_3_diff1 + value: 76.55984367763459 + - type: nauc_recall_at_3_max + value: 69.86960822047196 + - type: nauc_recall_at_3_std + value: -10.230045456623337 + - type: nauc_recall_at_5_diff1 + value: 76.32796311102106 + - type: nauc_recall_at_5_max + value: 73.88101356144475 + - type: nauc_recall_at_5_std + value: -6.151086787440187 + - type: ndcg_at_1 + value: 78.84100000000001 + - type: ndcg_at_10 + value: 87.26899999999999 + - type: ndcg_at_100 + value: 87.977 + - type: ndcg_at_1000 + value: 88.133 + - type: ndcg_at_20 + value: 87.599 + - type: ndcg_at_3 + value: 85.521 + - type: ndcg_at_5 + value: 86.531 + - type: precision_at_1 + value: 78.84100000000001 + - type: precision_at_10 + value: 9.474 + - type: precision_at_100 + value: 0.9809999999999999 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 4.802 + - type: precision_at_3 + value: 30.014000000000003 + - type: precision_at_5 + value: 18.496000000000002 + - type: recall_at_1 + value: 78.84100000000001 + - type: recall_at_10 + value: 94.742 + - type: recall_at_100 + value: 98.056 + - type: recall_at_1000 + value: 99.288 + - type: recall_at_20 + value: 96.04700000000001 + - type: recall_at_3 + value: 90.04100000000001 + - type: recall_at_5 + value: 92.47800000000001 + task: + type: Retrieval + - dataset: + config: php + name: MTEB COIRCodeSearchNetRetrieval (php) + revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 + split: test + type: CoIR-Retrieval/CodeSearchNet + metrics: + - type: main_score + value: 89.384 + - type: map_at_1 + value: 81.997 + - type: map_at_10 + value: 87.221 + - type: map_at_100 + value: 87.375 + - type: map_at_1000 + value: 87.378 + - type: map_at_20 + value: 87.32799999999999 + - type: map_at_3 + value: 86.429 + - type: map_at_5 + value: 86.91600000000001 + - type: mrr_at_1 + value: 81.9965748537177 + - type: mrr_at_10 + value: 87.22106125167394 + - type: mrr_at_100 + value: 87.37512926935148 + - type: mrr_at_1000 + value: 87.37841767615039 + - type: mrr_at_20 + value: 87.32764206073328 + - type: mrr_at_3 + value: 86.42904714333318 + - type: mrr_at_5 + value: 86.9157033442753 + - type: nauc_map_at_1000_diff1 + value: 84.98375890205129 + - type: nauc_map_at_1000_max + value: 52.169083096188054 + - type: nauc_map_at_1000_std + value: -35.027524704460916 + - type: nauc_map_at_100_diff1 + value: 84.98366301880897 + - type: nauc_map_at_100_max + value: 52.177207768384626 + - type: nauc_map_at_100_std + value: -35.01620836480514 + - type: nauc_map_at_10_diff1 + value: 84.95191430457028 + - type: nauc_map_at_10_max + value: 52.31158183948366 + - type: nauc_map_at_10_std + value: -35.13714349074112 + - type: nauc_map_at_1_diff1 + value: 86.2487043950067 + - type: nauc_map_at_1_max + value: 49.20239525557377 + - type: nauc_map_at_1_std + value: -33.71661281670688 + - type: nauc_map_at_20_diff1 + value: 84.973933039348 + - type: nauc_map_at_20_max + value: 52.2266012979943 + - type: nauc_map_at_20_std + value: -35.04159739087031 + - type: nauc_map_at_3_diff1 + value: 84.69068283876815 + - type: nauc_map_at_3_max + value: 52.343743449585524 + - type: nauc_map_at_3_std + value: -35.69366581732263 + - type: nauc_map_at_5_diff1 + value: 84.84059835122619 + - type: nauc_map_at_5_max + value: 52.29585814594443 + - type: nauc_map_at_5_std + value: -35.5113867651616 + - type: nauc_mrr_at_1000_diff1 + value: 84.98375890205129 + - type: nauc_mrr_at_1000_max + value: 52.169083096188054 + - type: nauc_mrr_at_1000_std + value: -35.027524704460916 + - type: nauc_mrr_at_100_diff1 + value: 84.98366301880897 + - type: nauc_mrr_at_100_max + value: 52.177207768384626 + - type: nauc_mrr_at_100_std + value: -35.01620836480514 + - type: nauc_mrr_at_10_diff1 + value: 84.95191430457028 + - type: nauc_mrr_at_10_max + value: 52.31158183948366 + - type: nauc_mrr_at_10_std + value: -35.13714349074112 + - type: nauc_mrr_at_1_diff1 + value: 86.2487043950067 + - type: nauc_mrr_at_1_max + value: 49.20239525557377 + - type: nauc_mrr_at_1_std + value: -33.71661281670688 + - type: nauc_mrr_at_20_diff1 + value: 84.973933039348 + - type: nauc_mrr_at_20_max + value: 52.2266012979943 + - type: nauc_mrr_at_20_std + value: -35.04159739087031 + - type: nauc_mrr_at_3_diff1 + value: 84.69068283876815 + - type: nauc_mrr_at_3_max + value: 52.343743449585524 + - type: nauc_mrr_at_3_std + value: -35.69366581732263 + - type: nauc_mrr_at_5_diff1 + value: 84.84059835122619 + - type: nauc_mrr_at_5_max + value: 52.29585814594443 + - type: nauc_mrr_at_5_std + value: -35.5113867651616 + - type: nauc_ndcg_at_1000_diff1 + value: 84.8458448603008 + - type: nauc_ndcg_at_1000_max + value: 52.77266190305249 + - type: nauc_ndcg_at_1000_std + value: -34.888069489815415 + - type: nauc_ndcg_at_100_diff1 + value: 84.84381394537597 + - type: nauc_ndcg_at_100_max + value: 53.01563101161627 + - type: nauc_ndcg_at_100_std + value: -34.509211628586314 + - type: nauc_ndcg_at_10_diff1 + value: 84.66548583706869 + - type: nauc_ndcg_at_10_max + value: 53.69134601286215 + - type: nauc_ndcg_at_10_std + value: -35.23415287527687 + - type: nauc_ndcg_at_1_diff1 + value: 86.2487043950067 + - type: nauc_ndcg_at_1_max + value: 49.20239525557377 + - type: nauc_ndcg_at_1_std + value: -33.71661281670688 + - type: nauc_ndcg_at_20_diff1 + value: 84.76038572414957 + - type: nauc_ndcg_at_20_max + value: 53.36813159950785 + - type: nauc_ndcg_at_20_std + value: -34.821410711460324 + - type: nauc_ndcg_at_3_diff1 + value: 84.1222575178288 + - type: nauc_ndcg_at_3_max + value: 53.5458597204313 + - type: nauc_ndcg_at_3_std + value: -36.57805029126134 + - type: nauc_ndcg_at_5_diff1 + value: 84.38081551698716 + - type: nauc_ndcg_at_5_max + value: 53.5309540463121 + - type: nauc_ndcg_at_5_std + value: -36.26371526873363 + - type: nauc_precision_at_1000_diff1 + value: 85.52761499735973 + - type: nauc_precision_at_1000_max + value: 85.57039121692812 + - type: nauc_precision_at_1000_std + value: 26.076840198712997 + - type: nauc_precision_at_100_diff1 + value: 84.71683047722709 + - type: nauc_precision_at_100_max + value: 78.6718703783981 + - type: nauc_precision_at_100_std + value: 7.427629853478685 + - type: nauc_precision_at_10_diff1 + value: 82.66742923426185 + - type: nauc_precision_at_10_max + value: 65.87095967797794 + - type: nauc_precision_at_10_std + value: -34.79989719383861 + - type: nauc_precision_at_1_diff1 + value: 86.2487043950067 + - type: nauc_precision_at_1_max + value: 49.20239525557377 + - type: nauc_precision_at_1_std + value: -33.71661281670688 + - type: nauc_precision_at_20_diff1 + value: 82.99219098545612 + - type: nauc_precision_at_20_max + value: 68.30891426964217 + - type: nauc_precision_at_20_std + value: -28.159133053121295 + - type: nauc_precision_at_3_diff1 + value: 81.61302076796814 + - type: nauc_precision_at_3_max + value: 58.883669048247576 + - type: nauc_precision_at_3_std + value: -40.57293703314752 + - type: nauc_precision_at_5_diff1 + value: 81.84587610537042 + - type: nauc_precision_at_5_max + value: 60.58183720409386 + - type: nauc_precision_at_5_std + value: -40.54776077088821 + - type: nauc_recall_at_1000_diff1 + value: 85.52761499733693 + - type: nauc_recall_at_1000_max + value: 85.57039121690828 + - type: nauc_recall_at_1000_std + value: 26.07684019872863 + - type: nauc_recall_at_100_diff1 + value: 84.7168304772257 + - type: nauc_recall_at_100_max + value: 78.67187037839709 + - type: nauc_recall_at_100_std + value: 7.427629853478298 + - type: nauc_recall_at_10_diff1 + value: 82.6674292342617 + - type: nauc_recall_at_10_max + value: 65.87095967797796 + - type: nauc_recall_at_10_std + value: -34.7998971938389 + - type: nauc_recall_at_1_diff1 + value: 86.2487043950067 + - type: nauc_recall_at_1_max + value: 49.20239525557377 + - type: nauc_recall_at_1_std + value: -33.71661281670688 + - type: nauc_recall_at_20_diff1 + value: 82.9921909854569 + - type: nauc_recall_at_20_max + value: 68.30891426964256 + - type: nauc_recall_at_20_std + value: -28.159133053120634 + - type: nauc_recall_at_3_diff1 + value: 81.61302076796801 + - type: nauc_recall_at_3_max + value: 58.88366904824765 + - type: nauc_recall_at_3_std + value: -40.57293703314801 + - type: nauc_recall_at_5_diff1 + value: 81.84587610537068 + - type: nauc_recall_at_5_max + value: 60.5818372040938 + - type: nauc_recall_at_5_std + value: -40.54776077088785 + - type: ndcg_at_1 + value: 81.997 + - type: ndcg_at_10 + value: 89.384 + - type: ndcg_at_100 + value: 90.091 + - type: ndcg_at_1000 + value: 90.18 + - type: ndcg_at_20 + value: 89.77000000000001 + - type: ndcg_at_3 + value: 87.788 + - type: ndcg_at_5 + value: 88.66000000000001 + - type: precision_at_1 + value: 81.997 + - type: precision_at_10 + value: 9.6 + - type: precision_at_100 + value: 0.992 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.8759999999999994 + - type: precision_at_3 + value: 30.564999999999998 + - type: precision_at_5 + value: 18.759999999999998 + - type: recall_at_1 + value: 81.997 + - type: recall_at_10 + value: 96.004 + - type: recall_at_100 + value: 99.208 + - type: recall_at_1000 + value: 99.914 + - type: recall_at_20 + value: 97.524 + - type: recall_at_3 + value: 91.694 + - type: recall_at_5 + value: 93.79899999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CodeFeedbackMT (default) + revision: b0f12fa0c0dd67f59c95a5c33d02aeeb4c398c5f + split: test + type: CoIR-Retrieval/codefeedback-mt + metrics: + - type: main_score + value: 93.581 + - type: map_at_1 + value: 89.17699999999999 + - type: map_at_10 + value: 92.293 + - type: map_at_100 + value: 92.39 + - type: map_at_1000 + value: 92.392 + - type: map_at_20 + value: 92.357 + - type: map_at_3 + value: 91.805 + - type: map_at_5 + value: 92.124 + - type: mrr_at_1 + value: 89.1767718611132 + - type: mrr_at_10 + value: 92.29094710867727 + - type: mrr_at_100 + value: 92.38822901927145 + - type: mrr_at_1000 + value: 92.39035562398948 + - type: mrr_at_20 + value: 92.35530678666272 + - type: mrr_at_3 + value: 91.80412241721295 + - type: mrr_at_5 + value: 92.122718485602 + - type: nauc_map_at_1000_diff1 + value: 91.32273096224377 + - type: nauc_map_at_1000_max + value: 67.91710992417916 + - type: nauc_map_at_1000_std + value: -74.42383054373403 + - type: nauc_map_at_100_diff1 + value: 91.32246480919676 + - type: nauc_map_at_100_max + value: 67.91988217232155 + - type: nauc_map_at_100_std + value: -74.42982893903614 + - type: nauc_map_at_10_diff1 + value: 91.33091844723225 + - type: nauc_map_at_10_max + value: 67.99200052857573 + - type: nauc_map_at_10_std + value: -74.86264731079909 + - type: nauc_map_at_1_diff1 + value: 91.19451424067374 + - type: nauc_map_at_1_max + value: 66.25392135499182 + - type: nauc_map_at_1_std + value: -66.71862444546531 + - type: nauc_map_at_20_diff1 + value: 91.338096532563 + - type: nauc_map_at_20_max + value: 67.9501381727388 + - type: nauc_map_at_20_std + value: -74.6541487409306 + - type: nauc_map_at_3_diff1 + value: 91.33378646831018 + - type: nauc_map_at_3_max + value: 68.07178423980443 + - type: nauc_map_at_3_std + value: -74.8448215489277 + - type: nauc_map_at_5_diff1 + value: 91.31919404747947 + - type: nauc_map_at_5_max + value: 67.9846196631338 + - type: nauc_map_at_5_std + value: -75.1772163451522 + - type: nauc_mrr_at_1000_diff1 + value: 91.3199902377502 + - type: nauc_mrr_at_1000_max + value: 67.9218672510084 + - type: nauc_mrr_at_1000_std + value: -74.43167854141912 + - type: nauc_mrr_at_100_diff1 + value: 91.31972491331909 + - type: nauc_mrr_at_100_max + value: 67.92463756203038 + - type: nauc_mrr_at_100_std + value: -74.43767343342054 + - type: nauc_mrr_at_10_diff1 + value: 91.32821156688269 + - type: nauc_mrr_at_10_max + value: 67.9966801151313 + - type: nauc_mrr_at_10_std + value: -74.87029985666574 + - type: nauc_mrr_at_1_diff1 + value: 91.19451424067374 + - type: nauc_mrr_at_1_max + value: 66.25392135499182 + - type: nauc_mrr_at_1_std + value: -66.71862444546531 + - type: nauc_mrr_at_20_diff1 + value: 91.33536515912955 + - type: nauc_mrr_at_20_max + value: 67.95486643404077 + - type: nauc_mrr_at_20_std + value: -74.66191099031441 + - type: nauc_mrr_at_3_diff1 + value: 91.330840031252 + - type: nauc_mrr_at_3_max + value: 68.07469130953064 + - type: nauc_mrr_at_3_std + value: -74.86010698346193 + - type: nauc_mrr_at_5_diff1 + value: 91.31654781777661 + - type: nauc_mrr_at_5_max + value: 67.98920016571047 + - type: nauc_mrr_at_5_std + value: -75.18463872642327 + - type: nauc_ndcg_at_1000_diff1 + value: 91.34667276431432 + - type: nauc_ndcg_at_1000_max + value: 68.21491424591551 + - type: nauc_ndcg_at_1000_std + value: -75.66992744128785 + - type: nauc_ndcg_at_100_diff1 + value: 91.32392901146443 + - type: nauc_ndcg_at_100_max + value: 68.29729595248577 + - type: nauc_ndcg_at_100_std + value: -75.71771214390895 + - type: nauc_ndcg_at_10_diff1 + value: 91.42077595372994 + - type: nauc_ndcg_at_10_max + value: 68.68694743767313 + - type: nauc_ndcg_at_10_std + value: -78.47824539632552 + - type: nauc_ndcg_at_1_diff1 + value: 91.19451424067374 + - type: nauc_ndcg_at_1_max + value: 66.25392135499182 + - type: nauc_ndcg_at_1_std + value: -66.71862444546531 + - type: nauc_ndcg_at_20_diff1 + value: 91.45155994342801 + - type: nauc_ndcg_at_20_max + value: 68.53325586053745 + - type: nauc_ndcg_at_20_std + value: -77.61552755050272 + - type: nauc_ndcg_at_3_diff1 + value: 91.40048288131362 + - type: nauc_ndcg_at_3_max + value: 68.77879241382503 + - type: nauc_ndcg_at_3_std + value: -78.145772024284 + - type: nauc_ndcg_at_5_diff1 + value: 91.38284059773454 + - type: nauc_ndcg_at_5_max + value: 68.65210990623613 + - type: nauc_ndcg_at_5_std + value: -79.12029324998576 + - type: nauc_precision_at_1000_diff1 + value: 93.46388819144805 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 63.81917689216168 + - type: nauc_precision_at_100_diff1 + value: 88.14968521318447 + - type: nauc_precision_at_100_max + value: 79.85443666227313 + - type: nauc_precision_at_100_std + value: -71.60524535547103 + - type: nauc_precision_at_10_diff1 + value: 92.26837092071085 + - type: nauc_precision_at_10_max + value: 74.6573995994554 + - type: nauc_precision_at_10_std + value: -110.01042456501374 + - type: nauc_precision_at_1_diff1 + value: 91.19451424067374 + - type: nauc_precision_at_1_max + value: 66.25392135499182 + - type: nauc_precision_at_1_std + value: -66.71862444546531 + - type: nauc_precision_at_20_diff1 + value: 93.20186301743918 + - type: nauc_precision_at_20_max + value: 75.80862507034404 + - type: nauc_precision_at_20_std + value: -114.47170564758076 + - type: nauc_precision_at_3_diff1 + value: 91.7095571991953 + - type: nauc_precision_at_3_max + value: 71.9834034178247 + - type: nauc_precision_at_3_std + value: -93.18361438847609 + - type: nauc_precision_at_5_diff1 + value: 91.7773542794046 + - type: nauc_precision_at_5_max + value: 72.60584615957593 + - type: nauc_precision_at_5_std + value: -103.21989758142311 + - type: nauc_recall_at_1000_diff1 + value: 93.46388819147347 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 63.819176892126386 + - type: nauc_recall_at_100_diff1 + value: 88.14968521317887 + - type: nauc_recall_at_100_max + value: 79.85443666226894 + - type: nauc_recall_at_100_std + value: -71.60524535547005 + - type: nauc_recall_at_10_diff1 + value: 92.2683709207107 + - type: nauc_recall_at_10_max + value: 74.65739959945586 + - type: nauc_recall_at_10_std + value: -110.01042456501415 + - type: nauc_recall_at_1_diff1 + value: 91.19451424067374 + - type: nauc_recall_at_1_max + value: 66.25392135499182 + - type: nauc_recall_at_1_std + value: -66.71862444546531 + - type: nauc_recall_at_20_diff1 + value: 93.20186301743836 + - type: nauc_recall_at_20_max + value: 75.8086250703445 + - type: nauc_recall_at_20_std + value: -114.47170564758093 + - type: nauc_recall_at_3_diff1 + value: 91.70955719919579 + - type: nauc_recall_at_3_max + value: 71.9834034178246 + - type: nauc_recall_at_3_std + value: -93.1836143884767 + - type: nauc_recall_at_5_diff1 + value: 91.77735427940513 + - type: nauc_recall_at_5_max + value: 72.60584615957615 + - type: nauc_recall_at_5_std + value: -103.21989758142347 + - type: ndcg_at_1 + value: 89.17699999999999 + - type: ndcg_at_10 + value: 93.581 + - type: ndcg_at_100 + value: 94.041 + - type: ndcg_at_1000 + value: 94.087 + - type: ndcg_at_20 + value: 93.813 + - type: ndcg_at_3 + value: 92.608 + - type: ndcg_at_5 + value: 93.183 + - type: precision_at_1 + value: 89.17699999999999 + - type: precision_at_10 + value: 9.751999999999999 + - type: precision_at_100 + value: 0.996 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.922 + - type: precision_at_3 + value: 31.639 + - type: precision_at_5 + value: 19.262 + - type: recall_at_1 + value: 89.17699999999999 + - type: recall_at_10 + value: 97.52199999999999 + - type: recall_at_100 + value: 99.638 + - type: recall_at_1000 + value: 99.985 + - type: recall_at_20 + value: 98.433 + - type: recall_at_3 + value: 94.916 + - type: recall_at_5 + value: 96.309 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CodeFeedbackST (default) + revision: d213819e87aab9010628da8b73ab4eb337c89340 + split: test + type: CoIR-Retrieval/codefeedback-st + metrics: + - type: main_score + value: 90.672 + - type: map_at_1 + value: 80.17 + - type: map_at_10 + value: 87.945 + - type: map_at_100 + value: 88.004 + - type: map_at_1000 + value: 88.004 + - type: map_at_20 + value: 87.99 + - type: map_at_3 + value: 87.248 + - type: map_at_5 + value: 87.79599999999999 + - type: mrr_at_1 + value: 80.08369002747078 + - type: mrr_at_10 + value: 87.8921544021677 + - type: mrr_at_100 + value: 87.95039935983601 + - type: mrr_at_1000 + value: 87.95113333851275 + - type: mrr_at_20 + value: 87.93646217104668 + - type: mrr_at_3 + value: 87.1787090866489 + - type: mrr_at_5 + value: 87.73579079622561 + - type: nauc_map_at_1000_diff1 + value: 84.59599547053739 + - type: nauc_map_at_1000_max + value: 11.563061625907412 + - type: nauc_map_at_1000_std + value: -60.044443468702234 + - type: nauc_map_at_100_diff1 + value: 84.59537761332842 + - type: nauc_map_at_100_max + value: 11.565642697551821 + - type: nauc_map_at_100_std + value: -60.04854322246466 + - type: nauc_map_at_10_diff1 + value: 84.54000249494548 + - type: nauc_map_at_10_max + value: 11.55440805584509 + - type: nauc_map_at_10_std + value: -60.437831701830824 + - type: nauc_map_at_1_diff1 + value: 86.11748953192901 + - type: nauc_map_at_1_max + value: 11.5654030601593 + - type: nauc_map_at_1_std + value: -53.511088508152106 + - type: nauc_map_at_20_diff1 + value: 84.58313379439 + - type: nauc_map_at_20_max + value: 11.570044402077079 + - type: nauc_map_at_20_std + value: -60.15198207708651 + - type: nauc_map_at_3_diff1 + value: 84.32880719653019 + - type: nauc_map_at_3_max + value: 11.306870549359703 + - type: nauc_map_at_3_std + value: -61.66972599149546 + - type: nauc_map_at_5_diff1 + value: 84.44481140471144 + - type: nauc_map_at_5_max + value: 11.431107322656027 + - type: nauc_map_at_5_std + value: -61.07717915800889 + - type: nauc_mrr_at_1000_diff1 + value: 84.68763078545194 + - type: nauc_mrr_at_1000_max + value: 11.654873092581681 + - type: nauc_mrr_at_1000_std + value: -59.336986725737205 + - type: nauc_mrr_at_100_diff1 + value: 84.68700936878848 + - type: nauc_mrr_at_100_max + value: 11.657439639783314 + - type: nauc_mrr_at_100_std + value: -59.34111569519106 + - type: nauc_mrr_at_10_diff1 + value: 84.6310183439079 + - type: nauc_mrr_at_10_max + value: 11.644216926632295 + - type: nauc_mrr_at_10_std + value: -59.72967384330525 + - type: nauc_mrr_at_1_diff1 + value: 86.20926531200153 + - type: nauc_mrr_at_1_max + value: 11.798672092672408 + - type: nauc_mrr_at_1_std + value: -52.65061182155048 + - type: nauc_mrr_at_20_diff1 + value: 84.67473718323151 + - type: nauc_mrr_at_20_max + value: 11.66210308966798 + - type: nauc_mrr_at_20_std + value: -59.44638680106128 + - type: nauc_mrr_at_3_diff1 + value: 84.43542937601633 + - type: nauc_mrr_at_3_max + value: 11.386252739390274 + - type: nauc_mrr_at_3_std + value: -60.847260123302824 + - type: nauc_mrr_at_5_diff1 + value: 84.54163679900752 + - type: nauc_mrr_at_5_max + value: 11.507249429826652 + - type: nauc_mrr_at_5_std + value: -60.37227397638133 + - type: nauc_ndcg_at_1000_diff1 + value: 84.38515659117648 + - type: nauc_ndcg_at_1000_max + value: 11.696388104993815 + - type: nauc_ndcg_at_1000_std + value: -60.99717188632636 + - type: nauc_ndcg_at_100_diff1 + value: 84.36252666208128 + - type: nauc_ndcg_at_100_max + value: 11.779193272629582 + - type: nauc_ndcg_at_100_std + value: -61.07767732450027 + - type: nauc_ndcg_at_10_diff1 + value: 84.04954066703365 + - type: nauc_ndcg_at_10_max + value: 11.779974489837082 + - type: nauc_ndcg_at_10_std + value: -63.216690095927774 + - type: nauc_ndcg_at_1_diff1 + value: 86.11748953192901 + - type: nauc_ndcg_at_1_max + value: 11.5654030601593 + - type: nauc_ndcg_at_1_std + value: -53.511088508152106 + - type: nauc_ndcg_at_20_diff1 + value: 84.24823772848971 + - type: nauc_ndcg_at_20_max + value: 11.863360518928207 + - type: nauc_ndcg_at_20_std + value: -61.93361896101012 + - type: nauc_ndcg_at_3_diff1 + value: 83.59193584331861 + - type: nauc_ndcg_at_3_max + value: 11.101961347273388 + - type: nauc_ndcg_at_3_std + value: -65.94436298501897 + - type: nauc_ndcg_at_5_diff1 + value: 83.78223501020723 + - type: nauc_ndcg_at_5_max + value: 11.381166299845784 + - type: nauc_ndcg_at_5_std + value: -65.04889907197641 + - type: nauc_precision_at_1000_diff1 + value: 76.06066445979025 + - type: nauc_precision_at_1000_max + value: 88.80070662604956 + - type: nauc_precision_at_1000_std + value: 52.334059183043905 + - type: nauc_precision_at_100_diff1 + value: 72.04652343451335 + - type: nauc_precision_at_100_max + value: 63.89605474602823 + - type: nauc_precision_at_100_std + value: -69.43370560244632 + - type: nauc_precision_at_10_diff1 + value: 72.89952199107627 + - type: nauc_precision_at_10_max + value: 19.311114535110576 + - type: nauc_precision_at_10_std + value: -128.84033294430404 + - type: nauc_precision_at_1_diff1 + value: 86.11748953192901 + - type: nauc_precision_at_1_max + value: 11.5654030601593 + - type: nauc_precision_at_1_std + value: -53.511088508152106 + - type: nauc_precision_at_20_diff1 + value: 72.67081355507392 + - type: nauc_precision_at_20_max + value: 31.810009027751327 + - type: nauc_precision_at_20_std + value: -125.0241647763245 + - type: nauc_precision_at_3_diff1 + value: 78.25481403127574 + - type: nauc_precision_at_3_max + value: 9.522751941864358 + - type: nauc_precision_at_3_std + value: -97.78790351049261 + - type: nauc_precision_at_5_diff1 + value: 74.76998378870415 + - type: nauc_precision_at_5_max + value: 10.813166077171799 + - type: nauc_precision_at_5_std + value: -121.15471825739507 + - type: nauc_recall_at_1000_diff1 + value: 76.06066445979108 + - type: nauc_recall_at_1000_max + value: 88.80070662604261 + - type: nauc_recall_at_1000_std + value: 52.334059183053746 + - type: nauc_recall_at_100_diff1 + value: 72.04652343451711 + - type: nauc_recall_at_100_max + value: 63.896054746020795 + - type: nauc_recall_at_100_std + value: -69.43370560243693 + - type: nauc_recall_at_10_diff1 + value: 72.89952199107772 + - type: nauc_recall_at_10_max + value: 19.311114535110473 + - type: nauc_recall_at_10_std + value: -128.8403329443047 + - type: nauc_recall_at_1_diff1 + value: 86.11748953192901 + - type: nauc_recall_at_1_max + value: 11.5654030601593 + - type: nauc_recall_at_1_std + value: -53.511088508152106 + - type: nauc_recall_at_20_diff1 + value: 72.6708135550744 + - type: nauc_recall_at_20_max + value: 31.81000902775227 + - type: nauc_recall_at_20_std + value: -125.02416477631772 + - type: nauc_recall_at_3_diff1 + value: 78.25481403127577 + - type: nauc_recall_at_3_max + value: 9.522751941864486 + - type: nauc_recall_at_3_std + value: -97.78790351049261 + - type: nauc_recall_at_5_diff1 + value: 74.76998378870401 + - type: nauc_recall_at_5_max + value: 10.813166077172303 + - type: nauc_recall_at_5_std + value: -121.15471825739255 + - type: ndcg_at_1 + value: 80.17 + - type: ndcg_at_10 + value: 90.672 + - type: ndcg_at_100 + value: 90.923 + - type: ndcg_at_1000 + value: 90.93900000000001 + - type: ndcg_at_20 + value: 90.833 + - type: ndcg_at_3 + value: 89.347 + - type: ndcg_at_5 + value: 90.321 + - type: precision_at_1 + value: 80.17 + - type: precision_at_10 + value: 9.875 + - type: precision_at_100 + value: 0.9990000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.968999999999999 + - type: precision_at_3 + value: 31.785999999999998 + - type: precision_at_5 + value: 19.538 + - type: recall_at_1 + value: 80.17 + - type: recall_at_10 + value: 98.75399999999999 + - type: recall_at_100 + value: 99.85300000000001 + - type: recall_at_1000 + value: 99.971 + - type: recall_at_20 + value: 99.384 + - type: recall_at_3 + value: 95.35900000000001 + - type: recall_at_5 + value: 97.69099999999999 + task: + type: Retrieval + - dataset: + config: python + name: MTEB CodeSearchNetCCRetrieval (python) + revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 + split: test + type: CoIR-Retrieval/CodeSearchNet-ccr + metrics: + - type: main_score + value: 94.433 + - type: map_at_1 + value: 89.19399999999999 + - type: map_at_10 + value: 93.028 + - type: map_at_100 + value: 93.082 + - type: map_at_1000 + value: 93.083 + - type: map_at_20 + value: 93.063 + - type: map_at_3 + value: 92.635 + - type: map_at_5 + value: 92.914 + - type: mrr_at_1 + value: 89.19426196541092 + - type: mrr_at_10 + value: 93.02709840248427 + - type: mrr_at_100 + value: 93.08116003424139 + - type: mrr_at_1000 + value: 93.08217593910118 + - type: mrr_at_20 + value: 93.06190896089124 + - type: mrr_at_3 + value: 92.63529516914724 + - type: mrr_at_5 + value: 92.91147160030434 + - type: nauc_map_at_1000_diff1 + value: 88.83311452663212 + - type: nauc_map_at_1000_max + value: 60.95017820522245 + - type: nauc_map_at_1000_std + value: -27.89769485046986 + - type: nauc_map_at_100_diff1 + value: 88.83290610426658 + - type: nauc_map_at_100_max + value: 60.95500551380402 + - type: nauc_map_at_100_std + value: -27.89016844487898 + - type: nauc_map_at_10_diff1 + value: 88.85572581806649 + - type: nauc_map_at_10_max + value: 61.15711675036097 + - type: nauc_map_at_10_std + value: -27.863737251156817 + - type: nauc_map_at_1_diff1 + value: 88.75783103896558 + - type: nauc_map_at_1_max + value: 56.05082536501996 + - type: nauc_map_at_1_std + value: -26.762271272974058 + - type: nauc_map_at_20_diff1 + value: 88.83684101691745 + - type: nauc_map_at_20_max + value: 61.005770466214415 + - type: nauc_map_at_20_std + value: -27.93102109901855 + - type: nauc_map_at_3_diff1 + value: 88.8117321224026 + - type: nauc_map_at_3_max + value: 61.93756889954432 + - type: nauc_map_at_3_std + value: -28.688755234894575 + - type: nauc_map_at_5_diff1 + value: 88.80829639130586 + - type: nauc_map_at_5_max + value: 61.453270288714315 + - type: nauc_map_at_5_std + value: -28.349980140724064 + - type: nauc_mrr_at_1000_diff1 + value: 88.84020090540668 + - type: nauc_mrr_at_1000_max + value: 60.889945118911925 + - type: nauc_mrr_at_1000_std + value: -27.98205495040352 + - type: nauc_mrr_at_100_diff1 + value: 88.8399914575814 + - type: nauc_mrr_at_100_max + value: 60.894780806340286 + - type: nauc_mrr_at_100_std + value: -27.974516894245998 + - type: nauc_mrr_at_10_diff1 + value: 88.86278756786537 + - type: nauc_mrr_at_10_max + value: 61.097374379118406 + - type: nauc_mrr_at_10_std + value: -27.94782127554896 + - type: nauc_mrr_at_1_diff1 + value: 88.75783103896558 + - type: nauc_mrr_at_1_max + value: 56.00537716973826 + - type: nauc_mrr_at_1_std + value: -26.855708242461617 + - type: nauc_mrr_at_20_diff1 + value: 88.84394060587955 + - type: nauc_mrr_at_20_max + value: 60.94574319151849 + - type: nauc_mrr_at_20_std + value: -28.015522604336574 + - type: nauc_mrr_at_3_diff1 + value: 88.8197212493924 + - type: nauc_mrr_at_3_max + value: 61.864940406758166 + - type: nauc_mrr_at_3_std + value: -28.785412568295914 + - type: nauc_mrr_at_5_diff1 + value: 88.82086414470385 + - type: nauc_mrr_at_5_max + value: 61.39568301316898 + - type: nauc_mrr_at_5_std + value: -28.470440741112203 + - type: nauc_ndcg_at_1000_diff1 + value: 88.87772673122608 + - type: nauc_ndcg_at_1000_max + value: 61.72947423485932 + - type: nauc_ndcg_at_1000_std + value: -27.73716470116156 + - type: nauc_ndcg_at_100_diff1 + value: 88.88494233648349 + - type: nauc_ndcg_at_100_max + value: 61.89646036981659 + - type: nauc_ndcg_at_100_std + value: -27.485345609558276 + - type: nauc_ndcg_at_10_diff1 + value: 89.0023293800841 + - type: nauc_ndcg_at_10_max + value: 62.94119991860201 + - type: nauc_ndcg_at_10_std + value: -27.460194148873917 + - type: nauc_ndcg_at_1_diff1 + value: 88.75783103896558 + - type: nauc_ndcg_at_1_max + value: 56.05082536501996 + - type: nauc_ndcg_at_1_std + value: -26.762271272974058 + - type: nauc_ndcg_at_20_diff1 + value: 88.91820321119603 + - type: nauc_ndcg_at_20_max + value: 62.296584649286466 + - type: nauc_ndcg_at_20_std + value: -27.766244930438113 + - type: nauc_ndcg_at_3_diff1 + value: 88.8645684782722 + - type: nauc_ndcg_at_3_max + value: 64.56115657789144 + - type: nauc_ndcg_at_3_std + value: -29.624561516864517 + - type: nauc_ndcg_at_5_diff1 + value: 88.86373245096877 + - type: nauc_ndcg_at_5_max + value: 63.72960904786197 + - type: nauc_ndcg_at_5_std + value: -28.9714090724707 + - type: nauc_precision_at_1000_diff1 + value: 93.46395492502747 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 79.16615094256558 + - type: nauc_precision_at_100_diff1 + value: 91.18620004808257 + - type: nauc_precision_at_100_max + value: 93.22441208688358 + - type: nauc_precision_at_100_std + value: 24.541027605722128 + - type: nauc_precision_at_10_diff1 + value: 91.39898569182662 + - type: nauc_precision_at_10_max + value: 86.36227958399685 + - type: nauc_precision_at_10_std + value: -18.10242240819358 + - type: nauc_precision_at_1_diff1 + value: 88.75783103896558 + - type: nauc_precision_at_1_max + value: 56.05082536501996 + - type: nauc_precision_at_1_std + value: -26.762271272974058 + - type: nauc_precision_at_20_diff1 + value: 90.67031369822483 + - type: nauc_precision_at_20_max + value: 83.74877222150064 + - type: nauc_precision_at_20_std + value: -20.408972788884217 + - type: nauc_precision_at_3_diff1 + value: 89.20609963322217 + - type: nauc_precision_at_3_max + value: 80.51313561745297 + - type: nauc_precision_at_3_std + value: -35.36879832053998 + - type: nauc_precision_at_5_diff1 + value: 89.40146615001582 + - type: nauc_precision_at_5_max + value: 83.94859838264775 + - type: nauc_precision_at_5_std + value: -34.3716686751886 + - type: nauc_recall_at_1000_diff1 + value: 93.46395492500885 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 79.1661509425585 + - type: nauc_recall_at_100_diff1 + value: 91.18620004808852 + - type: nauc_recall_at_100_max + value: 93.22441208689622 + - type: nauc_recall_at_100_std + value: 24.54102760573055 + - type: nauc_recall_at_10_diff1 + value: 91.39898569182566 + - type: nauc_recall_at_10_max + value: 86.36227958399564 + - type: nauc_recall_at_10_std + value: -18.102422408192854 + - type: nauc_recall_at_1_diff1 + value: 88.75783103896558 + - type: nauc_recall_at_1_max + value: 56.05082536501996 + - type: nauc_recall_at_1_std + value: -26.762271272974058 + - type: nauc_recall_at_20_diff1 + value: 90.67031369822419 + - type: nauc_recall_at_20_max + value: 83.7487722215012 + - type: nauc_recall_at_20_std + value: -20.408972788880643 + - type: nauc_recall_at_3_diff1 + value: 89.20609963322205 + - type: nauc_recall_at_3_max + value: 80.51313561745232 + - type: nauc_recall_at_3_std + value: -35.36879832053958 + - type: nauc_recall_at_5_diff1 + value: 89.40146615001574 + - type: nauc_recall_at_5_max + value: 83.94859838264772 + - type: nauc_recall_at_5_std + value: -34.371668675188346 + - type: ndcg_at_1 + value: 89.19399999999999 + - type: ndcg_at_10 + value: 94.433 + - type: ndcg_at_100 + value: 94.679 + - type: ndcg_at_1000 + value: 94.707 + - type: ndcg_at_20 + value: 94.556 + - type: ndcg_at_3 + value: 93.661 + - type: ndcg_at_5 + value: 94.161 + - type: precision_at_1 + value: 89.19399999999999 + - type: precision_at_10 + value: 9.864 + - type: precision_at_100 + value: 0.997 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.956 + - type: precision_at_3 + value: 32.2 + - type: precision_at_5 + value: 19.561999999999998 + - type: recall_at_1 + value: 89.19399999999999 + - type: recall_at_10 + value: 98.639 + - type: recall_at_100 + value: 99.74499999999999 + - type: recall_at_1000 + value: 99.973 + - type: recall_at_20 + value: 99.122 + - type: recall_at_3 + value: 96.601 + - type: recall_at_5 + value: 97.80799999999999 + task: + type: Retrieval + - dataset: + config: javascript + name: MTEB CodeSearchNetCCRetrieval (javascript) + revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 + split: test + type: CoIR-Retrieval/CodeSearchNet-ccr + metrics: + - type: main_score + value: 91.532 + - type: map_at_1 + value: 84.04700000000001 + - type: map_at_10 + value: 89.50399999999999 + - type: map_at_100 + value: 89.581 + - type: map_at_1000 + value: 89.584 + - type: map_at_20 + value: 89.548 + - type: map_at_3 + value: 88.914 + - type: map_at_5 + value: 89.271 + - type: mrr_at_1 + value: 83.98663020358553 + - type: mrr_at_10 + value: 89.46699271220692 + - type: mrr_at_100 + value: 89.54429619092761 + - type: mrr_at_1000 + value: 89.54715966900696 + - type: mrr_at_20 + value: 89.51131350599995 + - type: mrr_at_3 + value: 88.8787602552416 + - type: mrr_at_5 + value: 89.23427529626254 + - type: nauc_map_at_1000_diff1 + value: 81.33895244348352 + - type: nauc_map_at_1000_max + value: 51.005827566402075 + - type: nauc_map_at_1000_std + value: -22.252578589937457 + - type: nauc_map_at_100_diff1 + value: 81.33433499029653 + - type: nauc_map_at_100_max + value: 51.01323653352282 + - type: nauc_map_at_100_std + value: -22.24639503900029 + - type: nauc_map_at_10_diff1 + value: 81.30647802788602 + - type: nauc_map_at_10_max + value: 51.211145238065484 + - type: nauc_map_at_10_std + value: -22.30319656889033 + - type: nauc_map_at_1_diff1 + value: 82.57372620311462 + - type: nauc_map_at_1_max + value: 44.54905481211588 + - type: nauc_map_at_1_std + value: -21.962071475865756 + - type: nauc_map_at_20_diff1 + value: 81.31928022528241 + - type: nauc_map_at_20_max + value: 51.13238604171399 + - type: nauc_map_at_20_std + value: -22.20703220818948 + - type: nauc_map_at_3_diff1 + value: 81.10849311491785 + - type: nauc_map_at_3_max + value: 51.8208019735269 + - type: nauc_map_at_3_std + value: -22.96789103735518 + - type: nauc_map_at_5_diff1 + value: 81.1494438340007 + - type: nauc_map_at_5_max + value: 51.54484151827443 + - type: nauc_map_at_5_std + value: -22.39835418757183 + - type: nauc_mrr_at_1000_diff1 + value: 81.42887177105008 + - type: nauc_mrr_at_1000_max + value: 50.78456409105162 + - type: nauc_mrr_at_1000_std + value: -22.810803121918408 + - type: nauc_mrr_at_100_diff1 + value: 81.42424440922578 + - type: nauc_mrr_at_100_max + value: 50.792010710475864 + - type: nauc_mrr_at_100_std + value: -22.804483531306143 + - type: nauc_mrr_at_10_diff1 + value: 81.39579265182797 + - type: nauc_mrr_at_10_max + value: 50.99096194207801 + - type: nauc_mrr_at_10_std + value: -22.856845472961556 + - type: nauc_mrr_at_1_diff1 + value: 82.68635369279959 + - type: nauc_mrr_at_1_max + value: 44.2393421717323 + - type: nauc_mrr_at_1_std + value: -22.775172610293808 + - type: nauc_mrr_at_20_diff1 + value: 81.40894511054444 + - type: nauc_mrr_at_20_max + value: 50.91149353347425 + - type: nauc_mrr_at_20_std + value: -22.763426041817088 + - type: nauc_mrr_at_3_diff1 + value: 81.1947777094554 + - type: nauc_mrr_at_3_max + value: 51.60584449653166 + - type: nauc_mrr_at_3_std + value: -23.48440779958821 + - type: nauc_mrr_at_5_diff1 + value: 81.23725416306374 + - type: nauc_mrr_at_5_max + value: 51.32853667650834 + - type: nauc_mrr_at_5_std + value: -22.939278713896133 + - type: nauc_ndcg_at_1000_diff1 + value: 81.12746150854298 + - type: nauc_ndcg_at_1000_max + value: 52.1786963598477 + - type: nauc_ndcg_at_1000_std + value: -21.66735535060729 + - type: nauc_ndcg_at_100_diff1 + value: 81.00850024967238 + - type: nauc_ndcg_at_100_max + value: 52.376900809779855 + - type: nauc_ndcg_at_100_std + value: -21.437187821641885 + - type: nauc_ndcg_at_10_diff1 + value: 80.87079398487964 + - type: nauc_ndcg_at_10_max + value: 53.719012858465916 + - type: nauc_ndcg_at_10_std + value: -21.57272074203413 + - type: nauc_ndcg_at_1_diff1 + value: 82.57372620311462 + - type: nauc_ndcg_at_1_max + value: 44.54905481211588 + - type: nauc_ndcg_at_1_std + value: -21.962071475865756 + - type: nauc_ndcg_at_20_diff1 + value: 80.89584525085091 + - type: nauc_ndcg_at_20_max + value: 53.401415874657296 + - type: nauc_ndcg_at_20_std + value: -21.129352577941283 + - type: nauc_ndcg_at_3_diff1 + value: 80.46103065917166 + - type: nauc_ndcg_at_3_max + value: 54.91263156860439 + - type: nauc_ndcg_at_3_std + value: -23.0656484747823 + - type: nauc_ndcg_at_5_diff1 + value: 80.50385469743831 + - type: nauc_ndcg_at_5_max + value: 54.53609881176663 + - type: nauc_ndcg_at_5_std + value: -21.826145336827878 + - type: nauc_precision_at_1000_diff1 + value: 81.4825203906492 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 100.0 + - type: nauc_precision_at_100_diff1 + value: 69.4979677737838 + - type: nauc_precision_at_100_max + value: 79.81453242811645 + - type: nauc_precision_at_100_std + value: 22.735882735419853 + - type: nauc_precision_at_10_diff1 + value: 76.02578489916326 + - type: nauc_precision_at_10_max + value: 82.20220582823438 + - type: nauc_precision_at_10_std + value: -11.133888291415301 + - type: nauc_precision_at_1_diff1 + value: 82.57372620311462 + - type: nauc_precision_at_1_max + value: 44.54905481211588 + - type: nauc_precision_at_1_std + value: -21.962071475865756 + - type: nauc_precision_at_20_diff1 + value: 74.57535506743962 + - type: nauc_precision_at_20_max + value: 86.22474553094298 + - type: nauc_precision_at_20_std + value: 0.37124480616714156 + - type: nauc_precision_at_3_diff1 + value: 76.90725827318549 + - type: nauc_precision_at_3_max + value: 71.80786851596808 + - type: nauc_precision_at_3_std + value: -23.38779609088237 + - type: nauc_precision_at_5_diff1 + value: 75.76703985132757 + - type: nauc_precision_at_5_max + value: 76.30757619721807 + - type: nauc_precision_at_5_std + value: -16.772096432698 + - type: nauc_recall_at_1000_diff1 + value: 81.48252039064056 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 100.0 + - type: nauc_recall_at_100_diff1 + value: 69.49796777378361 + - type: nauc_recall_at_100_max + value: 79.81453242811749 + - type: nauc_recall_at_100_std + value: 22.73588273542357 + - type: nauc_recall_at_10_diff1 + value: 76.0257848991628 + - type: nauc_recall_at_10_max + value: 82.20220582823433 + - type: nauc_recall_at_10_std + value: -11.13388829141429 + - type: nauc_recall_at_1_diff1 + value: 82.57372620311462 + - type: nauc_recall_at_1_max + value: 44.54905481211588 + - type: nauc_recall_at_1_std + value: -21.962071475865756 + - type: nauc_recall_at_20_diff1 + value: 74.57535506744009 + - type: nauc_recall_at_20_max + value: 86.22474553094256 + - type: nauc_recall_at_20_std + value: 0.3712448061678841 + - type: nauc_recall_at_3_diff1 + value: 76.90725827318568 + - type: nauc_recall_at_3_max + value: 71.80786851596824 + - type: nauc_recall_at_3_std + value: -23.38779609088207 + - type: nauc_recall_at_5_diff1 + value: 75.76703985132794 + - type: nauc_recall_at_5_max + value: 76.30757619721837 + - type: nauc_recall_at_5_std + value: -16.772096432697477 + - type: ndcg_at_1 + value: 84.04700000000001 + - type: ndcg_at_10 + value: 91.532 + - type: ndcg_at_100 + value: 91.914 + - type: ndcg_at_1000 + value: 91.979 + - type: ndcg_at_20 + value: 91.688 + - type: ndcg_at_3 + value: 90.35300000000001 + - type: ndcg_at_5 + value: 90.993 + - type: precision_at_1 + value: 84.04700000000001 + - type: precision_at_10 + value: 9.763 + - type: precision_at_100 + value: 0.9939999999999999 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.912 + - type: precision_at_3 + value: 31.490000000000002 + - type: precision_at_5 + value: 19.204 + - type: recall_at_1 + value: 84.04700000000001 + - type: recall_at_10 + value: 97.63 + - type: recall_at_100 + value: 99.423 + - type: recall_at_1000 + value: 99.909 + - type: recall_at_20 + value: 98.238 + - type: recall_at_3 + value: 94.47 + - type: recall_at_5 + value: 96.019 + task: + type: Retrieval + - dataset: + config: go + name: MTEB CodeSearchNetCCRetrieval (go) + revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 + split: test + type: CoIR-Retrieval/CodeSearchNet-ccr + metrics: + - type: main_score + value: 91.75999999999999 + - type: map_at_1 + value: 84.28999999999999 + - type: map_at_10 + value: 89.689 + - type: map_at_100 + value: 89.773 + - type: map_at_1000 + value: 89.774 + - type: map_at_20 + value: 89.747 + - type: map_at_3 + value: 89.02600000000001 + - type: map_at_5 + value: 89.484 + - type: mrr_at_1 + value: 84.27727160797834 + - type: mrr_at_10 + value: 89.67515722532931 + - type: mrr_at_100 + value: 89.75855757010807 + - type: mrr_at_1000 + value: 89.7601837108608 + - type: mrr_at_20 + value: 89.73302201153147 + - type: mrr_at_3 + value: 89.01337929902301 + - type: mrr_at_5 + value: 89.47262579003495 + - type: nauc_map_at_1000_diff1 + value: 85.17428924238607 + - type: nauc_map_at_1000_max + value: 48.685961107484225 + - type: nauc_map_at_1000_std + value: -30.709156511297298 + - type: nauc_map_at_100_diff1 + value: 85.17188089036237 + - type: nauc_map_at_100_max + value: 48.6913539422267 + - type: nauc_map_at_100_std + value: -30.704177527491318 + - type: nauc_map_at_10_diff1 + value: 85.16218860131448 + - type: nauc_map_at_10_max + value: 48.8187190573415 + - type: nauc_map_at_10_std + value: -30.769000848055377 + - type: nauc_map_at_1_diff1 + value: 86.00618812234512 + - type: nauc_map_at_1_max + value: 45.7845929261639 + - type: nauc_map_at_1_std + value: -29.077490123642914 + - type: nauc_map_at_20_diff1 + value: 85.17186125725058 + - type: nauc_map_at_20_max + value: 48.7405854117929 + - type: nauc_map_at_20_std + value: -30.71428055810858 + - type: nauc_map_at_3_diff1 + value: 85.04460665149782 + - type: nauc_map_at_3_max + value: 49.06920858786511 + - type: nauc_map_at_3_std + value: -31.417225141010363 + - type: nauc_map_at_5_diff1 + value: 85.11442620241468 + - type: nauc_map_at_5_max + value: 48.834954058956406 + - type: nauc_map_at_5_std + value: -31.287907384502528 + - type: nauc_mrr_at_1000_diff1 + value: 85.21632400557363 + - type: nauc_mrr_at_1000_max + value: 48.84146877454467 + - type: nauc_mrr_at_1000_std + value: -30.7223552994054 + - type: nauc_mrr_at_100_diff1 + value: 85.2139127988098 + - type: nauc_mrr_at_100_max + value: 48.84683704195986 + - type: nauc_mrr_at_100_std + value: -30.717390859364478 + - type: nauc_mrr_at_10_diff1 + value: 85.20230661077468 + - type: nauc_mrr_at_10_max + value: 48.957929042209614 + - type: nauc_mrr_at_10_std + value: -30.78597710748554 + - type: nauc_mrr_at_1_diff1 + value: 86.0269378207981 + - type: nauc_mrr_at_1_max + value: 45.94255127280741 + - type: nauc_mrr_at_1_std + value: -29.011204207077935 + - type: nauc_mrr_at_20_diff1 + value: 85.21586630373561 + - type: nauc_mrr_at_20_max + value: 48.88183732057669 + - type: nauc_mrr_at_20_std + value: -30.73183934707418 + - type: nauc_mrr_at_3_diff1 + value: 85.08479102826571 + - type: nauc_mrr_at_3_max + value: 49.23008482061558 + - type: nauc_mrr_at_3_std + value: -31.465046952057303 + - type: nauc_mrr_at_5_diff1 + value: 85.1584637044273 + - type: nauc_mrr_at_5_max + value: 49.00177049406493 + - type: nauc_mrr_at_5_std + value: -31.307066011998224 + - type: nauc_ndcg_at_1000_diff1 + value: 85.09478158429043 + - type: nauc_ndcg_at_1000_max + value: 49.25560935342975 + - type: nauc_ndcg_at_1000_std + value: -30.70761801312305 + - type: nauc_ndcg_at_100_diff1 + value: 85.03251451319845 + - type: nauc_ndcg_at_100_max + value: 49.42104394245281 + - type: nauc_ndcg_at_100_std + value: -30.551562494744168 + - type: nauc_ndcg_at_10_diff1 + value: 85.00106483349344 + - type: nauc_ndcg_at_10_max + value: 50.115231910648575 + - type: nauc_ndcg_at_10_std + value: -30.87312388259668 + - type: nauc_ndcg_at_1_diff1 + value: 86.00618812234512 + - type: nauc_ndcg_at_1_max + value: 45.7845929261639 + - type: nauc_ndcg_at_1_std + value: -29.077490123642914 + - type: nauc_ndcg_at_20_diff1 + value: 85.05361380063081 + - type: nauc_ndcg_at_20_max + value: 49.772465794293254 + - type: nauc_ndcg_at_20_std + value: -30.672595686871666 + - type: nauc_ndcg_at_3_diff1 + value: 84.77305397981563 + - type: nauc_ndcg_at_3_max + value: 50.52441698065767 + - type: nauc_ndcg_at_3_std + value: -32.46848973997472 + - type: nauc_ndcg_at_5_diff1 + value: 84.88420002224352 + - type: nauc_ndcg_at_5_max + value: 50.11133138392584 + - type: nauc_ndcg_at_5_std + value: -32.3270235813713 + - type: nauc_precision_at_1000_diff1 + value: 91.83038946833551 + - type: nauc_precision_at_1000_max + value: 88.46880842429545 + - type: nauc_precision_at_1000_std + value: 21.769969322436477 + - type: nauc_precision_at_100_diff1 + value: 76.47557733881311 + - type: nauc_precision_at_100_max + value: 84.28732698302338 + - type: nauc_precision_at_100_std + value: 5.283523585432388 + - type: nauc_precision_at_10_diff1 + value: 83.18181354333525 + - type: nauc_precision_at_10_max + value: 68.00168082360678 + - type: nauc_precision_at_10_std + value: -29.843581018958474 + - type: nauc_precision_at_1_diff1 + value: 86.00618812234512 + - type: nauc_precision_at_1_max + value: 45.7845929261639 + - type: nauc_precision_at_1_std + value: -29.077490123642914 + - type: nauc_precision_at_20_diff1 + value: 83.4769114952918 + - type: nauc_precision_at_20_max + value: 71.01630819186266 + - type: nauc_precision_at_20_std + value: -24.608779769104114 + - type: nauc_precision_at_3_diff1 + value: 83.38151337361876 + - type: nauc_precision_at_3_max + value: 58.48742413861129 + - type: nauc_precision_at_3_std + value: -38.23005616508197 + - type: nauc_precision_at_5_diff1 + value: 83.16792264231816 + - type: nauc_precision_at_5_max + value: 60.3244549312709 + - type: nauc_precision_at_5_std + value: -40.87299770895208 + - type: nauc_recall_at_1000_diff1 + value: 91.83038946832791 + - type: nauc_recall_at_1000_max + value: 88.46880842431686 + - type: nauc_recall_at_1000_std + value: 21.769969322480403 + - type: nauc_recall_at_100_diff1 + value: 76.47557733881604 + - type: nauc_recall_at_100_max + value: 84.28732698303843 + - type: nauc_recall_at_100_std + value: 5.283523585434394 + - type: nauc_recall_at_10_diff1 + value: 83.18181354333507 + - type: nauc_recall_at_10_max + value: 68.00168082360621 + - type: nauc_recall_at_10_std + value: -29.843581018957572 + - type: nauc_recall_at_1_diff1 + value: 86.00618812234512 + - type: nauc_recall_at_1_max + value: 45.7845929261639 + - type: nauc_recall_at_1_std + value: -29.077490123642914 + - type: nauc_recall_at_20_diff1 + value: 83.47691149529236 + - type: nauc_recall_at_20_max + value: 71.01630819186342 + - type: nauc_recall_at_20_std + value: -24.608779769102323 + - type: nauc_recall_at_3_diff1 + value: 83.38151337361859 + - type: nauc_recall_at_3_max + value: 58.48742413861102 + - type: nauc_recall_at_3_std + value: -38.23005616508199 + - type: nauc_recall_at_5_diff1 + value: 83.16792264231778 + - type: nauc_recall_at_5_max + value: 60.32445493127084 + - type: nauc_recall_at_5_std + value: -40.8729977089511 + - type: ndcg_at_1 + value: 84.28999999999999 + - type: ndcg_at_10 + value: 91.75999999999999 + - type: ndcg_at_100 + value: 92.128 + - type: ndcg_at_1000 + value: 92.166 + - type: ndcg_at_20 + value: 91.965 + - type: ndcg_at_3 + value: 90.441 + - type: ndcg_at_5 + value: 91.264 + - type: precision_at_1 + value: 84.28999999999999 + - type: precision_at_10 + value: 9.802 + - type: precision_at_100 + value: 0.997 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.941 + - type: precision_at_3 + value: 31.499 + - type: precision_at_5 + value: 19.298000000000002 + - type: recall_at_1 + value: 84.28999999999999 + - type: recall_at_10 + value: 98.018 + - type: recall_at_100 + value: 99.655 + - type: recall_at_1000 + value: 99.938 + - type: recall_at_20 + value: 98.818 + - type: recall_at_3 + value: 94.49600000000001 + - type: recall_at_5 + value: 96.491 + task: + type: Retrieval + - dataset: + config: ruby + name: MTEB CodeSearchNetCCRetrieval (ruby) + revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 + split: test + type: CoIR-Retrieval/CodeSearchNet-ccr + metrics: + - type: main_score + value: 89.25699999999999 + - type: map_at_1 + value: 80.967 + - type: map_at_10 + value: 86.978 + - type: map_at_100 + value: 87.113 + - type: map_at_1000 + value: 87.117 + - type: map_at_20 + value: 87.054 + - type: map_at_3 + value: 86.29400000000001 + - type: map_at_5 + value: 86.738 + - type: mrr_at_1 + value: 80.9674861221253 + - type: mrr_at_10 + value: 86.95775587528166 + - type: mrr_at_100 + value: 87.09311803235961 + - type: mrr_at_1000 + value: 87.09630839341808 + - type: mrr_at_20 + value: 87.03383454810835 + - type: mrr_at_3 + value: 86.25429553264607 + - type: mrr_at_5 + value: 86.71821305841928 + - type: nauc_map_at_1000_diff1 + value: 82.17808945938484 + - type: nauc_map_at_1000_max + value: 60.75779539267633 + - type: nauc_map_at_1000_std + value: -23.755896580375826 + - type: nauc_map_at_100_diff1 + value: 82.1794560801495 + - type: nauc_map_at_100_max + value: 60.76366486911127 + - type: nauc_map_at_100_std + value: -23.74131956559222 + - type: nauc_map_at_10_diff1 + value: 82.18782847323823 + - type: nauc_map_at_10_max + value: 60.91576878869061 + - type: nauc_map_at_10_std + value: -23.863677820347533 + - type: nauc_map_at_1_diff1 + value: 83.72039170136367 + - type: nauc_map_at_1_max + value: 57.25380612858921 + - type: nauc_map_at_1_std + value: -25.935791475913987 + - type: nauc_map_at_20_diff1 + value: 82.17796635750986 + - type: nauc_map_at_20_max + value: 60.83301592914025 + - type: nauc_map_at_20_std + value: -23.683955007198033 + - type: nauc_map_at_3_diff1 + value: 81.83929922286214 + - type: nauc_map_at_3_max + value: 61.30090311509839 + - type: nauc_map_at_3_std + value: -23.67250024837735 + - type: nauc_map_at_5_diff1 + value: 81.88267528957118 + - type: nauc_map_at_5_max + value: 61.06118360132867 + - type: nauc_map_at_5_std + value: -23.564202037953418 + - type: nauc_mrr_at_1000_diff1 + value: 82.24831373360206 + - type: nauc_mrr_at_1000_max + value: 60.547458216811776 + - type: nauc_mrr_at_1000_std + value: -24.62799535642754 + - type: nauc_mrr_at_100_diff1 + value: 82.24966010195052 + - type: nauc_mrr_at_100_max + value: 60.553373399867574 + - type: nauc_mrr_at_100_std + value: -24.613214185234447 + - type: nauc_mrr_at_10_diff1 + value: 82.25711255868444 + - type: nauc_mrr_at_10_max + value: 60.70674951461077 + - type: nauc_mrr_at_10_std + value: -24.723134285637112 + - type: nauc_mrr_at_1_diff1 + value: 83.72039170136367 + - type: nauc_mrr_at_1_max + value: 56.98766026206557 + - type: nauc_mrr_at_1_std + value: -26.606885807082385 + - type: nauc_mrr_at_20_diff1 + value: 82.24783610762223 + - type: nauc_mrr_at_20_max + value: 60.62362881960727 + - type: nauc_mrr_at_20_std + value: -24.551756212224145 + - type: nauc_mrr_at_3_diff1 + value: 81.93274024955835 + - type: nauc_mrr_at_3_max + value: 61.09367721118326 + - type: nauc_mrr_at_3_std + value: -24.713070452305637 + - type: nauc_mrr_at_5_diff1 + value: 81.95108632175992 + - type: nauc_mrr_at_5_max + value: 60.85590424882734 + - type: nauc_mrr_at_5_std + value: -24.40785466160273 + - type: nauc_ndcg_at_1000_diff1 + value: 82.13638313064818 + - type: nauc_ndcg_at_1000_max + value: 61.36229533164094 + - type: nauc_ndcg_at_1000_std + value: -23.263619782617027 + - type: nauc_ndcg_at_100_diff1 + value: 82.17708460929015 + - type: nauc_ndcg_at_100_max + value: 61.55039283837963 + - type: nauc_ndcg_at_100_std + value: -22.82475534090763 + - type: nauc_ndcg_at_10_diff1 + value: 82.19581572094847 + - type: nauc_ndcg_at_10_max + value: 62.250754196853194 + - type: nauc_ndcg_at_10_std + value: -23.522643567705575 + - type: nauc_ndcg_at_1_diff1 + value: 83.72039170136367 + - type: nauc_ndcg_at_1_max + value: 57.25380612858921 + - type: nauc_ndcg_at_1_std + value: -25.935791475913987 + - type: nauc_ndcg_at_20_diff1 + value: 82.14559730614958 + - type: nauc_ndcg_at_20_max + value: 61.93079238970315 + - type: nauc_ndcg_at_20_std + value: -22.719105026052123 + - type: nauc_ndcg_at_3_diff1 + value: 81.36414038291007 + - type: nauc_ndcg_at_3_max + value: 62.97389940233761 + - type: nauc_ndcg_at_3_std + value: -22.958679785157702 + - type: nauc_ndcg_at_5_diff1 + value: 81.41077106697202 + - type: nauc_ndcg_at_5_max + value: 62.605999596646335 + - type: nauc_ndcg_at_5_std + value: -22.684305831996262 + - type: nauc_precision_at_1000_diff1 + value: 100.0 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 55.42527628232039 + - type: nauc_precision_at_100_diff1 + value: 87.74666354712059 + - type: nauc_precision_at_100_max + value: 83.08111330779417 + - type: nauc_precision_at_100_std + value: 25.395348413223466 + - type: nauc_precision_at_10_diff1 + value: 83.35598753622322 + - type: nauc_precision_at_10_max + value: 73.70169446342423 + - type: nauc_precision_at_10_std + value: -21.615549803915652 + - type: nauc_precision_at_1_diff1 + value: 83.72039170136367 + - type: nauc_precision_at_1_max + value: 57.25380612858921 + - type: nauc_precision_at_1_std + value: -25.935791475913987 + - type: nauc_precision_at_20_diff1 + value: 83.03143013817835 + - type: nauc_precision_at_20_max + value: 73.49113429650389 + - type: nauc_precision_at_20_std + value: -9.826961687536869 + - type: nauc_precision_at_3_diff1 + value: 79.16945277065827 + - type: nauc_precision_at_3_max + value: 71.31564419236823 + - type: nauc_precision_at_3_std + value: -19.52745553766472 + - type: nauc_precision_at_5_diff1 + value: 78.5956097040408 + - type: nauc_precision_at_5_max + value: 72.48547478245246 + - type: nauc_precision_at_5_std + value: -17.07438252522886 + - type: nauc_recall_at_1000_diff1 + value: 100.0 + - type: nauc_recall_at_1000_max + value: 100.0 + - type: nauc_recall_at_1000_std + value: 55.42527628229722 + - type: nauc_recall_at_100_diff1 + value: 87.74666354712278 + - type: nauc_recall_at_100_max + value: 83.08111330779381 + - type: nauc_recall_at_100_std + value: 25.39534841322166 + - type: nauc_recall_at_10_diff1 + value: 83.35598753622315 + - type: nauc_recall_at_10_max + value: 73.70169446342429 + - type: nauc_recall_at_10_std + value: -21.615549803915744 + - type: nauc_recall_at_1_diff1 + value: 83.72039170136367 + - type: nauc_recall_at_1_max + value: 57.25380612858921 + - type: nauc_recall_at_1_std + value: -25.935791475913987 + - type: nauc_recall_at_20_diff1 + value: 83.03143013817795 + - type: nauc_recall_at_20_max + value: 73.49113429650306 + - type: nauc_recall_at_20_std + value: -9.826961687536835 + - type: nauc_recall_at_3_diff1 + value: 79.16945277065781 + - type: nauc_recall_at_3_max + value: 71.31564419236814 + - type: nauc_recall_at_3_std + value: -19.527455537664128 + - type: nauc_recall_at_5_diff1 + value: 78.59560970404043 + - type: nauc_recall_at_5_max + value: 72.4854747824523 + - type: nauc_recall_at_5_std + value: -17.074382525229172 + - type: ndcg_at_1 + value: 80.967 + - type: ndcg_at_10 + value: 89.25699999999999 + - type: ndcg_at_100 + value: 89.91799999999999 + - type: ndcg_at_1000 + value: 90.007 + - type: ndcg_at_20 + value: 89.524 + - type: ndcg_at_3 + value: 87.909 + - type: ndcg_at_5 + value: 88.701 + - type: precision_at_1 + value: 80.967 + - type: precision_at_10 + value: 9.611 + - type: precision_at_100 + value: 0.992 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.857 + - type: precision_at_3 + value: 30.849 + - type: precision_at_5 + value: 18.89 + - type: recall_at_1 + value: 80.967 + - type: recall_at_10 + value: 96.114 + - type: recall_at_100 + value: 99.207 + - type: recall_at_1000 + value: 99.921 + - type: recall_at_20 + value: 97.14500000000001 + - type: recall_at_3 + value: 92.54599999999999 + - type: recall_at_5 + value: 94.44900000000001 + task: + type: Retrieval + - dataset: + config: java + name: MTEB CodeSearchNetCCRetrieval (java) + revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 + split: test + type: CoIR-Retrieval/CodeSearchNet-ccr + metrics: + - type: main_score + value: 90.187 + - type: map_at_1 + value: 83.04899999999999 + - type: map_at_10 + value: 88.2 + - type: map_at_100 + value: 88.294 + - type: map_at_1000 + value: 88.297 + - type: map_at_20 + value: 88.268 + - type: map_at_3 + value: 87.581 + - type: map_at_5 + value: 87.997 + - type: mrr_at_1 + value: 83.03057964399817 + - type: mrr_at_10 + value: 88.19186353987806 + - type: mrr_at_100 + value: 88.2852857857312 + - type: mrr_at_1000 + value: 88.28810205243452 + - type: mrr_at_20 + value: 88.25951626896247 + - type: mrr_at_3 + value: 87.5749277346721 + - type: mrr_at_5 + value: 87.9875247223489 + - type: nauc_map_at_1000_diff1 + value: 84.10730394433992 + - type: nauc_map_at_1000_max + value: 61.52233212127969 + - type: nauc_map_at_1000_std + value: -13.441699805696281 + - type: nauc_map_at_100_diff1 + value: 84.10552079587292 + - type: nauc_map_at_100_max + value: 61.52588029468992 + - type: nauc_map_at_100_std + value: -13.438768702376994 + - type: nauc_map_at_10_diff1 + value: 84.06622800697097 + - type: nauc_map_at_10_max + value: 61.62852938331853 + - type: nauc_map_at_10_std + value: -13.540123911279858 + - type: nauc_map_at_1_diff1 + value: 84.58928110942394 + - type: nauc_map_at_1_max + value: 56.15285789003811 + - type: nauc_map_at_1_std + value: -17.221403594342735 + - type: nauc_map_at_20_diff1 + value: 84.09390965699994 + - type: nauc_map_at_20_max + value: 61.569980648560886 + - type: nauc_map_at_20_std + value: -13.445035300847309 + - type: nauc_map_at_3_diff1 + value: 83.88744798135446 + - type: nauc_map_at_3_max + value: 61.88368527363093 + - type: nauc_map_at_3_std + value: -14.711176741863827 + - type: nauc_map_at_5_diff1 + value: 83.98674004934382 + - type: nauc_map_at_5_max + value: 61.74583336524621 + - type: nauc_map_at_5_std + value: -13.965194914228382 + - type: nauc_mrr_at_1000_diff1 + value: 84.07413270132193 + - type: nauc_mrr_at_1000_max + value: 61.703650435638856 + - type: nauc_mrr_at_1000_std + value: -13.350251134092783 + - type: nauc_mrr_at_100_diff1 + value: 84.07235854433452 + - type: nauc_mrr_at_100_max + value: 61.707169972354 + - type: nauc_mrr_at_100_std + value: -13.347338815823864 + - type: nauc_mrr_at_10_diff1 + value: 84.03062092695845 + - type: nauc_mrr_at_10_max + value: 61.80570291957679 + - type: nauc_mrr_at_10_std + value: -13.438890580569844 + - type: nauc_mrr_at_1_diff1 + value: 84.61937189362341 + - type: nauc_mrr_at_1_max + value: 56.41943088748145 + - type: nauc_mrr_at_1_std + value: -17.177446751086197 + - type: nauc_mrr_at_20_diff1 + value: 84.06068844396994 + - type: nauc_mrr_at_20_max + value: 61.750926871721475 + - type: nauc_mrr_at_20_std + value: -13.35357387841223 + - type: nauc_mrr_at_3_diff1 + value: 83.85345674418377 + - type: nauc_mrr_at_3_max + value: 62.043812033960535 + - type: nauc_mrr_at_3_std + value: -14.6395621624969 + - type: nauc_mrr_at_5_diff1 + value: 83.95764939870173 + - type: nauc_mrr_at_5_max + value: 61.92516669938368 + - type: nauc_mrr_at_5_std + value: -13.886532596685806 + - type: nauc_ndcg_at_1000_diff1 + value: 84.42408974340631 + - type: nauc_ndcg_at_1000_max + value: 63.32980075889167 + - type: nauc_ndcg_at_1000_std + value: -9.859850962580813 + - type: nauc_ndcg_at_100_diff1 + value: 84.3765359347518 + - type: nauc_ndcg_at_100_max + value: 63.39891417852467 + - type: nauc_ndcg_at_100_std + value: -9.822808442396664 + - type: nauc_ndcg_at_10_diff1 + value: 84.14419181720488 + - type: nauc_ndcg_at_10_max + value: 63.950601713958136 + - type: nauc_ndcg_at_10_std + value: -10.490521537540003 + - type: nauc_ndcg_at_1_diff1 + value: 84.58928110942394 + - type: nauc_ndcg_at_1_max + value: 56.15285789003811 + - type: nauc_ndcg_at_1_std + value: -17.221403594342735 + - type: nauc_ndcg_at_20_diff1 + value: 84.26722898082026 + - type: nauc_ndcg_at_20_max + value: 63.749766448071746 + - type: nauc_ndcg_at_20_std + value: -10.004576105283306 + - type: nauc_ndcg_at_3_diff1 + value: 83.72365997941274 + - type: nauc_ndcg_at_3_max + value: 64.3195097894205 + - type: nauc_ndcg_at_3_std + value: -13.41750319915828 + - type: nauc_ndcg_at_5_diff1 + value: 83.91163247137122 + - type: nauc_ndcg_at_5_max + value: 64.19792463538528 + - type: nauc_ndcg_at_5_std + value: -11.76492800063572 + - type: nauc_precision_at_1000_diff1 + value: 97.13217959743038 + - type: nauc_precision_at_1000_max + value: 98.46911971679215 + - type: nauc_precision_at_1000_std + value: 90.2347995052506 + - type: nauc_precision_at_100_diff1 + value: 91.63251470011757 + - type: nauc_precision_at_100_max + value: 90.17514956130655 + - type: nauc_precision_at_100_std + value: 61.0806660186732 + - type: nauc_precision_at_10_diff1 + value: 85.28726061856946 + - type: nauc_precision_at_10_max + value: 83.23195279280301 + - type: nauc_precision_at_10_std + value: 17.658421447762464 + - type: nauc_precision_at_1_diff1 + value: 84.58928110942394 + - type: nauc_precision_at_1_max + value: 56.15285789003811 + - type: nauc_precision_at_1_std + value: -17.221403594342735 + - type: nauc_precision_at_20_diff1 + value: 87.21693331269776 + - type: nauc_precision_at_20_max + value: 86.97422806229869 + - type: nauc_precision_at_20_std + value: 32.947726980858015 + - type: nauc_precision_at_3_diff1 + value: 83.00721034174204 + - type: nauc_precision_at_3_max + value: 76.03478845540039 + - type: nauc_precision_at_3_std + value: -7.075307314296672 + - type: nauc_precision_at_5_diff1 + value: 83.59241003022704 + - type: nauc_precision_at_5_max + value: 79.35871830248998 + - type: nauc_precision_at_5_std + value: 2.590360587580312 + - type: nauc_recall_at_1000_diff1 + value: 97.13217959743086 + - type: nauc_recall_at_1000_max + value: 98.46911971679087 + - type: nauc_recall_at_1000_std + value: 90.23479950524963 + - type: nauc_recall_at_100_diff1 + value: 91.6325147001174 + - type: nauc_recall_at_100_max + value: 90.17514956130668 + - type: nauc_recall_at_100_std + value: 61.08066601867284 + - type: nauc_recall_at_10_diff1 + value: 85.28726061856851 + - type: nauc_recall_at_10_max + value: 83.23195279280242 + - type: nauc_recall_at_10_std + value: 17.65842144776203 + - type: nauc_recall_at_1_diff1 + value: 84.58928110942394 + - type: nauc_recall_at_1_max + value: 56.15285789003811 + - type: nauc_recall_at_1_std + value: -17.221403594342735 + - type: nauc_recall_at_20_diff1 + value: 87.21693331269785 + - type: nauc_recall_at_20_max + value: 86.97422806229821 + - type: nauc_recall_at_20_std + value: 32.94772698085802 + - type: nauc_recall_at_3_diff1 + value: 83.007210341742 + - type: nauc_recall_at_3_max + value: 76.03478845540012 + - type: nauc_recall_at_3_std + value: -7.075307314297068 + - type: nauc_recall_at_5_diff1 + value: 83.59241003022677 + - type: nauc_recall_at_5_max + value: 79.35871830248969 + - type: nauc_recall_at_5_std + value: 2.5903605875805535 + - type: ndcg_at_1 + value: 83.04899999999999 + - type: ndcg_at_10 + value: 90.187 + - type: ndcg_at_100 + value: 90.61399999999999 + - type: ndcg_at_1000 + value: 90.686 + - type: ndcg_at_20 + value: 90.429 + - type: ndcg_at_3 + value: 88.957 + - type: ndcg_at_5 + value: 89.7 + - type: precision_at_1 + value: 83.04899999999999 + - type: precision_at_10 + value: 9.619 + - type: precision_at_100 + value: 0.9809999999999999 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 4.857 + - type: precision_at_3 + value: 30.969 + - type: precision_at_5 + value: 18.939 + - type: recall_at_1 + value: 83.04899999999999 + - type: recall_at_10 + value: 96.194 + - type: recall_at_100 + value: 98.129 + - type: recall_at_1000 + value: 98.695 + - type: recall_at_20 + value: 97.134 + - type: recall_at_3 + value: 92.907 + - type: recall_at_5 + value: 94.696 + task: + type: Retrieval + - dataset: + config: php + name: MTEB CodeSearchNetCCRetrieval (php) + revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 + split: test + type: CoIR-Retrieval/CodeSearchNet-ccr + metrics: + - type: main_score + value: 83.38799999999999 + - type: map_at_1 + value: 73.25500000000001 + - type: map_at_10 + value: 80.387 + - type: map_at_100 + value: 80.64099999999999 + - type: map_at_1000 + value: 80.648 + - type: map_at_20 + value: 80.55799999999999 + - type: map_at_3 + value: 79.256 + - type: map_at_5 + value: 79.97800000000001 + - type: mrr_at_1 + value: 73.24818039103754 + - type: mrr_at_10 + value: 80.37402789953873 + - type: mrr_at_100 + value: 80.62749361355517 + - type: mrr_at_1000 + value: 80.63492699285514 + - type: mrr_at_20 + value: 80.5450937498758 + - type: mrr_at_3 + value: 79.25170068027252 + - type: mrr_at_5 + value: 79.96063460349235 + - type: nauc_map_at_1000_diff1 + value: 77.55679733254976 + - type: nauc_map_at_1000_max + value: 47.30416473970588 + - type: nauc_map_at_1000_std + value: -26.244493881093646 + - type: nauc_map_at_100_diff1 + value: 77.55521791705124 + - type: nauc_map_at_100_max + value: 47.30912923045687 + - type: nauc_map_at_100_std + value: -26.243222353374808 + - type: nauc_map_at_10_diff1 + value: 77.46319200229681 + - type: nauc_map_at_10_max + value: 47.42459632571874 + - type: nauc_map_at_10_std + value: -26.359798504203013 + - type: nauc_map_at_1_diff1 + value: 79.527051162636 + - type: nauc_map_at_1_max + value: 44.89631724552463 + - type: nauc_map_at_1_std + value: -25.451766234360097 + - type: nauc_map_at_20_diff1 + value: 77.52788532848108 + - type: nauc_map_at_20_max + value: 47.33830573278101 + - type: nauc_map_at_20_std + value: -26.257482321628622 + - type: nauc_map_at_3_diff1 + value: 77.41062263238054 + - type: nauc_map_at_3_max + value: 47.499244399918155 + - type: nauc_map_at_3_std + value: -26.523866758341992 + - type: nauc_map_at_5_diff1 + value: 77.35129682203655 + - type: nauc_map_at_5_max + value: 47.47681231525524 + - type: nauc_map_at_5_std + value: -26.5173636683384 + - type: nauc_mrr_at_1000_diff1 + value: 77.57769206330354 + - type: nauc_mrr_at_1000_max + value: 47.34367340151292 + - type: nauc_mrr_at_1000_std + value: -26.1670476224202 + - type: nauc_mrr_at_100_diff1 + value: 77.57602878422465 + - type: nauc_mrr_at_100_max + value: 47.34876516670401 + - type: nauc_mrr_at_100_std + value: -26.16596878042115 + - type: nauc_mrr_at_10_diff1 + value: 77.48926507168112 + - type: nauc_mrr_at_10_max + value: 47.46252680868696 + - type: nauc_mrr_at_10_std + value: -26.282201461393317 + - type: nauc_mrr_at_1_diff1 + value: 79.53993768139632 + - type: nauc_mrr_at_1_max + value: 44.95304522643992 + - type: nauc_mrr_at_1_std + value: -25.32633885677888 + - type: nauc_mrr_at_20_diff1 + value: 77.54890591729169 + - type: nauc_mrr_at_20_max + value: 47.37631632390287 + - type: nauc_mrr_at_20_std + value: -26.183017296393896 + - type: nauc_mrr_at_3_diff1 + value: 77.4534326297116 + - type: nauc_mrr_at_3_max + value: 47.54929766473431 + - type: nauc_mrr_at_3_std + value: -26.432202942463785 + - type: nauc_mrr_at_5_diff1 + value: 77.38739995097214 + - type: nauc_mrr_at_5_max + value: 47.50110390682936 + - type: nauc_mrr_at_5_std + value: -26.463416200561753 + - type: nauc_ndcg_at_1000_diff1 + value: 77.20842500889081 + - type: nauc_ndcg_at_1000_max + value: 47.70813895735421 + - type: nauc_ndcg_at_1000_std + value: -26.023069702012446 + - type: nauc_ndcg_at_100_diff1 + value: 77.16015320575458 + - type: nauc_ndcg_at_100_max + value: 47.88272658215937 + - type: nauc_ndcg_at_100_std + value: -25.893271327002005 + - type: nauc_ndcg_at_10_diff1 + value: 76.68338764268012 + - type: nauc_ndcg_at_10_max + value: 48.4217483255242 + - type: nauc_ndcg_at_10_std + value: -26.36940839993425 + - type: nauc_ndcg_at_1_diff1 + value: 79.527051162636 + - type: nauc_ndcg_at_1_max + value: 44.89631724552463 + - type: nauc_ndcg_at_1_std + value: -25.451766234360097 + - type: nauc_ndcg_at_20_diff1 + value: 76.9371111655833 + - type: nauc_ndcg_at_20_max + value: 48.10138804970404 + - type: nauc_ndcg_at_20_std + value: -25.97419280800229 + - type: nauc_ndcg_at_3_diff1 + value: 76.61248461087024 + - type: nauc_ndcg_at_3_max + value: 48.51663194946113 + - type: nauc_ndcg_at_3_std + value: -26.789177280257793 + - type: nauc_ndcg_at_5_diff1 + value: 76.42970850899898 + - type: nauc_ndcg_at_5_max + value: 48.52719429311673 + - type: nauc_ndcg_at_5_std + value: -26.78716430555766 + - type: nauc_precision_at_1000_diff1 + value: 70.03887145024319 + - type: nauc_precision_at_1000_max + value: 42.00416767646647 + - type: nauc_precision_at_1000_std + value: -2.156778197450252 + - type: nauc_precision_at_100_diff1 + value: 72.9743402702392 + - type: nauc_precision_at_100_max + value: 56.4734539374456 + - type: nauc_precision_at_100_std + value: -12.25156037378497 + - type: nauc_precision_at_10_diff1 + value: 71.50550227914846 + - type: nauc_precision_at_10_max + value: 55.096376602923144 + - type: nauc_precision_at_10_std + value: -25.929290375863356 + - type: nauc_precision_at_1_diff1 + value: 79.527051162636 + - type: nauc_precision_at_1_max + value: 44.89631724552463 + - type: nauc_precision_at_1_std + value: -25.451766234360097 + - type: nauc_precision_at_20_diff1 + value: 71.99203963125369 + - type: nauc_precision_at_20_max + value: 54.66712448432459 + - type: nauc_precision_at_20_std + value: -21.478852653786358 + - type: nauc_precision_at_3_diff1 + value: 73.33785677931962 + - type: nauc_precision_at_3_max + value: 52.70785427475565 + - type: nauc_precision_at_3_std + value: -27.81147380600097 + - type: nauc_precision_at_5_diff1 + value: 71.76130552913553 + - type: nauc_precision_at_5_max + value: 53.78865964066962 + - type: nauc_precision_at_5_std + value: -28.044309543143616 + - type: nauc_recall_at_1000_diff1 + value: 70.03887145024234 + - type: nauc_recall_at_1000_max + value: 42.00416767647104 + - type: nauc_recall_at_1000_std + value: -2.15677819744854 + - type: nauc_recall_at_100_diff1 + value: 72.974340270239 + - type: nauc_recall_at_100_max + value: 56.473453937445306 + - type: nauc_recall_at_100_std + value: -12.25156037378462 + - type: nauc_recall_at_10_diff1 + value: 71.50550227914843 + - type: nauc_recall_at_10_max + value: 55.096376602923 + - type: nauc_recall_at_10_std + value: -25.929290375863673 + - type: nauc_recall_at_1_diff1 + value: 79.527051162636 + - type: nauc_recall_at_1_max + value: 44.89631724552463 + - type: nauc_recall_at_1_std + value: -25.451766234360097 + - type: nauc_recall_at_20_diff1 + value: 71.992039631253 + - type: nauc_recall_at_20_max + value: 54.66712448432437 + - type: nauc_recall_at_20_std + value: -21.478852653786607 + - type: nauc_recall_at_3_diff1 + value: 73.33785677931951 + - type: nauc_recall_at_3_max + value: 52.707854274755626 + - type: nauc_recall_at_3_std + value: -27.811473806001057 + - type: nauc_recall_at_5_diff1 + value: 71.76130552913536 + - type: nauc_recall_at_5_max + value: 53.788659640669344 + - type: nauc_recall_at_5_std + value: -28.044309543143715 + - type: ndcg_at_1 + value: 73.25500000000001 + - type: ndcg_at_10 + value: 83.38799999999999 + - type: ndcg_at_100 + value: 84.571 + - type: ndcg_at_1000 + value: 84.768 + - type: ndcg_at_20 + value: 83.99799999999999 + - type: ndcg_at_3 + value: 81.118 + - type: ndcg_at_5 + value: 82.40899999999999 + - type: precision_at_1 + value: 73.25500000000001 + - type: precision_at_10 + value: 9.259 + - type: precision_at_100 + value: 0.98 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.749 + - type: precision_at_3 + value: 28.826 + - type: precision_at_5 + value: 17.918 + - type: recall_at_1 + value: 73.25500000000001 + - type: recall_at_10 + value: 92.586 + - type: recall_at_100 + value: 98.016 + - type: recall_at_1000 + value: 99.572 + - type: recall_at_20 + value: 94.976 + - type: recall_at_3 + value: 86.478 + - type: recall_at_5 + value: 89.589 + task: + type: Retrieval + - dataset: + config: python + name: MTEB CodeSearchNetRetrieval (python) + revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 + split: test + type: code-search-net/code_search_net + metrics: + - type: main_score + value: 96.688 + - type: map_at_1 + value: 92.7 + - type: map_at_10 + value: 95.643 + - type: map_at_100 + value: 95.649 + - type: map_at_1000 + value: 95.649 + - type: map_at_20 + value: 95.643 + - type: map_at_3 + value: 95.417 + - type: map_at_5 + value: 95.562 + - type: mrr_at_1 + value: 92.7 + - type: mrr_at_10 + value: 95.64289682539683 + - type: mrr_at_100 + value: 95.64912393162395 + - type: mrr_at_1000 + value: 95.64912393162395 + - type: mrr_at_20 + value: 95.64289682539683 + - type: mrr_at_3 + value: 95.41666666666669 + - type: mrr_at_5 + value: 95.56166666666668 + - type: nauc_map_at_1000_diff1 + value: 93.2086804630001 + - type: nauc_map_at_1000_max + value: 71.36336949003405 + - type: nauc_map_at_1000_std + value: 7.542855305507389 + - type: nauc_map_at_100_diff1 + value: 93.2086804630001 + - type: nauc_map_at_100_max + value: 71.36336949003405 + - type: nauc_map_at_100_std + value: 7.542855305507389 + - type: nauc_map_at_10_diff1 + value: 93.21838651458631 + - type: nauc_map_at_10_max + value: 71.40429653101195 + - type: nauc_map_at_10_std + value: 7.674993664044225 + - type: nauc_map_at_1_diff1 + value: 92.97225739610897 + - type: nauc_map_at_1_max + value: 69.06680480411342 + - type: nauc_map_at_1_std + value: 8.97062021155494 + - type: nauc_map_at_20_diff1 + value: 93.21838651458631 + - type: nauc_map_at_20_max + value: 71.40429653101195 + - type: nauc_map_at_20_std + value: 7.674993664044225 + - type: nauc_map_at_3_diff1 + value: 93.22689075630251 + - type: nauc_map_at_3_max + value: 71.83278159748735 + - type: nauc_map_at_3_std + value: 5.838383838384187 + - type: nauc_map_at_5_diff1 + value: 93.07924446534143 + - type: nauc_map_at_5_max + value: 71.48845769375428 + - type: nauc_map_at_5_std + value: 6.123107648366922 + - type: nauc_mrr_at_1000_diff1 + value: 93.2086804630001 + - type: nauc_mrr_at_1000_max + value: 71.36336949003405 + - type: nauc_mrr_at_1000_std + value: 7.542855305507389 + - type: nauc_mrr_at_100_diff1 + value: 93.2086804630001 + - type: nauc_mrr_at_100_max + value: 71.36336949003405 + - type: nauc_mrr_at_100_std + value: 7.542855305507389 + - type: nauc_mrr_at_10_diff1 + value: 93.21838651458631 + - type: nauc_mrr_at_10_max + value: 71.40429653101195 + - type: nauc_mrr_at_10_std + value: 7.674993664044225 + - type: nauc_mrr_at_1_diff1 + value: 92.97225739610897 + - type: nauc_mrr_at_1_max + value: 69.06680480411342 + - type: nauc_mrr_at_1_std + value: 8.97062021155494 + - type: nauc_mrr_at_20_diff1 + value: 93.21838651458631 + - type: nauc_mrr_at_20_max + value: 71.40429653101195 + - type: nauc_mrr_at_20_std + value: 7.674993664044225 + - type: nauc_mrr_at_3_diff1 + value: 93.22689075630251 + - type: nauc_mrr_at_3_max + value: 71.83278159748735 + - type: nauc_mrr_at_3_std + value: 5.838383838384187 + - type: nauc_mrr_at_5_diff1 + value: 93.07924446534143 + - type: nauc_mrr_at_5_max + value: 71.48845769375428 + - type: nauc_mrr_at_5_std + value: 6.123107648366922 + - type: nauc_ndcg_at_1000_diff1 + value: 93.23304542841751 + - type: nauc_ndcg_at_1000_max + value: 71.63083509229389 + - type: nauc_ndcg_at_1000_std + value: 7.571755253398231 + - type: nauc_ndcg_at_100_diff1 + value: 93.23304542841751 + - type: nauc_ndcg_at_100_max + value: 71.63083509229389 + - type: nauc_ndcg_at_100_std + value: 7.571755253398231 + - type: nauc_ndcg_at_10_diff1 + value: 93.31367660323025 + - type: nauc_ndcg_at_10_max + value: 71.96886589636927 + - type: nauc_ndcg_at_10_std + value: 8.673077551486251 + - type: nauc_ndcg_at_1_diff1 + value: 92.97225739610897 + - type: nauc_ndcg_at_1_max + value: 69.06680480411342 + - type: nauc_ndcg_at_1_std + value: 8.97062021155494 + - type: nauc_ndcg_at_20_diff1 + value: 93.31367660323025 + - type: nauc_ndcg_at_20_max + value: 71.96886589636927 + - type: nauc_ndcg_at_20_std + value: 8.673077551486251 + - type: nauc_ndcg_at_3_diff1 + value: 93.23894522522178 + - type: nauc_ndcg_at_3_max + value: 72.902496295325 + - type: nauc_ndcg_at_3_std + value: 3.6178051090847845 + - type: nauc_ndcg_at_5_diff1 + value: 92.90972948390606 + - type: nauc_ndcg_at_5_max + value: 72.19526067100512 + - type: nauc_ndcg_at_5_std + value: 4.041582233975721 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 100.0 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 100.0 + - type: nauc_precision_at_10_diff1 + value: 100.0 + - type: nauc_precision_at_10_max + value: 100.0 + - type: nauc_precision_at_10_std + value: 100.0 + - type: nauc_precision_at_1_diff1 + value: 92.97225739610897 + - type: nauc_precision_at_1_max + value: 69.06680480411342 + - type: nauc_precision_at_1_std + value: 8.97062021155494 + - type: nauc_precision_at_20_diff1 + value: 100.0 + - type: nauc_precision_at_20_max + value: 100.0 + - type: nauc_precision_at_20_std + value: 100.0 + - type: nauc_precision_at_3_diff1 + value: 93.2306255835672 + - type: nauc_precision_at_3_max + value: 82.46298519407813 + - type: nauc_precision_at_3_std + value: -17.583700146723764 + - type: nauc_precision_at_5_diff1 + value: 89.78758169934665 + - type: nauc_precision_at_5_max + value: 82.09033613445358 + - type: nauc_precision_at_5_std + value: -29.580999066296027 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: .nan + - type: nauc_recall_at_100_max + value: .nan + - type: nauc_recall_at_100_std + value: .nan + - type: nauc_recall_at_10_diff1 + value: 100.0 + - type: nauc_recall_at_10_max + value: 100.0 + - type: nauc_recall_at_10_std + value: 100.0 + - type: nauc_recall_at_1_diff1 + value: 92.97225739610897 + - type: nauc_recall_at_1_max + value: 69.06680480411342 + - type: nauc_recall_at_1_std + value: 8.97062021155494 + - type: nauc_recall_at_20_diff1 + value: 100.0 + - type: nauc_recall_at_20_max + value: 100.0 + - type: nauc_recall_at_20_std + value: 100.0 + - type: nauc_recall_at_3_diff1 + value: 93.2306255835672 + - type: nauc_recall_at_3_max + value: 82.4629851940786 + - type: nauc_recall_at_3_std + value: -17.5837001467247 + - type: nauc_recall_at_5_diff1 + value: 89.78758169934649 + - type: nauc_recall_at_5_max + value: 82.09033613445361 + - type: nauc_recall_at_5_std + value: -29.580999066293263 + - type: ndcg_at_1 + value: 92.7 + - type: ndcg_at_10 + value: 96.688 + - type: ndcg_at_100 + value: 96.72800000000001 + - type: ndcg_at_1000 + value: 96.72800000000001 + - type: ndcg_at_20 + value: 96.688 + - type: ndcg_at_3 + value: 96.23899999999999 + - type: ndcg_at_5 + value: 96.493 + - type: precision_at_1 + value: 92.7 + - type: precision_at_10 + value: 9.98 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.99 + - type: precision_at_3 + value: 32.867000000000004 + - type: precision_at_5 + value: 19.84 + - type: recall_at_1 + value: 92.7 + - type: recall_at_10 + value: 99.8 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.8 + - type: recall_at_3 + value: 98.6 + - type: recall_at_5 + value: 99.2 + task: + type: Retrieval + - dataset: + config: javascript + name: MTEB CodeSearchNetRetrieval (javascript) + revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 + split: test + type: code-search-net/code_search_net + metrics: + - type: main_score + value: 89.97800000000001 + - type: map_at_1 + value: 85.0 + - type: map_at_10 + value: 88.427 + - type: map_at_100 + value: 88.627 + - type: map_at_1000 + value: 88.631 + - type: map_at_20 + value: 88.588 + - type: map_at_3 + value: 87.767 + - type: map_at_5 + value: 88.19200000000001 + - type: mrr_at_1 + value: 85.0 + - type: mrr_at_10 + value: 88.42742063492065 + - type: mrr_at_100 + value: 88.62702226900807 + - type: mrr_at_1000 + value: 88.63072320739236 + - type: mrr_at_20 + value: 88.58809151632684 + - type: mrr_at_3 + value: 87.76666666666667 + - type: mrr_at_5 + value: 88.1916666666667 + - type: nauc_map_at_1000_diff1 + value: 90.72581837961151 + - type: nauc_map_at_1000_max + value: 62.96996529181109 + - type: nauc_map_at_1000_std + value: 5.4686257905510445 + - type: nauc_map_at_100_diff1 + value: 90.7251318399043 + - type: nauc_map_at_100_max + value: 62.98186646957673 + - type: nauc_map_at_100_std + value: 5.499359385640601 + - type: nauc_map_at_10_diff1 + value: 90.7954839398183 + - type: nauc_map_at_10_max + value: 63.22150406130067 + - type: nauc_map_at_10_std + value: 5.413600198301859 + - type: nauc_map_at_1_diff1 + value: 91.52066384640422 + - type: nauc_map_at_1_max + value: 57.639765701269155 + - type: nauc_map_at_1_std + value: 1.4959323136999483 + - type: nauc_map_at_20_diff1 + value: 90.71434972102585 + - type: nauc_map_at_20_max + value: 63.07611119431469 + - type: nauc_map_at_20_std + value: 5.640274998494253 + - type: nauc_map_at_3_diff1 + value: 90.33141255622978 + - type: nauc_map_at_3_max + value: 62.34164710561914 + - type: nauc_map_at_3_std + value: 3.5068574882828925 + - type: nauc_map_at_5_diff1 + value: 90.79552786535547 + - type: nauc_map_at_5_max + value: 63.119944203478106 + - type: nauc_map_at_5_std + value: 4.953420352568366 + - type: nauc_mrr_at_1000_diff1 + value: 90.72581837961151 + - type: nauc_mrr_at_1000_max + value: 62.96996529181109 + - type: nauc_mrr_at_1000_std + value: 5.4686257905510445 + - type: nauc_mrr_at_100_diff1 + value: 90.7251318399043 + - type: nauc_mrr_at_100_max + value: 62.98186646957673 + - type: nauc_mrr_at_100_std + value: 5.499359385640601 + - type: nauc_mrr_at_10_diff1 + value: 90.7954839398183 + - type: nauc_mrr_at_10_max + value: 63.22150406130067 + - type: nauc_mrr_at_10_std + value: 5.413600198301859 + - type: nauc_mrr_at_1_diff1 + value: 91.52066384640422 + - type: nauc_mrr_at_1_max + value: 57.639765701269155 + - type: nauc_mrr_at_1_std + value: 1.4959323136999483 + - type: nauc_mrr_at_20_diff1 + value: 90.71434972102585 + - type: nauc_mrr_at_20_max + value: 63.07611119431469 + - type: nauc_mrr_at_20_std + value: 5.640274998494253 + - type: nauc_mrr_at_3_diff1 + value: 90.33141255622978 + - type: nauc_mrr_at_3_max + value: 62.34164710561914 + - type: nauc_mrr_at_3_std + value: 3.5068574882828925 + - type: nauc_mrr_at_5_diff1 + value: 90.79552786535547 + - type: nauc_mrr_at_5_max + value: 63.119944203478106 + - type: nauc_mrr_at_5_std + value: 4.953420352568366 + - type: nauc_ndcg_at_1000_diff1 + value: 90.6732163208453 + - type: nauc_ndcg_at_1000_max + value: 64.27542517763368 + - type: nauc_ndcg_at_1000_std + value: 7.423716824052952 + - type: nauc_ndcg_at_100_diff1 + value: 90.65475476530615 + - type: nauc_ndcg_at_100_max + value: 64.89222263529967 + - type: nauc_ndcg_at_100_std + value: 9.036733154889918 + - type: nauc_ndcg_at_10_diff1 + value: 90.88391240350151 + - type: nauc_ndcg_at_10_max + value: 66.10798661293683 + - type: nauc_ndcg_at_10_std + value: 8.662637793277094 + - type: nauc_ndcg_at_1_diff1 + value: 91.52066384640422 + - type: nauc_ndcg_at_1_max + value: 57.639765701269155 + - type: nauc_ndcg_at_1_std + value: 1.4959323136999483 + - type: nauc_ndcg_at_20_diff1 + value: 90.55301735430407 + - type: nauc_ndcg_at_20_max + value: 65.65827895142017 + - type: nauc_ndcg_at_20_std + value: 9.990699968426656 + - type: nauc_ndcg_at_3_diff1 + value: 89.96356996211253 + - type: nauc_ndcg_at_3_max + value: 64.03584890384899 + - type: nauc_ndcg_at_3_std + value: 4.295929350921648 + - type: nauc_ndcg_at_5_diff1 + value: 90.89636835182858 + - type: nauc_ndcg_at_5_max + value: 65.69109133325729 + - type: nauc_ndcg_at_5_std + value: 7.251888023539761 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 89.89595838335227 + - type: nauc_precision_at_100_max + value: 98.13258636787991 + - type: nauc_precision_at_100_std + value: 95.88168600773692 + - type: nauc_precision_at_10_diff1 + value: 91.80313150901407 + - type: nauc_precision_at_10_max + value: 84.73120017237689 + - type: nauc_precision_at_10_std + value: 30.963154492565963 + - type: nauc_precision_at_1_diff1 + value: 91.52066384640422 + - type: nauc_precision_at_1_max + value: 57.639765701269155 + - type: nauc_precision_at_1_std + value: 1.4959323136999483 + - type: nauc_precision_at_20_diff1 + value: 88.51379632312695 + - type: nauc_precision_at_20_max + value: 93.89870890885065 + - type: nauc_precision_at_20_std + value: 66.06458675424169 + - type: nauc_precision_at_3_diff1 + value: 88.5919911035577 + - type: nauc_precision_at_3_max + value: 70.47755431760744 + - type: nauc_precision_at_3_std + value: 7.324877516549845 + - type: nauc_precision_at_5_diff1 + value: 91.68000533546777 + - type: nauc_precision_at_5_max + value: 78.08323329331722 + - type: nauc_precision_at_5_std + value: 18.784847272242132 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 89.8959583833538 + - type: nauc_recall_at_100_max + value: 98.13258636788038 + - type: nauc_recall_at_100_std + value: 95.88168600773625 + - type: nauc_recall_at_10_diff1 + value: 91.80313150901372 + - type: nauc_recall_at_10_max + value: 84.73120017237645 + - type: nauc_recall_at_10_std + value: 30.963154492566204 + - type: nauc_recall_at_1_diff1 + value: 91.52066384640422 + - type: nauc_recall_at_1_max + value: 57.639765701269155 + - type: nauc_recall_at_1_std + value: 1.4959323136999483 + - type: nauc_recall_at_20_diff1 + value: 88.51379632312701 + - type: nauc_recall_at_20_max + value: 93.89870890885045 + - type: nauc_recall_at_20_std + value: 66.0645867542416 + - type: nauc_recall_at_3_diff1 + value: 88.59199110355753 + - type: nauc_recall_at_3_max + value: 70.47755431760724 + - type: nauc_recall_at_3_std + value: 7.324877516549637 + - type: nauc_recall_at_5_diff1 + value: 91.68000533546758 + - type: nauc_recall_at_5_max + value: 78.08323329331743 + - type: nauc_recall_at_5_std + value: 18.784847272242207 + - type: ndcg_at_1 + value: 85.0 + - type: ndcg_at_10 + value: 89.97800000000001 + - type: ndcg_at_100 + value: 90.842 + - type: ndcg_at_1000 + value: 91.00399999999999 + - type: ndcg_at_20 + value: 90.561 + - type: ndcg_at_3 + value: 88.626 + - type: ndcg_at_5 + value: 89.401 + - type: precision_at_1 + value: 85.0 + - type: precision_at_10 + value: 9.48 + - type: precision_at_100 + value: 0.9860000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.855 + - type: precision_at_3 + value: 30.367 + - type: precision_at_5 + value: 18.6 + - type: recall_at_1 + value: 85.0 + - type: recall_at_10 + value: 94.8 + - type: recall_at_100 + value: 98.6 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 97.1 + - type: recall_at_3 + value: 91.10000000000001 + - type: recall_at_5 + value: 93.0 + task: + type: Retrieval + - dataset: + config: go + name: MTEB CodeSearchNetRetrieval (go) + revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 + split: test + type: code-search-net/code_search_net + metrics: + - type: main_score + value: 97.505 + - type: map_at_1 + value: 94.39999999999999 + - type: map_at_10 + value: 96.808 + - type: map_at_100 + value: 96.82300000000001 + - type: map_at_1000 + value: 96.824 + - type: map_at_20 + value: 96.813 + - type: map_at_3 + value: 96.733 + - type: map_at_5 + value: 96.808 + - type: mrr_at_1 + value: 94.39999999999999 + - type: mrr_at_10 + value: 96.80833333333334 + - type: mrr_at_100 + value: 96.82307275541795 + - type: mrr_at_1000 + value: 96.82390608875127 + - type: mrr_at_20 + value: 96.81333333333333 + - type: mrr_at_3 + value: 96.73333333333333 + - type: mrr_at_5 + value: 96.80833333333334 + - type: nauc_map_at_1000_diff1 + value: 97.27333921447281 + - type: nauc_map_at_1000_max + value: 40.53177582941515 + - type: nauc_map_at_1000_std + value: -9.68497853001627 + - type: nauc_map_at_100_diff1 + value: 97.27405443932574 + - type: nauc_map_at_100_max + value: 40.54737481852009 + - type: nauc_map_at_100_std + value: -9.663493618536894 + - type: nauc_map_at_10_diff1 + value: 97.28664311677696 + - type: nauc_map_at_10_max + value: 40.81115474910615 + - type: nauc_map_at_10_std + value: -10.076963770712538 + - type: nauc_map_at_1_diff1 + value: 96.90709617180208 + - type: nauc_map_at_1_max + value: 33.33833533413366 + - type: nauc_map_at_1_std + value: -6.367547018807825 + - type: nauc_map_at_20_diff1 + value: 97.28238575765059 + - type: nauc_map_at_20_max + value: 40.71828522203857 + - type: nauc_map_at_20_std + value: -9.912064546879208 + - type: nauc_map_at_3_diff1 + value: 97.34893957583056 + - type: nauc_map_at_3_max + value: 41.67881438289635 + - type: nauc_map_at_3_std + value: -9.509041711922595 + - type: nauc_map_at_5_diff1 + value: 97.28664311677696 + - type: nauc_map_at_5_max + value: 40.81115474910615 + - type: nauc_map_at_5_std + value: -10.076963770712538 + - type: nauc_mrr_at_1000_diff1 + value: 97.27333921447281 + - type: nauc_mrr_at_1000_max + value: 40.53177582941515 + - type: nauc_mrr_at_1000_std + value: -9.68497853001627 + - type: nauc_mrr_at_100_diff1 + value: 97.27405443932574 + - type: nauc_mrr_at_100_max + value: 40.54737481852009 + - type: nauc_mrr_at_100_std + value: -9.663493618536894 + - type: nauc_mrr_at_10_diff1 + value: 97.28664311677696 + - type: nauc_mrr_at_10_max + value: 40.81115474910615 + - type: nauc_mrr_at_10_std + value: -10.076963770712538 + - type: nauc_mrr_at_1_diff1 + value: 96.90709617180208 + - type: nauc_mrr_at_1_max + value: 33.33833533413366 + - type: nauc_mrr_at_1_std + value: -6.367547018807825 + - type: nauc_mrr_at_20_diff1 + value: 97.28238575765059 + - type: nauc_mrr_at_20_max + value: 40.71828522203857 + - type: nauc_mrr_at_20_std + value: -9.912064546879208 + - type: nauc_mrr_at_3_diff1 + value: 97.34893957583056 + - type: nauc_mrr_at_3_max + value: 41.67881438289635 + - type: nauc_mrr_at_3_std + value: -9.509041711922595 + - type: nauc_mrr_at_5_diff1 + value: 97.28664311677696 + - type: nauc_mrr_at_5_max + value: 40.81115474910615 + - type: nauc_mrr_at_5_std + value: -10.076963770712538 + - type: nauc_ndcg_at_1000_diff1 + value: 97.33376561125955 + - type: nauc_ndcg_at_1000_max + value: 41.75737985174807 + - type: nauc_ndcg_at_1000_std + value: -10.511570027083764 + - type: nauc_ndcg_at_100_diff1 + value: 97.34974235237087 + - type: nauc_ndcg_at_100_max + value: 42.106384150737576 + - type: nauc_ndcg_at_100_std + value: -10.015808519562162 + - type: nauc_ndcg_at_10_diff1 + value: 97.43761675418024 + - type: nauc_ndcg_at_10_max + value: 43.92682037582593 + - type: nauc_ndcg_at_10_std + value: -13.11593389665566 + - type: nauc_ndcg_at_1_diff1 + value: 96.90709617180208 + - type: nauc_ndcg_at_1_max + value: 33.33833533413366 + - type: nauc_ndcg_at_1_std + value: -6.367547018807825 + - type: nauc_ndcg_at_20_diff1 + value: 97.41401672160926 + - type: nauc_ndcg_at_20_max + value: 43.410375824598866 + - type: nauc_ndcg_at_20_std + value: -12.175971519065602 + - type: nauc_ndcg_at_3_diff1 + value: 97.56379014982639 + - type: nauc_ndcg_at_3_max + value: 45.63427566353211 + - type: nauc_ndcg_at_3_std + value: -11.748268623706998 + - type: nauc_ndcg_at_5_diff1 + value: 97.43761675418024 + - type: nauc_ndcg_at_5_max + value: 43.92682037582593 + - type: nauc_ndcg_at_5_std + value: -13.11593389665566 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 100.0 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 72.22222222222335 + - type: nauc_precision_at_10_diff1 + value: 100.0 + - type: nauc_precision_at_10_max + value: 97.38562091503177 + - type: nauc_precision_at_10_std + value: -69.85060690943635 + - type: nauc_precision_at_1_diff1 + value: 96.90709617180208 + - type: nauc_precision_at_1_max + value: 33.33833533413366 + - type: nauc_precision_at_1_std + value: -6.367547018807825 + - type: nauc_precision_at_20_diff1 + value: 100.0 + - type: nauc_precision_at_20_max + value: 96.73202614378965 + - type: nauc_precision_at_20_std + value: -58.52007469655789 + - type: nauc_precision_at_3_diff1 + value: 100.0 + - type: nauc_precision_at_3_max + value: 90.34197012138252 + - type: nauc_precision_at_3_std + value: -38.159430438841106 + - type: nauc_precision_at_5_diff1 + value: 100.0 + - type: nauc_precision_at_5_max + value: 97.38562091503177 + - type: nauc_precision_at_5_std + value: -69.85060690943635 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 100.0 + - type: nauc_recall_at_100_max + value: 100.0 + - type: nauc_recall_at_100_std + value: 72.22222222222368 + - type: nauc_recall_at_10_diff1 + value: 100.0 + - type: nauc_recall_at_10_max + value: 97.38562091503128 + - type: nauc_recall_at_10_std + value: -69.85060690942723 + - type: nauc_recall_at_1_diff1 + value: 96.90709617180208 + - type: nauc_recall_at_1_max + value: 33.33833533413366 + - type: nauc_recall_at_1_std + value: -6.367547018807825 + - type: nauc_recall_at_20_diff1 + value: 100.0 + - type: nauc_recall_at_20_max + value: 96.7320261437891 + - type: nauc_recall_at_20_std + value: -58.52007469654495 + - type: nauc_recall_at_3_diff1 + value: 100.0 + - type: nauc_recall_at_3_max + value: 90.34197012138186 + - type: nauc_recall_at_3_std + value: -38.15943043884293 + - type: nauc_recall_at_5_diff1 + value: 100.0 + - type: nauc_recall_at_5_max + value: 97.38562091503128 + - type: nauc_recall_at_5_std + value: -69.85060690942723 + - type: ndcg_at_1 + value: 94.39999999999999 + - type: ndcg_at_10 + value: 97.505 + - type: ndcg_at_100 + value: 97.588 + - type: ndcg_at_1000 + value: 97.602 + - type: ndcg_at_20 + value: 97.528 + - type: ndcg_at_3 + value: 97.37599999999999 + - type: ndcg_at_5 + value: 97.505 + - type: precision_at_1 + value: 94.39999999999999 + - type: precision_at_10 + value: 9.950000000000001 + - type: precision_at_100 + value: 0.9990000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.9799999999999995 + - type: precision_at_3 + value: 33.067 + - type: precision_at_5 + value: 19.900000000000002 + - type: recall_at_1 + value: 94.39999999999999 + - type: recall_at_10 + value: 99.5 + - type: recall_at_100 + value: 99.9 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.6 + - type: recall_at_3 + value: 99.2 + - type: recall_at_5 + value: 99.5 + task: + type: Retrieval + - dataset: + config: ruby + name: MTEB CodeSearchNetRetrieval (ruby) + revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 + split: test + type: code-search-net/code_search_net + metrics: + - type: main_score + value: 92.83200000000001 + - type: map_at_1 + value: 87.7 + - type: map_at_10 + value: 91.496 + - type: map_at_100 + value: 91.582 + - type: map_at_1000 + value: 91.58699999999999 + - type: map_at_20 + value: 91.55 + - type: map_at_3 + value: 91.167 + - type: map_at_5 + value: 91.39699999999999 + - type: mrr_at_1 + value: 87.7 + - type: mrr_at_10 + value: 91.49646825396826 + - type: mrr_at_100 + value: 91.58154350757961 + - type: mrr_at_1000 + value: 91.58711452954685 + - type: mrr_at_20 + value: 91.54990637415563 + - type: mrr_at_3 + value: 91.16666666666669 + - type: mrr_at_5 + value: 91.39666666666668 + - type: nauc_map_at_1000_diff1 + value: 90.01079432013319 + - type: nauc_map_at_1000_max + value: 76.25110199586136 + - type: nauc_map_at_1000_std + value: 21.95406993804367 + - type: nauc_map_at_100_diff1 + value: 90.01415529036015 + - type: nauc_map_at_100_max + value: 76.26710387746687 + - type: nauc_map_at_100_std + value: 21.991489847497782 + - type: nauc_map_at_10_diff1 + value: 89.88101575524061 + - type: nauc_map_at_10_max + value: 76.40985435650425 + - type: nauc_map_at_10_std + value: 22.554999001780367 + - type: nauc_map_at_1_diff1 + value: 90.45505123995243 + - type: nauc_map_at_1_max + value: 70.27343537229697 + - type: nauc_map_at_1_std + value: 10.378716550076664 + - type: nauc_map_at_20_diff1 + value: 90.03751820546869 + - type: nauc_map_at_20_max + value: 76.35756916603282 + - type: nauc_map_at_20_std + value: 22.15447723869247 + - type: nauc_map_at_3_diff1 + value: 89.90841801530556 + - type: nauc_map_at_3_max + value: 76.90028137713493 + - type: nauc_map_at_3_std + value: 21.843737118482988 + - type: nauc_map_at_5_diff1 + value: 89.93455699311205 + - type: nauc_map_at_5_max + value: 76.52511956217161 + - type: nauc_map_at_5_std + value: 22.235121325826523 + - type: nauc_mrr_at_1000_diff1 + value: 90.01079432013319 + - type: nauc_mrr_at_1000_max + value: 76.25110199586136 + - type: nauc_mrr_at_1000_std + value: 21.95406993804367 + - type: nauc_mrr_at_100_diff1 + value: 90.01415529036015 + - type: nauc_mrr_at_100_max + value: 76.26710387746687 + - type: nauc_mrr_at_100_std + value: 21.991489847497782 + - type: nauc_mrr_at_10_diff1 + value: 89.88101575524061 + - type: nauc_mrr_at_10_max + value: 76.40985435650425 + - type: nauc_mrr_at_10_std + value: 22.554999001780367 + - type: nauc_mrr_at_1_diff1 + value: 90.45505123995243 + - type: nauc_mrr_at_1_max + value: 70.27343537229697 + - type: nauc_mrr_at_1_std + value: 10.378716550076664 + - type: nauc_mrr_at_20_diff1 + value: 90.03751820546869 + - type: nauc_mrr_at_20_max + value: 76.35756916603282 + - type: nauc_mrr_at_20_std + value: 22.15447723869247 + - type: nauc_mrr_at_3_diff1 + value: 89.90841801530556 + - type: nauc_mrr_at_3_max + value: 76.90028137713493 + - type: nauc_mrr_at_3_std + value: 21.843737118482988 + - type: nauc_mrr_at_5_diff1 + value: 89.93455699311205 + - type: nauc_mrr_at_5_max + value: 76.52511956217161 + - type: nauc_mrr_at_5_std + value: 22.235121325826523 + - type: nauc_ndcg_at_1000_diff1 + value: 90.0388376789719 + - type: nauc_ndcg_at_1000_max + value: 77.4286773752732 + - type: nauc_ndcg_at_1000_std + value: 24.905596869880494 + - type: nauc_ndcg_at_100_diff1 + value: 90.1570526184995 + - type: nauc_ndcg_at_100_max + value: 77.97459201928656 + - type: nauc_ndcg_at_100_std + value: 26.36432448776628 + - type: nauc_ndcg_at_10_diff1 + value: 89.60631662977974 + - type: nauc_ndcg_at_10_max + value: 78.82415924492551 + - type: nauc_ndcg_at_10_std + value: 29.259897960486477 + - type: nauc_ndcg_at_1_diff1 + value: 90.45505123995243 + - type: nauc_ndcg_at_1_max + value: 70.27343537229697 + - type: nauc_ndcg_at_1_std + value: 10.378716550076664 + - type: nauc_ndcg_at_20_diff1 + value: 90.34712330725388 + - type: nauc_ndcg_at_20_max + value: 78.65263241067265 + - type: nauc_ndcg_at_20_std + value: 27.634547611895844 + - type: nauc_ndcg_at_3_diff1 + value: 89.69525439370882 + - type: nauc_ndcg_at_3_max + value: 79.68227200895333 + - type: nauc_ndcg_at_3_std + value: 27.16978363425553 + - type: nauc_ndcg_at_5_diff1 + value: 89.74542043203509 + - type: nauc_ndcg_at_5_max + value: 79.0789328591088 + - type: nauc_ndcg_at_5_std + value: 28.35584817342069 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 94.84665661136168 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 86.96760755584259 + - type: nauc_precision_at_10_diff1 + value: 87.527719421102 + - type: nauc_precision_at_10_max + value: 96.1922268907559 + - type: nauc_precision_at_10_std + value: 80.32942343604061 + - type: nauc_precision_at_1_diff1 + value: 90.45505123995243 + - type: nauc_precision_at_1_max + value: 70.27343537229697 + - type: nauc_precision_at_1_std + value: 10.378716550076664 + - type: nauc_precision_at_20_diff1 + value: 95.5746187363837 + - type: nauc_precision_at_20_max + value: 100.0 + - type: nauc_precision_at_20_std + value: 78.69397759103582 + - type: nauc_precision_at_3_diff1 + value: 88.63450141961569 + - type: nauc_precision_at_3_max + value: 93.64984088873672 + - type: nauc_precision_at_3_std + value: 54.208350006669704 + - type: nauc_precision_at_5_diff1 + value: 88.62314156431772 + - type: nauc_precision_at_5_max + value: 94.5892887069356 + - type: nauc_precision_at_5_std + value: 66.80552562905466 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 94.8466566113629 + - type: nauc_recall_at_100_max + value: 100.0 + - type: nauc_recall_at_100_std + value: 86.96760755584259 + - type: nauc_recall_at_10_diff1 + value: 87.52771942110216 + - type: nauc_recall_at_10_max + value: 96.19222689075643 + - type: nauc_recall_at_10_std + value: 80.32942343604144 + - type: nauc_recall_at_1_diff1 + value: 90.45505123995243 + - type: nauc_recall_at_1_max + value: 70.27343537229697 + - type: nauc_recall_at_1_std + value: 10.378716550076664 + - type: nauc_recall_at_20_diff1 + value: 95.57461873638331 + - type: nauc_recall_at_20_max + value: 100.0 + - type: nauc_recall_at_20_std + value: 78.69397759103617 + - type: nauc_recall_at_3_diff1 + value: 88.63450141961552 + - type: nauc_recall_at_3_max + value: 93.64984088873658 + - type: nauc_recall_at_3_std + value: 54.20835000666945 + - type: nauc_recall_at_5_diff1 + value: 88.62314156431815 + - type: nauc_recall_at_5_max + value: 94.58928870693578 + - type: nauc_recall_at_5_std + value: 66.80552562905518 + - type: ndcg_at_1 + value: 87.7 + - type: ndcg_at_10 + value: 92.83200000000001 + - type: ndcg_at_100 + value: 93.242 + - type: ndcg_at_1000 + value: 93.403 + - type: ndcg_at_20 + value: 93.03099999999999 + - type: ndcg_at_3 + value: 92.186 + - type: ndcg_at_5 + value: 92.599 + - type: precision_at_1 + value: 87.7 + - type: precision_at_10 + value: 9.68 + - type: precision_at_100 + value: 0.987 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.88 + - type: precision_at_3 + value: 31.7 + - type: precision_at_5 + value: 19.220000000000002 + - type: recall_at_1 + value: 87.7 + - type: recall_at_10 + value: 96.8 + - type: recall_at_100 + value: 98.7 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 97.6 + - type: recall_at_3 + value: 95.1 + - type: recall_at_5 + value: 96.1 + task: + type: Retrieval + - dataset: + config: java + name: MTEB CodeSearchNetRetrieval (java) + revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 + split: test + type: code-search-net/code_search_net + metrics: + - type: main_score + value: 94.50699999999999 + - type: map_at_1 + value: 91.4 + - type: map_at_10 + value: 93.67699999999999 + - type: map_at_100 + value: 93.746 + - type: map_at_1000 + value: 93.753 + - type: map_at_20 + value: 93.706 + - type: map_at_3 + value: 93.43299999999999 + - type: map_at_5 + value: 93.593 + - type: mrr_at_1 + value: 91.4 + - type: mrr_at_10 + value: 93.67706349206348 + - type: mrr_at_100 + value: 93.74605961033865 + - type: mrr_at_1000 + value: 93.75307690063596 + - type: mrr_at_20 + value: 93.70575802250723 + - type: mrr_at_3 + value: 93.43333333333334 + - type: mrr_at_5 + value: 93.59333333333333 + - type: nauc_map_at_1000_diff1 + value: 91.96988867062096 + - type: nauc_map_at_1000_max + value: 80.8753031650805 + - type: nauc_map_at_1000_std + value: 54.38722696006999 + - type: nauc_map_at_100_diff1 + value: 91.97889892956285 + - type: nauc_map_at_100_max + value: 80.89676220389073 + - type: nauc_map_at_100_std + value: 54.4132005408842 + - type: nauc_map_at_10_diff1 + value: 92.00847920033857 + - type: nauc_map_at_10_max + value: 81.10521744502199 + - type: nauc_map_at_10_std + value: 54.68790929661031 + - type: nauc_map_at_1_diff1 + value: 89.60328317373455 + - type: nauc_map_at_1_max + value: 72.47899159663876 + - type: nauc_map_at_1_std + value: 44.26584587323289 + - type: nauc_map_at_20_diff1 + value: 92.02464449562623 + - type: nauc_map_at_20_max + value: 81.01907888930855 + - type: nauc_map_at_20_std + value: 54.571031073682384 + - type: nauc_map_at_3_diff1 + value: 92.09358870451744 + - type: nauc_map_at_3_max + value: 81.7007682937812 + - type: nauc_map_at_3_std + value: 55.27046690080435 + - type: nauc_map_at_5_diff1 + value: 92.11292217199042 + - type: nauc_map_at_5_max + value: 81.3521575817287 + - type: nauc_map_at_5_std + value: 54.99813695856418 + - type: nauc_mrr_at_1000_diff1 + value: 91.96988867062096 + - type: nauc_mrr_at_1000_max + value: 80.8753031650805 + - type: nauc_mrr_at_1000_std + value: 54.38722696006999 + - type: nauc_mrr_at_100_diff1 + value: 91.97889892956285 + - type: nauc_mrr_at_100_max + value: 80.89676220389073 + - type: nauc_mrr_at_100_std + value: 54.4132005408842 + - type: nauc_mrr_at_10_diff1 + value: 92.00847920033857 + - type: nauc_mrr_at_10_max + value: 81.10521744502199 + - type: nauc_mrr_at_10_std + value: 54.68790929661031 + - type: nauc_mrr_at_1_diff1 + value: 89.60328317373455 + - type: nauc_mrr_at_1_max + value: 72.47899159663876 + - type: nauc_mrr_at_1_std + value: 44.26584587323289 + - type: nauc_mrr_at_20_diff1 + value: 92.02464449562623 + - type: nauc_mrr_at_20_max + value: 81.01907888930855 + - type: nauc_mrr_at_20_std + value: 54.571031073682384 + - type: nauc_mrr_at_3_diff1 + value: 92.09358870451744 + - type: nauc_mrr_at_3_max + value: 81.7007682937812 + - type: nauc_mrr_at_3_std + value: 55.27046690080435 + - type: nauc_mrr_at_5_diff1 + value: 92.11292217199042 + - type: nauc_mrr_at_5_max + value: 81.3521575817287 + - type: nauc_mrr_at_5_std + value: 54.99813695856418 + - type: nauc_ndcg_at_1000_diff1 + value: 92.39514411202762 + - type: nauc_ndcg_at_1000_max + value: 82.1789464930351 + - type: nauc_ndcg_at_1000_std + value: 55.98331155582365 + - type: nauc_ndcg_at_100_diff1 + value: 92.60315730234768 + - type: nauc_ndcg_at_100_max + value: 82.66640006867893 + - type: nauc_ndcg_at_100_std + value: 56.60969071497668 + - type: nauc_ndcg_at_10_diff1 + value: 92.84846723213055 + - type: nauc_ndcg_at_10_max + value: 83.93273402590029 + - type: nauc_ndcg_at_10_std + value: 58.180048945638184 + - type: nauc_ndcg_at_1_diff1 + value: 89.60328317373455 + - type: nauc_ndcg_at_1_max + value: 72.47899159663876 + - type: nauc_ndcg_at_1_std + value: 44.26584587323289 + - type: nauc_ndcg_at_20_diff1 + value: 92.9455259704201 + - type: nauc_ndcg_at_20_max + value: 83.57732205562041 + - type: nauc_ndcg_at_20_std + value: 57.695601177913225 + - type: nauc_ndcg_at_3_diff1 + value: 93.02972756100402 + - type: nauc_ndcg_at_3_max + value: 85.03899149427752 + - type: nauc_ndcg_at_3_std + value: 59.29245146091644 + - type: nauc_ndcg_at_5_diff1 + value: 93.09702309051526 + - type: nauc_ndcg_at_5_max + value: 84.49116159880194 + - type: nauc_ndcg_at_5_std + value: 58.887701004976 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 100.0 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 79.24836601307065 + - type: nauc_precision_at_10_diff1 + value: 97.6423902894492 + - type: nauc_precision_at_10_max + value: 100.0 + - type: nauc_precision_at_10_std + value: 78.07345160286333 + - type: nauc_precision_at_1_diff1 + value: 89.60328317373455 + - type: nauc_precision_at_1_max + value: 72.47899159663876 + - type: nauc_precision_at_1_std + value: 44.26584587323289 + - type: nauc_precision_at_20_diff1 + value: 99.47712418300702 + - type: nauc_precision_at_20_max + value: 100.0 + - type: nauc_precision_at_20_std + value: 77.69374416433266 + - type: nauc_precision_at_3_diff1 + value: 97.06316635181199 + - type: nauc_precision_at_3_max + value: 99.35400516795852 + - type: nauc_precision_at_3_std + value: 76.56070180010008 + - type: nauc_precision_at_5_diff1 + value: 98.03532524120767 + - type: nauc_precision_at_5_max + value: 100.0 + - type: nauc_precision_at_5_std + value: 78.16163502438044 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 100.0 + - type: nauc_recall_at_100_max + value: 100.0 + - type: nauc_recall_at_100_std + value: 79.24836601307193 + - type: nauc_recall_at_10_diff1 + value: 97.64239028944921 + - type: nauc_recall_at_10_max + value: 100.0 + - type: nauc_recall_at_10_std + value: 78.07345160286356 + - type: nauc_recall_at_1_diff1 + value: 89.60328317373455 + - type: nauc_recall_at_1_max + value: 72.47899159663876 + - type: nauc_recall_at_1_std + value: 44.26584587323289 + - type: nauc_recall_at_20_diff1 + value: 99.47712418300625 + - type: nauc_recall_at_20_max + value: 100.0 + - type: nauc_recall_at_20_std + value: 77.69374416433251 + - type: nauc_recall_at_3_diff1 + value: 97.06316635181186 + - type: nauc_recall_at_3_max + value: 99.35400516795868 + - type: nauc_recall_at_3_std + value: 76.56070180009995 + - type: nauc_recall_at_5_diff1 + value: 98.03532524120769 + - type: nauc_recall_at_5_max + value: 100.0 + - type: nauc_recall_at_5_std + value: 78.16163502438029 + - type: ndcg_at_1 + value: 91.4 + - type: ndcg_at_10 + value: 94.50699999999999 + - type: ndcg_at_100 + value: 94.90899999999999 + - type: ndcg_at_1000 + value: 95.048 + - type: ndcg_at_20 + value: 94.626 + - type: ndcg_at_3 + value: 94.021 + - type: ndcg_at_5 + value: 94.31 + - type: precision_at_1 + value: 91.4 + - type: precision_at_10 + value: 9.700000000000001 + - type: precision_at_100 + value: 0.9900000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.875 + - type: precision_at_3 + value: 31.900000000000002 + - type: precision_at_5 + value: 19.28 + - type: recall_at_1 + value: 91.4 + - type: recall_at_10 + value: 97.0 + - type: recall_at_100 + value: 99.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 97.5 + - type: recall_at_3 + value: 95.7 + - type: recall_at_5 + value: 96.39999999999999 + task: + type: Retrieval + - dataset: + config: php + name: MTEB CodeSearchNetRetrieval (php) + revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 + split: test + type: code-search-net/code_search_net + metrics: + - type: main_score + value: 92.288 + - type: map_at_1 + value: 85.5 + - type: map_at_10 + value: 90.312 + - type: map_at_100 + value: 90.351 + - type: map_at_1000 + value: 90.35300000000001 + - type: map_at_20 + value: 90.335 + - type: map_at_3 + value: 89.583 + - type: map_at_5 + value: 90.098 + - type: mrr_at_1 + value: 85.5 + - type: mrr_at_10 + value: 90.31230158730165 + - type: mrr_at_100 + value: 90.35130514931716 + - type: mrr_at_1000 + value: 90.3533725076738 + - type: mrr_at_20 + value: 90.33520202020209 + - type: mrr_at_3 + value: 89.58333333333339 + - type: mrr_at_5 + value: 90.09833333333341 + - type: nauc_map_at_1000_diff1 + value: 93.85544381917798 + - type: nauc_map_at_1000_max + value: 40.90472409247008 + - type: nauc_map_at_1000_std + value: -16.463521873713017 + - type: nauc_map_at_100_diff1 + value: 93.85516451221955 + - type: nauc_map_at_100_max + value: 40.917784128196594 + - type: nauc_map_at_100_std + value: -16.437783475285407 + - type: nauc_map_at_10_diff1 + value: 93.821658302016 + - type: nauc_map_at_10_max + value: 41.1631023862988 + - type: nauc_map_at_10_std + value: -15.954316262724616 + - type: nauc_map_at_1_diff1 + value: 94.51659451659448 + - type: nauc_map_at_1_max + value: 35.95691130574863 + - type: nauc_map_at_1_std + value: -25.156884459210044 + - type: nauc_map_at_20_diff1 + value: 93.83850503661382 + - type: nauc_map_at_20_max + value: 41.019314488512634 + - type: nauc_map_at_20_std + value: -16.237689928755426 + - type: nauc_map_at_3_diff1 + value: 93.80837080594951 + - type: nauc_map_at_3_max + value: 40.635458703255324 + - type: nauc_map_at_3_std + value: -17.033014335677798 + - type: nauc_map_at_5_diff1 + value: 93.9020242116256 + - type: nauc_map_at_5_max + value: 41.174017150832654 + - type: nauc_map_at_5_std + value: -16.30029652673213 + - type: nauc_mrr_at_1000_diff1 + value: 93.85544381917798 + - type: nauc_mrr_at_1000_max + value: 40.90472409247008 + - type: nauc_mrr_at_1000_std + value: -16.463521873713017 + - type: nauc_mrr_at_100_diff1 + value: 93.85516451221955 + - type: nauc_mrr_at_100_max + value: 40.917784128196594 + - type: nauc_mrr_at_100_std + value: -16.437783475285407 + - type: nauc_mrr_at_10_diff1 + value: 93.821658302016 + - type: nauc_mrr_at_10_max + value: 41.1631023862988 + - type: nauc_mrr_at_10_std + value: -15.954316262724616 + - type: nauc_mrr_at_1_diff1 + value: 94.51659451659448 + - type: nauc_mrr_at_1_max + value: 35.95691130574863 + - type: nauc_mrr_at_1_std + value: -25.156884459210044 + - type: nauc_mrr_at_20_diff1 + value: 93.83850503661382 + - type: nauc_mrr_at_20_max + value: 41.019314488512634 + - type: nauc_mrr_at_20_std + value: -16.237689928755426 + - type: nauc_mrr_at_3_diff1 + value: 93.80837080594951 + - type: nauc_mrr_at_3_max + value: 40.635458703255324 + - type: nauc_mrr_at_3_std + value: -17.033014335677798 + - type: nauc_mrr_at_5_diff1 + value: 93.9020242116256 + - type: nauc_mrr_at_5_max + value: 41.174017150832654 + - type: nauc_mrr_at_5_std + value: -16.30029652673213 + - type: nauc_ndcg_at_1000_diff1 + value: 93.77150134331873 + - type: nauc_ndcg_at_1000_max + value: 42.34949722008596 + - type: nauc_ndcg_at_1000_std + value: -13.688355172335143 + - type: nauc_ndcg_at_100_diff1 + value: 93.77562254293132 + - type: nauc_ndcg_at_100_max + value: 42.992049705674965 + - type: nauc_ndcg_at_100_std + value: -12.42122424240366 + - type: nauc_ndcg_at_10_diff1 + value: 93.54925638165832 + - type: nauc_ndcg_at_10_max + value: 44.555550459089986 + - type: nauc_ndcg_at_10_std + value: -9.337958348868717 + - type: nauc_ndcg_at_1_diff1 + value: 94.51659451659448 + - type: nauc_ndcg_at_1_max + value: 35.95691130574863 + - type: nauc_ndcg_at_1_std + value: -25.156884459210044 + - type: nauc_ndcg_at_20_diff1 + value: 93.61932694899781 + - type: nauc_ndcg_at_20_max + value: 43.96879091200643 + - type: nauc_ndcg_at_20_std + value: -10.495064090760449 + - type: nauc_ndcg_at_3_diff1 + value: 93.62155771108924 + - type: nauc_ndcg_at_3_max + value: 42.893378116364104 + - type: nauc_ndcg_at_3_std + value: -12.73029168651973 + - type: nauc_ndcg_at_5_diff1 + value: 93.78967591680556 + - type: nauc_ndcg_at_5_max + value: 44.44155136825693 + - type: nauc_ndcg_at_5_std + value: -10.579261977804965 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 94.39775910363963 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 100.0 + - type: nauc_precision_at_10_diff1 + value: 89.1387927720107 + - type: nauc_precision_at_10_max + value: 100.0 + - type: nauc_precision_at_10_std + value: 100.0 + - type: nauc_precision_at_1_diff1 + value: 94.51659451659448 + - type: nauc_precision_at_1_max + value: 35.95691130574863 + - type: nauc_precision_at_1_std + value: -25.156884459210044 + - type: nauc_precision_at_20_diff1 + value: 89.61251167133568 + - type: nauc_precision_at_20_max + value: 100.0 + - type: nauc_precision_at_20_std + value: 100.0 + - type: nauc_precision_at_3_diff1 + value: 92.71284271284276 + - type: nauc_precision_at_3_max + value: 54.854426619132425 + - type: nauc_precision_at_3_std + value: 10.262286732874971 + - type: nauc_precision_at_5_diff1 + value: 93.00449346405235 + - type: nauc_precision_at_5_max + value: 73.52795284780564 + - type: nauc_precision_at_5_std + value: 40.204831932772336 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 94.39775910364197 + - type: nauc_recall_at_100_max + value: 100.0 + - type: nauc_recall_at_100_std + value: 100.0 + - type: nauc_recall_at_10_diff1 + value: 89.13879277201103 + - type: nauc_recall_at_10_max + value: 100.0 + - type: nauc_recall_at_10_std + value: 100.0 + - type: nauc_recall_at_1_diff1 + value: 94.51659451659448 + - type: nauc_recall_at_1_max + value: 35.95691130574863 + - type: nauc_recall_at_1_std + value: -25.156884459210044 + - type: nauc_recall_at_20_diff1 + value: 89.61251167133527 + - type: nauc_recall_at_20_max + value: 100.0 + - type: nauc_recall_at_20_std + value: 100.0 + - type: nauc_recall_at_3_diff1 + value: 92.71284271284289 + - type: nauc_recall_at_3_max + value: 54.85442661913248 + - type: nauc_recall_at_3_std + value: 10.262286732875078 + - type: nauc_recall_at_5_diff1 + value: 93.00449346405259 + - type: nauc_recall_at_5_max + value: 73.52795284780608 + - type: nauc_recall_at_5_std + value: 40.20483193277305 + - type: ndcg_at_1 + value: 85.5 + - type: ndcg_at_10 + value: 92.288 + - type: ndcg_at_100 + value: 92.49300000000001 + - type: ndcg_at_1000 + value: 92.574 + - type: ndcg_at_20 + value: 92.366 + - type: ndcg_at_3 + value: 90.851 + - type: ndcg_at_5 + value: 91.789 + - type: precision_at_1 + value: 85.5 + - type: precision_at_10 + value: 9.83 + - type: precision_at_100 + value: 0.993 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.93 + - type: precision_at_3 + value: 31.5 + - type: precision_at_5 + value: 19.36 + - type: recall_at_1 + value: 85.5 + - type: recall_at_10 + value: 98.3 + - type: recall_at_100 + value: 99.3 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 98.6 + - type: recall_at_3 + value: 94.5 + - type: recall_at_5 + value: 96.8 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CodeTransOceanContest (default) + revision: 20da4eb20a4b17300c0986ee148c90867a7f2a4d + split: test + type: CoIR-Retrieval/codetrans-contest + metrics: + - type: main_score + value: 94.96 + - type: map_at_1 + value: 91.855 + - type: map_at_10 + value: 94.054 + - type: map_at_100 + value: 94.14099999999999 + - type: map_at_1000 + value: 94.145 + - type: map_at_20 + value: 94.124 + - type: map_at_3 + value: 93.741 + - type: map_at_5 + value: 93.92200000000001 + - type: mrr_at_1 + value: 91.8552036199095 + - type: mrr_at_10 + value: 94.05354449472097 + - type: mrr_at_100 + value: 94.14134449135241 + - type: mrr_at_1000 + value: 94.14490739440583 + - type: mrr_at_20 + value: 94.12357250592545 + - type: mrr_at_3 + value: 93.74057315233786 + - type: mrr_at_5 + value: 93.92156862745098 + - type: nauc_map_at_1000_diff1 + value: 92.55002145543314 + - type: nauc_map_at_1000_max + value: 83.9544206201828 + - type: nauc_map_at_1000_std + value: 13.818812515717433 + - type: nauc_map_at_100_diff1 + value: 92.55455211118941 + - type: nauc_map_at_100_max + value: 83.96417863440325 + - type: nauc_map_at_100_std + value: 13.871223041576354 + - type: nauc_map_at_10_diff1 + value: 92.58163562742963 + - type: nauc_map_at_10_max + value: 84.20094910390932 + - type: nauc_map_at_10_std + value: 14.827570038179012 + - type: nauc_map_at_1_diff1 + value: 90.8130775986731 + - type: nauc_map_at_1_max + value: 81.11634573555125 + - type: nauc_map_at_1_std + value: 6.18089047774425 + - type: nauc_map_at_20_diff1 + value: 92.57706925983298 + - type: nauc_map_at_20_max + value: 84.01267551178432 + - type: nauc_map_at_20_std + value: 13.990683573503478 + - type: nauc_map_at_3_diff1 + value: 92.76365755738576 + - type: nauc_map_at_3_max + value: 84.18142235516605 + - type: nauc_map_at_3_std + value: 11.473399520154405 + - type: nauc_map_at_5_diff1 + value: 92.74270433278942 + - type: nauc_map_at_5_max + value: 84.12361744519418 + - type: nauc_map_at_5_std + value: 14.668669421165939 + - type: nauc_mrr_at_1000_diff1 + value: 92.55002145543314 + - type: nauc_mrr_at_1000_max + value: 83.9544206201828 + - type: nauc_mrr_at_1000_std + value: 13.818812515717433 + - type: nauc_mrr_at_100_diff1 + value: 92.55455211118941 + - type: nauc_mrr_at_100_max + value: 83.96417863440325 + - type: nauc_mrr_at_100_std + value: 13.871223041576354 + - type: nauc_mrr_at_10_diff1 + value: 92.58163562742963 + - type: nauc_mrr_at_10_max + value: 84.20094910390932 + - type: nauc_mrr_at_10_std + value: 14.827570038179012 + - type: nauc_mrr_at_1_diff1 + value: 90.8130775986731 + - type: nauc_mrr_at_1_max + value: 81.11634573555125 + - type: nauc_mrr_at_1_std + value: 6.18089047774425 + - type: nauc_mrr_at_20_diff1 + value: 92.57706925983298 + - type: nauc_mrr_at_20_max + value: 84.01267551178432 + - type: nauc_mrr_at_20_std + value: 13.990683573503478 + - type: nauc_mrr_at_3_diff1 + value: 92.76365755738576 + - type: nauc_mrr_at_3_max + value: 84.18142235516605 + - type: nauc_mrr_at_3_std + value: 11.473399520154405 + - type: nauc_mrr_at_5_diff1 + value: 92.74270433278942 + - type: nauc_mrr_at_5_max + value: 84.12361744519418 + - type: nauc_mrr_at_5_std + value: 14.668669421165939 + - type: nauc_ndcg_at_1000_diff1 + value: 92.84906743720337 + - type: nauc_ndcg_at_1000_max + value: 84.57965223804021 + - type: nauc_ndcg_at_1000_std + value: 15.875602814166239 + - type: nauc_ndcg_at_100_diff1 + value: 92.94862546375188 + - type: nauc_ndcg_at_100_max + value: 84.7943402354994 + - type: nauc_ndcg_at_100_std + value: 17.046814945571057 + - type: nauc_ndcg_at_10_diff1 + value: 93.18693605775724 + - type: nauc_ndcg_at_10_max + value: 85.99181169093023 + - type: nauc_ndcg_at_10_std + value: 21.453785415429227 + - type: nauc_ndcg_at_1_diff1 + value: 90.8130775986731 + - type: nauc_ndcg_at_1_max + value: 81.11634573555125 + - type: nauc_ndcg_at_1_std + value: 6.18089047774425 + - type: nauc_ndcg_at_20_diff1 + value: 93.18182308386798 + - type: nauc_ndcg_at_20_max + value: 85.29721008749047 + - type: nauc_ndcg_at_20_std + value: 18.265825551265774 + - type: nauc_ndcg_at_3_diff1 + value: 93.5678392571751 + - type: nauc_ndcg_at_3_max + value: 85.68193097329507 + - type: nauc_ndcg_at_3_std + value: 13.252390138276315 + - type: nauc_ndcg_at_5_diff1 + value: 93.57442357672642 + - type: nauc_ndcg_at_5_max + value: 85.65305420139138 + - type: nauc_ndcg_at_5_std + value: 20.399096483631958 + - type: nauc_precision_at_1000_diff1 + value: 100.0 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 100.0 + - type: nauc_precision_at_100_diff1 + value: 100.0 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 100.0 + - type: nauc_precision_at_10_diff1 + value: 97.38693467336647 + - type: nauc_precision_at_10_max + value: 100.0 + - type: nauc_precision_at_10_std + value: 76.08263155229871 + - type: nauc_precision_at_1_diff1 + value: 90.8130775986731 + - type: nauc_precision_at_1_max + value: 81.11634573555125 + - type: nauc_precision_at_1_std + value: 6.18089047774425 + - type: nauc_precision_at_20_diff1 + value: 100.0 + - type: nauc_precision_at_20_max + value: 100.0 + - type: nauc_precision_at_20_std + value: 69.38924997712752 + - type: nauc_precision_at_3_diff1 + value: 97.09659408151863 + - type: nauc_precision_at_3_max + value: 92.38060989189518 + - type: nauc_precision_at_3_std + value: 20.8638963856334 + - type: nauc_precision_at_5_diff1 + value: 98.13352476669084 + - type: nauc_precision_at_5_max + value: 94.1685830281352 + - type: nauc_precision_at_5_std + value: 54.20627096881086 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 100.0 + - type: nauc_recall_at_100_max + value: 100.0 + - type: nauc_recall_at_100_std + value: 100.0 + - type: nauc_recall_at_10_diff1 + value: 97.38693467336678 + - type: nauc_recall_at_10_max + value: 100.0 + - type: nauc_recall_at_10_std + value: 76.08263155229885 + - type: nauc_recall_at_1_diff1 + value: 90.8130775986731 + - type: nauc_recall_at_1_max + value: 81.11634573555125 + - type: nauc_recall_at_1_std + value: 6.18089047774425 + - type: nauc_recall_at_20_diff1 + value: 100.0 + - type: nauc_recall_at_20_max + value: 100.0 + - type: nauc_recall_at_20_std + value: 69.3892499771279 + - type: nauc_recall_at_3_diff1 + value: 97.09659408151863 + - type: nauc_recall_at_3_max + value: 92.38060989189518 + - type: nauc_recall_at_3_std + value: 20.863896385634074 + - type: nauc_recall_at_5_diff1 + value: 98.13352476669056 + - type: nauc_recall_at_5_max + value: 94.16858302813482 + - type: nauc_recall_at_5_std + value: 54.20627096881058 + - type: ndcg_at_1 + value: 91.855 + - type: ndcg_at_10 + value: 94.96 + - type: ndcg_at_100 + value: 95.357 + - type: ndcg_at_1000 + value: 95.422 + - type: ndcg_at_20 + value: 95.19800000000001 + - type: ndcg_at_3 + value: 94.306 + - type: ndcg_at_5 + value: 94.65599999999999 + - type: precision_at_1 + value: 91.855 + - type: precision_at_10 + value: 9.774 + - type: precision_at_100 + value: 0.9950000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.932 + - type: precision_at_3 + value: 31.976 + - type: precision_at_5 + value: 19.367 + - type: recall_at_1 + value: 91.855 + - type: recall_at_10 + value: 97.738 + - type: recall_at_100 + value: 99.548 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 98.643 + - type: recall_at_3 + value: 95.928 + - type: recall_at_5 + value: 96.833 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CodeTransOceanDL (default) + revision: 281562cb8a1265ab5c0824bfa6ddcd9b0a15618f + split: test + type: CoIR-Retrieval/codetrans-dl + metrics: + - type: main_score + value: 38.566 + - type: map_at_1 + value: 8.889 + - type: map_at_10 + value: 24.311 + - type: map_at_100 + value: 25.413999999999998 + - type: map_at_1000 + value: 25.413999999999998 + - type: map_at_20 + value: 25.413999999999998 + - type: map_at_3 + value: 14.352 + - type: map_at_5 + value: 18.296 + - type: mrr_at_1 + value: 7.777777777777778 + - type: mrr_at_10 + value: 23.692460317460355 + - type: mrr_at_100 + value: 24.94246031746036 + - type: mrr_at_1000 + value: 24.94246031746036 + - type: mrr_at_20 + value: 24.94246031746036 + - type: mrr_at_3 + value: 13.611111111111104 + - type: mrr_at_5 + value: 18.13888888888888 + - type: nauc_map_at_1000_diff1 + value: -11.995791794456052 + - type: nauc_map_at_1000_max + value: -37.4290224077524 + - type: nauc_map_at_1000_std + value: -8.685310117692485 + - type: nauc_map_at_100_diff1 + value: -11.995791794456052 + - type: nauc_map_at_100_max + value: -37.4290224077524 + - type: nauc_map_at_100_std + value: -8.685310117692485 + - type: nauc_map_at_10_diff1 + value: -12.904069625259378 + - type: nauc_map_at_10_max + value: -36.469676015537445 + - type: nauc_map_at_10_std + value: -8.18556060930571 + - type: nauc_map_at_1_diff1 + value: -22.632143815401133 + - type: nauc_map_at_1_max + value: -38.05674805473572 + - type: nauc_map_at_1_std + value: -5.114703514891332 + - type: nauc_map_at_20_diff1 + value: -11.995791794456052 + - type: nauc_map_at_20_max + value: -37.4290224077524 + - type: nauc_map_at_20_std + value: -8.685310117692485 + - type: nauc_map_at_3_diff1 + value: -9.430569631049272 + - type: nauc_map_at_3_max + value: -40.31306127773153 + - type: nauc_map_at_3_std + value: -9.279256680145705 + - type: nauc_map_at_5_diff1 + value: -11.17196963640011 + - type: nauc_map_at_5_max + value: -36.71021151035073 + - type: nauc_map_at_5_std + value: -10.227972134518213 + - type: nauc_mrr_at_1000_diff1 + value: 7.811397652262392 + - type: nauc_mrr_at_1000_max + value: -35.827281345154454 + - type: nauc_mrr_at_1000_std + value: -10.625492656056048 + - type: nauc_mrr_at_100_diff1 + value: 7.811397652262392 + - type: nauc_mrr_at_100_max + value: -35.827281345154454 + - type: nauc_mrr_at_100_std + value: -10.625492656056048 + - type: nauc_mrr_at_10_diff1 + value: 7.0457873475325155 + - type: nauc_mrr_at_10_max + value: -34.091630838685006 + - type: nauc_mrr_at_10_std + value: -10.015877703911025 + - type: nauc_mrr_at_1_diff1 + value: 6.72505653723791 + - type: nauc_mrr_at_1_max + value: -39.05860707577908 + - type: nauc_mrr_at_1_std + value: -12.350032580781182 + - type: nauc_mrr_at_20_diff1 + value: 7.811397652262392 + - type: nauc_mrr_at_20_max + value: -35.827281345154454 + - type: nauc_mrr_at_20_std + value: -10.625492656056048 + - type: nauc_mrr_at_3_diff1 + value: 9.229424174654557 + - type: nauc_mrr_at_3_max + value: -39.823960095369024 + - type: nauc_mrr_at_3_std + value: -12.365662926663418 + - type: nauc_mrr_at_5_diff1 + value: 7.455314880931567 + - type: nauc_mrr_at_5_max + value: -34.4606595687379 + - type: nauc_mrr_at_5_std + value: -9.748438671046124 + - type: nauc_ndcg_at_1000_diff1 + value: -11.393187924360975 + - type: nauc_ndcg_at_1000_max + value: -37.14382596573424 + - type: nauc_ndcg_at_1000_std + value: -8.714465550011122 + - type: nauc_ndcg_at_100_diff1 + value: -11.393187924360975 + - type: nauc_ndcg_at_100_max + value: -37.14382596573424 + - type: nauc_ndcg_at_100_std + value: -8.714465550011122 + - type: nauc_ndcg_at_10_diff1 + value: -14.50305018622005 + - type: nauc_ndcg_at_10_max + value: -34.183276513019024 + - type: nauc_ndcg_at_10_std + value: -6.91808119392616 + - type: nauc_ndcg_at_1_diff1 + value: -22.632143815401133 + - type: nauc_ndcg_at_1_max + value: -38.05674805473572 + - type: nauc_ndcg_at_1_std + value: -5.114703514891332 + - type: nauc_ndcg_at_20_diff1 + value: -11.393187924360975 + - type: nauc_ndcg_at_20_max + value: -37.14382596573424 + - type: nauc_ndcg_at_20_std + value: -8.714465550011122 + - type: nauc_ndcg_at_3_diff1 + value: -6.28820826432392 + - type: nauc_ndcg_at_3_max + value: -40.515698550813426 + - type: nauc_ndcg_at_3_std + value: -9.763678652070176 + - type: nauc_ndcg_at_5_diff1 + value: -9.784255160012018 + - type: nauc_ndcg_at_5_max + value: -35.547401094483774 + - type: nauc_ndcg_at_5_std + value: -11.253198818827153 + - type: nauc_precision_at_1000_diff1 + value: 100.0 + - type: nauc_precision_at_1000_max + value: 100.0 + - type: nauc_precision_at_1000_std + value: 100.0 + - type: nauc_precision_at_100_diff1 + value: 100.0 + - type: nauc_precision_at_100_max + value: 100.0 + - type: nauc_precision_at_100_std + value: 100.0 + - type: nauc_precision_at_10_diff1 + value: -28.569339405841387 + - type: nauc_precision_at_10_max + value: -21.317845652446533 + - type: nauc_precision_at_10_std + value: 1.6546191451133108 + - type: nauc_precision_at_1_diff1 + value: -22.632143815401133 + - type: nauc_precision_at_1_max + value: -38.05674805473572 + - type: nauc_precision_at_1_std + value: -5.114703514891332 + - type: nauc_precision_at_20_diff1 + value: .nan + - type: nauc_precision_at_20_max + value: .nan + - type: nauc_precision_at_20_std + value: .nan + - type: nauc_precision_at_3_diff1 + value: 0.21101534115564527 + - type: nauc_precision_at_3_max + value: -40.8638015094404 + - type: nauc_precision_at_3_std + value: -10.657976464982523 + - type: nauc_precision_at_5_diff1 + value: -7.729717358965525 + - type: nauc_precision_at_5_max + value: -33.27683373383679 + - type: nauc_precision_at_5_std + value: -13.203741846893195 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: .nan + - type: nauc_recall_at_100_max + value: .nan + - type: nauc_recall_at_100_std + value: .nan + - type: nauc_recall_at_10_diff1 + value: -28.569339405841294 + - type: nauc_recall_at_10_max + value: -21.317845652446415 + - type: nauc_recall_at_10_std + value: 1.6546191451134153 + - type: nauc_recall_at_1_diff1 + value: -22.632143815401133 + - type: nauc_recall_at_1_max + value: -38.05674805473572 + - type: nauc_recall_at_1_std + value: -5.114703514891332 + - type: nauc_recall_at_20_diff1 + value: .nan + - type: nauc_recall_at_20_max + value: .nan + - type: nauc_recall_at_20_std + value: .nan + - type: nauc_recall_at_3_diff1 + value: 0.2110153411556453 + - type: nauc_recall_at_3_max + value: -40.863801509440385 + - type: nauc_recall_at_3_std + value: -10.65797646498251 + - type: nauc_recall_at_5_diff1 + value: -7.729717358965576 + - type: nauc_recall_at_5_max + value: -33.276833733836845 + - type: nauc_recall_at_5_std + value: -13.203741846893251 + - type: ndcg_at_1 + value: 8.889 + - type: ndcg_at_10 + value: 38.566 + - type: ndcg_at_100 + value: 42.063 + - type: ndcg_at_1000 + value: 42.063 + - type: ndcg_at_20 + value: 42.063 + - type: ndcg_at_3 + value: 16.488 + - type: ndcg_at_5 + value: 23.589 + - type: precision_at_1 + value: 8.889 + - type: precision_at_10 + value: 8.722000000000001 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 7.593 + - type: precision_at_5 + value: 8.0 + - type: recall_at_1 + value: 8.889 + - type: recall_at_10 + value: 87.222 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 22.778000000000002 + - type: recall_at_5 + value: 40.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CosQA (default) + revision: bc5efb7e9d437246ce393ed19d772e08e4a79535 + split: test + type: CoIR-Retrieval/cosqa + metrics: + - type: main_score + value: 34.452 + - type: map_at_1 + value: 15.2 + - type: map_at_10 + value: 26.473000000000003 + - type: map_at_100 + value: 28.01 + - type: map_at_1000 + value: 28.038 + - type: map_at_20 + value: 27.48 + - type: map_at_3 + value: 20.833 + - type: map_at_5 + value: 24.153 + - type: mrr_at_1 + value: 19.2 + - type: mrr_at_10 + value: 28.281111111111105 + - type: mrr_at_100 + value: 29.8632158616367 + - type: mrr_at_1000 + value: 29.891199207757197 + - type: mrr_at_20 + value: 29.27876414692977 + - type: mrr_at_3 + value: 23.966666666666658 + - type: mrr_at_5 + value: 25.636666666666663 + - type: nauc_map_at_1000_diff1 + value: 24.10146846046148 + - type: nauc_map_at_1000_max + value: 10.598410702361434 + - type: nauc_map_at_1000_std + value: -20.144523237787222 + - type: nauc_map_at_100_diff1 + value: 24.053508672795086 + - type: nauc_map_at_100_max + value: 10.62060173891184 + - type: nauc_map_at_100_std + value: -20.123151261597684 + - type: nauc_map_at_10_diff1 + value: 24.03401073979636 + - type: nauc_map_at_10_max + value: 10.204901836438324 + - type: nauc_map_at_10_std + value: -20.723915647527605 + - type: nauc_map_at_1_diff1 + value: 34.9707775098606 + - type: nauc_map_at_1_max + value: 8.792695974322537 + - type: nauc_map_at_1_std + value: -17.679606534340426 + - type: nauc_map_at_20_diff1 + value: 23.635390281247904 + - type: nauc_map_at_20_max + value: 10.51091267854992 + - type: nauc_map_at_20_std + value: -20.503982338903548 + - type: nauc_map_at_3_diff1 + value: 27.08078023869985 + - type: nauc_map_at_3_max + value: 8.624902998628762 + - type: nauc_map_at_3_std + value: -21.16290389423999 + - type: nauc_map_at_5_diff1 + value: 24.375227072550878 + - type: nauc_map_at_5_max + value: 10.446847054213334 + - type: nauc_map_at_5_std + value: -20.416673549725004 + - type: nauc_mrr_at_1000_diff1 + value: 19.78435935694305 + - type: nauc_mrr_at_1000_max + value: 6.389276526733645 + - type: nauc_mrr_at_1000_std + value: -24.87029202656466 + - type: nauc_mrr_at_100_diff1 + value: 19.741897116785875 + - type: nauc_mrr_at_100_max + value: 6.399527259062092 + - type: nauc_mrr_at_100_std + value: -24.86454789316169 + - type: nauc_mrr_at_10_diff1 + value: 19.82954896251409 + - type: nauc_mrr_at_10_max + value: 5.841515376599231 + - type: nauc_mrr_at_10_std + value: -25.585472957584926 + - type: nauc_mrr_at_1_diff1 + value: 22.61366365164493 + - type: nauc_mrr_at_1_max + value: 5.888630966674663 + - type: nauc_mrr_at_1_std + value: -23.551522185757364 + - type: nauc_mrr_at_20_diff1 + value: 19.610569618359854 + - type: nauc_mrr_at_20_max + value: 6.388923736326994 + - type: nauc_mrr_at_20_std + value: -25.022579250123666 + - type: nauc_mrr_at_3_diff1 + value: 21.689261589102053 + - type: nauc_mrr_at_3_max + value: 5.433511684692397 + - type: nauc_mrr_at_3_std + value: -25.22461843203615 + - type: nauc_mrr_at_5_diff1 + value: 20.769666984905864 + - type: nauc_mrr_at_5_max + value: 5.103054542191035 + - type: nauc_mrr_at_5_std + value: -25.62916796715137 + - type: nauc_ndcg_at_1000_diff1 + value: 21.724475676486286 + - type: nauc_ndcg_at_1000_max + value: 11.92044108916809 + - type: nauc_ndcg_at_1000_std + value: -19.560471367028224 + - type: nauc_ndcg_at_100_diff1 + value: 20.65779971541893 + - type: nauc_ndcg_at_100_max + value: 12.772564966149327 + - type: nauc_ndcg_at_100_std + value: -18.53040956973807 + - type: nauc_ndcg_at_10_diff1 + value: 19.715388368513594 + - type: nauc_ndcg_at_10_max + value: 11.290056737017848 + - type: nauc_ndcg_at_10_std + value: -21.48987713396045 + - type: nauc_ndcg_at_1_diff1 + value: 34.9707775098606 + - type: nauc_ndcg_at_1_max + value: 8.792695974322537 + - type: nauc_ndcg_at_1_std + value: -17.679606534340426 + - type: nauc_ndcg_at_20_diff1 + value: 18.26515065287157 + - type: nauc_ndcg_at_20_max + value: 12.420581957680346 + - type: nauc_ndcg_at_20_std + value: -20.560776758982286 + - type: nauc_ndcg_at_3_diff1 + value: 24.86486238625192 + - type: nauc_ndcg_at_3_max + value: 8.51016243931613 + - type: nauc_ndcg_at_3_std + value: -22.38799975521682 + - type: nauc_ndcg_at_5_diff1 + value: 20.38709195941383 + - type: nauc_ndcg_at_5_max + value: 11.692332914550612 + - type: nauc_ndcg_at_5_std + value: -21.095128458420735 + - type: nauc_precision_at_1000_diff1 + value: -2.427637721760636 + - type: nauc_precision_at_1000_max + value: 93.46405228758199 + - type: nauc_precision_at_1000_std + value: 56.139122315591536 + - type: nauc_precision_at_100_diff1 + value: -12.075404874593668 + - type: nauc_precision_at_100_max + value: 46.41166811552213 + - type: nauc_precision_at_100_std + value: 21.43662062526193 + - type: nauc_precision_at_10_diff1 + value: 7.927965224591165 + - type: nauc_precision_at_10_max + value: 14.499053516090232 + - type: nauc_precision_at_10_std + value: -23.215405338109257 + - type: nauc_precision_at_1_diff1 + value: 34.9707775098606 + - type: nauc_precision_at_1_max + value: 8.792695974322537 + - type: nauc_precision_at_1_std + value: -17.679606534340426 + - type: nauc_precision_at_20_diff1 + value: -2.2220305852593376 + - type: nauc_precision_at_20_max + value: 20.688168333908145 + - type: nauc_precision_at_20_std + value: -19.112501847927895 + - type: nauc_precision_at_3_diff1 + value: 19.523274018213513 + - type: nauc_precision_at_3_max + value: 8.217018367841955 + - type: nauc_precision_at_3_std + value: -25.40158982161363 + - type: nauc_precision_at_5_diff1 + value: 10.681426032017447 + - type: nauc_precision_at_5_max + value: 14.961711488638585 + - type: nauc_precision_at_5_std + value: -22.588397199647623 + - type: nauc_recall_at_1000_diff1 + value: -2.4276377217553624 + - type: nauc_recall_at_1000_max + value: 93.46405228758147 + - type: nauc_recall_at_1000_std + value: 56.13912231559195 + - type: nauc_recall_at_100_diff1 + value: -12.075404874593449 + - type: nauc_recall_at_100_max + value: 46.41166811552219 + - type: nauc_recall_at_100_std + value: 21.436620625261668 + - type: nauc_recall_at_10_diff1 + value: 7.927965224591155 + - type: nauc_recall_at_10_max + value: 14.499053516090227 + - type: nauc_recall_at_10_std + value: -23.21540533810927 + - type: nauc_recall_at_1_diff1 + value: 34.9707775098606 + - type: nauc_recall_at_1_max + value: 8.792695974322537 + - type: nauc_recall_at_1_std + value: -17.679606534340426 + - type: nauc_recall_at_20_diff1 + value: -2.22203058525925 + - type: nauc_recall_at_20_max + value: 20.688168333908262 + - type: nauc_recall_at_20_std + value: -19.11250184792784 + - type: nauc_recall_at_3_diff1 + value: 19.523274018213503 + - type: nauc_recall_at_3_max + value: 8.217018367841955 + - type: nauc_recall_at_3_std + value: -25.40158982161363 + - type: nauc_recall_at_5_diff1 + value: 10.681426032017423 + - type: nauc_recall_at_5_max + value: 14.961711488638551 + - type: nauc_recall_at_5_std + value: -22.58839719964763 + - type: ndcg_at_1 + value: 15.2 + - type: ndcg_at_10 + value: 34.452 + - type: ndcg_at_100 + value: 41.723 + - type: ndcg_at_1000 + value: 42.421 + - type: ndcg_at_20 + value: 38.002 + - type: ndcg_at_3 + value: 22.864 + - type: ndcg_at_5 + value: 28.904999999999998 + - type: precision_at_1 + value: 15.2 + - type: precision_at_10 + value: 6.0600000000000005 + - type: precision_at_100 + value: 0.942 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 3.7199999999999998 + - type: precision_at_3 + value: 9.6 + - type: precision_at_5 + value: 8.72 + - type: recall_at_1 + value: 15.2 + - type: recall_at_10 + value: 60.6 + - type: recall_at_100 + value: 94.19999999999999 + - type: recall_at_1000 + value: 99.6 + - type: recall_at_20 + value: 74.4 + - type: recall_at_3 + value: 28.799999999999997 + - type: recall_at_5 + value: 43.6 + task: + type: Retrieval + - dataset: + config: default + name: MTEB StackOverflowQA (default) + revision: db8f169f3894c14a00251061f957b2063eef2bd5 + split: test + type: CoIR-Retrieval/stackoverflow-qa + metrics: + - type: main_score + value: 97.16799999999999 + - type: map_at_1 + value: 94.985 + - type: map_at_10 + value: 96.533 + - type: map_at_100 + value: 96.568 + - type: map_at_1000 + value: 96.569 + - type: map_at_20 + value: 96.554 + - type: map_at_3 + value: 96.272 + - type: map_at_5 + value: 96.48 + - type: mrr_at_1 + value: 94.98495486459379 + - type: mrr_at_10 + value: 96.53325054528665 + - type: mrr_at_100 + value: 96.56849512537934 + - type: mrr_at_1000 + value: 96.56899560792449 + - type: mrr_at_20 + value: 96.55354401474338 + - type: mrr_at_3 + value: 96.27214978268137 + - type: mrr_at_5 + value: 96.48027415580073 + - type: nauc_map_at_1000_diff1 + value: 96.7701081739238 + - type: nauc_map_at_1000_max + value: 89.92147554128405 + - type: nauc_map_at_1000_std + value: -11.367860416596368 + - type: nauc_map_at_100_diff1 + value: 96.7701266461378 + - type: nauc_map_at_100_max + value: 89.92249288169609 + - type: nauc_map_at_100_std + value: -11.365088267343506 + - type: nauc_map_at_10_diff1 + value: 96.80296303411313 + - type: nauc_map_at_10_max + value: 89.97144795621024 + - type: nauc_map_at_10_std + value: -11.225883175842885 + - type: nauc_map_at_1_diff1 + value: 96.33579234749389 + - type: nauc_map_at_1_max + value: 88.14831631499773 + - type: nauc_map_at_1_std + value: -11.043584429943797 + - type: nauc_map_at_20_diff1 + value: 96.7841381971511 + - type: nauc_map_at_20_max + value: 89.96621023207143 + - type: nauc_map_at_20_std + value: -11.236908272386724 + - type: nauc_map_at_3_diff1 + value: 96.77341739930384 + - type: nauc_map_at_3_max + value: 90.18226749615211 + - type: nauc_map_at_3_std + value: -12.405699503933928 + - type: nauc_map_at_5_diff1 + value: 96.79677069369804 + - type: nauc_map_at_5_max + value: 90.04406323888772 + - type: nauc_map_at_5_std + value: -11.09579820828818 + - type: nauc_mrr_at_1000_diff1 + value: 96.7701081739238 + - type: nauc_mrr_at_1000_max + value: 89.92147554128405 + - type: nauc_mrr_at_1000_std + value: -11.367860416596368 + - type: nauc_mrr_at_100_diff1 + value: 96.7701266461378 + - type: nauc_mrr_at_100_max + value: 89.92249288169609 + - type: nauc_mrr_at_100_std + value: -11.365088267343506 + - type: nauc_mrr_at_10_diff1 + value: 96.80296303411313 + - type: nauc_mrr_at_10_max + value: 89.97144795621024 + - type: nauc_mrr_at_10_std + value: -11.225883175842885 + - type: nauc_mrr_at_1_diff1 + value: 96.33579234749389 + - type: nauc_mrr_at_1_max + value: 88.14831631499773 + - type: nauc_mrr_at_1_std + value: -11.043584429943797 + - type: nauc_mrr_at_20_diff1 + value: 96.7841381971511 + - type: nauc_mrr_at_20_max + value: 89.96621023207143 + - type: nauc_mrr_at_20_std + value: -11.236908272386724 + - type: nauc_mrr_at_3_diff1 + value: 96.77341739930384 + - type: nauc_mrr_at_3_max + value: 90.18226749615211 + - type: nauc_mrr_at_3_std + value: -12.405699503933928 + - type: nauc_mrr_at_5_diff1 + value: 96.79677069369804 + - type: nauc_mrr_at_5_max + value: 90.04406323888772 + - type: nauc_mrr_at_5_std + value: -11.09579820828818 + - type: nauc_ndcg_at_1000_diff1 + value: 96.83153135375731 + - type: nauc_ndcg_at_1000_max + value: 90.17747818029089 + - type: nauc_ndcg_at_1000_std + value: -11.09225768112506 + - type: nauc_ndcg_at_100_diff1 + value: 96.82500676553634 + - type: nauc_ndcg_at_100_max + value: 90.21640577830735 + - type: nauc_ndcg_at_100_std + value: -11.051187241816189 + - type: nauc_ndcg_at_10_diff1 + value: 97.01040984737975 + - type: nauc_ndcg_at_10_max + value: 90.53590661634338 + - type: nauc_ndcg_at_10_std + value: -10.181364845409425 + - type: nauc_ndcg_at_1_diff1 + value: 96.33579234749389 + - type: nauc_ndcg_at_1_max + value: 88.14831631499773 + - type: nauc_ndcg_at_1_std + value: -11.043584429943797 + - type: nauc_ndcg_at_20_diff1 + value: 96.92876646822131 + - type: nauc_ndcg_at_20_max + value: 90.53613648406619 + - type: nauc_ndcg_at_20_std + value: -10.211015921117655 + - type: nauc_ndcg_at_3_diff1 + value: 96.92301963415002 + - type: nauc_ndcg_at_3_max + value: 90.96341737957319 + - type: nauc_ndcg_at_3_std + value: -12.397871226017621 + - type: nauc_ndcg_at_5_diff1 + value: 96.99223706029623 + - type: nauc_ndcg_at_5_max + value: 90.7297142361567 + - type: nauc_ndcg_at_5_std + value: -9.78987977360544 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: 95.64367251079331 + - type: nauc_precision_at_100_max + value: 95.64367251079331 + - type: nauc_precision_at_100_std + value: -7.9737174815413105 + - type: nauc_precision_at_10_diff1 + value: 99.27394541846478 + - type: nauc_precision_at_10_max + value: 96.07116729984303 + - type: nauc_precision_at_10_std + value: 2.384368710766077 + - type: nauc_precision_at_1_diff1 + value: 96.33579234749389 + - type: nauc_precision_at_1_max + value: 88.14831631499773 + - type: nauc_precision_at_1_std + value: -11.043584429943797 + - type: nauc_precision_at_20_diff1 + value: 98.91091812769709 + - type: nauc_precision_at_20_max + value: 98.91091812769709 + - type: nauc_precision_at_20_std + value: 8.043758582511515 + - type: nauc_precision_at_3_diff1 + value: 97.6238213695205 + - type: nauc_precision_at_3_max + value: 94.67950559666251 + - type: nauc_precision_at_3_std + value: -12.047680267059418 + - type: nauc_precision_at_5_diff1 + value: 98.49204048450248 + - type: nauc_precision_at_5_max + value: 95.77207938439504 + - type: nauc_precision_at_5_std + value: 1.5409353000901833 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 95.64367251078416 + - type: nauc_recall_at_100_max + value: 95.64367251078416 + - type: nauc_recall_at_100_std + value: -7.973717481540185 + - type: nauc_recall_at_10_diff1 + value: 99.27394541846407 + - type: nauc_recall_at_10_max + value: 96.07116729984456 + - type: nauc_recall_at_10_std + value: 2.3843687107667484 + - type: nauc_recall_at_1_diff1 + value: 96.33579234749389 + - type: nauc_recall_at_1_max + value: 88.14831631499773 + - type: nauc_recall_at_1_std + value: -11.043584429943797 + - type: nauc_recall_at_20_diff1 + value: 98.91091812769824 + - type: nauc_recall_at_20_max + value: 98.91091812769824 + - type: nauc_recall_at_20_std + value: 8.043758582515963 + - type: nauc_recall_at_3_diff1 + value: 97.62382136952051 + - type: nauc_recall_at_3_max + value: 94.67950559666313 + - type: nauc_recall_at_3_std + value: -12.047680267059349 + - type: nauc_recall_at_5_diff1 + value: 98.49204048450328 + - type: nauc_recall_at_5_max + value: 95.77207938439567 + - type: nauc_recall_at_5_std + value: 1.54093530009189 + - type: ndcg_at_1 + value: 94.985 + - type: ndcg_at_10 + value: 97.16799999999999 + - type: ndcg_at_100 + value: 97.333 + - type: ndcg_at_1000 + value: 97.351 + - type: ndcg_at_20 + value: 97.243 + - type: ndcg_at_3 + value: 96.665 + - type: ndcg_at_5 + value: 97.038 + - type: precision_at_1 + value: 94.985 + - type: precision_at_10 + value: 9.91 + - type: precision_at_100 + value: 0.9979999999999999 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.97 + - type: precision_at_3 + value: 32.598 + - type: precision_at_5 + value: 19.739 + - type: recall_at_1 + value: 94.985 + - type: recall_at_10 + value: 99.09700000000001 + - type: recall_at_100 + value: 99.85000000000001 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.398 + - type: recall_at_3 + value: 97.79299999999999 + - type: recall_at_5 + value: 98.696 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SyntheticText2SQL (default) + revision: 686b87296c3a0191b5d9415a00526c62db9fce09 + split: test + type: CoIR-Retrieval/synthetic-text2sql + metrics: + - type: main_score + value: 62.871 + - type: map_at_1 + value: 5.931 + - type: map_at_10 + value: 50.172000000000004 + - type: map_at_100 + value: 50.221000000000004 + - type: map_at_1000 + value: 50.221000000000004 + - type: map_at_20 + value: 50.217 + - type: map_at_3 + value: 49.126 + - type: map_at_5 + value: 49.966 + - type: mrr_at_1 + value: 46.84669287301316 + - type: mrr_at_10 + value: 71.28829558371541 + - type: mrr_at_100 + value: 71.33501465192626 + - type: mrr_at_1000 + value: 71.33501465192626 + - type: mrr_at_20 + value: 71.33145566423184 + - type: mrr_at_3 + value: 70.51501167891556 + - type: mrr_at_5 + value: 71.10978180368052 + - type: nauc_map_at_1000_diff1 + value: 16.85059525629914 + - type: nauc_map_at_1000_max + value: 7.2095269963540725 + - type: nauc_map_at_1000_std + value: -31.394151564475674 + - type: nauc_map_at_100_diff1 + value: 16.85059525629914 + - type: nauc_map_at_100_max + value: 7.2095269963540725 + - type: nauc_map_at_100_std + value: -31.394151564475674 + - type: nauc_map_at_10_diff1 + value: 16.005030500086086 + - type: nauc_map_at_10_max + value: 7.264507684893849 + - type: nauc_map_at_10_std + value: -31.715470923775076 + - type: nauc_map_at_1_diff1 + value: 88.69496772604496 + - type: nauc_map_at_1_max + value: 2.919830321081601 + - type: nauc_map_at_1_std + value: -12.316988109913101 + - type: nauc_map_at_20_diff1 + value: 16.782304707510036 + - type: nauc_map_at_20_max + value: 7.245475061050962 + - type: nauc_map_at_20_std + value: -31.399548485799105 + - type: nauc_map_at_3_diff1 + value: 0.7297119245486146 + - type: nauc_map_at_3_max + value: 8.421968652977062 + - type: nauc_map_at_3_std + value: -36.37536179543213 + - type: nauc_map_at_5_diff1 + value: 12.348535589464996 + - type: nauc_map_at_5_max + value: 7.675248350287251 + - type: nauc_map_at_5_std + value: -33.03849075726066 + - type: nauc_mrr_at_1000_diff1 + value: -19.145171066076582 + - type: nauc_mrr_at_1000_max + value: 3.456240689954698 + - type: nauc_mrr_at_1000_std + value: -16.911698657686483 + - type: nauc_mrr_at_100_diff1 + value: -19.145171066076582 + - type: nauc_mrr_at_100_max + value: 3.456240689954698 + - type: nauc_mrr_at_100_std + value: -16.911698657686483 + - type: nauc_mrr_at_10_diff1 + value: -19.383555832224395 + - type: nauc_mrr_at_10_max + value: 3.4956712263663245 + - type: nauc_mrr_at_10_std + value: -17.088724852125562 + - type: nauc_mrr_at_1_diff1 + value: -7.248689506871619 + - type: nauc_mrr_at_1_max + value: 1.6279165510593367 + - type: nauc_mrr_at_1_std + value: -10.76401320944665 + - type: nauc_mrr_at_20_diff1 + value: -19.166540939871457 + - type: nauc_mrr_at_20_max + value: 3.471598334930003 + - type: nauc_mrr_at_20_std + value: -16.91659546762138 + - type: nauc_mrr_at_3_diff1 + value: -23.156601373830686 + - type: nauc_mrr_at_3_max + value: 4.041242636869845 + - type: nauc_mrr_at_3_std + value: -19.05754313395974 + - type: nauc_mrr_at_5_diff1 + value: -20.312683178010765 + - type: nauc_mrr_at_5_max + value: 3.617044817759521 + - type: nauc_mrr_at_5_std + value: -17.611736247571123 + - type: nauc_ndcg_at_1000_diff1 + value: 12.200580013571008 + - type: nauc_ndcg_at_1000_max + value: 7.5728136486801105 + - type: nauc_ndcg_at_1000_std + value: -32.773584410427645 + - type: nauc_ndcg_at_100_diff1 + value: 12.200580013571008 + - type: nauc_ndcg_at_100_max + value: 7.5728136486801105 + - type: nauc_ndcg_at_100_std + value: -32.773584410427645 + - type: nauc_ndcg_at_10_diff1 + value: 8.346563719126793 + - type: nauc_ndcg_at_10_max + value: 7.984058762338721 + - type: nauc_ndcg_at_10_std + value: -34.20079429928775 + - type: nauc_ndcg_at_1_diff1 + value: 88.69496772604496 + - type: nauc_ndcg_at_1_max + value: 2.919830321081601 + - type: nauc_ndcg_at_1_std + value: -12.316988109913101 + - type: nauc_ndcg_at_20_diff1 + value: 11.618780757172203 + - type: nauc_ndcg_at_20_max + value: 7.879670436948672 + - type: nauc_ndcg_at_20_std + value: -32.832497615096266 + - type: nauc_ndcg_at_3_diff1 + value: -21.635858761258074 + - type: nauc_ndcg_at_3_max + value: 10.142935236930697 + - type: nauc_ndcg_at_3_std + value: -43.48822107044396 + - type: nauc_ndcg_at_5_diff1 + value: -1.4973636275297837 + - type: nauc_ndcg_at_5_max + value: 8.981298991414445 + - type: nauc_ndcg_at_5_std + value: -37.87476407099174 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: .nan + - type: nauc_precision_at_100_max + value: .nan + - type: nauc_precision_at_100_std + value: .nan + - type: nauc_precision_at_10_diff1 + value: -121.58511191301635 + - type: nauc_precision_at_10_max + value: 25.248667404560425 + - type: nauc_precision_at_10_std + value: -81.22406477400408 + - type: nauc_precision_at_1_diff1 + value: 88.69496772604496 + - type: nauc_precision_at_1_max + value: 2.919830321081601 + - type: nauc_precision_at_1_std + value: -12.316988109913101 + - type: nauc_precision_at_20_diff1 + value: -140.40859643368495 + - type: nauc_precision_at_20_max + value: 87.46836627148437 + - type: nauc_precision_at_20_std + value: -50.273460814697444 + - type: nauc_precision_at_3_diff1 + value: -116.53772258202089 + - type: nauc_precision_at_3_max + value: 17.67074016899205 + - type: nauc_precision_at_3_std + value: -74.36661207778327 + - type: nauc_precision_at_5_diff1 + value: -123.86337036708947 + - type: nauc_precision_at_5_max + value: 21.909069776986158 + - type: nauc_precision_at_5_std + value: -83.69114367867371 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: .nan + - type: nauc_recall_at_100_max + value: .nan + - type: nauc_recall_at_100_std + value: .nan + - type: nauc_recall_at_10_diff1 + value: -121.58511191302262 + - type: nauc_recall_at_10_max + value: 25.24866740455998 + - type: nauc_recall_at_10_std + value: -81.22406477400794 + - type: nauc_recall_at_1_diff1 + value: 88.69496772604496 + - type: nauc_recall_at_1_max + value: 2.919830321081601 + - type: nauc_recall_at_1_std + value: -12.316988109913101 + - type: nauc_recall_at_20_diff1 + value: -140.40859643373693 + - type: nauc_recall_at_20_max + value: 87.46836627148096 + - type: nauc_recall_at_20_std + value: -50.27346081473573 + - type: nauc_recall_at_3_diff1 + value: -116.53772258202 + - type: nauc_recall_at_3_max + value: 17.670740168992516 + - type: nauc_recall_at_3_std + value: -74.3666120777825 + - type: nauc_recall_at_5_diff1 + value: -123.86337036708981 + - type: nauc_recall_at_5_max + value: 21.909069776986765 + - type: nauc_recall_at_5_std + value: -83.69114367867463 + - type: ndcg_at_1 + value: 5.931 + - type: ndcg_at_10 + value: 62.871 + - type: ndcg_at_100 + value: 63.05 + - type: ndcg_at_1000 + value: 63.05 + - type: ndcg_at_20 + value: 63.027 + - type: ndcg_at_3 + value: 60.89 + - type: ndcg_at_5 + value: 62.382000000000005 + - type: precision_at_1 + value: 5.931 + - type: precision_at_10 + value: 9.927999999999999 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.994 + - type: precision_at_3 + value: 31.408 + - type: precision_at_5 + value: 19.559 + - type: recall_at_1 + value: 5.931 + - type: recall_at_10 + value: 99.28200000000001 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.88 + - type: recall_at_3 + value: 94.223 + - type: recall_at_5 + value: 97.795 + task: + type: Retrieval +tags: +- mteb +--- +# voyage-code-3 + +This repo contains the tokenizer of the `voyage-code-3` embedding model. `voyage-code-3` is optimized for code retrieval, outperforming OpenAI-v3-large and CodeSage-large by an average of 13.80% and 16.81% on a suite of 238 code retrieval datasets, respectively. By supporting smaller dimensions with Matryoshka learning and quantized formats like int8 and binary, voyage-code-3 can also dramatically reduce storage and search costs with minimal impact on retrieval quality. Please refer to [our blogpost](https://blog.voyageai.com/2024/12/04/voyage-code-3) for more details about this model. + +You can use `voyage-code-3` via the [Voyage API](https://docs.voyageai.com/docs/embeddings), [AWS SageMaker](https://aws.amazon.com/marketplace/pp/prodview-d5nri3kbddsrw) or on-prem deplotment. + +## Voyage API +The following code snippet shows the usage of the Voyage API. Install the Voyage AI Python SDK via: + +``` +pip install -U voyageai +``` + +Voyage AI utilizes API keys to monitor usage and manage permissions. To obtain your key, please sign in with your Voyage AI account and click the "Create new API key" button in the dashboard. We recommend setting the API key as an environment variable. For example, in MacOS or Linux, type the following command in the terminal, replacing with your actual API key: + +```python +import voyageai + +vo = voyageai.Client() +# This will automatically use the environment variable VOYAGE_API_KEY. +# Alternatively, you can use vo = voyageai.Client(api_key="") + +result = vo.embed( + texts=["hello world"], + model="voyage-code-3", + input_type="document", + output_dimension=2048, + output_dtype="float" +) +``` +## AWS SageMaker + +The embedding model can be privately deployed in your AWS Cloud using our [AWS SageMaker marketplace offering](https://aws.amazon.com/marketplace/pp/prodview-d5nri3kbddsrw). It runs privately in your VPC. + +## On-prem Deployment + +Want to run `voyage-code-3` on your own hardware? Feel free to reach out to contact@voyageai.com to learn more.