--- library_name: sentence-transformers pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - transformers - sentence-embedding - mteb - mteb model-index: - name: bilingual-embedding-base results: - task: type: Clustering dataset: type: lyon-nlp/alloprof name: MTEB AlloProfClusteringP2P config: default split: test revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b metrics: - type: v_measure value: 64.71347977413274 - type: v_measures value: [0.6276406100583347, 0.6380984027637511, 0.6322659733117469, 0.652473343931587, 0.6411502615838207] - task: type: Clustering dataset: type: lyon-nlp/alloprof name: MTEB AlloProfClusteringS2S config: default split: test revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b metrics: - type: v_measure value: 45.56886694296516 - type: v_measures value: [0.4359417286866465, 0.4218168523895086, 0.419693469863105, 0.4981808644314091, 0.4546120704986696] - task: type: Reranking dataset: type: lyon-nlp/mteb-fr-reranking-alloprof-s2p name: MTEB AlloprofReranking config: default split: test revision: 65393d0d7a08a10b4e348135e824f385d420b0fd metrics: - type: map value: 70.45979049191963 - type: mrr value: 71.61250582558874 - type: nAUC_map_diff1 value: 53.81172404763562 - type: nAUC_map_max value: 11.344605627946006 - type: nAUC_mrr_diff1 value: 53.73572411920392 - type: nAUC_mrr_max value: 11.900785644756448 - task: type: Retrieval dataset: type: lyon-nlp/alloprof name: MTEB AlloprofRetrieval config: default split: test revision: fcf295ea64c750f41fadbaa37b9b861558e1bfbd metrics: - type: map_at_1 value: 24.956999999999997 - type: map_at_10 value: 35.519 - type: map_at_100 value: 36.602000000000004 - type: map_at_1000 value: 36.667 - type: map_at_20 value: 36.199 - type: map_at_3 value: 32.397999999999996 - type: map_at_5 value: 34.121 - type: mrr_at_1 value: 24.956822107081177 - type: mrr_at_10 value: 35.5187515420676 - type: mrr_at_100 value: 36.60233410773095 - type: mrr_at_1000 value: 36.6673425266803 - type: mrr_at_20 value: 36.19886807640962 - type: mrr_at_3 value: 32.39781232009212 - type: mrr_at_5 value: 34.120610247553294 - type: nauc_map_at_1000_diff1 value: 35.47549954610535 - type: nauc_map_at_1000_max value: 32.90078190148035 - type: nauc_map_at_100_diff1 value: 35.450977423889455 - type: nauc_map_at_100_max value: 32.94163322687819 - type: nauc_map_at_10_diff1 value: 35.33565787959566 - type: nauc_map_at_10_max value: 32.64293321320758 - type: nauc_map_at_1_diff1 value: 41.36063963274824 - type: nauc_map_at_1_max value: 29.231198539339754 - type: nauc_map_at_20_diff1 value: 35.43536321023612 - type: nauc_map_at_20_max value: 32.88345868813372 - type: nauc_map_at_3_diff1 value: 35.686505961183194 - type: nauc_map_at_3_max value: 32.00806639149525 - type: nauc_map_at_5_diff1 value: 35.34083004527267 - type: nauc_map_at_5_max value: 32.34459863266794 - type: nauc_mrr_at_1000_diff1 value: 35.47549954610535 - type: nauc_mrr_at_1000_max value: 32.90078190148035 - type: nauc_mrr_at_100_diff1 value: 35.450977423889455 - type: nauc_mrr_at_100_max value: 32.94163322687819 - type: nauc_mrr_at_10_diff1 value: 35.33565787959566 - type: nauc_mrr_at_10_max value: 32.64293321320758 - type: nauc_mrr_at_1_diff1 value: 41.36063963274824 - type: nauc_mrr_at_1_max value: 29.231198539339754 - type: nauc_mrr_at_20_diff1 value: 35.43536321023612 - type: nauc_mrr_at_20_max value: 32.88345868813372 - type: nauc_mrr_at_3_diff1 value: 35.686505961183194 - type: nauc_mrr_at_3_max value: 32.00806639149525 - type: nauc_mrr_at_5_diff1 value: 35.34083004527267 - type: nauc_mrr_at_5_max value: 32.34459863266794 - type: nauc_ndcg_at_1000_diff1 value: 34.4325639393338 - type: nauc_ndcg_at_1000_max value: 34.44147157978809 - type: nauc_ndcg_at_100_diff1 value: 33.88653913244061 - type: nauc_ndcg_at_100_max value: 35.59739772150559 - type: nauc_ndcg_at_10_diff1 value: 33.51057326557921 - type: nauc_ndcg_at_10_max value: 34.18300339583124 - type: nauc_ndcg_at_1_diff1 value: 41.36063963274824 - type: nauc_ndcg_at_1_max value: 29.231198539339754 - type: nauc_ndcg_at_20_diff1 value: 33.804940889687714 - type: nauc_ndcg_at_20_max value: 35.13066047975507 - type: nauc_ndcg_at_3_diff1 value: 34.167179830350406 - type: nauc_ndcg_at_3_max value: 32.94370726673571 - type: nauc_ndcg_at_5_diff1 value: 33.52138730976869 - type: nauc_ndcg_at_5_max value: 33.48723912009657 - type: nauc_precision_at_1000_diff1 value: 22.67827963096167 - type: nauc_precision_at_1000_max value: 67.17958434804059 - type: nauc_precision_at_100_diff1 value: 24.83989747177019 - type: nauc_precision_at_100_max value: 56.86858468981111 - type: nauc_precision_at_10_diff1 value: 27.973507280697774 - type: nauc_precision_at_10_max value: 39.14247264250278 - type: nauc_precision_at_1_diff1 value: 41.36063963274824 - type: nauc_precision_at_1_max value: 29.231198539339754 - type: nauc_precision_at_20_diff1 value: 28.244479044808852 - type: nauc_precision_at_20_max value: 44.5538108461975 - type: nauc_precision_at_3_diff1 value: 30.09786756288918 - type: nauc_precision_at_3_max value: 35.51252285506707 - type: nauc_precision_at_5_diff1 value: 28.444019525872083 - type: nauc_precision_at_5_max value: 36.71010320598834 - type: nauc_recall_at_1000_diff1 value: 22.67827963096142 - type: nauc_recall_at_1000_max value: 67.17958434804105 - type: nauc_recall_at_100_diff1 value: 24.83989747177035 - type: nauc_recall_at_100_max value: 56.868584689811186 - type: nauc_recall_at_10_diff1 value: 27.973507280697774 - type: nauc_recall_at_10_max value: 39.14247264250283 - type: nauc_recall_at_1_diff1 value: 41.36063963274824 - type: nauc_recall_at_1_max value: 29.231198539339754 - type: nauc_recall_at_20_diff1 value: 28.244479044808823 - type: nauc_recall_at_20_max value: 44.553810846197486 - type: nauc_recall_at_3_diff1 value: 30.097867562889213 - type: nauc_recall_at_3_max value: 35.51252285506708 - type: nauc_recall_at_5_diff1 value: 28.444019525872054 - type: nauc_recall_at_5_max value: 36.71010320598833 - type: ndcg_at_1 value: 24.956999999999997 - type: ndcg_at_10 value: 41.260999999999996 - type: ndcg_at_100 value: 46.628 - type: ndcg_at_1000 value: 48.339999999999996 - type: ndcg_at_20 value: 43.71 - type: ndcg_at_3 value: 34.771 - type: ndcg_at_5 value: 37.862 - type: precision_at_1 value: 24.956999999999997 - type: precision_at_10 value: 5.963 - type: precision_at_100 value: 0.8500000000000001 - type: precision_at_1000 value: 0.098 - type: precision_at_20 value: 3.463 - type: precision_at_3 value: 13.874 - type: precision_at_5 value: 9.818999999999999 - type: recall_at_1 value: 24.956999999999997 - type: recall_at_10 value: 59.629 - type: recall_at_100 value: 84.974 - type: recall_at_1000 value: 98.402 - type: recall_at_20 value: 69.257 - type: recall_at_3 value: 41.623 - type: recall_at_5 value: 49.092999999999996 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (fr) config: fr split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 40.354 - type: f1 value: 38.96127209677864 - type: f1_weighted value: 38.96127209677863 - task: type: Retrieval dataset: type: maastrichtlawtech/bsard name: MTEB BSARDRetrieval config: default split: test revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59 metrics: - type: map_at_1 value: 5.405 - type: map_at_10 value: 9.715 - type: map_at_100 value: 10.852 - type: map_at_1000 value: 10.986 - type: map_at_20 value: 10.324 - type: map_at_3 value: 8.183 - type: map_at_5 value: 9.129 - type: mrr_at_1 value: 5.405405405405405 - type: mrr_at_10 value: 9.715250965250963 - type: mrr_at_100 value: 10.852488713991455 - type: mrr_at_1000 value: 10.986128810251913 - type: mrr_at_20 value: 10.324372731493472 - type: mrr_at_3 value: 8.183183183183184 - type: mrr_at_5 value: 9.12912912912913 - type: nauc_map_at_1000_diff1 value: 27.989147369852507 - type: nauc_map_at_1000_max value: 13.956676274011212 - type: nauc_map_at_100_diff1 value: 27.97931691839789 - type: nauc_map_at_100_max value: 13.885054931557967 - type: nauc_map_at_10_diff1 value: 28.39484917161119 - type: nauc_map_at_10_max value: 12.859153645903026 - type: nauc_map_at_1_diff1 value: 40.37326707123 - type: nauc_map_at_1_max value: 19.53225223707901 - type: nauc_map_at_20_diff1 value: 27.969553431914463 - type: nauc_map_at_20_max value: 13.993555023169318 - type: nauc_map_at_3_diff1 value: 31.130088078198526 - type: nauc_map_at_3_max value: 11.59332560179606 - type: nauc_map_at_5_diff1 value: 28.51357198186801 - type: nauc_map_at_5_max value: 12.629395513278464 - type: nauc_mrr_at_1000_diff1 value: 27.989147369852507 - type: nauc_mrr_at_1000_max value: 13.956676274011212 - type: nauc_mrr_at_100_diff1 value: 27.97931691839789 - type: nauc_mrr_at_100_max value: 13.885054931557967 - type: nauc_mrr_at_10_diff1 value: 28.39484917161119 - type: nauc_mrr_at_10_max value: 12.859153645903026 - type: nauc_mrr_at_1_diff1 value: 40.37326707123 - type: nauc_mrr_at_1_max value: 19.53225223707901 - type: nauc_mrr_at_20_diff1 value: 27.969553431914463 - type: nauc_mrr_at_20_max value: 13.993555023169318 - type: nauc_mrr_at_3_diff1 value: 31.130088078198526 - type: nauc_mrr_at_3_max value: 11.59332560179606 - type: nauc_mrr_at_5_diff1 value: 28.51357198186801 - type: nauc_mrr_at_5_max value: 12.629395513278464 - type: nauc_ndcg_at_1000_diff1 value: 23.808642111518818 - type: nauc_ndcg_at_1000_max value: 16.627566094152723 - type: nauc_ndcg_at_100_diff1 value: 24.15753828838301 - type: nauc_ndcg_at_100_max value: 15.687219543962943 - type: nauc_ndcg_at_10_diff1 value: 25.37534871155007 - type: nauc_ndcg_at_10_max value: 12.129150838266701 - type: nauc_ndcg_at_1_diff1 value: 40.37326707123 - type: nauc_ndcg_at_1_max value: 19.53225223707901 - type: nauc_ndcg_at_20_diff1 value: 24.446964236295017 - type: nauc_ndcg_at_20_max value: 15.718345074121808 - type: nauc_ndcg_at_3_diff1 value: 29.121862169292296 - type: nauc_ndcg_at_3_max value: 9.481317109332187 - type: nauc_ndcg_at_5_diff1 value: 25.17815567035254 - type: nauc_ndcg_at_5_max value: 11.187689974665869 - type: nauc_precision_at_1000_diff1 value: 10.131813130975075 - type: nauc_precision_at_1000_max value: 30.42776840947068 - type: nauc_precision_at_100_diff1 value: 17.8621079715631 - type: nauc_precision_at_100_max value: 20.81364427037172 - type: nauc_precision_at_10_diff1 value: 20.451314767316635 - type: nauc_precision_at_10_max value: 11.400840293532708 - type: nauc_precision_at_1_diff1 value: 40.37326707123 - type: nauc_precision_at_1_max value: 19.53225223707901 - type: nauc_precision_at_20_diff1 value: 19.17170129809007 - type: nauc_precision_at_20_max value: 20.190489899791007 - type: nauc_precision_at_3_diff1 value: 24.905337103765735 - type: nauc_precision_at_3_max value: 4.960457155777402 - type: nauc_precision_at_5_diff1 value: 18.725510703139488 - type: nauc_precision_at_5_max value: 8.555964364751343 - type: nauc_recall_at_1000_diff1 value: 10.131813130975143 - type: nauc_recall_at_1000_max value: 30.427768409470673 - type: nauc_recall_at_100_diff1 value: 17.862107971563105 - type: nauc_recall_at_100_max value: 20.813644270371707 - type: nauc_recall_at_10_diff1 value: 20.45131476731657 - type: nauc_recall_at_10_max value: 11.400840293532651 - type: nauc_recall_at_1_diff1 value: 40.37326707123 - type: nauc_recall_at_1_max value: 19.53225223707901 - type: nauc_recall_at_20_diff1 value: 19.171701298090017 - type: nauc_recall_at_20_max value: 20.19048989979099 - type: nauc_recall_at_3_diff1 value: 24.905337103765717 - type: nauc_recall_at_3_max value: 4.9604571557773935 - type: nauc_recall_at_5_diff1 value: 18.72551070313952 - type: nauc_recall_at_5_max value: 8.55596436475138 - type: ndcg_at_1 value: 5.405 - type: ndcg_at_10 value: 12.217 - type: ndcg_at_100 value: 18.512999999999998 - type: ndcg_at_1000 value: 22.002 - type: ndcg_at_20 value: 14.551 - type: ndcg_at_3 value: 9.089 - type: ndcg_at_5 value: 10.776 - type: precision_at_1 value: 5.405 - type: precision_at_10 value: 2.027 - type: precision_at_100 value: 0.514 - type: precision_at_1000 value: 0.079 - type: precision_at_20 value: 1.486 - type: precision_at_3 value: 3.904 - type: precision_at_5 value: 3.1530000000000005 - type: recall_at_1 value: 5.405 - type: recall_at_10 value: 20.27 - type: recall_at_100 value: 51.351 - type: recall_at_1000 value: 78.82900000000001 - type: recall_at_20 value: 29.73 - type: recall_at_3 value: 11.712 - type: recall_at_5 value: 15.766 - task: type: Clustering dataset: type: lyon-nlp/clustering-hal-s2s name: MTEB HALClusteringS2S config: default split: test revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915 metrics: - type: v_measure value: 25.366610995664384 - type: v_measures value: [0.2853826102888054, 0.27554329622230517, 0.2659387504290534, 0.272421074779971, 0.23780511730712292] - task: type: Clustering dataset: type: reciTAL/mlsum name: MTEB MLSUMClusteringP2P config: fr split: test revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7 metrics: - type: v_measure value: 44.22735442638259 - type: v_measures value: [0.446401643334608, 0.45940182243882194, 0.4535014442023603, 0.4315840671354274, 0.40219963696623473] - task: type: Clustering dataset: type: reciTAL/mlsum name: MTEB MLSUMClusteringS2S config: fr split: test revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7 metrics: - type: v_measure value: 44.57521454657146 - type: v_measures value: [0.4501823523199927, 0.45800459800343646, 0.4535621236055794, 0.4353844290212204, 0.40539074932285035] - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (fr) config: fr split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 86.82743501409333 - type: f1 value: 87.07777165714171 - type: f1_weighted value: 86.70555382175719 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (fr) config: fr split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 59.129345443156886 - type: f1 value: 40.374753319633946 - type: f1_weighted value: 61.735222244513906 - task: type: Classification dataset: type: mteb/masakhanews name: MTEB MasakhaNEWSClassification (fra) config: fra split: test revision: 18193f187b92da67168c655c9973a165ed9593dd metrics: - type: accuracy value: 77.44075829383887 - type: f1 value: 73.59099795975669 - type: f1_weighted value: 77.53638597300765 - task: type: Clustering dataset: type: masakhane/masakhanews name: MTEB MasakhaNEWSClusteringP2P (fra) config: fra split: test revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60 metrics: - type: v_measure value: 61.57579457743856 - type: v_measures value: [1.0, 0.19333147584460073, 0.1918473189330316, 0.9180419176341081, 0.7755690164601873] - task: type: Clustering dataset: type: masakhane/masakhanews name: MTEB MasakhaNEWSClusteringS2S (fra) config: fra split: test revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60 metrics: - type: v_measure value: 46.09882977546196 - type: v_measures value: [1.0, 0.0, 0.4881603944307428, 0.1341399033659235, 0.6826411909764316] - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fr) config: fr split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 64.98655010087424 - type: f1 value: 62.07892690857404 - type: f1_weighted value: 64.04916798028313 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fr) config: fr split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 71.72494956287828 - type: f1 value: 70.7613627592262 - type: f1_weighted value: 71.59023734198762 - task: type: Retrieval dataset: type: jinaai/mintakaqa name: MTEB MintakaRetrieval (fr) config: fr split: test revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e metrics: - type: map_at_1 value: 15.643 - type: map_at_10 value: 23.263 - type: map_at_100 value: 24.282 - type: map_at_1000 value: 24.386 - type: map_at_20 value: 23.810000000000002 - type: map_at_3 value: 20.857 - type: map_at_5 value: 22.127 - type: mrr_at_1 value: 15.642915642915641 - type: mrr_at_10 value: 23.26313326313326 - type: mrr_at_100 value: 24.28242740941141 - type: mrr_at_1000 value: 24.38579869502159 - type: mrr_at_20 value: 23.809503335080308 - type: mrr_at_3 value: 20.857220857220817 - type: mrr_at_5 value: 22.126672126672087 - type: nauc_map_at_1000_diff1 value: 22.791507278613405 - type: nauc_map_at_1000_max value: 34.49738411163668 - type: nauc_map_at_100_diff1 value: 22.753348275392053 - type: nauc_map_at_100_max value: 34.503329867035156 - type: nauc_map_at_10_diff1 value: 22.949396903228543 - type: nauc_map_at_10_max value: 34.59635858547973 - type: nauc_map_at_1_diff1 value: 29.68671981414948 - type: nauc_map_at_1_max value: 32.40476256531568 - type: nauc_map_at_20_diff1 value: 22.839943329322214 - type: nauc_map_at_20_max value: 34.5512812666387 - type: nauc_map_at_3_diff1 value: 24.531567517208703 - type: nauc_map_at_3_max value: 34.63194411307717 - type: nauc_map_at_5_diff1 value: 23.785928597532532 - type: nauc_map_at_5_max value: 34.49711117096583 - type: nauc_mrr_at_1000_diff1 value: 22.791507278613405 - type: nauc_mrr_at_1000_max value: 34.49738411163668 - type: nauc_mrr_at_100_diff1 value: 22.753348275392053 - type: nauc_mrr_at_100_max value: 34.503329867035156 - type: nauc_mrr_at_10_diff1 value: 22.949396903228543 - type: nauc_mrr_at_10_max value: 34.59635858547973 - type: nauc_mrr_at_1_diff1 value: 29.68671981414948 - type: nauc_mrr_at_1_max value: 32.40476256531568 - type: nauc_mrr_at_20_diff1 value: 22.839943329322214 - type: nauc_mrr_at_20_max value: 34.5512812666387 - type: nauc_mrr_at_3_diff1 value: 24.531567517208703 - type: nauc_mrr_at_3_max value: 34.63194411307717 - type: nauc_mrr_at_5_diff1 value: 23.785928597532532 - type: nauc_mrr_at_5_max value: 34.49711117096583 - type: nauc_ndcg_at_1000_diff1 value: 20.36281863938966 - type: nauc_ndcg_at_1000_max value: 34.39195365895218 - type: nauc_ndcg_at_100_diff1 value: 19.226093736031558 - type: nauc_ndcg_at_100_max value: 34.598730272263424 - type: nauc_ndcg_at_10_diff1 value: 20.175947236861635 - type: nauc_ndcg_at_10_max value: 35.06289054957336 - type: nauc_ndcg_at_1_diff1 value: 29.68671981414948 - type: nauc_ndcg_at_1_max value: 32.40476256531568 - type: nauc_ndcg_at_20_diff1 value: 19.76832604541867 - type: nauc_ndcg_at_20_max value: 34.88676463098282 - type: nauc_ndcg_at_3_diff1 value: 23.165684344826936 - type: nauc_ndcg_at_3_max value: 35.058555779606806 - type: nauc_ndcg_at_5_diff1 value: 22.008327776501197 - type: nauc_ndcg_at_5_max value: 34.81586957799795 - type: nauc_precision_at_1000_diff1 value: -0.17116637411109903 - type: nauc_precision_at_1000_max value: 22.79067675552558 - type: nauc_precision_at_100_diff1 value: 6.442518254575118 - type: nauc_precision_at_100_max value: 34.22531735083736 - type: nauc_precision_at_10_diff1 value: 13.139149147711764 - type: nauc_precision_at_10_max value: 36.17629609592965 - type: nauc_precision_at_1_diff1 value: 29.68671981414948 - type: nauc_precision_at_1_max value: 32.40476256531568 - type: nauc_precision_at_20_diff1 value: 11.54118462661454 - type: nauc_precision_at_20_max value: 35.56967988819633 - type: nauc_precision_at_3_diff1 value: 19.81196490555276 - type: nauc_precision_at_3_max value: 36.06171743276854 - type: nauc_precision_at_5_diff1 value: 17.72477129059423 - type: nauc_precision_at_5_max value: 35.49631000776651 - type: nauc_recall_at_1000_diff1 value: -0.17116637411164665 - type: nauc_recall_at_1000_max value: 22.790676755525375 - type: nauc_recall_at_100_diff1 value: 6.442518254575061 - type: nauc_recall_at_100_max value: 34.22531735083735 - type: nauc_recall_at_10_diff1 value: 13.139149147711779 - type: nauc_recall_at_10_max value: 36.17629609592963 - type: nauc_recall_at_1_diff1 value: 29.68671981414948 - type: nauc_recall_at_1_max value: 32.40476256531568 - type: nauc_recall_at_20_diff1 value: 11.541184626614548 - type: nauc_recall_at_20_max value: 35.56967988819635 - type: nauc_recall_at_3_diff1 value: 19.811964905552767 - type: nauc_recall_at_3_max value: 36.06171743276855 - type: nauc_recall_at_5_diff1 value: 17.72477129059422 - type: nauc_recall_at_5_max value: 35.49631000776651 - type: ndcg_at_1 value: 15.643 - type: ndcg_at_10 value: 27.596999999999998 - type: ndcg_at_100 value: 33.036 - type: ndcg_at_1000 value: 36.348 - type: ndcg_at_20 value: 29.586000000000002 - type: ndcg_at_3 value: 22.567 - type: ndcg_at_5 value: 24.858 - type: precision_at_1 value: 15.643 - type: precision_at_10 value: 4.152 - type: precision_at_100 value: 0.681 - type: precision_at_1000 value: 0.095 - type: precision_at_20 value: 2.469 - type: precision_at_3 value: 9.173 - type: precision_at_5 value: 6.618 - type: recall_at_1 value: 15.643 - type: recall_at_10 value: 41.522999999999996 - type: recall_at_100 value: 68.059 - type: recall_at_1000 value: 95.332 - type: recall_at_20 value: 49.386 - type: recall_at_3 value: 27.517999999999997 - type: recall_at_5 value: 33.088 - task: type: PairClassification dataset: type: GEM/opusparcus name: MTEB OpusparcusPC (fr) config: fr split: test revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a metrics: - type: cos_sim_accuracy value: 82.08446866485014 - type: cos_sim_ap value: 93.63779278112501 - type: cos_sim_f1 value: 87.31307284129282 - type: cos_sim_precision value: 84.8968105065666 - type: cos_sim_recall value: 89.87090367428004 - type: dot_accuracy value: 82.08446866485014 - type: dot_ap value: 93.63779278112501 - type: dot_f1 value: 87.31307284129282 - type: dot_precision value: 84.8968105065666 - type: dot_recall value: 89.87090367428004 - type: euclidean_accuracy value: 82.08446866485014 - type: euclidean_ap value: 93.63779278112501 - type: euclidean_f1 value: 87.31307284129282 - type: euclidean_precision value: 84.8968105065666 - type: euclidean_recall value: 89.87090367428004 - type: manhattan_accuracy value: 82.15258855585831 - type: manhattan_ap value: 93.6291276133462 - type: manhattan_f1 value: 87.36740597878496 - type: manhattan_precision value: 84.91096532333646 - type: manhattan_recall value: 89.97020854021847 - type: max_accuracy value: 82.15258855585831 - type: max_ap value: 93.63779278112501 - type: max_f1 value: 87.36740597878496 - task: type: PairClassification dataset: type: google-research-datasets/paws-x name: MTEB PawsX (fr) config: fr split: test revision: 8a04d940a42cd40658986fdd8e3da561533a3646 metrics: - type: cos_sim_accuracy value: 60.85 - type: cos_sim_ap value: 60.37878110577144 - type: cos_sim_f1 value: 62.67978803936413 - type: cos_sim_precision value: 47.61357101782634 - type: cos_sim_recall value: 91.69435215946844 - type: dot_accuracy value: 60.85 - type: dot_ap value: 60.282876235548 - type: dot_f1 value: 62.67978803936413 - type: dot_precision value: 47.61357101782634 - type: dot_recall value: 91.69435215946844 - type: euclidean_accuracy value: 60.85 - type: euclidean_ap value: 60.37878110577144 - type: euclidean_f1 value: 62.67978803936413 - type: euclidean_precision value: 47.61357101782634 - type: euclidean_recall value: 91.69435215946844 - type: manhattan_accuracy value: 61.0 - type: manhattan_ap value: 60.32592616688752 - type: manhattan_f1 value: 62.70871985157699 - type: manhattan_precision value: 47.154017857142854 - type: manhattan_recall value: 93.57696566998892 - type: max_accuracy value: 61.0 - type: max_ap value: 60.37878110577144 - type: max_f1 value: 62.70871985157699 - task: type: STS dataset: type: Lajavaness/SICK-fr name: MTEB SICKFr config: default split: test revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a metrics: - type: cos_sim_pearson value: 82.41331853949572 - type: cos_sim_spearman value: 75.4815169821101 - type: euclidean_pearson value: 79.81380205322172 - type: euclidean_spearman value: 75.48076002322844 - type: manhattan_pearson value: 79.86035354304643 - type: manhattan_spearman value: 75.51770448845875 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr) config: fr split: test revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 metrics: - type: cos_sim_pearson value: 78.63723398201921 - type: cos_sim_spearman value: 81.1413239539243 - type: euclidean_pearson value: 79.65250044199216 - type: euclidean_spearman value: 81.1413239539243 - type: manhattan_pearson value: 79.4690995487022 - type: manhattan_spearman value: 80.61411938663267 - task: type: STS dataset: type: mteb/stsb_multi_mt name: MTEB STSBenchmarkMultilingualSTS (fr) config: fr split: test revision: 29afa2569dcedaaa2fe6a3dcfebab33d28b82e8c metrics: - type: cos_sim_pearson value: 83.34057641357742 - type: cos_sim_spearman value: 83.74771008807434 - type: euclidean_pearson value: 83.22245971062527 - type: euclidean_spearman value: 83.74894925253341 - type: manhattan_pearson value: 83.06789257151542 - type: manhattan_spearman value: 83.52796149940158 - task: type: Summarization dataset: type: lyon-nlp/summarization-summeval-fr-p2p name: MTEB SummEvalFr config: default split: test revision: b385812de6a9577b6f4d0f88c6a6e35395a94054 metrics: - type: cos_sim_pearson value: 29.949370780736455 - type: cos_sim_spearman value: 30.16472987232583 - type: dot_pearson value: 29.94937008799093 - type: dot_spearman value: 30.16472987232583 - task: type: Reranking dataset: type: lyon-nlp/mteb-fr-reranking-syntec-s2p name: MTEB SyntecReranking config: default split: test revision: daf0863838cd9e3ba50544cdce3ac2b338a1b0ad metrics: - type: map value: 83.82619047619046 - type: mrr value: 83.82619047619046 - type: nAUC_map_diff1 value: 56.291594426865686 - type: nAUC_map_max value: 9.006252496368798 - type: nAUC_mrr_diff1 value: 56.291594426865686 - type: nAUC_mrr_max value: 9.006252496368798 - task: type: Retrieval dataset: type: lyon-nlp/mteb-fr-retrieval-syntec-s2p name: MTEB SyntecRetrieval config: default split: test revision: 19661ccdca4dfc2d15122d776b61685f48c68ca9 metrics: - type: map_at_1 value: 62.0 - type: map_at_10 value: 74.369 - type: map_at_100 value: 74.626 - type: map_at_1000 value: 74.626 - type: map_at_20 value: 74.626 - type: map_at_3 value: 72.333 - type: map_at_5 value: 73.533 - type: mrr_at_1 value: 62.0 - type: mrr_at_10 value: 74.36944444444444 - type: mrr_at_100 value: 74.62553418803418 - type: mrr_at_1000 value: 74.62553418803418 - type: mrr_at_20 value: 74.62553418803418 - type: mrr_at_3 value: 72.33333333333333 - type: mrr_at_5 value: 73.53333333333333 - type: nauc_map_at_1000_diff1 value: 59.84829658893367 - type: nauc_map_at_1000_max value: 25.92680320484747 - type: nauc_map_at_100_diff1 value: 59.84829658893367 - type: nauc_map_at_100_max value: 25.92680320484747 - type: nauc_map_at_10_diff1 value: 60.14734510056334 - type: nauc_map_at_10_max value: 26.4859845903765 - type: nauc_map_at_1_diff1 value: 61.15589330215675 - type: nauc_map_at_1_max value: 25.502261329723385 - type: nauc_map_at_20_diff1 value: 59.84829658893367 - type: nauc_map_at_20_max value: 25.92680320484747 - type: nauc_map_at_3_diff1 value: 60.664125602445864 - type: nauc_map_at_3_max value: 24.598480701707597 - type: nauc_map_at_5_diff1 value: 59.96933672856163 - type: nauc_map_at_5_max value: 26.87050847362874 - type: nauc_mrr_at_1000_diff1 value: 59.84829658893367 - type: nauc_mrr_at_1000_max value: 25.92680320484747 - type: nauc_mrr_at_100_diff1 value: 59.84829658893367 - type: nauc_mrr_at_100_max value: 25.92680320484747 - type: nauc_mrr_at_10_diff1 value: 60.14734510056334 - type: nauc_mrr_at_10_max value: 26.4859845903765 - type: nauc_mrr_at_1_diff1 value: 61.15589330215675 - type: nauc_mrr_at_1_max value: 25.502261329723385 - type: nauc_mrr_at_20_diff1 value: 59.84829658893367 - type: nauc_mrr_at_20_max value: 25.92680320484747 - type: nauc_mrr_at_3_diff1 value: 60.664125602445864 - type: nauc_mrr_at_3_max value: 24.598480701707597 - type: nauc_mrr_at_5_diff1 value: 59.96933672856163 - type: nauc_mrr_at_5_max value: 26.87050847362874 - type: nauc_ndcg_at_1000_diff1 value: 60.04965149586935 - type: nauc_ndcg_at_1000_max value: 26.34908378184259 - type: nauc_ndcg_at_100_diff1 value: 60.04965149586935 - type: nauc_ndcg_at_100_max value: 26.34908378184259 - type: nauc_ndcg_at_10_diff1 value: 61.517946155950945 - type: nauc_ndcg_at_10_max value: 29.005286712766882 - type: nauc_ndcg_at_1_diff1 value: 61.15589330215675 - type: nauc_ndcg_at_1_max value: 25.502261329723385 - type: nauc_ndcg_at_20_diff1 value: 60.04965149586935 - type: nauc_ndcg_at_20_max value: 26.34908378184259 - type: nauc_ndcg_at_3_diff1 value: 62.138907107716314 - type: nauc_ndcg_at_3_max value: 24.66905359423539 - type: nauc_ndcg_at_5_diff1 value: 60.81453858060155 - type: nauc_ndcg_at_5_max value: 29.652166992041785 - type: nauc_precision_at_1000_diff1 value: nan - type: nauc_precision_at_1000_max value: nan - type: nauc_precision_at_100_diff1 value: nan - type: nauc_precision_at_100_max value: nan - type: nauc_precision_at_10_diff1 value: 86.11111111111084 - type: nauc_precision_at_10_max value: 72.80578898225937 - type: nauc_precision_at_1_diff1 value: 61.15589330215675 - type: nauc_precision_at_1_max value: 25.502261329723385 - type: nauc_precision_at_20_diff1 value: 100.0 - type: nauc_precision_at_20_max value: 100.0 - type: nauc_precision_at_3_diff1 value: 69.83729254799863 - type: nauc_precision_at_3_max value: 25.2229092092417 - type: nauc_precision_at_5_diff1 value: 66.95144724556499 - type: nauc_precision_at_5_max value: 49.640522875817005 - type: nauc_recall_at_1000_diff1 value: nan - type: nauc_recall_at_1000_max value: nan - type: nauc_recall_at_100_diff1 value: nan - type: nauc_recall_at_100_max value: nan - type: nauc_recall_at_10_diff1 value: 86.11111111111124 - type: nauc_recall_at_10_max value: 72.8057889822595 - type: nauc_recall_at_1_diff1 value: 61.15589330215675 - type: nauc_recall_at_1_max value: 25.502261329723385 - type: nauc_recall_at_20_diff1 value: nan - type: nauc_recall_at_20_max value: nan - type: nauc_recall_at_3_diff1 value: 69.83729254799871 - type: nauc_recall_at_3_max value: 25.22290920924175 - type: nauc_recall_at_5_diff1 value: 66.95144724556485 - type: nauc_recall_at_5_max value: 49.64052287581686 - type: ndcg_at_1 value: 62.0 - type: ndcg_at_10 value: 79.679 - type: ndcg_at_100 value: 80.664 - type: ndcg_at_1000 value: 80.664 - type: ndcg_at_20 value: 80.664 - type: ndcg_at_3 value: 75.595 - type: ndcg_at_5 value: 77.704 - type: precision_at_1 value: 62.0 - type: precision_at_10 value: 9.6 - type: precision_at_100 value: 1.0 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 5.0 - type: precision_at_3 value: 28.333000000000002 - type: precision_at_5 value: 18.0 - type: recall_at_1 value: 62.0 - type: recall_at_10 value: 96.0 - type: recall_at_100 value: 100.0 - type: recall_at_1000 value: 100.0 - type: recall_at_20 value: 100.0 - type: recall_at_3 value: 85.0 - type: recall_at_5 value: 90.0 - task: type: Retrieval dataset: type: jinaai/xpqa name: MTEB XPQARetrieval (fr) config: fr split: test revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f metrics: - type: map_at_1 value: 37.597 - type: map_at_10 value: 57.888 - type: map_at_100 value: 59.277 - type: map_at_1000 value: 59.352000000000004 - type: map_at_20 value: 58.717 - type: map_at_3 value: 51.957 - type: map_at_5 value: 55.772999999999996 - type: mrr_at_1 value: 59.813084112149525 - type: mrr_at_10 value: 67.10031364570752 - type: mrr_at_100 value: 67.66545041236235 - type: mrr_at_1000 value: 67.6835494818551 - type: mrr_at_20 value: 67.49393600922676 - type: mrr_at_3 value: 65.17578994214506 - type: mrr_at_5 value: 66.18380062305292 - type: nauc_map_at_1000_diff1 value: 48.73204134706741 - type: nauc_map_at_1000_max value: 48.167144249595445 - type: nauc_map_at_100_diff1 value: 48.678967361326784 - type: nauc_map_at_100_max value: 48.150876586146055 - type: nauc_map_at_10_diff1 value: 48.49819455901337 - type: nauc_map_at_10_max value: 47.419803565618324 - type: nauc_map_at_1_diff1 value: 58.76698174343172 - type: nauc_map_at_1_max value: 32.18211527608151 - type: nauc_map_at_20_diff1 value: 48.40197639399712 - type: nauc_map_at_20_max value: 47.81717012073508 - type: nauc_map_at_3_diff1 value: 50.73932443832007 - type: nauc_map_at_3_max value: 42.24066337784361 - type: nauc_map_at_5_diff1 value: 49.46975560894318 - type: nauc_map_at_5_max value: 45.80443987601635 - type: nauc_mrr_at_1000_diff1 value: 57.835936362591525 - type: nauc_mrr_at_1000_max value: 58.573550643855896 - type: nauc_mrr_at_100_diff1 value: 57.82678446452145 - type: nauc_mrr_at_100_max value: 58.5764861850602 - type: nauc_mrr_at_10_diff1 value: 57.76879895710224 - type: nauc_mrr_at_10_max value: 58.64968053975452 - type: nauc_mrr_at_1_diff1 value: 59.856533986276574 - type: nauc_mrr_at_1_max value: 58.730867813105725 - type: nauc_mrr_at_20_diff1 value: 57.795349563018746 - type: nauc_mrr_at_20_max value: 58.516079879105256 - type: nauc_mrr_at_3_diff1 value: 57.76489841294852 - type: nauc_mrr_at_3_max value: 58.58658769928032 - type: nauc_mrr_at_5_diff1 value: 57.694777424382515 - type: nauc_mrr_at_5_max value: 58.56316065282314 - type: nauc_ndcg_at_1000_diff1 value: 50.69160796479768 - type: nauc_ndcg_at_1000_max value: 52.199577971960785 - type: nauc_ndcg_at_100_diff1 value: 49.86099689038677 - type: nauc_ndcg_at_100_max value: 52.151790872414125 - type: nauc_ndcg_at_10_diff1 value: 48.72866627696869 - type: nauc_ndcg_at_10_max value: 50.06953156417251 - type: nauc_ndcg_at_1_diff1 value: 59.856533986276574 - type: nauc_ndcg_at_1_max value: 58.730867813105725 - type: nauc_ndcg_at_20_diff1 value: 48.588995750275565 - type: nauc_ndcg_at_20_max value: 50.49019072586609 - type: nauc_ndcg_at_3_diff1 value: 50.45588351336764 - type: nauc_ndcg_at_3_max value: 48.984274202014916 - type: nauc_ndcg_at_5_diff1 value: 50.006960016438505 - type: nauc_ndcg_at_5_max value: 48.303806275166735 - type: nauc_precision_at_1000_diff1 value: -16.164680055782775 - type: nauc_precision_at_1000_max value: 16.124503094722208 - type: nauc_precision_at_100_diff1 value: -13.36404724754223 - type: nauc_precision_at_100_max value: 23.29679787821716 - type: nauc_precision_at_10_diff1 value: -3.7402851742774788 - type: nauc_precision_at_10_max value: 32.14138887961609 - type: nauc_precision_at_1_diff1 value: 59.856533986276574 - type: nauc_precision_at_1_max value: 58.730867813105725 - type: nauc_precision_at_20_diff1 value: -8.971829372748005 - type: nauc_precision_at_20_max value: 27.87664599166348 - type: nauc_precision_at_3_diff1 value: 11.427083566719917 - type: nauc_precision_at_3_max value: 39.94075723264166 - type: nauc_precision_at_5_diff1 value: 3.5428167965724233 - type: nauc_precision_at_5_max value: 35.65970898316591 - type: nauc_recall_at_1000_diff1 value: 42.34179228230109 - type: nauc_recall_at_1000_max value: 69.84746848920715 - type: nauc_recall_at_100_diff1 value: 30.47165287340051 - type: nauc_recall_at_100_max value: 47.90447689942959 - type: nauc_recall_at_10_diff1 value: 36.65046615294308 - type: nauc_recall_at_10_max value: 41.81737771961874 - type: nauc_recall_at_1_diff1 value: 58.76698174343172 - type: nauc_recall_at_1_max value: 32.18211527608151 - type: nauc_recall_at_20_diff1 value: 33.33461286517975 - type: nauc_recall_at_20_max value: 40.033064434150155 - type: nauc_recall_at_3_diff1 value: 45.94521561906703 - type: nauc_recall_at_3_max value: 37.46948921295656 - type: nauc_recall_at_5_diff1 value: 42.66425368847329 - type: nauc_recall_at_5_max value: 40.64657773118315 - type: ndcg_at_1 value: 59.813 - type: ndcg_at_10 value: 64.208 - type: ndcg_at_100 value: 69.002 - type: ndcg_at_1000 value: 70.23700000000001 - type: ndcg_at_20 value: 66.29899999999999 - type: ndcg_at_3 value: 59.099999999999994 - type: ndcg_at_5 value: 60.763999999999996 - type: precision_at_1 value: 59.813 - type: precision_at_10 value: 14.766000000000002 - type: precision_at_100 value: 1.8870000000000002 - type: precision_at_1000 value: 0.20600000000000002 - type: precision_at_20 value: 8.117 - type: precision_at_3 value: 35.781 - type: precision_at_5 value: 25.688 - type: recall_at_1 value: 37.597 - type: recall_at_10 value: 72.919 - type: recall_at_100 value: 91.526 - type: recall_at_1000 value: 99.421 - type: recall_at_20 value: 79.64 - type: recall_at_3 value: 56.836 - type: recall_at_5 value: 64.364 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 77.41791044776119 - type: ap value: 41.50313649654625 - type: ap_weighted value: 41.50313649654625 - type: f1 value: 71.69242302886543 - type: f1_weighted value: 79.4051024757404 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification config: default split: test revision: e2d317d38cd51312af73b3d32a06d1a08b442046 metrics: - type: accuracy value: 89.51284999999999 - type: ap value: 85.78657792017795 - type: ap_weighted value: 85.78657792017795 - type: f1 value: 89.48680521118494 - type: f1_weighted value: 89.48680521118494 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) config: en split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 46.096000000000004 - type: f1 value: 44.75176024696358 - type: f1_weighted value: 44.75176024696358 - task: type: Retrieval dataset: type: mteb/arguana name: MTEB ArguAna config: default split: test revision: c22ab2a51041ffd869aaddef7af8d8215647e41a metrics: - type: map_at_1 value: 27.738000000000003 - type: map_at_10 value: 42.992999999999995 - type: map_at_100 value: 44.025999999999996 - type: map_at_1000 value: 44.037 - type: map_at_20 value: 43.838 - type: map_at_3 value: 37.707 - type: map_at_5 value: 40.851 - type: mrr_at_1 value: 28.02275960170697 - type: mrr_at_10 value: 43.129445234708385 - type: mrr_at_100 value: 44.14842568065762 - type: mrr_at_1000 value: 44.15944224906741 - type: mrr_at_20 value: 43.9604803125253 - type: mrr_at_3 value: 37.79042200094829 - type: mrr_at_5 value: 40.937648174490285 - type: nauc_map_at_1000_diff1 value: 7.819134326851951 - type: nauc_map_at_1000_max value: -5.363208805959698 - type: nauc_map_at_100_diff1 value: 7.827581788081036 - type: nauc_map_at_100_max value: -5.343758430445495 - type: nauc_map_at_10_diff1 value: 7.788779237579437 - type: nauc_map_at_10_max value: -5.212502829074271 - type: nauc_map_at_1_diff1 value: 8.676582434483777 - type: nauc_map_at_1_max value: -9.028400174777206 - type: nauc_map_at_20_diff1 value: 7.843647300531639 - type: nauc_map_at_20_max value: -5.171628909368918 - type: nauc_map_at_3_diff1 value: 7.580280985257144 - type: nauc_map_at_3_max value: -6.508188744493309 - type: nauc_map_at_5_diff1 value: 7.461255107228164 - type: nauc_map_at_5_max value: -5.150343094282473 - type: nauc_mrr_at_1000_diff1 value: 6.911455227219296 - type: nauc_mrr_at_1000_max value: -5.866891482528428 - type: nauc_mrr_at_100_diff1 value: 6.920215930616841 - type: nauc_mrr_at_100_max value: -5.847302915503507 - type: nauc_mrr_at_10_diff1 value: 6.89637694159382 - type: nauc_mrr_at_10_max value: -5.7267515522713985 - type: nauc_mrr_at_1_diff1 value: 7.784148492423344 - type: nauc_mrr_at_1_max value: -9.261177003486702 - type: nauc_mrr_at_20_diff1 value: 6.941892867312857 - type: nauc_mrr_at_20_max value: -5.672366841914248 - type: nauc_mrr_at_3_diff1 value: 6.655156176778528 - type: nauc_mrr_at_3_max value: -7.08130312539927 - type: nauc_mrr_at_5_diff1 value: 6.474672753475723 - type: nauc_mrr_at_5_max value: -5.7771100192539455 - type: nauc_ndcg_at_1000_diff1 value: 7.82257902156683 - type: nauc_ndcg_at_1000_max value: -4.284182821313092 - type: nauc_ndcg_at_100_diff1 value: 7.982663390398444 - type: nauc_ndcg_at_100_max value: -3.829916815248607 - type: nauc_ndcg_at_10_diff1 value: 7.940691283173258 - type: nauc_ndcg_at_10_max value: -2.759653230358356 - type: nauc_ndcg_at_1_diff1 value: 8.676582434483777 - type: nauc_ndcg_at_1_max value: -9.028400174777206 - type: nauc_ndcg_at_20_diff1 value: 8.216154898914834 - type: nauc_ndcg_at_20_max value: -2.5281250069887644 - type: nauc_ndcg_at_3_diff1 value: 7.397192517335338 - type: nauc_ndcg_at_3_max value: -5.506567290248059 - type: nauc_ndcg_at_5_diff1 value: 7.224597118942196 - type: nauc_ndcg_at_5_max value: -2.8306642442626635 - type: nauc_precision_at_1000_diff1 value: -28.424446830488918 - type: nauc_precision_at_1000_max value: 15.2996011292648 - type: nauc_precision_at_100_diff1 value: 13.817321356393311 - type: nauc_precision_at_100_max value: 35.62226207701842 - type: nauc_precision_at_10_diff1 value: 9.129870306379654 - type: nauc_precision_at_10_max value: 10.371721765203494 - type: nauc_precision_at_1_diff1 value: 8.676582434483777 - type: nauc_precision_at_1_max value: -9.028400174777206 - type: nauc_precision_at_20_diff1 value: 14.378273723959786 - type: nauc_precision_at_20_max value: 29.411949476113275 - type: nauc_precision_at_3_diff1 value: 6.924474252093112 - type: nauc_precision_at_3_max value: -2.589098226592611 - type: nauc_precision_at_5_diff1 value: 6.523266339112172 - type: nauc_precision_at_5_max value: 5.387183619653168 - type: nauc_recall_at_1000_diff1 value: -28.424446830487355 - type: nauc_recall_at_1000_max value: 15.299601129265062 - type: nauc_recall_at_100_diff1 value: 13.817321356392966 - type: nauc_recall_at_100_max value: 35.62226207701895 - type: nauc_recall_at_10_diff1 value: 9.129870306379667 - type: nauc_recall_at_10_max value: 10.371721765203487 - type: nauc_recall_at_1_diff1 value: 8.676582434483777 - type: nauc_recall_at_1_max value: -9.028400174777206 - type: nauc_recall_at_20_diff1 value: 14.378273723959634 - type: nauc_recall_at_20_max value: 29.411949476113342 - type: nauc_recall_at_3_diff1 value: 6.924474252093177 - type: nauc_recall_at_3_max value: -2.589098226592573 - type: nauc_recall_at_5_diff1 value: 6.5232663391122045 - type: nauc_recall_at_5_max value: 5.38718361965314 - type: ndcg_at_1 value: 27.738000000000003 - type: ndcg_at_10 value: 51.867 - type: ndcg_at_100 value: 56.010000000000005 - type: ndcg_at_1000 value: 56.25599999999999 - type: ndcg_at_20 value: 54.872 - type: ndcg_at_3 value: 41.041 - type: ndcg_at_5 value: 46.7 - type: precision_at_1 value: 27.738000000000003 - type: precision_at_10 value: 8.036999999999999 - type: precision_at_100 value: 0.9780000000000001 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 4.605 - type: precision_at_3 value: 16.904 - type: precision_at_5 value: 12.888 - type: recall_at_1 value: 27.738000000000003 - type: recall_at_10 value: 80.36999999999999 - type: recall_at_100 value: 97.795 - type: recall_at_1000 value: 99.644 - type: recall_at_20 value: 92.105 - type: recall_at_3 value: 50.711 - type: recall_at_5 value: 64.43799999999999 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P config: default split: test revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d metrics: - type: v_measure value: 44.25989960324142 - type: v_measures value: [0.4355201593114565, 0.43400969136008477, 0.4621501390953121, 0.4368716556310582, 0.46142659337392417] - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S config: default split: test revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 metrics: - type: v_measure value: 36.00514206925075 - type: v_measures value: [0.38194577110919753, 0.3641834175557571, 0.3594175019099288, 0.35392802860850453, 0.36877397641003723] - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions config: default split: test revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 metrics: - type: map value: 59.2761176425466 - type: mrr value: 73.2264872708086 - type: nAUC_map_diff1 value: 7.0068523566823835 - type: nAUC_map_max value: 21.51466581178718 - type: nAUC_mrr_diff1 value: 16.619643437951563 - type: nAUC_mrr_max value: 32.302108972143714 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES config: default split: test revision: d3fb88f8f02e40887cd149695127462bbcf29b4a metrics: - type: cos_sim_pearson value: 87.84186171115664 - type: cos_sim_spearman value: 87.13150342270266 - type: euclidean_pearson value: 86.93703588206957 - type: euclidean_spearman value: 87.13150342270266 - type: manhattan_pearson value: 86.78921932668315 - type: manhattan_spearman value: 86.73631369514506 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification config: default split: test revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 metrics: - type: accuracy value: 78.46103896103897 - type: f1 value: 77.72189862815705 - type: f1_weighted value: 77.72189862815705 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P config: default split: test revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 metrics: - type: v_measure value: 36.380764168313746 - type: v_measures value: [0.37382307264671194, 0.36301531251106245, 0.3793738872327412, 0.35770158136125185, 0.3555364559997305] - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S config: default split: test revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 metrics: - type: v_measure value: 31.538017334550577 - type: v_measures value: [0.30690945509666684, 0.31808990661791575, 0.31514802723414864, 0.3159451399149567, 0.3171249261521223] - task: type: Retrieval dataset: type: mteb/cqadupstack-android name: MTEB CQADupstackAndroidRetrieval config: default split: test revision: f46a197baaae43b4f621051089b82a364682dfeb metrics: - type: map_at_1 value: 32.464 - type: map_at_10 value: 42.91 - type: map_at_100 value: 44.253 - type: map_at_1000 value: 44.378 - type: map_at_20 value: 43.608000000000004 - type: map_at_3 value: 39.21 - type: map_at_5 value: 41.243 - type: mrr_at_1 value: 40.05722460658083 - type: mrr_at_10 value: 49.06050593818831 - type: mrr_at_100 value: 49.75550694943466 - type: mrr_at_1000 value: 49.80211926259268 - type: mrr_at_20 value: 49.466746239607275 - type: mrr_at_3 value: 46.256556986170715 - type: mrr_at_5 value: 48.05197901764422 - type: nauc_map_at_1000_diff1 value: 54.452516877596366 - type: nauc_map_at_1000_max value: 42.37830524228681 - type: nauc_map_at_100_diff1 value: 54.455120677207944 - type: nauc_map_at_100_max value: 42.35365218254892 - type: nauc_map_at_10_diff1 value: 54.411476414486735 - type: nauc_map_at_10_max value: 41.799626583253286 - type: nauc_map_at_1_diff1 value: 56.844131977626574 - type: nauc_map_at_1_max value: 39.50909847050082 - type: nauc_map_at_20_diff1 value: 54.474114038349 - type: nauc_map_at_20_max value: 41.95011324047797 - type: nauc_map_at_3_diff1 value: 54.94801787552844 - type: nauc_map_at_3_max value: 40.93589777136806 - type: nauc_map_at_5_diff1 value: 54.51000824622664 - type: nauc_map_at_5_max value: 41.942988719761516 - type: nauc_mrr_at_1000_diff1 value: 54.73077714753202 - type: nauc_mrr_at_1000_max value: 44.79399425080995 - type: nauc_mrr_at_100_diff1 value: 54.72534798039509 - type: nauc_mrr_at_100_max value: 44.784407506240214 - type: nauc_mrr_at_10_diff1 value: 54.60375865684595 - type: nauc_mrr_at_10_max value: 44.557364932034154 - type: nauc_mrr_at_1_diff1 value: 57.776230077156235 - type: nauc_mrr_at_1_max value: 46.471474228551344 - type: nauc_mrr_at_20_diff1 value: 54.70304613396688 - type: nauc_mrr_at_20_max value: 44.66734285522038 - type: nauc_mrr_at_3_diff1 value: 55.3933778729286 - type: nauc_mrr_at_3_max value: 44.696022513397615 - type: nauc_mrr_at_5_diff1 value: 54.71004626344184 - type: nauc_mrr_at_5_max value: 44.965744262291004 - type: nauc_ndcg_at_1000_diff1 value: 53.26255776636424 - type: nauc_ndcg_at_1000_max value: 43.58732162869603 - type: nauc_ndcg_at_100_diff1 value: 52.99574164185918 - type: nauc_ndcg_at_100_max value: 43.35845196216733 - type: nauc_ndcg_at_10_diff1 value: 52.660009377886766 - type: nauc_ndcg_at_10_max value: 41.93002636395951 - type: nauc_ndcg_at_1_diff1 value: 57.776230077156235 - type: nauc_ndcg_at_1_max value: 46.471474228551344 - type: nauc_ndcg_at_20_diff1 value: 52.925283797059066 - type: nauc_ndcg_at_20_max value: 41.96413582256493 - type: nauc_ndcg_at_3_diff1 value: 53.86123945887276 - type: nauc_ndcg_at_3_max value: 42.5192092394243 - type: nauc_ndcg_at_5_diff1 value: 53.02739573145395 - type: nauc_ndcg_at_5_max value: 42.86255544029417 - type: nauc_precision_at_1000_diff1 value: -14.01444880844629 - type: nauc_precision_at_1000_max value: -3.2426344768649065 - type: nauc_precision_at_100_diff1 value: -2.665892254195872 - type: nauc_precision_at_100_max value: 11.174117765610346 - type: nauc_precision_at_10_diff1 value: 17.760500367118006 - type: nauc_precision_at_10_max value: 25.819513742057314 - type: nauc_precision_at_1_diff1 value: 57.776230077156235 - type: nauc_precision_at_1_max value: 46.471474228551344 - type: nauc_precision_at_20_diff1 value: 10.720961197841934 - type: nauc_precision_at_20_max value: 20.104016753843656 - type: nauc_precision_at_3_diff1 value: 38.00682945145973 - type: nauc_precision_at_3_max value: 39.91552880079303 - type: nauc_precision_at_5_diff1 value: 29.195186929472932 - type: nauc_precision_at_5_max value: 36.060771452887344 - type: nauc_recall_at_1000_diff1 value: 34.24272104794043 - type: nauc_recall_at_1000_max value: 56.510230841605825 - type: nauc_recall_at_100_diff1 value: 39.42477153393114 - type: nauc_recall_at_100_max value: 41.44622822460404 - type: nauc_recall_at_10_diff1 value: 42.98765339932259 - type: nauc_recall_at_10_max value: 34.34817326152696 - type: nauc_recall_at_1_diff1 value: 56.844131977626574 - type: nauc_recall_at_1_max value: 39.50909847050082 - type: nauc_recall_at_20_diff1 value: 42.49763875384549 - type: nauc_recall_at_20_max value: 34.211320392734436 - type: nauc_recall_at_3_diff1 value: 49.54385449610674 - type: nauc_recall_at_3_max value: 37.050307605313755 - type: nauc_recall_at_5_diff1 value: 45.79369932076432 - type: nauc_recall_at_5_max value: 38.06187420388636 - type: ndcg_at_1 value: 40.056999999999995 - type: ndcg_at_10 value: 49.228 - type: ndcg_at_100 value: 54.162 - type: ndcg_at_1000 value: 56.205000000000005 - type: ndcg_at_20 value: 51.034 - type: ndcg_at_3 value: 43.94 - type: ndcg_at_5 value: 46.504 - type: precision_at_1 value: 40.056999999999995 - type: precision_at_10 value: 9.528 - type: precision_at_100 value: 1.472 - type: precision_at_1000 value: 0.192 - type: precision_at_20 value: 5.494000000000001 - type: precision_at_3 value: 20.887 - type: precision_at_5 value: 15.193000000000001 - type: recall_at_1 value: 32.464 - type: recall_at_10 value: 60.831 - type: recall_at_100 value: 81.85900000000001 - type: recall_at_1000 value: 95.15 - type: recall_at_20 value: 67.657 - type: recall_at_3 value: 45.489000000000004 - type: recall_at_5 value: 52.839000000000006 - task: type: Retrieval dataset: type: mteb/cqadupstack-english name: MTEB CQADupstackEnglishRetrieval config: default split: test revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 metrics: - type: map_at_1 value: 27.204 - type: map_at_10 value: 36.565999999999995 - type: map_at_100 value: 37.706 - type: map_at_1000 value: 37.827 - type: map_at_20 value: 37.124 - type: map_at_3 value: 33.629999999999995 - type: map_at_5 value: 35.345 - type: mrr_at_1 value: 33.88535031847134 - type: mrr_at_10 value: 42.00993327267216 - type: mrr_at_100 value: 42.69324389051348 - type: mrr_at_1000 value: 42.742001707727674 - type: mrr_at_20 value: 42.383657095896574 - type: mrr_at_3 value: 39.61783439490446 - type: mrr_at_5 value: 41.02547770700639 - type: nauc_map_at_1000_diff1 value: 51.2434173008466 - type: nauc_map_at_1000_max value: 42.89551471851398 - type: nauc_map_at_100_diff1 value: 51.25803151639599 - type: nauc_map_at_100_max value: 42.84835058448656 - type: nauc_map_at_10_diff1 value: 51.392478648037475 - type: nauc_map_at_10_max value: 42.520436932382275 - type: nauc_map_at_1_diff1 value: 58.311410816649435 - type: nauc_map_at_1_max value: 38.78632287563295 - type: nauc_map_at_20_diff1 value: 51.2961213976287 - type: nauc_map_at_20_max value: 42.5853171119173 - type: nauc_map_at_3_diff1 value: 52.988281664633696 - type: nauc_map_at_3_max value: 41.255285265369714 - type: nauc_map_at_5_diff1 value: 51.90811230897579 - type: nauc_map_at_5_max value: 42.25025338907201 - type: nauc_mrr_at_1000_diff1 value: 49.968831428382956 - type: nauc_mrr_at_1000_max value: 44.835796996668066 - type: nauc_mrr_at_100_diff1 value: 49.96427305359113 - type: nauc_mrr_at_100_max value: 44.83231841203824 - type: nauc_mrr_at_10_diff1 value: 49.94029375694121 - type: nauc_mrr_at_10_max value: 44.88750685573963 - type: nauc_mrr_at_1_diff1 value: 56.40160291266728 - type: nauc_mrr_at_1_max value: 45.557456279594454 - type: nauc_mrr_at_20_diff1 value: 49.948501688516814 - type: nauc_mrr_at_20_max value: 44.86680460708911 - type: nauc_mrr_at_3_diff1 value: 50.70198183915884 - type: nauc_mrr_at_3_max value: 44.69764399444459 - type: nauc_mrr_at_5_diff1 value: 50.171095819167164 - type: nauc_mrr_at_5_max value: 44.81579964530846 - type: nauc_ndcg_at_1000_diff1 value: 48.15540595763338 - type: nauc_ndcg_at_1000_max value: 43.71537508558133 - type: nauc_ndcg_at_100_diff1 value: 48.272332711054126 - type: nauc_ndcg_at_100_max value: 43.6322161272428 - type: nauc_ndcg_at_10_diff1 value: 48.18746476758319 - type: nauc_ndcg_at_10_max value: 43.36809828847912 - type: nauc_ndcg_at_1_diff1 value: 56.40160291266728 - type: nauc_ndcg_at_1_max value: 45.557456279594454 - type: nauc_ndcg_at_20_diff1 value: 48.17338076733571 - type: nauc_ndcg_at_20_max value: 43.20321056220099 - type: nauc_ndcg_at_3_diff1 value: 50.105385726603714 - type: nauc_ndcg_at_3_max value: 43.120332022171915 - type: nauc_ndcg_at_5_diff1 value: 48.8630439359171 - type: nauc_ndcg_at_5_max value: 43.370066685949446 - type: nauc_precision_at_1000_diff1 value: -11.350357006977681 - type: nauc_precision_at_1000_max value: 12.582096763421525 - type: nauc_precision_at_100_diff1 value: -2.958036676255927 - type: nauc_precision_at_100_max value: 22.696487876398155 - type: nauc_precision_at_10_diff1 value: 16.189353401009978 - type: nauc_precision_at_10_max value: 38.13442143062204 - type: nauc_precision_at_1_diff1 value: 56.40160291266728 - type: nauc_precision_at_1_max value: 45.557456279594454 - type: nauc_precision_at_20_diff1 value: 8.797894276494759 - type: nauc_precision_at_20_max value: 32.53273333292465 - type: nauc_precision_at_3_diff1 value: 33.166335907173284 - type: nauc_precision_at_3_max value: 43.54274244894697 - type: nauc_precision_at_5_diff1 value: 24.890638457308835 - type: nauc_precision_at_5_max value: 41.85350446231859 - type: nauc_recall_at_1000_diff1 value: 30.966590369789042 - type: nauc_recall_at_1000_max value: 40.05217189462629 - type: nauc_recall_at_100_diff1 value: 36.50185764515 - type: nauc_recall_at_100_max value: 40.40225246071926 - type: nauc_recall_at_10_diff1 value: 39.21222014929766 - type: nauc_recall_at_10_max value: 39.510881532659106 - type: nauc_recall_at_1_diff1 value: 58.311410816649435 - type: nauc_recall_at_1_max value: 38.78632287563295 - type: nauc_recall_at_20_diff1 value: 37.90625645817586 - type: nauc_recall_at_20_max value: 38.83095083812718 - type: nauc_recall_at_3_diff1 value: 45.67833862057349 - type: nauc_recall_at_3_max value: 38.301387193837385 - type: nauc_recall_at_5_diff1 value: 41.97377098035378 - type: nauc_recall_at_5_max value: 39.44428303279494 - type: ndcg_at_1 value: 33.885 - type: ndcg_at_10 value: 41.959999999999994 - type: ndcg_at_100 value: 46.444 - type: ndcg_at_1000 value: 48.542 - type: ndcg_at_20 value: 43.516 - type: ndcg_at_3 value: 37.662 - type: ndcg_at_5 value: 39.694 - type: precision_at_1 value: 33.885 - type: precision_at_10 value: 7.911 - type: precision_at_100 value: 1.306 - type: precision_at_1000 value: 0.17600000000000002 - type: precision_at_20 value: 4.634 - type: precision_at_3 value: 18.195 - type: precision_at_5 value: 13.032 - type: recall_at_1 value: 27.204 - type: recall_at_10 value: 51.964999999999996 - type: recall_at_100 value: 70.977 - type: recall_at_1000 value: 84.48 - type: recall_at_20 value: 57.568 - type: recall_at_3 value: 39.292 - type: recall_at_5 value: 45.051 - task: type: Retrieval dataset: type: mteb/cqadupstack-gaming name: MTEB CQADupstackGamingRetrieval config: default split: test revision: 4885aa143210c98657558c04aaf3dc47cfb54340 metrics: - type: map_at_1 value: 40.092 - type: map_at_10 value: 51.94499999999999 - type: map_at_100 value: 52.952999999999996 - type: map_at_1000 value: 53.010999999999996 - type: map_at_20 value: 52.544000000000004 - type: map_at_3 value: 48.798 - type: map_at_5 value: 50.575 - type: mrr_at_1 value: 45.57993730407524 - type: mrr_at_10 value: 55.19226750261236 - type: mrr_at_100 value: 55.86648188807898 - type: mrr_at_1000 value: 55.893179256732914 - type: mrr_at_20 value: 55.605862164423684 - type: mrr_at_3 value: 52.737722048066914 - type: mrr_at_5 value: 54.142110762800485 - type: nauc_map_at_1000_diff1 value: 56.4825391687947 - type: nauc_map_at_1000_max value: 42.69718253696078 - type: nauc_map_at_100_diff1 value: 56.472594143313714 - type: nauc_map_at_100_max value: 42.68390703868357 - type: nauc_map_at_10_diff1 value: 56.60851591326905 - type: nauc_map_at_10_max value: 42.545318648853254 - type: nauc_map_at_1_diff1 value: 58.78580013920528 - type: nauc_map_at_1_max value: 37.250129371959034 - type: nauc_map_at_20_diff1 value: 56.522847775596496 - type: nauc_map_at_20_max value: 42.67436226258157 - type: nauc_map_at_3_diff1 value: 56.41058843854863 - type: nauc_map_at_3_max value: 41.1447087205128 - type: nauc_map_at_5_diff1 value: 56.615742462460375 - type: nauc_map_at_5_max value: 42.18432838422091 - type: nauc_mrr_at_1000_diff1 value: 57.02890931424712 - type: nauc_mrr_at_1000_max value: 44.542419456217395 - type: nauc_mrr_at_100_diff1 value: 57.015358825202966 - type: nauc_mrr_at_100_max value: 44.53905706111591 - type: nauc_mrr_at_10_diff1 value: 57.04585925976531 - type: nauc_mrr_at_10_max value: 44.61348989967417 - type: nauc_mrr_at_1_diff1 value: 59.97805122993276 - type: nauc_mrr_at_1_max value: 43.74889272537995 - type: nauc_mrr_at_20_diff1 value: 57.006157936095484 - type: nauc_mrr_at_20_max value: 44.59603635627128 - type: nauc_mrr_at_3_diff1 value: 56.917680357532504 - type: nauc_mrr_at_3_max value: 44.16899447567816 - type: nauc_mrr_at_5_diff1 value: 56.92085593489732 - type: nauc_mrr_at_5_max value: 44.6067245655727 - type: nauc_ndcg_at_1000_diff1 value: 55.99301213747579 - type: nauc_ndcg_at_1000_max value: 44.13571318751295 - type: nauc_ndcg_at_100_diff1 value: 55.69267672766463 - type: nauc_ndcg_at_100_max value: 44.01363451781653 - type: nauc_ndcg_at_10_diff1 value: 56.08101977835497 - type: nauc_ndcg_at_10_max value: 44.04565223998733 - type: nauc_ndcg_at_1_diff1 value: 59.97805122993276 - type: nauc_ndcg_at_1_max value: 43.74889272537995 - type: nauc_ndcg_at_20_diff1 value: 55.9395678717101 - type: nauc_ndcg_at_20_max value: 44.276016640316584 - type: nauc_ndcg_at_3_diff1 value: 55.61181442897005 - type: nauc_ndcg_at_3_max value: 42.505752873203875 - type: nauc_ndcg_at_5_diff1 value: 55.931533774058074 - type: nauc_ndcg_at_5_max value: 43.62473544458933 - type: nauc_precision_at_1000_diff1 value: -13.91854408201959 - type: nauc_precision_at_1000_max value: 8.959796412269117 - type: nauc_precision_at_100_diff1 value: -6.051946111858287 - type: nauc_precision_at_100_max value: 15.919914740220339 - type: nauc_precision_at_10_diff1 value: 19.584103811469795 - type: nauc_precision_at_10_max value: 32.679327751531886 - type: nauc_precision_at_1_diff1 value: 59.97805122993276 - type: nauc_precision_at_1_max value: 43.74889272537995 - type: nauc_precision_at_20_diff1 value: 10.708778552460565 - type: nauc_precision_at_20_max value: 27.76302369902412 - type: nauc_precision_at_3_diff1 value: 37.72812007268646 - type: nauc_precision_at_3_max value: 40.9098215392736 - type: nauc_precision_at_5_diff1 value: 29.8927353855664 - type: nauc_precision_at_5_max value: 38.77105720875548 - type: nauc_recall_at_1000_diff1 value: 49.43352356696205 - type: nauc_recall_at_1000_max value: 54.87661194579149 - type: nauc_recall_at_100_diff1 value: 46.097079017519164 - type: nauc_recall_at_100_max value: 44.71576750940437 - type: nauc_recall_at_10_diff1 value: 52.66222731151876 - type: nauc_recall_at_10_max value: 44.98829150528362 - type: nauc_recall_at_1_diff1 value: 58.78580013920528 - type: nauc_recall_at_1_max value: 37.250129371959034 - type: nauc_recall_at_20_diff1 value: 51.55229424448056 - type: nauc_recall_at_20_max value: 46.82438704609937 - type: nauc_recall_at_3_diff1 value: 52.87567308584833 - type: nauc_recall_at_3_max value: 41.13816110816809 - type: nauc_recall_at_5_diff1 value: 52.83904035972527 - type: nauc_recall_at_5_max value: 43.519187012248025 - type: ndcg_at_1 value: 45.58 - type: ndcg_at_10 value: 57.534 - type: ndcg_at_100 value: 61.6 - type: ndcg_at_1000 value: 62.742 - type: ndcg_at_20 value: 59.270999999999994 - type: ndcg_at_3 value: 52.193 - type: ndcg_at_5 value: 54.725 - type: precision_at_1 value: 45.58 - type: precision_at_10 value: 9.129 - type: precision_at_100 value: 1.203 - type: precision_at_1000 value: 0.136 - type: precision_at_20 value: 5.082 - type: precision_at_3 value: 23.051 - type: precision_at_5 value: 15.748999999999999 - type: recall_at_1 value: 40.092 - type: recall_at_10 value: 70.889 - type: recall_at_100 value: 88.45100000000001 - type: recall_at_1000 value: 96.447 - type: recall_at_20 value: 77.239 - type: recall_at_3 value: 56.489999999999995 - type: recall_at_5 value: 62.647 - task: type: Retrieval dataset: type: mteb/cqadupstack-gis name: MTEB CQADupstackGisRetrieval config: default split: test revision: 5003b3064772da1887988e05400cf3806fe491f2 metrics: - type: map_at_1 value: 20.277 - type: map_at_10 value: 27.969 - type: map_at_100 value: 28.933999999999997 - type: map_at_1000 value: 29.038000000000004 - type: map_at_20 value: 28.486 - type: map_at_3 value: 25.418000000000003 - type: map_at_5 value: 26.709 - type: mrr_at_1 value: 21.694915254237287 - type: mrr_at_10 value: 29.60447493498341 - type: mrr_at_100 value: 30.512620484407922 - type: mrr_at_1000 value: 30.591656063000244 - type: mrr_at_20 value: 30.121039448386362 - type: mrr_at_3 value: 27.19397363465162 - type: mrr_at_5 value: 28.448210922787204 - type: nauc_map_at_1000_diff1 value: 45.363822324742415 - type: nauc_map_at_1000_max value: 29.509295922731372 - type: nauc_map_at_100_diff1 value: 45.331606636295966 - type: nauc_map_at_100_max value: 29.505487961576044 - type: nauc_map_at_10_diff1 value: 45.38885724901821 - type: nauc_map_at_10_max value: 28.946567703884206 - type: nauc_map_at_1_diff1 value: 54.7423053253451 - type: nauc_map_at_1_max value: 30.202680999900373 - type: nauc_map_at_20_diff1 value: 45.43050910968979 - type: nauc_map_at_20_max value: 29.35034157464228 - type: nauc_map_at_3_diff1 value: 47.51058437623272 - type: nauc_map_at_3_max value: 29.740666890218183 - type: nauc_map_at_5_diff1 value: 45.90224481327457 - type: nauc_map_at_5_max value: 28.80931616822233 - type: nauc_mrr_at_1000_diff1 value: 45.38410126007919 - type: nauc_mrr_at_1000_max value: 31.462744626905998 - type: nauc_mrr_at_100_diff1 value: 45.36092538448443 - type: nauc_mrr_at_100_max value: 31.48088167063395 - type: nauc_mrr_at_10_diff1 value: 45.38915659941786 - type: nauc_mrr_at_10_max value: 31.07285932737546 - type: nauc_mrr_at_1_diff1 value: 54.930775252405226 - type: nauc_mrr_at_1_max value: 33.5116859460449 - type: nauc_mrr_at_20_diff1 value: 45.418208062544515 - type: nauc_mrr_at_20_max value: 31.423541265829346 - type: nauc_mrr_at_3_diff1 value: 47.521366857933685 - type: nauc_mrr_at_3_max value: 32.27219903173878 - type: nauc_mrr_at_5_diff1 value: 45.77958904462302 - type: nauc_mrr_at_5_max value: 31.03500930751467 - type: nauc_ndcg_at_1000_diff1 value: 41.635595953748044 - type: nauc_ndcg_at_1000_max value: 29.99928035763284 - type: nauc_ndcg_at_100_diff1 value: 40.88371796938874 - type: nauc_ndcg_at_100_max value: 30.281800111940075 - type: nauc_ndcg_at_10_diff1 value: 41.21530633973889 - type: nauc_ndcg_at_10_max value: 28.03405573161477 - type: nauc_ndcg_at_1_diff1 value: 54.930775252405226 - type: nauc_ndcg_at_1_max value: 33.5116859460449 - type: nauc_ndcg_at_20_diff1 value: 41.354735186387494 - type: nauc_ndcg_at_20_max value: 29.290000578859498 - type: nauc_ndcg_at_3_diff1 value: 45.31600437511932 - type: nauc_ndcg_at_3_max value: 30.107259401213447 - type: nauc_ndcg_at_5_diff1 value: 42.46548676756585 - type: nauc_ndcg_at_5_max value: 28.066140473016777 - type: nauc_precision_at_1000_diff1 value: 3.3001494044534 - type: nauc_precision_at_1000_max value: 23.86920410371473 - type: nauc_precision_at_100_diff1 value: 14.600936720175225 - type: nauc_precision_at_100_max value: 34.79005291009276 - type: nauc_precision_at_10_diff1 value: 25.191040884814313 - type: nauc_precision_at_10_max value: 27.787630029931737 - type: nauc_precision_at_1_diff1 value: 54.930775252405226 - type: nauc_precision_at_1_max value: 33.5116859460449 - type: nauc_precision_at_20_diff1 value: 23.94526878532444 - type: nauc_precision_at_20_max value: 31.64356816310904 - type: nauc_precision_at_3_diff1 value: 37.36686654535447 - type: nauc_precision_at_3_max value: 31.809307942763166 - type: nauc_precision_at_5_diff1 value: 30.658077015337877 - type: nauc_precision_at_5_max value: 27.987876687409614 - type: nauc_recall_at_1000_diff1 value: 16.313332570062347 - type: nauc_recall_at_1000_max value: 24.611193005552156 - type: nauc_recall_at_100_diff1 value: 22.517502540871675 - type: nauc_recall_at_100_max value: 30.270048758028008 - type: nauc_recall_at_10_diff1 value: 28.942161278215146 - type: nauc_recall_at_10_max value: 22.367333726084272 - type: nauc_recall_at_1_diff1 value: 54.7423053253451 - type: nauc_recall_at_1_max value: 30.202680999900373 - type: nauc_recall_at_20_diff1 value: 28.593619502228517 - type: nauc_recall_at_20_max value: 26.029280134895316 - type: nauc_recall_at_3_diff1 value: 38.74661393996696 - type: nauc_recall_at_3_max value: 27.640169897015266 - type: nauc_recall_at_5_diff1 value: 32.83863931854332 - type: nauc_recall_at_5_max value: 23.190141862761386 - type: ndcg_at_1 value: 21.695 - type: ndcg_at_10 value: 32.698 - type: ndcg_at_100 value: 37.641000000000005 - type: ndcg_at_1000 value: 40.245 - type: ndcg_at_20 value: 34.55 - type: ndcg_at_3 value: 27.632 - type: ndcg_at_5 value: 29.814 - type: precision_at_1 value: 21.695 - type: precision_at_10 value: 5.311 - type: precision_at_100 value: 0.819 - type: precision_at_1000 value: 0.108 - type: precision_at_20 value: 3.073 - type: precision_at_3 value: 11.977 - type: precision_at_5 value: 8.497 - type: recall_at_1 value: 20.277 - type: recall_at_10 value: 45.751999999999995 - type: recall_at_100 value: 68.72500000000001 - type: recall_at_1000 value: 88.307 - type: recall_at_20 value: 52.845 - type: recall_at_3 value: 31.928 - type: recall_at_5 value: 37.129 - task: type: Retrieval dataset: type: mteb/cqadupstack-mathematica name: MTEB CQADupstackMathematicaRetrieval config: default split: test revision: 90fceea13679c63fe563ded68f3b6f06e50061de metrics: - type: map_at_1 value: 13.061 - type: map_at_10 value: 19.952 - type: map_at_100 value: 21.177 - type: map_at_1000 value: 21.304000000000002 - type: map_at_20 value: 20.543 - type: map_at_3 value: 17.552 - type: map_at_5 value: 18.83 - type: mrr_at_1 value: 16.417910447761194 - type: mrr_at_10 value: 23.800146094922205 - type: mrr_at_100 value: 24.80931429553203 - type: mrr_at_1000 value: 24.883782140653754 - type: mrr_at_20 value: 24.263678541366147 - type: mrr_at_3 value: 21.22719734660033 - type: mrr_at_5 value: 22.763266998341624 - type: nauc_map_at_1000_diff1 value: 28.454223094067572 - type: nauc_map_at_1000_max value: 16.318450252339733 - type: nauc_map_at_100_diff1 value: 28.45468920929943 - type: nauc_map_at_100_max value: 16.280745093761492 - type: nauc_map_at_10_diff1 value: 28.475318654355913 - type: nauc_map_at_10_max value: 16.371297539246918 - type: nauc_map_at_1_diff1 value: 31.42806317607707 - type: nauc_map_at_1_max value: 16.441570076459733 - type: nauc_map_at_20_diff1 value: 28.39188164693206 - type: nauc_map_at_20_max value: 16.395102199165805 - type: nauc_map_at_3_diff1 value: 29.86515593563127 - type: nauc_map_at_3_max value: 15.942028463305657 - type: nauc_map_at_5_diff1 value: 28.801171298398238 - type: nauc_map_at_5_max value: 16.4938392502406 - type: nauc_mrr_at_1000_diff1 value: 29.404578525355905 - type: nauc_mrr_at_1000_max value: 17.06043073367033 - type: nauc_mrr_at_100_diff1 value: 29.36603539662459 - type: nauc_mrr_at_100_max value: 17.04266140433032 - type: nauc_mrr_at_10_diff1 value: 29.53064924621513 - type: nauc_mrr_at_10_max value: 17.262149332295344 - type: nauc_mrr_at_1_diff1 value: 34.4591422232893 - type: nauc_mrr_at_1_max value: 18.38441531072269 - type: nauc_mrr_at_20_diff1 value: 29.460658307695237 - type: nauc_mrr_at_20_max value: 17.177008114438692 - type: nauc_mrr_at_3_diff1 value: 31.401082424691413 - type: nauc_mrr_at_3_max value: 17.3600916517026 - type: nauc_mrr_at_5_diff1 value: 30.215482150264055 - type: nauc_mrr_at_5_max value: 17.113047363760682 - type: nauc_ndcg_at_1000_diff1 value: 26.56223239664528 - type: nauc_ndcg_at_1000_max value: 17.183346228015548 - type: nauc_ndcg_at_100_diff1 value: 26.452687521093587 - type: nauc_ndcg_at_100_max value: 16.227786533978918 - type: nauc_ndcg_at_10_diff1 value: 26.789816487503863 - type: nauc_ndcg_at_10_max value: 16.746307834455642 - type: nauc_ndcg_at_1_diff1 value: 34.4591422232893 - type: nauc_ndcg_at_1_max value: 18.38441531072269 - type: nauc_ndcg_at_20_diff1 value: 26.5048210959802 - type: nauc_ndcg_at_20_max value: 16.715661819049974 - type: nauc_ndcg_at_3_diff1 value: 29.856963970494903 - type: nauc_ndcg_at_3_max value: 16.30448334725527 - type: nauc_ndcg_at_5_diff1 value: 27.70865544302564 - type: nauc_ndcg_at_5_max value: 16.848739633174784 - type: nauc_precision_at_1000_diff1 value: 2.8134729866690966 - type: nauc_precision_at_1000_max value: 4.912645906722423 - type: nauc_precision_at_100_diff1 value: 12.322710301703319 - type: nauc_precision_at_100_max value: 7.326042531678355 - type: nauc_precision_at_10_diff1 value: 21.62505224748476 - type: nauc_precision_at_10_max value: 13.898621571795822 - type: nauc_precision_at_1_diff1 value: 34.4591422232893 - type: nauc_precision_at_1_max value: 18.38441531072269 - type: nauc_precision_at_20_diff1 value: 18.470843518995792 - type: nauc_precision_at_20_max value: 12.518698137323229 - type: nauc_precision_at_3_diff1 value: 29.716961833159882 - type: nauc_precision_at_3_max value: 15.75778937513801 - type: nauc_precision_at_5_diff1 value: 25.58620475567927 - type: nauc_precision_at_5_max value: 15.305405508622808 - type: nauc_recall_at_1000_diff1 value: 14.510689791158532 - type: nauc_recall_at_1000_max value: 28.477079172098108 - type: nauc_recall_at_100_diff1 value: 17.76094115011345 - type: nauc_recall_at_100_max value: 14.496601818232598 - type: nauc_recall_at_10_diff1 value: 20.649486228934894 - type: nauc_recall_at_10_max value: 16.53168596633394 - type: nauc_recall_at_1_diff1 value: 31.42806317607707 - type: nauc_recall_at_1_max value: 16.441570076459733 - type: nauc_recall_at_20_diff1 value: 19.624191122275327 - type: nauc_recall_at_20_max value: 16.252919740686675 - type: nauc_recall_at_3_diff1 value: 26.283117235001736 - type: nauc_recall_at_3_max value: 15.13584134035691 - type: nauc_recall_at_5_diff1 value: 22.423046028190434 - type: nauc_recall_at_5_max value: 16.432091896884675 - type: ndcg_at_1 value: 16.418 - type: ndcg_at_10 value: 24.565 - type: ndcg_at_100 value: 30.801000000000002 - type: ndcg_at_1000 value: 33.806999999999995 - type: ndcg_at_20 value: 26.512999999999998 - type: ndcg_at_3 value: 20.036 - type: ndcg_at_5 value: 22.131999999999998 - type: precision_at_1 value: 16.418 - type: precision_at_10 value: 4.776 - type: precision_at_100 value: 0.919 - type: precision_at_1000 value: 0.131 - type: precision_at_20 value: 2.91 - type: precision_at_3 value: 9.701 - type: precision_at_5 value: 7.289 - type: recall_at_1 value: 13.061 - type: recall_at_10 value: 34.961999999999996 - type: recall_at_100 value: 63.068000000000005 - type: recall_at_1000 value: 84.441 - type: recall_at_20 value: 42.013 - type: recall_at_3 value: 22.584 - type: recall_at_5 value: 27.950999999999997 - task: type: Retrieval dataset: type: mteb/cqadupstack-physics name: MTEB CQADupstackPhysicsRetrieval config: default split: test revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 metrics: - type: map_at_1 value: 24.813 - type: map_at_10 value: 33.593 - type: map_at_100 value: 34.909 - type: map_at_1000 value: 35.03 - type: map_at_20 value: 34.314 - type: map_at_3 value: 30.537999999999997 - type: map_at_5 value: 31.926 - type: mrr_at_1 value: 30.702598652550527 - type: mrr_at_10 value: 39.06545518431948 - type: mrr_at_100 value: 39.96716518258886 - type: mrr_at_1000 value: 40.01632070119006 - type: mrr_at_20 value: 39.530652551228556 - type: mrr_at_3 value: 36.58966955405838 - type: mrr_at_5 value: 37.75906320179657 - type: nauc_map_at_1000_diff1 value: 48.28735795160243 - type: nauc_map_at_1000_max value: 35.857827572549965 - type: nauc_map_at_100_diff1 value: 48.28389536809195 - type: nauc_map_at_100_max value: 35.8215491031998 - type: nauc_map_at_10_diff1 value: 48.4740670479628 - type: nauc_map_at_10_max value: 35.580153501789354 - type: nauc_map_at_1_diff1 value: 53.76162783827061 - type: nauc_map_at_1_max value: 35.987724515959904 - type: nauc_map_at_20_diff1 value: 48.39315138781542 - type: nauc_map_at_20_max value: 35.67428155833968 - type: nauc_map_at_3_diff1 value: 49.222529850894034 - type: nauc_map_at_3_max value: 35.79333062105859 - type: nauc_map_at_5_diff1 value: 49.39208013650273 - type: nauc_map_at_5_max value: 35.69664733704277 - type: nauc_mrr_at_1000_diff1 value: 49.444671325795056 - type: nauc_mrr_at_1000_max value: 38.3202725278323 - type: nauc_mrr_at_100_diff1 value: 49.42200393852792 - type: nauc_mrr_at_100_max value: 38.31143458434877 - type: nauc_mrr_at_10_diff1 value: 49.370146109325866 - type: nauc_mrr_at_10_max value: 38.160124098730236 - type: nauc_mrr_at_1_diff1 value: 56.1525302339788 - type: nauc_mrr_at_1_max value: 40.79527109574158 - type: nauc_mrr_at_20_diff1 value: 49.45102444653989 - type: nauc_mrr_at_20_max value: 38.25756256181159 - type: nauc_mrr_at_3_diff1 value: 50.14838465022493 - type: nauc_mrr_at_3_max value: 38.52708666540869 - type: nauc_mrr_at_5_diff1 value: 49.904819237426004 - type: nauc_mrr_at_5_max value: 38.38035220573582 - type: nauc_ndcg_at_1000_diff1 value: 45.58168600860826 - type: nauc_ndcg_at_1000_max value: 36.563706940380555 - type: nauc_ndcg_at_100_diff1 value: 45.04365109638583 - type: nauc_ndcg_at_100_max value: 36.033369265758196 - type: nauc_ndcg_at_10_diff1 value: 45.93349643770066 - type: nauc_ndcg_at_10_max value: 34.89977214117315 - type: nauc_ndcg_at_1_diff1 value: 56.1525302339788 - type: nauc_ndcg_at_1_max value: 40.79527109574158 - type: nauc_ndcg_at_20_diff1 value: 45.7339281351243 - type: nauc_ndcg_at_20_max value: 35.06101137124627 - type: nauc_ndcg_at_3_diff1 value: 47.98894511737239 - type: nauc_ndcg_at_3_max value: 36.848781920983924 - type: nauc_ndcg_at_5_diff1 value: 47.734244333699536 - type: nauc_ndcg_at_5_max value: 35.698338385055536 - type: nauc_precision_at_1000_diff1 value: -10.146987477879856 - type: nauc_precision_at_1000_max value: 4.5697671337926735 - type: nauc_precision_at_100_diff1 value: -0.23979892626648153 - type: nauc_precision_at_100_max value: 16.05933541503149 - type: nauc_precision_at_10_diff1 value: 21.283169904859836 - type: nauc_precision_at_10_max value: 26.180251486947377 - type: nauc_precision_at_1_diff1 value: 56.1525302339788 - type: nauc_precision_at_1_max value: 40.79527109574158 - type: nauc_precision_at_20_diff1 value: 15.36557800905963 - type: nauc_precision_at_20_max value: 23.405016571239443 - type: nauc_precision_at_3_diff1 value: 38.577821157959704 - type: nauc_precision_at_3_max value: 36.41036363960068 - type: nauc_precision_at_5_diff1 value: 34.52857379797606 - type: nauc_precision_at_5_max value: 33.5572774413252 - type: nauc_recall_at_1000_diff1 value: 17.59228181321267 - type: nauc_recall_at_1000_max value: 38.45146719434902 - type: nauc_recall_at_100_diff1 value: 23.718352007830674 - type: nauc_recall_at_100_max value: 30.191167925909102 - type: nauc_recall_at_10_diff1 value: 35.33833540132451 - type: nauc_recall_at_10_max value: 27.994679353081946 - type: nauc_recall_at_1_diff1 value: 53.76162783827061 - type: nauc_recall_at_1_max value: 35.987724515959904 - type: nauc_recall_at_20_diff1 value: 34.02078540569994 - type: nauc_recall_at_20_max value: 27.685543279801237 - type: nauc_recall_at_3_diff1 value: 42.17648880447465 - type: nauc_recall_at_3_max value: 32.62814119536231 - type: nauc_recall_at_5_diff1 value: 41.358303440502375 - type: nauc_recall_at_5_max value: 30.630510317531744 - type: ndcg_at_1 value: 30.703000000000003 - type: ndcg_at_10 value: 39.300000000000004 - type: ndcg_at_100 value: 45.1 - type: ndcg_at_1000 value: 47.386 - type: ndcg_at_20 value: 41.461999999999996 - type: ndcg_at_3 value: 34.277 - type: ndcg_at_5 value: 36.027 - type: precision_at_1 value: 30.703000000000003 - type: precision_at_10 value: 7.401000000000001 - type: precision_at_100 value: 1.218 - type: precision_at_1000 value: 0.16199999999999998 - type: precision_at_20 value: 4.398 - type: precision_at_3 value: 16.426 - type: precision_at_5 value: 11.472999999999999 - type: recall_at_1 value: 24.813 - type: recall_at_10 value: 51.044999999999995 - type: recall_at_100 value: 76.106 - type: recall_at_1000 value: 91.19 - type: recall_at_20 value: 58.744 - type: recall_at_3 value: 36.222 - type: recall_at_5 value: 41.349000000000004 - task: type: Retrieval dataset: type: mteb/cqadupstack-programmers name: MTEB CQADupstackProgrammersRetrieval config: default split: test revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 metrics: - type: map_at_1 value: 23.785999999999998 - type: map_at_10 value: 32.908 - type: map_at_100 value: 34.216 - type: map_at_1000 value: 34.345 - type: map_at_20 value: 33.669 - type: map_at_3 value: 30.084 - type: map_at_5 value: 31.692999999999998 - type: mrr_at_1 value: 28.538812785388128 - type: mrr_at_10 value: 37.57986337609625 - type: mrr_at_100 value: 38.51813582385385 - type: mrr_at_1000 value: 38.57741905983437 - type: mrr_at_20 value: 38.12193172703318 - type: mrr_at_3 value: 35.10273972602741 - type: mrr_at_5 value: 36.63812785388129 - type: nauc_map_at_1000_diff1 value: 50.257939242705326 - type: nauc_map_at_1000_max value: 40.82148200341322 - type: nauc_map_at_100_diff1 value: 50.2364465622847 - type: nauc_map_at_100_max value: 40.80019994119006 - type: nauc_map_at_10_diff1 value: 50.37138441336593 - type: nauc_map_at_10_max value: 40.50437470647407 - type: nauc_map_at_1_diff1 value: 56.771276390581896 - type: nauc_map_at_1_max value: 39.21668184007358 - type: nauc_map_at_20_diff1 value: 50.27381803917208 - type: nauc_map_at_20_max value: 40.64985824674605 - type: nauc_map_at_3_diff1 value: 51.75055615731813 - type: nauc_map_at_3_max value: 39.78757508816205 - type: nauc_map_at_5_diff1 value: 50.8734893320248 - type: nauc_map_at_5_max value: 40.02410017589514 - type: nauc_mrr_at_1000_diff1 value: 50.713285499907116 - type: nauc_mrr_at_1000_max value: 41.177149045516344 - type: nauc_mrr_at_100_diff1 value: 50.68880760672645 - type: nauc_mrr_at_100_max value: 41.15209896241968 - type: nauc_mrr_at_10_diff1 value: 50.74366283036783 - type: nauc_mrr_at_10_max value: 41.282470784533444 - type: nauc_mrr_at_1_diff1 value: 57.84445991591638 - type: nauc_mrr_at_1_max value: 43.215624153592266 - type: nauc_mrr_at_20_diff1 value: 50.64084950415708 - type: nauc_mrr_at_20_max value: 41.08394597751598 - type: nauc_mrr_at_3_diff1 value: 51.65659849102624 - type: nauc_mrr_at_3_max value: 41.60925465118275 - type: nauc_mrr_at_5_diff1 value: 50.950695800932365 - type: nauc_mrr_at_5_max value: 41.279261412729824 - type: nauc_ndcg_at_1000_diff1 value: 47.749991208061715 - type: nauc_ndcg_at_1000_max value: 41.127897310574845 - type: nauc_ndcg_at_100_diff1 value: 47.105720634992295 - type: nauc_ndcg_at_100_max value: 40.47798885234411 - type: nauc_ndcg_at_10_diff1 value: 47.531993133798885 - type: nauc_ndcg_at_10_max value: 40.31306642364056 - type: nauc_ndcg_at_1_diff1 value: 57.84445991591638 - type: nauc_ndcg_at_1_max value: 43.215624153592266 - type: nauc_ndcg_at_20_diff1 value: 47.312682165844976 - type: nauc_ndcg_at_20_max value: 40.288339506648256 - type: nauc_ndcg_at_3_diff1 value: 50.0455086369841 - type: nauc_ndcg_at_3_max value: 40.54078090764959 - type: nauc_ndcg_at_5_diff1 value: 48.579497493398414 - type: nauc_ndcg_at_5_max value: 40.12205334756227 - type: nauc_precision_at_1000_diff1 value: -10.060709814882319 - type: nauc_precision_at_1000_max value: 3.3172918415791113 - type: nauc_precision_at_100_diff1 value: 2.693932701389068 - type: nauc_precision_at_100_max value: 18.87077564079401 - type: nauc_precision_at_10_diff1 value: 25.065184074398722 - type: nauc_precision_at_10_max value: 37.59757586895368 - type: nauc_precision_at_1_diff1 value: 57.84445991591638 - type: nauc_precision_at_1_max value: 43.215624153592266 - type: nauc_precision_at_20_diff1 value: 18.242635445538607 - type: nauc_precision_at_20_max value: 32.84491156206929 - type: nauc_precision_at_3_diff1 value: 40.10849878825809 - type: nauc_precision_at_3_max value: 41.60542328360796 - type: nauc_precision_at_5_diff1 value: 33.704250787432464 - type: nauc_precision_at_5_max value: 39.62104984042399 - type: nauc_recall_at_1000_diff1 value: 20.865834650783867 - type: nauc_recall_at_1000_max value: 40.46838059326433 - type: nauc_recall_at_100_diff1 value: 27.936404516968814 - type: nauc_recall_at_100_max value: 31.42492046918158 - type: nauc_recall_at_10_diff1 value: 36.13832214819092 - type: nauc_recall_at_10_max value: 35.455673437705734 - type: nauc_recall_at_1_diff1 value: 56.771276390581896 - type: nauc_recall_at_1_max value: 39.21668184007358 - type: nauc_recall_at_20_diff1 value: 34.82064123770155 - type: nauc_recall_at_20_max value: 34.607469690168344 - type: nauc_recall_at_3_diff1 value: 44.00375363375299 - type: nauc_recall_at_3_max value: 36.44058673547244 - type: nauc_recall_at_5_diff1 value: 40.13678179130244 - type: nauc_recall_at_5_max value: 35.72641024766304 - type: ndcg_at_1 value: 28.538999999999998 - type: ndcg_at_10 value: 38.234 - type: ndcg_at_100 value: 44.025 - type: ndcg_at_1000 value: 46.611999999999995 - type: ndcg_at_20 value: 40.528999999999996 - type: ndcg_at_3 value: 33.603 - type: ndcg_at_5 value: 35.91 - type: precision_at_1 value: 28.538999999999998 - type: precision_at_10 value: 6.848999999999999 - type: precision_at_100 value: 1.146 - type: precision_at_1000 value: 0.155 - type: precision_at_20 value: 4.15 - type: precision_at_3 value: 16.02 - type: precision_at_5 value: 11.416 - type: recall_at_1 value: 23.785999999999998 - type: recall_at_10 value: 49.475 - type: recall_at_100 value: 74.211 - type: recall_at_1000 value: 91.69699999999999 - type: recall_at_20 value: 57.399 - type: recall_at_3 value: 36.716 - type: recall_at_5 value: 42.698 - task: type: Retrieval dataset: type: mteb/cqadupstack-stats name: MTEB CQADupstackStatsRetrieval config: default split: test revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a metrics: - type: map_at_1 value: 19.383 - type: map_at_10 value: 25.779999999999998 - type: map_at_100 value: 26.692 - type: map_at_1000 value: 26.790000000000003 - type: map_at_20 value: 26.288 - type: map_at_3 value: 23.286 - type: map_at_5 value: 24.881 - type: mrr_at_1 value: 21.625766871165645 - type: mrr_at_10 value: 28.134677183756935 - type: mrr_at_100 value: 28.990470265944246 - type: mrr_at_1000 value: 29.06015337908212 - type: mrr_at_20 value: 28.6217210745573 - type: mrr_at_3 value: 25.894683026584875 - type: mrr_at_5 value: 27.29038854805726 - type: nauc_map_at_1000_diff1 value: 49.911847914254295 - type: nauc_map_at_1000_max value: 35.48333790982145 - type: nauc_map_at_100_diff1 value: 49.90404203205197 - type: nauc_map_at_100_max value: 35.467667772052444 - type: nauc_map_at_10_diff1 value: 50.116677861558436 - type: nauc_map_at_10_max value: 35.22777804014755 - type: nauc_map_at_1_diff1 value: 54.43337006606851 - type: nauc_map_at_1_max value: 34.97044319260595 - type: nauc_map_at_20_diff1 value: 50.00485022476988 - type: nauc_map_at_20_max value: 35.27772351072898 - type: nauc_map_at_3_diff1 value: 51.101154626515466 - type: nauc_map_at_3_max value: 35.01188192392699 - type: nauc_map_at_5_diff1 value: 50.85455903915764 - type: nauc_map_at_5_max value: 35.222924790317556 - type: nauc_mrr_at_1000_diff1 value: 50.469777332868446 - type: nauc_mrr_at_1000_max value: 36.806239966091134 - type: nauc_mrr_at_100_diff1 value: 50.43774386023924 - type: nauc_mrr_at_100_max value: 36.81415281954135 - type: nauc_mrr_at_10_diff1 value: 50.65191632415702 - type: nauc_mrr_at_10_max value: 36.796669555237735 - type: nauc_mrr_at_1_diff1 value: 56.69444714733734 - type: nauc_mrr_at_1_max value: 37.950238556672936 - type: nauc_mrr_at_20_diff1 value: 50.58591151000893 - type: nauc_mrr_at_20_max value: 36.74486652850544 - type: nauc_mrr_at_3_diff1 value: 51.874376496561666 - type: nauc_mrr_at_3_max value: 37.17436215275038 - type: nauc_mrr_at_5_diff1 value: 51.23784797508226 - type: nauc_mrr_at_5_max value: 36.797843583446976 - type: nauc_ndcg_at_1000_diff1 value: 47.02453368687982 - type: nauc_ndcg_at_1000_max value: 35.645475773480015 - type: nauc_ndcg_at_100_diff1 value: 46.3919750229052 - type: nauc_ndcg_at_100_max value: 35.52540027815628 - type: nauc_ndcg_at_10_diff1 value: 47.783397540510954 - type: nauc_ndcg_at_10_max value: 34.978511185065415 - type: nauc_ndcg_at_1_diff1 value: 56.69444714733734 - type: nauc_ndcg_at_1_max value: 37.950238556672936 - type: nauc_ndcg_at_20_diff1 value: 47.54223429140186 - type: nauc_ndcg_at_20_max value: 34.96367876086221 - type: nauc_ndcg_at_3_diff1 value: 49.6543909342716 - type: nauc_ndcg_at_3_max value: 35.55329341746738 - type: nauc_ndcg_at_5_diff1 value: 49.27526999339532 - type: nauc_ndcg_at_5_max value: 35.269553382209025 - type: nauc_precision_at_1000_diff1 value: 4.152639847319254 - type: nauc_precision_at_1000_max value: 19.860239840663723 - type: nauc_precision_at_100_diff1 value: 20.5112056117488 - type: nauc_precision_at_100_max value: 33.986584990989556 - type: nauc_precision_at_10_diff1 value: 36.61149278468018 - type: nauc_precision_at_10_max value: 38.71670013842566 - type: nauc_precision_at_1_diff1 value: 56.69444714733734 - type: nauc_precision_at_1_max value: 37.950238556672936 - type: nauc_precision_at_20_diff1 value: 33.440399762319224 - type: nauc_precision_at_20_max value: 36.81534392845937 - type: nauc_precision_at_3_diff1 value: 46.57940283237786 - type: nauc_precision_at_3_max value: 40.2021686353609 - type: nauc_precision_at_5_diff1 value: 43.24878459897664 - type: nauc_precision_at_5_max value: 39.56338396559801 - type: nauc_recall_at_1000_diff1 value: 30.848441444485346 - type: nauc_recall_at_1000_max value: 29.206281789808227 - type: nauc_recall_at_100_diff1 value: 30.51257893193596 - type: nauc_recall_at_100_max value: 31.092511899847807 - type: nauc_recall_at_10_diff1 value: 39.87582825785208 - type: nauc_recall_at_10_max value: 31.21163188638994 - type: nauc_recall_at_1_diff1 value: 54.43337006606851 - type: nauc_recall_at_1_max value: 34.97044319260595 - type: nauc_recall_at_20_diff1 value: 38.87715004078399 - type: nauc_recall_at_20_max value: 30.6988866615607 - type: nauc_recall_at_3_diff1 value: 45.86720941947824 - type: nauc_recall_at_3_max value: 32.61976516194995 - type: nauc_recall_at_5_diff1 value: 44.153048811811296 - type: nauc_recall_at_5_max value: 32.11676236101473 - type: ndcg_at_1 value: 21.626 - type: ndcg_at_10 value: 29.859 - type: ndcg_at_100 value: 34.472 - type: ndcg_at_1000 value: 37.038 - type: ndcg_at_20 value: 31.635 - type: ndcg_at_3 value: 25.288 - type: ndcg_at_5 value: 27.855999999999998 - type: precision_at_1 value: 21.626 - type: precision_at_10 value: 4.954 - type: precision_at_100 value: 0.7779999999999999 - type: precision_at_1000 value: 0.108 - type: precision_at_20 value: 2.899 - type: precision_at_3 value: 11.094 - type: precision_at_5 value: 8.251999999999999 - type: recall_at_1 value: 19.383 - type: recall_at_10 value: 40.076 - type: recall_at_100 value: 61.242 - type: recall_at_1000 value: 80.314 - type: recall_at_20 value: 46.758 - type: recall_at_3 value: 27.962999999999997 - type: recall_at_5 value: 34.147 - task: type: Retrieval dataset: type: mteb/cqadupstack-tex name: MTEB CQADupstackTexRetrieval config: default split: test revision: 46989137a86843e03a6195de44b09deda022eec7 metrics: - type: map_at_1 value: 13.971 - type: map_at_10 value: 19.858 - type: map_at_100 value: 20.909 - type: map_at_1000 value: 21.047 - type: map_at_20 value: 20.426 - type: map_at_3 value: 17.888 - type: map_at_5 value: 18.948 - type: mrr_at_1 value: 17.377838953888507 - type: mrr_at_10 value: 23.3541168245229 - type: mrr_at_100 value: 24.28548028171366 - type: mrr_at_1000 value: 24.377827358449824 - type: mrr_at_20 value: 23.88638570357191 - type: mrr_at_3 value: 21.36958017894013 - type: mrr_at_5 value: 22.512044046799744 - type: nauc_map_at_1000_diff1 value: 37.755702284175236 - type: nauc_map_at_1000_max value: 29.985902989676845 - type: nauc_map_at_100_diff1 value: 37.75580303049384 - type: nauc_map_at_100_max value: 29.948110193131978 - type: nauc_map_at_10_diff1 value: 38.173556250676114 - type: nauc_map_at_10_max value: 29.977909805969883 - type: nauc_map_at_1_diff1 value: 46.397651367070175 - type: nauc_map_at_1_max value: 30.5126306843905 - type: nauc_map_at_20_diff1 value: 37.90916483077938 - type: nauc_map_at_20_max value: 29.968172386265085 - type: nauc_map_at_3_diff1 value: 39.72488280969923 - type: nauc_map_at_3_max value: 30.188740025946746 - type: nauc_map_at_5_diff1 value: 38.630985019403546 - type: nauc_map_at_5_max value: 29.879481875315722 - type: nauc_mrr_at_1000_diff1 value: 35.84642181893316 - type: nauc_mrr_at_1000_max value: 31.670254385165258 - type: nauc_mrr_at_100_diff1 value: 35.83045706635439 - type: nauc_mrr_at_100_max value: 31.66496126819178 - type: nauc_mrr_at_10_diff1 value: 36.09970407587168 - type: nauc_mrr_at_10_max value: 31.8396436118298 - type: nauc_mrr_at_1_diff1 value: 43.10155389869899 - type: nauc_mrr_at_1_max value: 34.16020753098804 - type: nauc_mrr_at_20_diff1 value: 35.896835477664894 - type: nauc_mrr_at_20_max value: 31.73239216068521 - type: nauc_mrr_at_3_diff1 value: 37.466645069277035 - type: nauc_mrr_at_3_max value: 32.49242404383092 - type: nauc_mrr_at_5_diff1 value: 36.405092730921915 - type: nauc_mrr_at_5_max value: 31.92760166562469 - type: nauc_ndcg_at_1000_diff1 value: 33.90603402215118 - type: nauc_ndcg_at_1000_max value: 29.729336831998786 - type: nauc_ndcg_at_100_diff1 value: 33.50850724053531 - type: nauc_ndcg_at_100_max value: 29.306492255119792 - type: nauc_ndcg_at_10_diff1 value: 34.97587417274376 - type: nauc_ndcg_at_10_max value: 29.85287300723461 - type: nauc_ndcg_at_1_diff1 value: 43.10155389869899 - type: nauc_ndcg_at_1_max value: 34.16020753098804 - type: nauc_ndcg_at_20_diff1 value: 34.20570916045498 - type: nauc_ndcg_at_20_max value: 29.60222341192852 - type: nauc_ndcg_at_3_diff1 value: 37.339592233518914 - type: nauc_ndcg_at_3_max value: 31.040627782319678 - type: nauc_ndcg_at_5_diff1 value: 35.77389927348168 - type: nauc_ndcg_at_5_max value: 29.914791666809233 - type: nauc_precision_at_1000_diff1 value: 2.899363456166234 - type: nauc_precision_at_1000_max value: 22.023156102458266 - type: nauc_precision_at_100_diff1 value: 11.303787835861796 - type: nauc_precision_at_100_max value: 26.139688179760295 - type: nauc_precision_at_10_diff1 value: 22.48116840886295 - type: nauc_precision_at_10_max value: 31.359997435260933 - type: nauc_precision_at_1_diff1 value: 43.10155389869899 - type: nauc_precision_at_1_max value: 34.16020753098804 - type: nauc_precision_at_20_diff1 value: 19.052342801259027 - type: nauc_precision_at_20_max value: 30.295818240552265 - type: nauc_precision_at_3_diff1 value: 29.98671296818897 - type: nauc_precision_at_3_max value: 33.518872240321095 - type: nauc_precision_at_5_diff1 value: 26.140183342571895 - type: nauc_precision_at_5_max value: 31.731967908857495 - type: nauc_recall_at_1000_diff1 value: 18.21949169710172 - type: nauc_recall_at_1000_max value: 21.981372080200686 - type: nauc_recall_at_100_diff1 value: 20.257733390464356 - type: nauc_recall_at_100_max value: 22.351366657877755 - type: nauc_recall_at_10_diff1 value: 27.478094748761023 - type: nauc_recall_at_10_max value: 25.693669844365168 - type: nauc_recall_at_1_diff1 value: 46.397651367070175 - type: nauc_recall_at_1_max value: 30.5126306843905 - type: nauc_recall_at_20_diff1 value: 24.426997641435758 - type: nauc_recall_at_20_max value: 24.371909403981874 - type: nauc_recall_at_3_diff1 value: 33.01864346236186 - type: nauc_recall_at_3_max value: 27.470654873232615 - type: nauc_recall_at_5_diff1 value: 29.70016272411814 - type: nauc_recall_at_5_max value: 26.045940185547146 - type: ndcg_at_1 value: 17.378 - type: ndcg_at_10 value: 23.798 - type: ndcg_at_100 value: 28.907 - type: ndcg_at_1000 value: 32.273 - type: ndcg_at_20 value: 25.707 - type: ndcg_at_3 value: 20.172 - type: ndcg_at_5 value: 21.802 - type: precision_at_1 value: 17.378 - type: precision_at_10 value: 4.425 - type: precision_at_100 value: 0.83 - type: precision_at_1000 value: 0.129 - type: precision_at_20 value: 2.796 - type: precision_at_3 value: 9.600999999999999 - type: precision_at_5 value: 6.999 - type: recall_at_1 value: 13.971 - type: recall_at_10 value: 32.24 - type: recall_at_100 value: 55.257 - type: recall_at_1000 value: 79.59 - type: recall_at_20 value: 39.229 - type: recall_at_3 value: 22.067999999999998 - type: recall_at_5 value: 26.293 - task: type: Retrieval dataset: type: mteb/cqadupstack-unix name: MTEB CQADupstackUnixRetrieval config: default split: test revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 metrics: - type: map_at_1 value: 22.913 - type: map_at_10 value: 31.479000000000003 - type: map_at_100 value: 32.552 - type: map_at_1000 value: 32.653 - type: map_at_20 value: 31.990000000000002 - type: map_at_3 value: 28.592000000000002 - type: map_at_5 value: 30.373 - type: mrr_at_1 value: 26.679104477611943 - type: mrr_at_10 value: 35.04575337597722 - type: mrr_at_100 value: 35.95382623195233 - type: mrr_at_1000 value: 36.010139872633495 - type: mrr_at_20 value: 35.53513175560514 - type: mrr_at_3 value: 32.50932835820894 - type: mrr_at_5 value: 34.029850746268615 - type: nauc_map_at_1000_diff1 value: 48.91111448779822 - type: nauc_map_at_1000_max value: 42.45001990527906 - type: nauc_map_at_100_diff1 value: 48.89190728335153 - type: nauc_map_at_100_max value: 42.42208663513962 - type: nauc_map_at_10_diff1 value: 49.18102301155058 - type: nauc_map_at_10_max value: 42.38922838539626 - type: nauc_map_at_1_diff1 value: 56.80065296129456 - type: nauc_map_at_1_max value: 44.49602807145463 - type: nauc_map_at_20_diff1 value: 48.8774198345571 - type: nauc_map_at_20_max value: 42.33035822233443 - type: nauc_map_at_3_diff1 value: 50.18578760557955 - type: nauc_map_at_3_max value: 42.049682832486525 - type: nauc_map_at_5_diff1 value: 49.486124069477185 - type: nauc_map_at_5_max value: 42.55783777242684 - type: nauc_mrr_at_1000_diff1 value: 49.36889165411647 - type: nauc_mrr_at_1000_max value: 44.77127656729937 - type: nauc_mrr_at_100_diff1 value: 49.334724602529384 - type: nauc_mrr_at_100_max value: 44.74054390604988 - type: nauc_mrr_at_10_diff1 value: 49.51295857027244 - type: nauc_mrr_at_10_max value: 44.9170190483006 - type: nauc_mrr_at_1_diff1 value: 58.082371939479195 - type: nauc_mrr_at_1_max value: 47.809669154624714 - type: nauc_mrr_at_20_diff1 value: 49.32453628990772 - type: nauc_mrr_at_20_max value: 44.85511763034043 - type: nauc_mrr_at_3_diff1 value: 50.28814557566801 - type: nauc_mrr_at_3_max value: 45.02918648542346 - type: nauc_mrr_at_5_diff1 value: 49.70083559303071 - type: nauc_mrr_at_5_max value: 45.24452033882872 - type: nauc_ndcg_at_1000_diff1 value: 45.98074456611413 - type: nauc_ndcg_at_1000_max value: 42.31080875816173 - type: nauc_ndcg_at_100_diff1 value: 45.40775721447589 - type: nauc_ndcg_at_100_max value: 41.66948346471324 - type: nauc_ndcg_at_10_diff1 value: 46.45033224294541 - type: nauc_ndcg_at_10_max value: 41.94587530702735 - type: nauc_ndcg_at_1_diff1 value: 58.082371939479195 - type: nauc_ndcg_at_1_max value: 47.809669154624714 - type: nauc_ndcg_at_20_diff1 value: 45.52119345567974 - type: nauc_ndcg_at_20_max value: 41.689091488310375 - type: nauc_ndcg_at_3_diff1 value: 47.7611351600258 - type: nauc_ndcg_at_3_max value: 42.12421531952486 - type: nauc_ndcg_at_5_diff1 value: 46.96076236146089 - type: nauc_ndcg_at_5_max value: 42.48883644550073 - type: nauc_precision_at_1000_diff1 value: -9.994796100724528 - type: nauc_precision_at_1000_max value: 4.537719884624942 - type: nauc_precision_at_100_diff1 value: 5.176562004472227 - type: nauc_precision_at_100_max value: 20.363881706230003 - type: nauc_precision_at_10_diff1 value: 28.13817373553328 - type: nauc_precision_at_10_max value: 35.95475620553222 - type: nauc_precision_at_1_diff1 value: 58.082371939479195 - type: nauc_precision_at_1_max value: 47.809669154624714 - type: nauc_precision_at_20_diff1 value: 20.22951508884242 - type: nauc_precision_at_20_max value: 31.772942110564383 - type: nauc_precision_at_3_diff1 value: 39.394928690349715 - type: nauc_precision_at_3_max value: 41.22337390889921 - type: nauc_precision_at_5_diff1 value: 33.3002766373711 - type: nauc_precision_at_5_max value: 39.798736765093004 - type: nauc_recall_at_1000_diff1 value: 23.754029850817286 - type: nauc_recall_at_1000_max value: 33.93123719288166 - type: nauc_recall_at_100_diff1 value: 27.544628904110198 - type: nauc_recall_at_100_max value: 30.334114782639638 - type: nauc_recall_at_10_diff1 value: 36.834663549355106 - type: nauc_recall_at_10_max value: 35.98186975365149 - type: nauc_recall_at_1_diff1 value: 56.80065296129456 - type: nauc_recall_at_1_max value: 44.49602807145463 - type: nauc_recall_at_20_diff1 value: 32.36845729763492 - type: nauc_recall_at_20_max value: 33.854227154499725 - type: nauc_recall_at_3_diff1 value: 41.3767634461603 - type: nauc_recall_at_3_max value: 37.614960977002795 - type: nauc_recall_at_5_diff1 value: 38.83337733461508 - type: nauc_recall_at_5_max value: 38.23953256920769 - type: ndcg_at_1 value: 26.679000000000002 - type: ndcg_at_10 value: 36.620999999999995 - type: ndcg_at_100 value: 41.942 - type: ndcg_at_1000 value: 44.374 - type: ndcg_at_20 value: 38.415 - type: ndcg_at_3 value: 31.538 - type: ndcg_at_5 value: 34.178999999999995 - type: precision_at_1 value: 26.679000000000002 - type: precision_at_10 value: 6.259 - type: precision_at_100 value: 1.006 - type: precision_at_1000 value: 0.133 - type: precision_at_20 value: 3.6290000000000004 - type: precision_at_3 value: 14.335 - type: precision_at_5 value: 10.485 - type: recall_at_1 value: 22.913 - type: recall_at_10 value: 48.716 - type: recall_at_100 value: 72.372 - type: recall_at_1000 value: 89.531 - type: recall_at_20 value: 55.269999999999996 - type: recall_at_3 value: 35.074 - type: recall_at_5 value: 41.537 - task: type: Retrieval dataset: type: mteb/cqadupstack-webmasters name: MTEB CQADupstackWebmastersRetrieval config: default split: test revision: 160c094312a0e1facb97e55eeddb698c0abe3571 metrics: - type: map_at_1 value: 24.111 - type: map_at_10 value: 32.586999999999996 - type: map_at_100 value: 34.12 - type: map_at_1000 value: 34.365 - type: map_at_20 value: 33.357 - type: map_at_3 value: 29.941000000000003 - type: map_at_5 value: 31.576999999999998 - type: mrr_at_1 value: 29.841897233201582 - type: mrr_at_10 value: 37.7974621996361 - type: mrr_at_100 value: 38.70855926216216 - type: mrr_at_1000 value: 38.78160750260466 - type: mrr_at_20 value: 38.28642292998807 - type: mrr_at_3 value: 35.540184453227944 - type: mrr_at_5 value: 36.9631093544137 - type: nauc_map_at_1000_diff1 value: 48.66304930719204 - type: nauc_map_at_1000_max value: 41.95032096153431 - type: nauc_map_at_100_diff1 value: 48.79765165759156 - type: nauc_map_at_100_max value: 42.07729091897765 - type: nauc_map_at_10_diff1 value: 48.857016210602815 - type: nauc_map_at_10_max value: 41.40602447689144 - type: nauc_map_at_1_diff1 value: 56.30285091540639 - type: nauc_map_at_1_max value: 41.50722701029977 - type: nauc_map_at_20_diff1 value: 48.81899029977082 - type: nauc_map_at_20_max value: 41.59748805021434 - type: nauc_map_at_3_diff1 value: 50.427376588223524 - type: nauc_map_at_3_max value: 42.16021881535689 - type: nauc_map_at_5_diff1 value: 49.30376475055564 - type: nauc_map_at_5_max value: 40.80672149673538 - type: nauc_mrr_at_1000_diff1 value: 46.345186006817585 - type: nauc_mrr_at_1000_max value: 43.50174997374244 - type: nauc_mrr_at_100_diff1 value: 46.295807235163586 - type: nauc_mrr_at_100_max value: 43.50031316913962 - type: nauc_mrr_at_10_diff1 value: 46.47773451307542 - type: nauc_mrr_at_10_max value: 43.458160190412194 - type: nauc_mrr_at_1_diff1 value: 52.74875572154898 - type: nauc_mrr_at_1_max value: 44.75642046390427 - type: nauc_mrr_at_20_diff1 value: 46.3900630839787 - type: nauc_mrr_at_20_max value: 43.50880808108073 - type: nauc_mrr_at_3_diff1 value: 48.19554935809951 - type: nauc_mrr_at_3_max value: 44.6333875693719 - type: nauc_mrr_at_5_diff1 value: 46.93570802196158 - type: nauc_mrr_at_5_max value: 43.125336922131 - type: nauc_ndcg_at_1000_diff1 value: 45.33511371430761 - type: nauc_ndcg_at_1000_max value: 42.71458917809584 - type: nauc_ndcg_at_100_diff1 value: 44.722449600774205 - type: nauc_ndcg_at_100_max value: 42.914978315188804 - type: nauc_ndcg_at_10_diff1 value: 45.227438065629464 - type: nauc_ndcg_at_10_max value: 41.683893995546676 - type: nauc_ndcg_at_1_diff1 value: 52.74875572154898 - type: nauc_ndcg_at_1_max value: 44.75642046390427 - type: nauc_ndcg_at_20_diff1 value: 44.55699203003113 - type: nauc_ndcg_at_20_max value: 41.415096016236674 - type: nauc_ndcg_at_3_diff1 value: 48.10326999177321 - type: nauc_ndcg_at_3_max value: 44.23088613607569 - type: nauc_ndcg_at_5_diff1 value: 46.38361769072502 - type: nauc_ndcg_at_5_max value: 41.022399131998824 - type: nauc_precision_at_1000_diff1 value: -14.552159603323588 - type: nauc_precision_at_1000_max value: -3.1254911408763344 - type: nauc_precision_at_100_diff1 value: -0.23909748335372277 - type: nauc_precision_at_100_max value: 16.89399096801419 - type: nauc_precision_at_10_diff1 value: 20.38093826207771 - type: nauc_precision_at_10_max value: 33.609116276999984 - type: nauc_precision_at_1_diff1 value: 52.74875572154898 - type: nauc_precision_at_1_max value: 44.75642046390427 - type: nauc_precision_at_20_diff1 value: 14.390742980745472 - type: nauc_precision_at_20_max value: 28.70169752553835 - type: nauc_precision_at_3_diff1 value: 35.95450638656823 - type: nauc_precision_at_3_max value: 43.164677486075306 - type: nauc_precision_at_5_diff1 value: 28.145486984369327 - type: nauc_precision_at_5_max value: 35.15010507177889 - type: nauc_recall_at_1000_diff1 value: 18.31197862072379 - type: nauc_recall_at_1000_max value: 39.21531256166399 - type: nauc_recall_at_100_diff1 value: 25.523825639252152 - type: nauc_recall_at_100_max value: 41.261468285876326 - type: nauc_recall_at_10_diff1 value: 33.25089536691914 - type: nauc_recall_at_10_max value: 35.62911707394771 - type: nauc_recall_at_1_diff1 value: 56.30285091540639 - type: nauc_recall_at_1_max value: 41.50722701029977 - type: nauc_recall_at_20_diff1 value: 29.960428793914986 - type: nauc_recall_at_20_max value: 35.185821233232936 - type: nauc_recall_at_3_diff1 value: 43.06694028157036 - type: nauc_recall_at_3_max value: 39.592097716636886 - type: nauc_recall_at_5_diff1 value: 37.78776183699997 - type: nauc_recall_at_5_max value: 33.29553795647041 - type: ndcg_at_1 value: 29.842000000000002 - type: ndcg_at_10 value: 38.17 - type: ndcg_at_100 value: 43.69 - type: ndcg_at_1000 value: 46.647 - type: ndcg_at_20 value: 40.071 - type: ndcg_at_3 value: 34.095 - type: ndcg_at_5 value: 36.24 - type: precision_at_1 value: 29.842000000000002 - type: precision_at_10 value: 7.273000000000001 - type: precision_at_100 value: 1.47 - type: precision_at_1000 value: 0.23800000000000002 - type: precision_at_20 value: 4.595 - type: precision_at_3 value: 16.073999999999998 - type: precision_at_5 value: 11.738999999999999 - type: recall_at_1 value: 24.111 - type: recall_at_10 value: 48.004000000000005 - type: recall_at_100 value: 72.94699999999999 - type: recall_at_1000 value: 91.888 - type: recall_at_20 value: 55.216 - type: recall_at_3 value: 35.811 - type: recall_at_5 value: 41.89 - task: type: Retrieval dataset: type: mteb/cqadupstack-wordpress name: MTEB CQADupstackWordpressRetrieval config: default split: test revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 metrics: - type: map_at_1 value: 18.028 - type: map_at_10 value: 24.82 - type: map_at_100 value: 25.768 - type: map_at_1000 value: 25.884 - type: map_at_20 value: 25.320999999999998 - type: map_at_3 value: 22.585 - type: map_at_5 value: 23.97 - type: mrr_at_1 value: 19.963031423290204 - type: mrr_at_10 value: 26.794589678138646 - type: mrr_at_100 value: 27.68954610737053 - type: mrr_at_1000 value: 27.779869498156962 - type: mrr_at_20 value: 27.282527769829105 - type: mrr_at_3 value: 24.522489217498453 - type: mrr_at_5 value: 25.89956869993838 - type: nauc_map_at_1000_diff1 value: 29.354656347993824 - type: nauc_map_at_1000_max value: 25.777632482963902 - type: nauc_map_at_100_diff1 value: 29.344606522701266 - type: nauc_map_at_100_max value: 25.74884197256006 - type: nauc_map_at_10_diff1 value: 29.263520383732228 - type: nauc_map_at_10_max value: 25.956046243549185 - type: nauc_map_at_1_diff1 value: 32.10083227622593 - type: nauc_map_at_1_max value: 26.082934151928335 - type: nauc_map_at_20_diff1 value: 29.252814048502522 - type: nauc_map_at_20_max value: 25.73021877076165 - type: nauc_map_at_3_diff1 value: 30.402946945740624 - type: nauc_map_at_3_max value: 25.479666391115586 - type: nauc_map_at_5_diff1 value: 30.112881326178037 - type: nauc_map_at_5_max value: 26.181377819243316 - type: nauc_mrr_at_1000_diff1 value: 29.36011085257817 - type: nauc_mrr_at_1000_max value: 26.512397267441905 - type: nauc_mrr_at_100_diff1 value: 29.34716126133153 - type: nauc_mrr_at_100_max value: 26.492126338060768 - type: nauc_mrr_at_10_diff1 value: 29.291501370936746 - type: nauc_mrr_at_10_max value: 26.720460613779785 - type: nauc_mrr_at_1_diff1 value: 32.59370226571806 - type: nauc_mrr_at_1_max value: 27.210974235559654 - type: nauc_mrr_at_20_diff1 value: 29.339683741214824 - type: nauc_mrr_at_20_max value: 26.565206125640618 - type: nauc_mrr_at_3_diff1 value: 30.615760229041822 - type: nauc_mrr_at_3_max value: 26.964923490024372 - type: nauc_mrr_at_5_diff1 value: 30.09283430145986 - type: nauc_mrr_at_5_max value: 26.97653024091469 - type: nauc_ndcg_at_1000_diff1 value: 28.1229347584593 - type: nauc_ndcg_at_1000_max value: 25.58333272731477 - type: nauc_ndcg_at_100_diff1 value: 27.785141959567877 - type: nauc_ndcg_at_100_max value: 25.262023849022043 - type: nauc_ndcg_at_10_diff1 value: 27.471970751404058 - type: nauc_ndcg_at_10_max value: 26.100161257581366 - type: nauc_ndcg_at_1_diff1 value: 32.59370226571806 - type: nauc_ndcg_at_1_max value: 27.210974235559654 - type: nauc_ndcg_at_20_diff1 value: 27.493348155478664 - type: nauc_ndcg_at_20_max value: 25.429920376418064 - type: nauc_ndcg_at_3_diff1 value: 30.049291118944282 - type: nauc_ndcg_at_3_max value: 25.87324242231816 - type: nauc_ndcg_at_5_diff1 value: 29.285225144397696 - type: nauc_ndcg_at_5_max value: 26.538239428331035 - type: nauc_precision_at_1000_diff1 value: 0.30786182255546335 - type: nauc_precision_at_1000_max value: -1.122583222669982 - type: nauc_precision_at_100_diff1 value: 16.643044236960975 - type: nauc_precision_at_100_max value: 16.898061651611094 - type: nauc_precision_at_10_diff1 value: 22.225745733008576 - type: nauc_precision_at_10_max value: 25.897094746875343 - type: nauc_precision_at_1_diff1 value: 32.59370226571806 - type: nauc_precision_at_1_max value: 27.210974235559654 - type: nauc_precision_at_20_diff1 value: 21.25369337146511 - type: nauc_precision_at_20_max value: 22.501776705458052 - type: nauc_precision_at_3_diff1 value: 29.752252136844444 - type: nauc_precision_at_3_max value: 26.54042152706389 - type: nauc_precision_at_5_diff1 value: 28.014614694166003 - type: nauc_precision_at_5_max value: 28.096365928752963 - type: nauc_recall_at_1000_diff1 value: 23.27380094527733 - type: nauc_recall_at_1000_max value: 21.84162629652965 - type: nauc_recall_at_100_diff1 value: 21.646953220120835 - type: nauc_recall_at_100_max value: 20.865427838218878 - type: nauc_recall_at_10_diff1 value: 21.50707716122318 - type: nauc_recall_at_10_max value: 25.11348571349948 - type: nauc_recall_at_1_diff1 value: 32.10083227622593 - type: nauc_recall_at_1_max value: 26.082934151928335 - type: nauc_recall_at_20_diff1 value: 21.149584140557366 - type: nauc_recall_at_20_max value: 22.687778440936103 - type: nauc_recall_at_3_diff1 value: 28.105583308602416 - type: nauc_recall_at_3_max value: 24.808247917706677 - type: nauc_recall_at_5_diff1 value: 26.268258977975016 - type: nauc_recall_at_5_max value: 26.14783487728073 - type: ndcg_at_1 value: 19.963 - type: ndcg_at_10 value: 28.988999999999997 - type: ndcg_at_100 value: 33.835 - type: ndcg_at_1000 value: 37.07 - type: ndcg_at_20 value: 30.808000000000003 - type: ndcg_at_3 value: 24.581 - type: ndcg_at_5 value: 26.979999999999997 - type: precision_at_1 value: 19.963 - type: precision_at_10 value: 4.621 - type: precision_at_100 value: 0.754 - type: precision_at_1000 value: 0.11199999999999999 - type: precision_at_20 value: 2.7449999999999997 - type: precision_at_3 value: 10.783 - type: precision_at_5 value: 7.837 - type: recall_at_1 value: 18.028 - type: recall_at_10 value: 39.997 - type: recall_at_100 value: 62.361 - type: recall_at_1000 value: 87.083 - type: recall_at_20 value: 46.936 - type: recall_at_3 value: 28.323999999999998 - type: recall_at_5 value: 34.091 - task: type: Retrieval dataset: type: mteb/climate-fever name: MTEB ClimateFEVER config: default split: test revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 metrics: - type: map_at_1 value: 6.18 - type: map_at_10 value: 11.277 - type: map_at_100 value: 12.873000000000001 - type: map_at_1000 value: 13.098 - type: map_at_20 value: 12.033000000000001 - type: map_at_3 value: 9.213000000000001 - type: map_at_5 value: 10.179 - type: mrr_at_1 value: 13.615635179153093 - type: mrr_at_10 value: 22.3759371283801 - type: mrr_at_100 value: 23.74043839617308 - type: mrr_at_1000 value: 23.807597594686023 - type: mrr_at_20 value: 23.172313674880147 - type: mrr_at_3 value: 19.15309446254072 - type: mrr_at_5 value: 20.902280130293132 - type: nauc_map_at_1000_diff1 value: 28.07875585225803 - type: nauc_map_at_1000_max value: 35.83708929348549 - type: nauc_map_at_100_diff1 value: 28.13253751629951 - type: nauc_map_at_100_max value: 35.74641807227035 - type: nauc_map_at_10_diff1 value: 28.494200987801044 - type: nauc_map_at_10_max value: 34.583975326049966 - type: nauc_map_at_1_diff1 value: 39.996859549061355 - type: nauc_map_at_1_max value: 39.270510397353526 - type: nauc_map_at_20_diff1 value: 28.025812489675044 - type: nauc_map_at_20_max value: 35.177169215298406 - type: nauc_map_at_3_diff1 value: 32.30909470003803 - type: nauc_map_at_3_max value: 36.37733076481426 - type: nauc_map_at_5_diff1 value: 29.921198050945296 - type: nauc_map_at_5_max value: 34.94586908456363 - type: nauc_mrr_at_1000_diff1 value: 26.08074340015364 - type: nauc_mrr_at_1000_max value: 33.02687297171295 - type: nauc_mrr_at_100_diff1 value: 26.054902916636646 - type: nauc_mrr_at_100_max value: 33.01617535618753 - type: nauc_mrr_at_10_diff1 value: 25.88148928931446 - type: nauc_mrr_at_10_max value: 32.61799742547017 - type: nauc_mrr_at_1_diff1 value: 36.03138484582729 - type: nauc_mrr_at_1_max value: 36.277950419702755 - type: nauc_mrr_at_20_diff1 value: 25.974259829372738 - type: nauc_mrr_at_20_max value: 32.797465492032124 - type: nauc_mrr_at_3_diff1 value: 28.465203801115152 - type: nauc_mrr_at_3_max value: 33.765630602498895 - type: nauc_mrr_at_5_diff1 value: 26.436793504159496 - type: nauc_mrr_at_5_max value: 32.62012479096292 - type: nauc_ndcg_at_1000_diff1 value: 23.672305808597077 - type: nauc_ndcg_at_1000_max value: 37.08958734194627 - type: nauc_ndcg_at_100_diff1 value: 23.944898359513253 - type: nauc_ndcg_at_100_max value: 35.90189193251265 - type: nauc_ndcg_at_10_diff1 value: 23.75418336772388 - type: nauc_ndcg_at_10_max value: 32.36026460519453 - type: nauc_ndcg_at_1_diff1 value: 36.03138484582729 - type: nauc_ndcg_at_1_max value: 36.277950419702755 - type: nauc_ndcg_at_20_diff1 value: 23.079788090123536 - type: nauc_ndcg_at_20_max value: 33.46154223206268 - type: nauc_ndcg_at_3_diff1 value: 28.612088697615302 - type: nauc_ndcg_at_3_max value: 34.42302593383679 - type: nauc_ndcg_at_5_diff1 value: 25.671378300813853 - type: nauc_ndcg_at_5_max value: 32.64433627531644 - type: nauc_precision_at_1000_diff1 value: -0.3052718758543331 - type: nauc_precision_at_1000_max value: 18.16387047369377 - type: nauc_precision_at_100_diff1 value: 7.956051604214289 - type: nauc_precision_at_100_max value: 25.73158237184035 - type: nauc_precision_at_10_diff1 value: 10.163750744754575 - type: nauc_precision_at_10_max value: 26.00696664055645 - type: nauc_precision_at_1_diff1 value: 36.03138484582729 - type: nauc_precision_at_1_max value: 36.277950419702755 - type: nauc_precision_at_20_diff1 value: 8.314279554483267 - type: nauc_precision_at_20_max value: 27.115560839727205 - type: nauc_precision_at_3_diff1 value: 21.690833394205402 - type: nauc_precision_at_3_max value: 32.156497547258645 - type: nauc_precision_at_5_diff1 value: 15.032164068271097 - type: nauc_precision_at_5_max value: 28.07574462716985 - type: nauc_recall_at_1000_diff1 value: 10.738783260564638 - type: nauc_recall_at_1000_max value: 35.56192865936298 - type: nauc_recall_at_100_diff1 value: 14.296129488159078 - type: nauc_recall_at_100_max value: 30.18732747422109 - type: nauc_recall_at_10_diff1 value: 15.306405656539345 - type: nauc_recall_at_10_max value: 25.621311074599586 - type: nauc_recall_at_1_diff1 value: 39.996859549061355 - type: nauc_recall_at_1_max value: 39.270510397353526 - type: nauc_recall_at_20_diff1 value: 13.106484977306085 - type: nauc_recall_at_20_max value: 26.123290353715582 - type: nauc_recall_at_3_diff1 value: 25.514669409942993 - type: nauc_recall_at_3_max value: 31.83833126301056 - type: nauc_recall_at_5_diff1 value: 18.916230651494935 - type: nauc_recall_at_5_max value: 27.401847297469374 - type: ndcg_at_1 value: 13.616 - type: ndcg_at_10 value: 17.005 - type: ndcg_at_100 value: 24.686 - type: ndcg_at_1000 value: 29.037000000000003 - type: ndcg_at_20 value: 19.575 - type: ndcg_at_3 value: 12.867 - type: ndcg_at_5 value: 14.313 - type: precision_at_1 value: 13.616 - type: precision_at_10 value: 5.603000000000001 - type: precision_at_100 value: 1.384 - type: precision_at_1000 value: 0.218 - type: precision_at_20 value: 3.8629999999999995 - type: precision_at_3 value: 9.62 - type: precision_at_5 value: 7.686999999999999 - type: recall_at_1 value: 6.18 - type: recall_at_10 value: 22.253999999999998 - type: recall_at_100 value: 49.857 - type: recall_at_1000 value: 74.509 - type: recall_at_20 value: 29.79 - type: recall_at_3 value: 12.471 - type: recall_at_5 value: 16.154 - task: type: Retrieval dataset: type: mteb/dbpedia name: MTEB DBPedia config: default split: test revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 metrics: - type: map_at_1 value: 8.291 - type: map_at_10 value: 16.830000000000002 - type: map_at_100 value: 23.034 - type: map_at_1000 value: 24.512999999999998 - type: map_at_20 value: 19.091 - type: map_at_3 value: 12.845999999999998 - type: map_at_5 value: 14.402000000000001 - type: mrr_at_1 value: 58.75 - type: mrr_at_10 value: 68.33035714285714 - type: mrr_at_100 value: 68.71251724781709 - type: mrr_at_1000 value: 68.72409210627302 - type: mrr_at_20 value: 68.47271810039805 - type: mrr_at_3 value: 66.25 - type: mrr_at_5 value: 67.575 - type: nauc_map_at_1000_diff1 value: 27.342350846844752 - type: nauc_map_at_1000_max value: 14.188457616813544 - type: nauc_map_at_100_diff1 value: 27.710778521551514 - type: nauc_map_at_100_max value: 11.818644063311075 - type: nauc_map_at_10_diff1 value: 28.06853470498657 - type: nauc_map_at_10_max value: -0.12310961533344932 - type: nauc_map_at_1_diff1 value: 36.28948693750488 - type: nauc_map_at_1_max value: -8.633353634959779 - type: nauc_map_at_20_diff1 value: 27.934994633916343 - type: nauc_map_at_20_max value: 4.66682483439594 - type: nauc_map_at_3_diff1 value: 28.558592385022912 - type: nauc_map_at_3_max value: -6.5695573701219905 - type: nauc_map_at_5_diff1 value: 28.15758211700583 - type: nauc_map_at_5_max value: -4.438837627387033 - type: nauc_mrr_at_1000_diff1 value: 32.88741935239777 - type: nauc_mrr_at_1000_max value: 36.85520170797062 - type: nauc_mrr_at_100_diff1 value: 32.89939422961155 - type: nauc_mrr_at_100_max value: 36.86316603211002 - type: nauc_mrr_at_10_diff1 value: 32.75700243218855 - type: nauc_mrr_at_10_max value: 37.04714292361367 - type: nauc_mrr_at_1_diff1 value: 34.13253827440505 - type: nauc_mrr_at_1_max value: 33.77037186076712 - type: nauc_mrr_at_20_diff1 value: 32.80415591728181 - type: nauc_mrr_at_20_max value: 36.99253314411835 - type: nauc_mrr_at_3_diff1 value: 32.586811805982016 - type: nauc_mrr_at_3_max value: 36.12324135058626 - type: nauc_mrr_at_5_diff1 value: 33.03671664465732 - type: nauc_mrr_at_5_max value: 36.97824557212217 - type: nauc_ndcg_at_1000_diff1 value: 28.337772149714162 - type: nauc_ndcg_at_1000_max value: 25.377883958551458 - type: nauc_ndcg_at_100_diff1 value: 28.41718749082906 - type: nauc_ndcg_at_100_max value: 18.123074545882446 - type: nauc_ndcg_at_10_diff1 value: 27.74101109260006 - type: nauc_ndcg_at_10_max value: 17.69386105978732 - type: nauc_ndcg_at_1_diff1 value: 31.913034148097413 - type: nauc_ndcg_at_1_max value: 26.268179898794834 - type: nauc_ndcg_at_20_diff1 value: 27.916849997321297 - type: nauc_ndcg_at_20_max value: 15.44589123370981 - type: nauc_ndcg_at_3_diff1 value: 26.337487002010345 - type: nauc_ndcg_at_3_max value: 22.565903592425173 - type: nauc_ndcg_at_5_diff1 value: 26.902286744591574 - type: nauc_ndcg_at_5_max value: 20.51307548413365 - type: nauc_precision_at_1000_diff1 value: -10.757573462902606 - type: nauc_precision_at_1000_max value: 33.20717701523462 - type: nauc_precision_at_100_diff1 value: 4.142228479859455 - type: nauc_precision_at_100_max value: 35.16404294730987 - type: nauc_precision_at_10_diff1 value: 12.469406451834095 - type: nauc_precision_at_10_max value: 37.9712993631463 - type: nauc_precision_at_1_diff1 value: 34.13253827440505 - type: nauc_precision_at_1_max value: 33.77037186076712 - type: nauc_precision_at_20_diff1 value: 9.319269643229944 - type: nauc_precision_at_20_max value: 37.327643501927774 - type: nauc_precision_at_3_diff1 value: 16.255946329046314 - type: nauc_precision_at_3_max value: 32.91012044795064 - type: nauc_precision_at_5_diff1 value: 14.45882910211086 - type: nauc_precision_at_5_max value: 37.335976736380175 - type: nauc_recall_at_1000_diff1 value: 21.567656720257148 - type: nauc_recall_at_1000_max value: 21.46402442471943 - type: nauc_recall_at_100_diff1 value: 22.39134946314963 - type: nauc_recall_at_100_max value: 11.668568431971867 - type: nauc_recall_at_10_diff1 value: 23.937310413594133 - type: nauc_recall_at_10_max value: -1.3011457352984097 - type: nauc_recall_at_1_diff1 value: 36.28948693750488 - type: nauc_recall_at_1_max value: -8.633353634959779 - type: nauc_recall_at_20_diff1 value: 23.638781595544128 - type: nauc_recall_at_20_max value: 5.460347213047418 - type: nauc_recall_at_3_diff1 value: 26.087367269108725 - type: nauc_recall_at_3_max value: -8.392789032179113 - type: nauc_recall_at_5_diff1 value: 24.741724571041253 - type: nauc_recall_at_5_max value: -6.393820851085444 - type: ndcg_at_1 value: 47.25 - type: ndcg_at_10 value: 35.081 - type: ndcg_at_100 value: 39.969 - type: ndcg_at_1000 value: 47.847 - type: ndcg_at_20 value: 34.547 - type: ndcg_at_3 value: 39.304 - type: ndcg_at_5 value: 36.79 - type: precision_at_1 value: 58.75 - type: precision_at_10 value: 26.85 - type: precision_at_100 value: 8.703 - type: precision_at_1000 value: 1.8980000000000001 - type: precision_at_20 value: 20.037 - type: precision_at_3 value: 42.083 - type: precision_at_5 value: 35.099999999999994 - type: recall_at_1 value: 8.291 - type: recall_at_10 value: 22.274 - type: recall_at_100 value: 46.904 - type: recall_at_1000 value: 71.71 - type: recall_at_20 value: 27.898 - type: recall_at_3 value: 14.213999999999999 - type: recall_at_5 value: 17.035 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification config: default split: test revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 metrics: - type: accuracy value: 47.11 - type: f1 value: 41.2294008856841 - type: f1_weighted value: 49.015064693968355 - task: type: Retrieval dataset: type: mteb/fever name: MTEB FEVER config: default split: test revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 metrics: - type: map_at_1 value: 40.014 - type: map_at_10 value: 54.081999999999994 - type: map_at_100 value: 54.698 - type: map_at_1000 value: 54.72 - type: map_at_20 value: 54.48 - type: map_at_3 value: 50.809000000000005 - type: map_at_5 value: 52.79899999999999 - type: mrr_at_1 value: 43.009300930093005 - type: mrr_at_10 value: 57.503827763729085 - type: mrr_at_100 value: 58.049446560229754 - type: mrr_at_1000 value: 58.0600709633308 - type: mrr_at_20 value: 57.875221009859665 - type: mrr_at_3 value: 54.25292529252964 - type: mrr_at_5 value: 56.24212421242145 - type: nauc_map_at_1000_diff1 value: 37.53159168913937 - type: nauc_map_at_1000_max value: 6.8836860808007465 - type: nauc_map_at_100_diff1 value: 37.528011745089835 - type: nauc_map_at_100_max value: 6.882491514393514 - type: nauc_map_at_10_diff1 value: 37.3772602442582 - type: nauc_map_at_10_max value: 6.941476394099929 - type: nauc_map_at_1_diff1 value: 41.86294360198839 - type: nauc_map_at_1_max value: 6.023892954634666 - type: nauc_map_at_20_diff1 value: 37.427127392559335 - type: nauc_map_at_20_max value: 6.882487617592033 - type: nauc_map_at_3_diff1 value: 37.20054603111901 - type: nauc_map_at_3_max value: 6.161906578752757 - type: nauc_map_at_5_diff1 value: 37.07890205012265 - type: nauc_map_at_5_max value: 6.693250003060114 - type: nauc_mrr_at_1000_diff1 value: 40.01409327276433 - type: nauc_mrr_at_1000_max value: 6.986140706019336 - type: nauc_mrr_at_100_diff1 value: 40.01603614242155 - type: nauc_mrr_at_100_max value: 6.994969349070441 - type: nauc_mrr_at_10_diff1 value: 39.88645059936797 - type: nauc_mrr_at_10_max value: 7.128874648135309 - type: nauc_mrr_at_1_diff1 value: 44.010610515801766 - type: nauc_mrr_at_1_max value: 5.9421616518924 - type: nauc_mrr_at_20_diff1 value: 39.91760958564354 - type: nauc_mrr_at_20_max value: 7.001496247992628 - type: nauc_mrr_at_3_diff1 value: 39.55814635404465 - type: nauc_mrr_at_3_max value: 6.348916278839159 - type: nauc_mrr_at_5_diff1 value: 39.53111951703581 - type: nauc_mrr_at_5_max value: 6.90608845386575 - type: nauc_ndcg_at_1000_diff1 value: 37.410083910100454 - type: nauc_ndcg_at_1000_max value: 7.702794760075665 - type: nauc_ndcg_at_100_diff1 value: 37.24367644094166 - type: nauc_ndcg_at_100_max value: 7.748369388619449 - type: nauc_ndcg_at_10_diff1 value: 36.42739864037634 - type: nauc_ndcg_at_10_max value: 8.023602519440981 - type: nauc_ndcg_at_1_diff1 value: 44.010610515801766 - type: nauc_ndcg_at_1_max value: 5.9421616518924 - type: nauc_ndcg_at_20_diff1 value: 36.532843433534836 - type: nauc_ndcg_at_20_max value: 7.736575556014484 - type: nauc_ndcg_at_3_diff1 value: 36.195062467683485 - type: nauc_ndcg_at_3_max value: 6.351990387166732 - type: nauc_ndcg_at_5_diff1 value: 35.823128734687835 - type: nauc_ndcg_at_5_max value: 7.397593069596111 - type: nauc_precision_at_1000_diff1 value: -3.5385176540536514 - type: nauc_precision_at_1000_max value: 6.514978387518318 - type: nauc_precision_at_100_diff1 value: 3.0567737052705435 - type: nauc_precision_at_100_max value: 9.197755692738543 - type: nauc_precision_at_10_diff1 value: 22.124829864658473 - type: nauc_precision_at_10_max value: 13.41228356156045 - type: nauc_precision_at_1_diff1 value: 44.010610515801766 - type: nauc_precision_at_1_max value: 5.9421616518924 - type: nauc_precision_at_20_diff1 value: 14.960320394382231 - type: nauc_precision_at_20_max value: 12.26648981111606 - type: nauc_precision_at_3_diff1 value: 31.130457883879092 - type: nauc_precision_at_3_max value: 6.82126659383764 - type: nauc_precision_at_5_diff1 value: 26.884225039660837 - type: nauc_precision_at_5_max value: 10.124900855966871 - type: nauc_recall_at_1000_diff1 value: 17.368406898729038 - type: nauc_recall_at_1000_max value: 20.346937130202836 - type: nauc_recall_at_100_diff1 value: 20.167019111696035 - type: nauc_recall_at_100_max value: 15.751973809238756 - type: nauc_recall_at_10_diff1 value: 24.579485696194457 - type: nauc_recall_at_10_max value: 12.597490151740628 - type: nauc_recall_at_1_diff1 value: 41.86294360198839 - type: nauc_recall_at_1_max value: 6.023892954634666 - type: nauc_recall_at_20_diff1 value: 21.09351386111729 - type: nauc_recall_at_20_max value: 11.699302732967896 - type: nauc_recall_at_3_diff1 value: 29.17762647595018 - type: nauc_recall_at_3_max value: 6.461951154139278 - type: nauc_recall_at_5_diff1 value: 26.266588024737242 - type: nauc_recall_at_5_max value: 9.344922507705471 - type: ndcg_at_1 value: 43.009 - type: ndcg_at_10 value: 61.541999999999994 - type: ndcg_at_100 value: 64.292 - type: ndcg_at_1000 value: 64.781 - type: ndcg_at_20 value: 62.925 - type: ndcg_at_3 value: 55.095000000000006 - type: ndcg_at_5 value: 58.557 - type: precision_at_1 value: 43.009 - type: precision_at_10 value: 8.863 - type: precision_at_100 value: 1.0370000000000001 - type: precision_at_1000 value: 0.11 - type: precision_at_20 value: 4.742 - type: precision_at_3 value: 23.077 - type: precision_at_5 value: 15.701 - type: recall_at_1 value: 40.014 - type: recall_at_10 value: 81.319 - type: recall_at_100 value: 93.362 - type: recall_at_1000 value: 96.72500000000001 - type: recall_at_20 value: 86.54899999999999 - type: recall_at_3 value: 63.975 - type: recall_at_5 value: 72.361 - task: type: Retrieval dataset: type: mteb/fiqa name: MTEB FiQA2018 config: default split: test revision: 27a168819829fe9bcd655c2df245fb19452e8e06 metrics: - type: map_at_1 value: 14.754999999999999 - type: map_at_10 value: 23.415 - type: map_at_100 value: 25.389 - type: map_at_1000 value: 25.583 - type: map_at_20 value: 24.471999999999998 - type: map_at_3 value: 20.706 - type: map_at_5 value: 22.137 - type: mrr_at_1 value: 29.938271604938272 - type: mrr_at_10 value: 37.82272682735643 - type: mrr_at_100 value: 38.93217561179166 - type: mrr_at_1000 value: 38.993419651389594 - type: mrr_at_20 value: 38.51688181126462 - type: mrr_at_3 value: 35.49382716049384 - type: mrr_at_5 value: 36.69753086419753 - type: nauc_map_at_1000_diff1 value: 39.87836835005687 - type: nauc_map_at_1000_max value: 27.964955726459046 - type: nauc_map_at_100_diff1 value: 39.85804813979803 - type: nauc_map_at_100_max value: 27.901085428248518 - type: nauc_map_at_10_diff1 value: 40.40910564822785 - type: nauc_map_at_10_max value: 27.072531521163906 - type: nauc_map_at_1_diff1 value: 45.20997932526616 - type: nauc_map_at_1_max value: 24.648676414998096 - type: nauc_map_at_20_diff1 value: 40.028106498724725 - type: nauc_map_at_20_max value: 27.60132226396262 - type: nauc_map_at_3_diff1 value: 41.05603264971634 - type: nauc_map_at_3_max value: 26.25426542015784 - type: nauc_map_at_5_diff1 value: 41.00888735158693 - type: nauc_map_at_5_max value: 27.191449534557897 - type: nauc_mrr_at_1000_diff1 value: 41.89281932766717 - type: nauc_mrr_at_1000_max value: 35.99284962374552 - type: nauc_mrr_at_100_diff1 value: 41.8809308098677 - type: nauc_mrr_at_100_max value: 35.99275706919067 - type: nauc_mrr_at_10_diff1 value: 42.034104043793334 - type: nauc_mrr_at_10_max value: 35.76467432283676 - type: nauc_mrr_at_1_diff1 value: 46.49733800476976 - type: nauc_mrr_at_1_max value: 37.4658949472576 - type: nauc_mrr_at_20_diff1 value: 41.748351174212104 - type: nauc_mrr_at_20_max value: 35.9625951228368 - type: nauc_mrr_at_3_diff1 value: 43.03598990961282 - type: nauc_mrr_at_3_max value: 36.15714656335092 - type: nauc_mrr_at_5_diff1 value: 42.82409168973717 - type: nauc_mrr_at_5_max value: 36.196423200277614 - type: nauc_ndcg_at_1000_diff1 value: 37.73166244969865 - type: nauc_ndcg_at_1000_max value: 31.381646846120077 - type: nauc_ndcg_at_100_diff1 value: 37.23016841045615 - type: nauc_ndcg_at_100_max value: 30.46125187550548 - type: nauc_ndcg_at_10_diff1 value: 38.973941698385175 - type: nauc_ndcg_at_10_max value: 28.50687601887441 - type: nauc_ndcg_at_1_diff1 value: 46.49733800476976 - type: nauc_ndcg_at_1_max value: 37.4658949472576 - type: nauc_ndcg_at_20_diff1 value: 37.789595359314184 - type: nauc_ndcg_at_20_max value: 29.22246454793801 - type: nauc_ndcg_at_3_diff1 value: 41.76335349611918 - type: nauc_ndcg_at_3_max value: 32.83475312409831 - type: nauc_ndcg_at_5_diff1 value: 41.107481866851366 - type: nauc_ndcg_at_5_max value: 30.504284365072547 - type: nauc_precision_at_1000_diff1 value: 2.5166465109243528 - type: nauc_precision_at_1000_max value: 24.899624433788485 - type: nauc_precision_at_100_diff1 value: 10.522599150582685 - type: nauc_precision_at_100_max value: 29.50873923975521 - type: nauc_precision_at_10_diff1 value: 27.120890973936778 - type: nauc_precision_at_10_max value: 32.955042844361046 - type: nauc_precision_at_1_diff1 value: 46.49733800476976 - type: nauc_precision_at_1_max value: 37.4658949472576 - type: nauc_precision_at_20_diff1 value: 20.299314592682165 - type: nauc_precision_at_20_max value: 33.25503440470617 - type: nauc_precision_at_3_diff1 value: 35.273519611653626 - type: nauc_precision_at_3_max value: 36.33062698057323 - type: nauc_precision_at_5_diff1 value: 33.590199611021006 - type: nauc_precision_at_5_max value: 36.68276540744115 - type: nauc_recall_at_1000_diff1 value: 8.618184855067168 - type: nauc_recall_at_1000_max value: 21.822927142441923 - type: nauc_recall_at_100_diff1 value: 17.055234719322666 - type: nauc_recall_at_100_max value: 19.555357018393373 - type: nauc_recall_at_10_diff1 value: 28.06102408728667 - type: nauc_recall_at_10_max value: 19.309443630849117 - type: nauc_recall_at_1_diff1 value: 45.20997932526616 - type: nauc_recall_at_1_max value: 24.648676414998096 - type: nauc_recall_at_20_diff1 value: 23.428917950250966 - type: nauc_recall_at_20_max value: 19.351922857081917 - type: nauc_recall_at_3_diff1 value: 35.290875728597044 - type: nauc_recall_at_3_max value: 22.777706184795857 - type: nauc_recall_at_5_diff1 value: 33.580206464404554 - type: nauc_recall_at_5_max value: 22.290191050692897 - type: ndcg_at_1 value: 29.938 - type: ndcg_at_10 value: 29.974 - type: ndcg_at_100 value: 37.791999999999994 - type: ndcg_at_1000 value: 41.525 - type: ndcg_at_20 value: 33.032000000000004 - type: ndcg_at_3 value: 27.418 - type: ndcg_at_5 value: 27.925 - type: precision_at_1 value: 29.938 - type: precision_at_10 value: 8.41 - type: precision_at_100 value: 1.608 - type: precision_at_1000 value: 0.22599999999999998 - type: precision_at_20 value: 5.409 - type: precision_at_3 value: 18.724 - type: precision_at_5 value: 13.456999999999999 - type: recall_at_1 value: 14.754999999999999 - type: recall_at_10 value: 34.863 - type: recall_at_100 value: 64.471 - type: recall_at_1000 value: 87.386 - type: recall_at_20 value: 44.574999999999996 - type: recall_at_3 value: 24.359 - type: recall_at_5 value: 28.442 - task: type: Retrieval dataset: type: mteb/hotpotqa name: MTEB HotpotQA config: default split: test revision: ab518f4d6fcca38d87c25209f94beba119d02014 metrics: - type: map_at_1 value: 35.476 - type: map_at_10 value: 55.208 - type: map_at_100 value: 56.152 - type: map_at_1000 value: 56.22200000000001 - type: map_at_20 value: 55.754000000000005 - type: map_at_3 value: 51.742999999999995 - type: map_at_5 value: 53.900000000000006 - type: mrr_at_1 value: 70.95205941931127 - type: mrr_at_10 value: 78.42775687812804 - type: mrr_at_100 value: 78.68481706123845 - type: mrr_at_1000 value: 78.69468662644054 - type: mrr_at_20 value: 78.59334220969656 - type: mrr_at_3 value: 77.21584514967346 - type: mrr_at_5 value: 78.01395453522359 - type: nauc_map_at_1000_diff1 value: 25.86438441197036 - type: nauc_map_at_1000_max value: 24.208328720722367 - type: nauc_map_at_100_diff1 value: 25.836404334583698 - type: nauc_map_at_100_max value: 24.195017226851576 - type: nauc_map_at_10_diff1 value: 25.722782423795227 - type: nauc_map_at_10_max value: 23.987656322283062 - type: nauc_map_at_1_diff1 value: 65.41482575591807 - type: nauc_map_at_1_max value: 36.750296668618084 - type: nauc_map_at_20_diff1 value: 25.748480864914974 - type: nauc_map_at_20_max value: 24.135897754765367 - type: nauc_map_at_3_diff1 value: 26.38356826015294 - type: nauc_map_at_3_max value: 23.621729891258973 - type: nauc_map_at_5_diff1 value: 26.0945105687306 - type: nauc_map_at_5_max value: 23.988655536143728 - type: nauc_mrr_at_1000_diff1 value: 64.17645353489316 - type: nauc_mrr_at_1000_max value: 39.676745401203924 - type: nauc_mrr_at_100_diff1 value: 64.17493074038984 - type: nauc_mrr_at_100_max value: 39.677377348237144 - type: nauc_mrr_at_10_diff1 value: 64.11955469392967 - type: nauc_mrr_at_10_max value: 39.82247333641905 - type: nauc_mrr_at_1_diff1 value: 65.41482575591807 - type: nauc_mrr_at_1_max value: 36.750296668618084 - type: nauc_mrr_at_20_diff1 value: 64.15026091520691 - type: nauc_mrr_at_20_max value: 39.73722517116609 - type: nauc_mrr_at_3_diff1 value: 63.9908644633026 - type: nauc_mrr_at_3_max value: 39.70016359929288 - type: nauc_mrr_at_5_diff1 value: 64.13062355820735 - type: nauc_mrr_at_5_max value: 39.90280825011535 - type: nauc_ndcg_at_1000_diff1 value: 31.23415779213086 - type: nauc_ndcg_at_1000_max value: 27.776674450652884 - type: nauc_ndcg_at_100_diff1 value: 30.35823573217142 - type: nauc_ndcg_at_100_max value: 27.34738082248897 - type: nauc_ndcg_at_10_diff1 value: 30.106808177162254 - type: nauc_ndcg_at_10_max value: 26.953412093415036 - type: nauc_ndcg_at_1_diff1 value: 65.41482575591807 - type: nauc_ndcg_at_1_max value: 36.750296668618084 - type: nauc_ndcg_at_20_diff1 value: 29.95512057702211 - type: nauc_ndcg_at_20_max value: 27.167767360883598 - type: nauc_ndcg_at_3_diff1 value: 31.769187019307232 - type: nauc_ndcg_at_3_max value: 26.73293484943561 - type: nauc_ndcg_at_5_diff1 value: 30.990681805979882 - type: nauc_ndcg_at_5_max value: 27.06261773043383 - type: nauc_precision_at_1000_diff1 value: 2.7411864902028804 - type: nauc_precision_at_1000_max value: 21.87546627838 - type: nauc_precision_at_100_diff1 value: 6.07948946116136 - type: nauc_precision_at_100_max value: 19.74129594300319 - type: nauc_precision_at_10_diff1 value: 12.773005438330138 - type: nauc_precision_at_10_max value: 21.240483881434862 - type: nauc_precision_at_1_diff1 value: 65.41482575591807 - type: nauc_precision_at_1_max value: 36.750296668618084 - type: nauc_precision_at_20_diff1 value: 10.059987751705838 - type: nauc_precision_at_20_max value: 20.886351066202028 - type: nauc_precision_at_3_diff1 value: 19.42571093704271 - type: nauc_precision_at_3_max value: 22.725022430676766 - type: nauc_precision_at_5_diff1 value: 16.630466686718133 - type: nauc_precision_at_5_max value: 22.43986381185507 - type: nauc_recall_at_1000_diff1 value: 2.741186490202987 - type: nauc_recall_at_1000_max value: 21.875466278380117 - type: nauc_recall_at_100_diff1 value: 6.079489461161415 - type: nauc_recall_at_100_max value: 19.74129594300323 - type: nauc_recall_at_10_diff1 value: 12.77300543833007 - type: nauc_recall_at_10_max value: 21.24048388143479 - type: nauc_recall_at_1_diff1 value: 65.41482575591807 - type: nauc_recall_at_1_max value: 36.750296668618084 - type: nauc_recall_at_20_diff1 value: 10.059987751705897 - type: nauc_recall_at_20_max value: 20.88635106620212 - type: nauc_recall_at_3_diff1 value: 19.42571093704268 - type: nauc_recall_at_3_max value: 22.725022430676713 - type: nauc_recall_at_5_diff1 value: 16.630466686718112 - type: nauc_recall_at_5_max value: 22.439863811855034 - type: ndcg_at_1 value: 70.952 - type: ndcg_at_10 value: 64.154 - type: ndcg_at_100 value: 67.54299999999999 - type: ndcg_at_1000 value: 68.931 - type: ndcg_at_20 value: 65.556 - type: ndcg_at_3 value: 59.10300000000001 - type: ndcg_at_5 value: 61.92100000000001 - type: precision_at_1 value: 70.952 - type: precision_at_10 value: 13.611999999999998 - type: precision_at_100 value: 1.627 - type: precision_at_1000 value: 0.181 - type: precision_at_20 value: 7.258000000000001 - type: precision_at_3 value: 37.929 - type: precision_at_5 value: 24.986 - type: recall_at_1 value: 35.476 - type: recall_at_10 value: 68.062 - type: recall_at_100 value: 81.34400000000001 - type: recall_at_1000 value: 90.567 - type: recall_at_20 value: 72.57900000000001 - type: recall_at_3 value: 56.894 - type: recall_at_5 value: 62.465 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification config: default split: test revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 metrics: - type: accuracy value: 87.392 - type: ap value: 82.30893640195251 - type: ap_weighted value: 82.30893640195251 - type: f1 value: 87.35348856878436 - type: f1_weighted value: 87.35348856878437 - task: type: Retrieval dataset: type: mteb/msmarco name: MTEB MSMARCO config: default split: test revision: c5a29a104738b98a9e76336939199e264163d4a0 metrics: - type: main_score value: 37.701 - type: map_at_1 value: 19.483 - type: map_at_10 value: 31.148 - type: map_at_100 value: 32.402 - type: map_at_1000 value: 32.462 - type: map_at_20 value: 31.921 - type: map_at_3 value: 27.509 - type: map_at_5 value: 29.521 - type: mrr_at_1 value: 19.95702005730659 - type: mrr_at_10 value: 31.629388957110927 - type: mrr_at_100 value: 32.835516469756676 - type: mrr_at_1000 value: 32.889168712361033 - type: mrr_at_20 value: 32.38456688305533 - type: mrr_at_3 value: 28.04680038204374 - type: mrr_at_5 value: 30.005969436485114 - type: nauc_map_at_1000_diff1 value: 32.10821100880179 - type: nauc_map_at_1000_max value: 0.7085504734279151 - type: nauc_map_at_1000_std value: -9.657081445961803 - type: nauc_map_at_100_diff1 value: 32.1022582818382 - type: nauc_map_at_100_max value: 0.7094263225288184 - type: nauc_map_at_100_std value: -9.615141441552195 - type: nauc_map_at_10_diff1 value: 32.123241507654395 - type: nauc_map_at_10_max value: 0.6031410503880614 - type: nauc_map_at_10_std value: -10.446787175853538 - type: nauc_map_at_1_diff1 value: 33.742442076477175 - type: nauc_map_at_1_max value: -0.4682625514356794 - type: nauc_map_at_1_std value: -10.128931659708205 - type: nauc_map_at_20_diff1 value: 32.0860174618957 - type: nauc_map_at_20_max value: 0.6702984410061132 - type: nauc_map_at_20_std value: -9.906667239123405 - type: nauc_map_at_3_diff1 value: 32.29226575652717 - type: nauc_map_at_3_max value: 0.0973438744261487 - type: nauc_map_at_3_std value: -11.124768365213317 - type: nauc_map_at_5_diff1 value: 32.08229672223041 - type: nauc_map_at_5_max value: 0.2176450911149463 - type: nauc_map_at_5_std value: -11.055413229428249 - type: nauc_mrr_at_1000_diff1 value: 31.895348209132734 - type: nauc_mrr_at_1000_max value: 0.7016011591304079 - type: nauc_mrr_at_1000_std value: -9.67662400401361 - type: nauc_mrr_at_100_diff1 value: 31.8866191175737 - type: nauc_mrr_at_100_max value: 0.7081721058731258 - type: nauc_mrr_at_100_std value: -9.633818059393717 - type: nauc_mrr_at_10_diff1 value: 31.89471221387619 - type: nauc_mrr_at_10_max value: 0.6339660994085835 - type: nauc_mrr_at_10_std value: -10.40302377816781 - type: nauc_mrr_at_1_diff1 value: 33.51497482281671 - type: nauc_mrr_at_1_max value: -0.38461440436551426 - type: nauc_mrr_at_1_std value: -10.229237135864577 - type: nauc_mrr_at_20_diff1 value: 31.858413430545546 - type: nauc_mrr_at_20_max value: 0.6976488035968785 - type: nauc_mrr_at_20_std value: -9.889412090836752 - type: nauc_mrr_at_3_diff1 value: 32.07820212397262 - type: nauc_mrr_at_3_max value: 0.023781593606752303 - type: nauc_mrr_at_3_std value: -11.225901208970873 - type: nauc_mrr_at_5_diff1 value: 31.880984335275736 - type: nauc_mrr_at_5_max value: 0.199684088046682 - type: nauc_mrr_at_5_std value: -11.076286377930138 - type: nauc_ndcg_at_1000_diff1 value: 31.54073203534171 - type: nauc_ndcg_at_1000_max value: 1.865885878739077 - type: nauc_ndcg_at_1000_std value: -7.111772836122038 - type: nauc_ndcg_at_100_diff1 value: 31.37901877227891 - type: nauc_ndcg_at_100_max value: 1.9480183608898675 - type: nauc_ndcg_at_100_std value: -5.691552787254062 - type: nauc_ndcg_at_10_diff1 value: 31.432464505042673 - type: nauc_ndcg_at_10_max value: 1.37534724565308 - type: nauc_ndcg_at_10_std value: -9.688757807073005 - type: nauc_ndcg_at_1_diff1 value: 33.44622804608476 - type: nauc_ndcg_at_1_max value: -0.4275117298270247 - type: nauc_ndcg_at_1_std value: -10.195443857766566 - type: nauc_ndcg_at_20_diff1 value: 31.27757208526727 - type: nauc_ndcg_at_20_max value: 1.6486846485417688 - type: nauc_ndcg_at_20_std value: -7.731364695550273 - type: nauc_ndcg_at_3_diff1 value: 31.833281451625756 - type: nauc_ndcg_at_3_max value: 0.248993633296169 - type: nauc_ndcg_at_3_std value: -11.375142227624853 - type: nauc_ndcg_at_5_diff1 value: 31.450773538855686 - type: nauc_ndcg_at_5_max value: 0.4439465859365249 - type: nauc_ndcg_at_5_std value: -11.21243082848607 - type: nauc_precision_at_1000_diff1 value: -3.189589259900143 - type: nauc_precision_at_1000_max value: 16.455244940685285 - type: nauc_precision_at_1000_std value: 14.259564795238847 - type: nauc_precision_at_100_diff1 value: 14.87654355875184 - type: nauc_precision_at_100_max value: 11.935690745346687 - type: nauc_precision_at_100_std value: 23.08831102598726 - type: nauc_precision_at_10_diff1 value: 27.37421025197246 - type: nauc_precision_at_10_max value: 4.166107809672891 - type: nauc_precision_at_10_std value: -6.881018792063406 - type: nauc_precision_at_1_diff1 value: 33.44622804608476 - type: nauc_precision_at_1_max value: -0.4275117298270247 - type: nauc_precision_at_1_std value: -10.195443857766566 - type: nauc_precision_at_20_diff1 value: 24.609029763068263 - type: nauc_precision_at_20_max value: 6.284294179586118 - type: nauc_precision_at_20_std value: 1.9907354399675788 - type: nauc_precision_at_3_diff1 value: 30.032708101293275 - type: nauc_precision_at_3_max value: 0.37236747435402825 - type: nauc_precision_at_3_std value: -12.337214267890699 - type: nauc_precision_at_5_diff1 value: 28.660692204010885 - type: nauc_precision_at_5_max value: 1.2268562134706205 - type: nauc_precision_at_5_std value: -11.633329112055832 - type: nauc_recall_at_1000_diff1 value: 25.817560788852197 - type: nauc_recall_at_1000_max value: 36.86227659755896 - type: nauc_recall_at_1000_std value: 63.94303180569221 - type: nauc_recall_at_100_diff1 value: 27.1728992415636 - type: nauc_recall_at_100_max value: 11.548635552653852 - type: nauc_recall_at_100_std value: 31.974599323501035 - type: nauc_recall_at_10_diff1 value: 29.294882367129405 - type: nauc_recall_at_10_max value: 3.763234551980347 - type: nauc_recall_at_10_std value: -6.861891907951573 - type: nauc_recall_at_1_diff1 value: 33.742442076477175 - type: nauc_recall_at_1_max value: -0.4682625514356794 - type: nauc_recall_at_1_std value: -10.128931659708205 - type: nauc_recall_at_20_diff1 value: 28.258984645981583 - type: nauc_recall_at_20_max value: 5.291852480348364 - type: nauc_recall_at_20_std value: 2.243022569638662 - type: nauc_recall_at_3_diff1 value: 30.6436214049594 - type: nauc_recall_at_3_max value: 0.7472464014018717 - type: nauc_recall_at_3_std value: -11.831114059680124 - type: nauc_recall_at_5_diff1 value: 29.74258077127522 - type: nauc_recall_at_5_max value: 1.0386095329679946 - type: nauc_recall_at_5_std value: -11.487480890097884 - type: ndcg_at_1 value: 19.971 - type: ndcg_at_10 value: 37.701 - type: ndcg_at_100 value: 43.71 - type: ndcg_at_1000 value: 45.191 - type: ndcg_at_20 value: 40.421 - type: ndcg_at_3 value: 30.25 - type: ndcg_at_5 value: 33.816 - type: precision_at_1 value: 19.971 - type: precision_at_10 value: 6.037 - type: precision_at_100 value: 0.904 - type: precision_at_1000 value: 0.103 - type: precision_at_20 value: 3.58 - type: precision_at_3 value: 12.97 - type: precision_at_5 value: 9.59 - type: recall_at_1 value: 19.483 - type: recall_at_10 value: 57.972 - type: recall_at_100 value: 85.639 - type: recall_at_1000 value: 96.954 - type: recall_at_20 value: 68.456 - type: recall_at_3 value: 37.628 - type: recall_at_5 value: 46.169 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) config: en split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 92.86365709074327 - type: f1 value: 92.6364668121745 - type: f1_weighted value: 92.86565911176659 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) config: en split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 64.8518011855905 - type: f1 value: 46.19074540696208 - type: f1_weighted value: 67.21842739260879 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 68.86348352387357 - type: f1 value: 66.61591434283466 - type: f1_weighted value: 67.89865692906163 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 75.24882313382649 - type: f1 value: 74.48847159335763 - type: f1_weighted value: 74.9792940543356 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P config: default split: test revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 metrics: - type: v_measure value: 31.112200560363956 - type: v_measures value: [0.2907912540071734, 0.30018368642906285, 0.2969957025066252, 0.3046412932920165, 0.3023643866062867] - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S config: default split: test revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 metrics: - type: v_measure value: 29.10468325586837 - type: v_measures value: [0.2762290794660957, 0.27088303670750474, 0.27869334502955523, 0.2877927347410669, 0.283558635107815] - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking config: default split: test revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 metrics: - type: map value: 31.65436388779704 - type: mrr value: 32.70570836057599 - type: nAUC_map_diff1 value: 11.29046170729857 - type: nAUC_map_max value: -21.303851901712402 - type: nAUC_mrr_diff1 value: 10.839304152084038 - type: nAUC_mrr_max value: -15.980370460373264 - task: type: Retrieval dataset: type: mteb/nfcorpus name: MTEB NFCorpus config: default split: test revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 metrics: - type: map_at_1 value: 5.349 - type: map_at_10 value: 11.943 - type: map_at_100 value: 14.838999999999999 - type: map_at_1000 value: 16.228 - type: map_at_20 value: 13.255 - type: map_at_3 value: 8.712 - type: map_at_5 value: 10.078 - type: mrr_at_1 value: 41.48606811145511 - type: mrr_at_10 value: 50.93174111749962 - type: mrr_at_100 value: 51.51424977382773 - type: mrr_at_1000 value: 51.54588577020572 - type: mrr_at_20 value: 51.23754793809378 - type: mrr_at_3 value: 48.14241486068111 - type: mrr_at_5 value: 50.123839009287906 - type: nauc_map_at_1000_diff1 value: 24.080224706969375 - type: nauc_map_at_1000_max value: 19.837568780211047 - type: nauc_map_at_100_diff1 value: 24.547349720088924 - type: nauc_map_at_100_max value: 18.76780474051508 - type: nauc_map_at_10_diff1 value: 27.36097408135702 - type: nauc_map_at_10_max value: 12.918546361288225 - type: nauc_map_at_1_diff1 value: 37.536183652718385 - type: nauc_map_at_1_max value: -1.1491546718720773 - type: nauc_map_at_20_diff1 value: 25.802748647118108 - type: nauc_map_at_20_max value: 15.721455228191134 - type: nauc_map_at_3_diff1 value: 34.211955433524196 - type: nauc_map_at_3_max value: 6.1567837982868 - type: nauc_map_at_5_diff1 value: 31.355144847293875 - type: nauc_map_at_5_max value: 9.248778872692705 - type: nauc_mrr_at_1000_diff1 value: 32.6636800918941 - type: nauc_mrr_at_1000_max value: 38.762875976765784 - type: nauc_mrr_at_100_diff1 value: 32.68119918270669 - type: nauc_mrr_at_100_max value: 38.78855177547366 - type: nauc_mrr_at_10_diff1 value: 32.6341499099646 - type: nauc_mrr_at_10_max value: 38.57304636650009 - type: nauc_mrr_at_1_diff1 value: 33.31813734214396 - type: nauc_mrr_at_1_max value: 32.615374725091584 - type: nauc_mrr_at_20_diff1 value: 32.70066434653636 - type: nauc_mrr_at_20_max value: 38.65649435307692 - type: nauc_mrr_at_3_diff1 value: 31.970185140740586 - type: nauc_mrr_at_3_max value: 37.36826662425585 - type: nauc_mrr_at_5_diff1 value: 32.75800999798963 - type: nauc_mrr_at_5_max value: 38.54932999541346 - type: nauc_ndcg_at_1000_diff1 value: 24.176402213150016 - type: nauc_ndcg_at_1000_max value: 37.64114068923691 - type: nauc_ndcg_at_100_diff1 value: 24.70407050925309 - type: nauc_ndcg_at_100_max value: 32.22741268616694 - type: nauc_ndcg_at_10_diff1 value: 22.754607233929026 - type: nauc_ndcg_at_10_max value: 31.40000991648274 - type: nauc_ndcg_at_1_diff1 value: 35.00398764062468 - type: nauc_ndcg_at_1_max value: 31.74496244016824 - type: nauc_ndcg_at_20_diff1 value: 22.663343998503983 - type: nauc_ndcg_at_20_max value: 29.9805490570407 - type: nauc_ndcg_at_3_diff1 value: 28.436841714938453 - type: nauc_ndcg_at_3_max value: 32.998271531027235 - type: nauc_ndcg_at_5_diff1 value: 25.421553164974465 - type: nauc_ndcg_at_5_max value: 32.18602590821988 - type: nauc_precision_at_1000_diff1 value: -0.3756162839426823 - type: nauc_precision_at_1000_max value: 10.910413856430598 - type: nauc_precision_at_100_diff1 value: 1.1336658089729603 - type: nauc_precision_at_100_max value: 22.634879988362034 - type: nauc_precision_at_10_diff1 value: 6.74407085117157 - type: nauc_precision_at_10_max value: 34.47976455090075 - type: nauc_precision_at_1_diff1 value: 33.31813734214396 - type: nauc_precision_at_1_max value: 32.615374725091584 - type: nauc_precision_at_20_diff1 value: 4.689841488175409 - type: nauc_precision_at_20_max value: 30.590963857775392 - type: nauc_precision_at_3_diff1 value: 20.55747518226839 - type: nauc_precision_at_3_max value: 37.32360882162725 - type: nauc_precision_at_5_diff1 value: 13.519365761095049 - type: nauc_precision_at_5_max value: 36.50427925753384 - type: nauc_recall_at_1000_diff1 value: 6.311137710079621 - type: nauc_recall_at_1000_max value: 17.25930964555457 - type: nauc_recall_at_100_diff1 value: 12.070904435192633 - type: nauc_recall_at_100_max value: 16.897787779896493 - type: nauc_recall_at_10_diff1 value: 18.611832015375775 - type: nauc_recall_at_10_max value: 11.574222141528578 - type: nauc_recall_at_1_diff1 value: 37.536183652718385 - type: nauc_recall_at_1_max value: -1.1491546718720773 - type: nauc_recall_at_20_diff1 value: 16.07623501237203 - type: nauc_recall_at_20_max value: 13.067066121164272 - type: nauc_recall_at_3_diff1 value: 30.081195021905327 - type: nauc_recall_at_3_max value: 5.842397752446413 - type: nauc_recall_at_5_diff1 value: 24.93661141138374 - type: nauc_recall_at_5_max value: 9.101458243965707 - type: ndcg_at_1 value: 40.402 - type: ndcg_at_10 value: 32.586999999999996 - type: ndcg_at_100 value: 29.584 - type: ndcg_at_1000 value: 38.547 - type: ndcg_at_20 value: 30.375999999999998 - type: ndcg_at_3 value: 36.538 - type: ndcg_at_5 value: 35.221999999999994 - type: precision_at_1 value: 41.486000000000004 - type: precision_at_10 value: 24.334 - type: precision_at_100 value: 7.424 - type: precision_at_1000 value: 2.0340000000000003 - type: precision_at_20 value: 17.91 - type: precision_at_3 value: 34.056 - type: precision_at_5 value: 30.464000000000002 - type: recall_at_1 value: 5.349 - type: recall_at_10 value: 16.242 - type: recall_at_100 value: 29.945 - type: recall_at_1000 value: 62.05199999999999 - type: recall_at_20 value: 19.902 - type: recall_at_3 value: 9.876 - type: recall_at_5 value: 12.572 - task: type: Retrieval dataset: type: mteb/nq name: MTEB NQ config: default split: test revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 metrics: - type: map_at_1 value: 29.425 - type: map_at_10 value: 44.198 - type: map_at_100 value: 45.295 - type: map_at_1000 value: 45.33 - type: map_at_20 value: 44.936 - type: map_at_3 value: 39.875 - type: map_at_5 value: 42.436 - type: mrr_at_1 value: 33.25608342989571 - type: mrr_at_10 value: 46.87629899391188 - type: mrr_at_100 value: 47.690164701118185 - type: mrr_at_1000 value: 47.71428403625546 - type: mrr_at_20 value: 47.42833555993071 - type: mrr_at_3 value: 43.34685206643475 - type: mrr_at_5 value: 45.51950560061793 - type: nauc_map_at_1000_diff1 value: 27.312226360073222 - type: nauc_map_at_1000_max value: 22.962888694992817 - type: nauc_map_at_100_diff1 value: 27.31236017939728 - type: nauc_map_at_100_max value: 22.98214732521748 - type: nauc_map_at_10_diff1 value: 27.124754062247607 - type: nauc_map_at_10_max value: 22.98523700527181 - type: nauc_map_at_1_diff1 value: 30.166695274339734 - type: nauc_map_at_1_max value: 16.94040610050655 - type: nauc_map_at_20_diff1 value: 27.347142209622294 - type: nauc_map_at_20_max value: 23.0670669691066 - type: nauc_map_at_3_diff1 value: 26.632306929470623 - type: nauc_map_at_3_max value: 21.481691542717822 - type: nauc_map_at_5_diff1 value: 27.287804705337535 - type: nauc_map_at_5_max value: 22.493844564235342 - type: nauc_mrr_at_1000_diff1 value: 27.30127737418056 - type: nauc_mrr_at_1000_max value: 22.787859374079716 - type: nauc_mrr_at_100_diff1 value: 27.303876802541748 - type: nauc_mrr_at_100_max value: 22.80721855197395 - type: nauc_mrr_at_10_diff1 value: 27.186611528500187 - type: nauc_mrr_at_10_max value: 22.859824168434336 - type: nauc_mrr_at_1_diff1 value: 29.671840768485236 - type: nauc_mrr_at_1_max value: 18.62641271291652 - type: nauc_mrr_at_20_diff1 value: 27.319675868007113 - type: nauc_mrr_at_20_max value: 22.87274340612363 - type: nauc_mrr_at_3_diff1 value: 26.61707654253056 - type: nauc_mrr_at_3_max value: 22.025508910524508 - type: nauc_mrr_at_5_diff1 value: 27.275599678159214 - type: nauc_mrr_at_5_max value: 22.817841124325863 - type: nauc_ndcg_at_1000_diff1 value: 27.024135971460566 - type: nauc_ndcg_at_1000_max value: 24.731731553474926 - type: nauc_ndcg_at_100_diff1 value: 27.04262454578955 - type: nauc_ndcg_at_100_max value: 25.345992976621467 - type: nauc_ndcg_at_10_diff1 value: 26.52062330830679 - type: nauc_ndcg_at_10_max value: 25.54987961260865 - type: nauc_ndcg_at_1_diff1 value: 29.75855826437188 - type: nauc_ndcg_at_1_max value: 18.533988613886347 - type: nauc_ndcg_at_20_diff1 value: 27.179672428049358 - type: nauc_ndcg_at_20_max value: 25.868852780159955 - type: nauc_ndcg_at_3_diff1 value: 25.73603612782681 - type: nauc_ndcg_at_3_max value: 22.822406256607508 - type: nauc_ndcg_at_5_diff1 value: 26.866765656608166 - type: nauc_ndcg_at_5_max value: 24.563327197364192 - type: nauc_precision_at_1000_diff1 value: -3.6671099515474546 - type: nauc_precision_at_1000_max value: 8.392883425148419 - type: nauc_precision_at_100_diff1 value: 1.6665941155151767 - type: nauc_precision_at_100_max value: 17.17133780287315 - type: nauc_precision_at_10_diff1 value: 12.52603911510069 - type: nauc_precision_at_10_max value: 27.829277569200784 - type: nauc_precision_at_1_diff1 value: 29.75855826437188 - type: nauc_precision_at_1_max value: 18.533988613886347 - type: nauc_precision_at_20_diff1 value: 10.394727205117283 - type: nauc_precision_at_20_max value: 25.940765842937154 - type: nauc_precision_at_3_diff1 value: 18.89131093283048 - type: nauc_precision_at_3_max value: 27.157524819932267 - type: nauc_precision_at_5_diff1 value: 18.540350280124553 - type: nauc_precision_at_5_max value: 28.899931159479568 - type: nauc_recall_at_1000_diff1 value: 17.289718955680986 - type: nauc_recall_at_1000_max value: 69.19421724583071 - type: nauc_recall_at_100_diff1 value: 24.72129337125488 - type: nauc_recall_at_100_max value: 50.247836461213026 - type: nauc_recall_at_10_diff1 value: 23.094557038973154 - type: nauc_recall_at_10_max value: 34.688707228391436 - type: nauc_recall_at_1_diff1 value: 30.166695274339734 - type: nauc_recall_at_1_max value: 16.94040610050655 - type: nauc_recall_at_20_diff1 value: 26.066983320875796 - type: nauc_recall_at_20_max value: 40.1137815499548 - type: nauc_recall_at_3_diff1 value: 22.431670311997458 - type: nauc_recall_at_3_max value: 25.214444970878137 - type: nauc_recall_at_5_diff1 value: 24.77117131128259 - type: nauc_recall_at_5_max value: 29.723938351180635 - type: ndcg_at_1 value: 33.227000000000004 - type: ndcg_at_10 value: 51.919000000000004 - type: ndcg_at_100 value: 56.484 - type: ndcg_at_1000 value: 57.282999999999994 - type: ndcg_at_20 value: 54.269999999999996 - type: ndcg_at_3 value: 43.828 - type: ndcg_at_5 value: 48.098 - type: precision_at_1 value: 33.227000000000004 - type: precision_at_10 value: 8.652999999999999 - type: precision_at_100 value: 1.123 - type: precision_at_1000 value: 0.12 - type: precision_at_20 value: 4.89 - type: precision_at_3 value: 20.133000000000003 - type: precision_at_5 value: 14.513000000000002 - type: recall_at_1 value: 29.425 - type: recall_at_10 value: 72.82 - type: recall_at_100 value: 92.538 - type: recall_at_1000 value: 98.424 - type: recall_at_20 value: 81.477 - type: recall_at_3 value: 51.815 - type: recall_at_5 value: 61.667 - task: type: Retrieval dataset: type: mteb/quora name: MTEB QuoraRetrieval config: default split: test revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 metrics: - type: map_at_1 value: 69.82000000000001 - type: map_at_10 value: 83.673 - type: map_at_100 value: 84.344 - type: map_at_1000 value: 84.36099999999999 - type: map_at_20 value: 84.112 - type: map_at_3 value: 80.729 - type: map_at_5 value: 82.61500000000001 - type: mrr_at_1 value: 80.53 - type: mrr_at_10 value: 86.92019047619019 - type: mrr_at_100 value: 87.04157853007743 - type: mrr_at_1000 value: 87.04266956945463 - type: mrr_at_20 value: 87.01226011501363 - type: mrr_at_3 value: 85.9499999999997 - type: mrr_at_5 value: 86.63349999999961 - type: nauc_map_at_1000_diff1 value: 75.65678265654154 - type: nauc_map_at_1000_max value: 31.033445730504262 - type: nauc_map_at_100_diff1 value: 75.665676378331 - type: nauc_map_at_100_max value: 31.01031193403524 - type: nauc_map_at_10_diff1 value: 75.92155211732666 - type: nauc_map_at_10_max value: 30.699553005920876 - type: nauc_map_at_1_diff1 value: 78.81016014060471 - type: nauc_map_at_1_max value: 22.814760471331834 - type: nauc_map_at_20_diff1 value: 75.74929629006137 - type: nauc_map_at_20_max value: 30.893554874139433 - type: nauc_map_at_3_diff1 value: 76.1764115683232 - type: nauc_map_at_3_max value: 28.16660891891235 - type: nauc_map_at_5_diff1 value: 76.09603035768426 - type: nauc_map_at_5_max value: 29.56936639444799 - type: nauc_mrr_at_1000_diff1 value: 75.78580796966573 - type: nauc_mrr_at_1000_max value: 34.02109834360725 - type: nauc_mrr_at_100_diff1 value: 75.78595527403508 - type: nauc_mrr_at_100_max value: 34.021087682823335 - type: nauc_mrr_at_10_diff1 value: 75.7979018845172 - type: nauc_mrr_at_10_max value: 34.135531496766916 - type: nauc_mrr_at_1_diff1 value: 76.43928247510952 - type: nauc_mrr_at_1_max value: 33.337959508505584 - type: nauc_mrr_at_20_diff1 value: 75.79212450365954 - type: nauc_mrr_at_20_max value: 34.06210079146971 - type: nauc_mrr_at_3_diff1 value: 75.5255736362254 - type: nauc_mrr_at_3_max value: 33.89721705659265 - type: nauc_mrr_at_5_diff1 value: 75.68937847088145 - type: nauc_mrr_at_5_max value: 34.02467565216077 - type: nauc_ndcg_at_1000_diff1 value: 75.4126317877968 - type: nauc_ndcg_at_1000_max value: 32.699594674546205 - type: nauc_ndcg_at_100_diff1 value: 75.47759102907804 - type: nauc_ndcg_at_100_max value: 32.66261940818275 - type: nauc_ndcg_at_10_diff1 value: 75.73588489669638 - type: nauc_ndcg_at_10_max value: 32.33245124470892 - type: nauc_ndcg_at_1_diff1 value: 76.45767684895692 - type: nauc_ndcg_at_1_max value: 33.20840934731383 - type: nauc_ndcg_at_20_diff1 value: 75.58603380019049 - type: nauc_ndcg_at_20_max value: 32.38108118316705 - type: nauc_ndcg_at_3_diff1 value: 74.93057981530794 - type: nauc_ndcg_at_3_max value: 30.454945230766768 - type: nauc_ndcg_at_5_diff1 value: 75.51622080687214 - type: nauc_ndcg_at_5_max value: 31.13850590682269 - type: nauc_precision_at_1000_diff1 value: -43.024967717775205 - type: nauc_precision_at_1000_max value: -3.1998109745927548 - type: nauc_precision_at_100_diff1 value: -42.515126578347335 - type: nauc_precision_at_100_max value: -2.9690115801151893 - type: nauc_precision_at_10_diff1 value: -35.9659880370386 - type: nauc_precision_at_10_max value: 1.9839275718355172 - type: nauc_precision_at_1_diff1 value: 76.45767684895692 - type: nauc_precision_at_1_max value: 33.20840934731383 - type: nauc_precision_at_20_diff1 value: -39.88046573159446 - type: nauc_precision_at_20_max value: -0.9048730965169257 - type: nauc_precision_at_3_diff1 value: -14.92778542739045 - type: nauc_precision_at_3_max value: 9.506709583490675 - type: nauc_precision_at_5_diff1 value: -28.135454012537338 - type: nauc_precision_at_5_max value: 4.920384611100925 - type: nauc_recall_at_1000_diff1 value: 50.96814010736587 - type: nauc_recall_at_1000_max value: 35.178726454686455 - type: nauc_recall_at_100_diff1 value: 73.73669235424917 - type: nauc_recall_at_100_max value: 37.233185728668374 - type: nauc_recall_at_10_diff1 value: 72.24423264106863 - type: nauc_recall_at_10_max value: 31.982200960724956 - type: nauc_recall_at_1_diff1 value: 78.81016014060471 - type: nauc_recall_at_1_max value: 22.814760471331834 - type: nauc_recall_at_20_diff1 value: 71.93499290798466 - type: nauc_recall_at_20_max value: 32.59815791665593 - type: nauc_recall_at_3_diff1 value: 72.3320862681771 - type: nauc_recall_at_3_max value: 25.07376413220408 - type: nauc_recall_at_5_diff1 value: 71.67188030305823 - type: nauc_recall_at_5_max value: 26.80994890402375 - type: ndcg_at_1 value: 80.52 - type: ndcg_at_10 value: 87.561 - type: ndcg_at_100 value: 88.90299999999999 - type: ndcg_at_1000 value: 89.01400000000001 - type: ndcg_at_20 value: 88.293 - type: ndcg_at_3 value: 84.682 - type: ndcg_at_5 value: 86.319 - type: precision_at_1 value: 80.52 - type: precision_at_10 value: 13.288 - type: precision_at_100 value: 1.529 - type: precision_at_1000 value: 0.157 - type: precision_at_20 value: 7.077 - type: precision_at_3 value: 37.047000000000004 - type: precision_at_5 value: 24.398 - type: recall_at_1 value: 69.82000000000001 - type: recall_at_10 value: 94.826 - type: recall_at_100 value: 99.451 - type: recall_at_1000 value: 99.97399999999999 - type: recall_at_20 value: 97.18900000000001 - type: recall_at_3 value: 86.514 - type: recall_at_5 value: 91.142 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering config: default split: test revision: 24640382cdbf8abc73003fb0fa6d111a705499eb metrics: - type: v_measure value: 51.343294599772406 - type: v_measures value: [0.524406904623621, 0.6101985612354274, 0.4897743980881878, 0.4778925279162627, 0.5135415040552358] - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P config: default split: test revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 metrics: - type: v_measure value: 62.5095092123788 - type: v_measures value: [0.6648234836097784, 0.6647202577739932, 0.6578622320980865, 0.3861971752849744, 0.702695555977878] - task: type: Retrieval dataset: type: mteb/scidocs name: MTEB SCIDOCS config: default split: test revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 metrics: - type: map_at_1 value: 4.077999999999999 - type: map_at_10 value: 9.778 - type: map_at_100 value: 11.82 - type: map_at_1000 value: 12.115 - type: map_at_20 value: 10.781 - type: map_at_3 value: 6.973999999999999 - type: map_at_5 value: 8.305 - type: mrr_at_1 value: 20.0 - type: mrr_at_10 value: 29.48242063492062 - type: mrr_at_100 value: 30.73048798111141 - type: mrr_at_1000 value: 30.80176607280717 - type: mrr_at_20 value: 30.25008647742002 - type: mrr_at_3 value: 26.200000000000035 - type: mrr_at_5 value: 28.159999999999997 - type: nauc_map_at_1000_diff1 value: 11.636747601961716 - type: nauc_map_at_1000_max value: 26.56203998243688 - type: nauc_map_at_100_diff1 value: 11.563351860766852 - type: nauc_map_at_100_max value: 26.424696755662403 - type: nauc_map_at_10_diff1 value: 11.60703222490682 - type: nauc_map_at_10_max value: 25.186183431711402 - type: nauc_map_at_1_diff1 value: 17.854436578277337 - type: nauc_map_at_1_max value: 18.26224581093851 - type: nauc_map_at_20_diff1 value: 12.180332838724942 - type: nauc_map_at_20_max value: 25.8033910164551 - type: nauc_map_at_3_diff1 value: 12.934911478090966 - type: nauc_map_at_3_max value: 23.735462709754234 - type: nauc_map_at_5_diff1 value: 12.555897350050973 - type: nauc_map_at_5_max value: 24.934261010188717 - type: nauc_mrr_at_1000_diff1 value: 15.322270869482562 - type: nauc_mrr_at_1000_max value: 21.176483776434594 - type: nauc_mrr_at_100_diff1 value: 15.313592100314633 - type: nauc_mrr_at_100_max value: 21.189701285575325 - type: nauc_mrr_at_10_diff1 value: 15.04116376326651 - type: nauc_mrr_at_10_max value: 20.925070769290475 - type: nauc_mrr_at_1_diff1 value: 17.964804649660955 - type: nauc_mrr_at_1_max value: 18.691637068130458 - type: nauc_mrr_at_20_diff1 value: 15.2990030423077 - type: nauc_mrr_at_20_max value: 21.217369625839176 - type: nauc_mrr_at_3_diff1 value: 15.528981019166746 - type: nauc_mrr_at_3_max value: 20.603019869260343 - type: nauc_mrr_at_5_diff1 value: 15.466774363671028 - type: nauc_mrr_at_5_max value: 20.975652450389344 - type: nauc_ndcg_at_1000_diff1 value: 12.038972353500306 - type: nauc_ndcg_at_1000_max value: 27.903890926586932 - type: nauc_ndcg_at_100_diff1 value: 11.319931937239677 - type: nauc_ndcg_at_100_max value: 27.027157611973795 - type: nauc_ndcg_at_10_diff1 value: 11.078446709122273 - type: nauc_ndcg_at_10_max value: 23.822320107334757 - type: nauc_ndcg_at_1_diff1 value: 17.964804649660955 - type: nauc_ndcg_at_1_max value: 18.691637068130458 - type: nauc_ndcg_at_20_diff1 value: 12.181596567416559 - type: nauc_ndcg_at_20_max value: 25.37079586511582 - type: nauc_ndcg_at_3_diff1 value: 13.160188878029732 - type: nauc_ndcg_at_3_max value: 23.08138307792905 - type: nauc_ndcg_at_5_diff1 value: 12.469734888035655 - type: nauc_ndcg_at_5_max value: 23.90995773496041 - type: nauc_precision_at_1000_diff1 value: 6.917224255812411 - type: nauc_precision_at_1000_max value: 25.68380359473843 - type: nauc_precision_at_100_diff1 value: 6.398565019403672 - type: nauc_precision_at_100_max value: 25.768394101162112 - type: nauc_precision_at_10_diff1 value: 6.984700975104491 - type: nauc_precision_at_10_max value: 22.95876859709157 - type: nauc_precision_at_1_diff1 value: 17.964804649660955 - type: nauc_precision_at_1_max value: 18.691637068130458 - type: nauc_precision_at_20_diff1 value: 9.297910278094806 - type: nauc_precision_at_20_max value: 25.047556185054287 - type: nauc_precision_at_3_diff1 value: 11.002794885922452 - type: nauc_precision_at_3_max value: 24.196674867665706 - type: nauc_precision_at_5_diff1 value: 10.022107536962826 - type: nauc_precision_at_5_max value: 25.168302306099687 - type: nauc_recall_at_1000_diff1 value: 7.394647014691793 - type: nauc_recall_at_1000_max value: 25.484913499786305 - type: nauc_recall_at_100_diff1 value: 6.725259371546906 - type: nauc_recall_at_100_max value: 25.516933161468554 - type: nauc_recall_at_10_diff1 value: 7.129477895634021 - type: nauc_recall_at_10_max value: 22.349457271477668 - type: nauc_recall_at_1_diff1 value: 17.854436578277337 - type: nauc_recall_at_1_max value: 18.26224581093851 - type: nauc_recall_at_20_diff1 value: 9.42187360268394 - type: nauc_recall_at_20_max value: 24.71382966903428 - type: nauc_recall_at_3_diff1 value: 10.921292270138911 - type: nauc_recall_at_3_max value: 23.694131354239822 - type: nauc_recall_at_5_diff1 value: 9.99247973152595 - type: nauc_recall_at_5_max value: 24.651761102821084 - type: ndcg_at_1 value: 20.0 - type: ndcg_at_10 value: 16.927 - type: ndcg_at_100 value: 25.258999999999997 - type: ndcg_at_1000 value: 30.653999999999996 - type: ndcg_at_20 value: 19.764 - type: ndcg_at_3 value: 15.744 - type: ndcg_at_5 value: 13.925 - type: precision_at_1 value: 20.0 - type: precision_at_10 value: 8.83 - type: precision_at_100 value: 2.095 - type: precision_at_1000 value: 0.33899999999999997 - type: precision_at_20 value: 6.0600000000000005 - type: precision_at_3 value: 14.6 - type: precision_at_5 value: 12.24 - type: recall_at_1 value: 4.077999999999999 - type: recall_at_10 value: 17.928 - type: recall_at_100 value: 42.518 - type: recall_at_1000 value: 68.845 - type: recall_at_20 value: 24.6 - type: recall_at_3 value: 8.898 - type: recall_at_5 value: 12.413 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R config: default split: test revision: 20a6d6f312dd54037fe07a32d58e5e168867909d metrics: - type: cos_sim_pearson value: 85.7559383087264 - type: cos_sim_spearman value: 79.49310740723678 - type: euclidean_pearson value: 83.65824145595195 - type: euclidean_spearman value: 79.49312535113798 - type: manhattan_pearson value: 83.66988365842141 - type: manhattan_spearman value: 79.47470834406325 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 config: default split: test revision: a0d554a64d88156834ff5ae9920b964011b16384 metrics: - type: cos_sim_pearson value: 87.5839985534636 - type: cos_sim_spearman value: 79.59450354665624 - type: euclidean_pearson value: 84.43014644677757 - type: euclidean_spearman value: 79.59390361649422 - type: manhattan_pearson value: 84.29576591448196 - type: manhattan_spearman value: 79.41063925463973 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 config: default split: test revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca metrics: - type: cos_sim_pearson value: 84.29076815493453 - type: cos_sim_spearman value: 84.70010174539925 - type: euclidean_pearson value: 84.61503525685455 - type: euclidean_spearman value: 84.70010174539925 - type: manhattan_pearson value: 84.49870751490295 - type: manhattan_spearman value: 84.56101687530112 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 config: default split: test revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 metrics: - type: cos_sim_pearson value: 85.9713152765247 - type: cos_sim_spearman value: 83.91546630697559 - type: euclidean_pearson value: 85.24770099001337 - type: euclidean_spearman value: 83.915468446163 - type: manhattan_pearson value: 85.10357473564157 - type: manhattan_spearman value: 83.73948507923797 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 config: default split: test revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 metrics: - type: cos_sim_pearson value: 88.93457583410698 - type: cos_sim_spearman value: 89.87899103812357 - type: euclidean_pearson value: 89.3577529833881 - type: euclidean_spearman value: 89.87899029911844 - type: manhattan_pearson value: 89.27376081191959 - type: manhattan_spearman value: 89.7855896343813 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 config: default split: test revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 metrics: - type: cos_sim_pearson value: 83.07870989011116 - type: cos_sim_spearman value: 84.87521134248455 - type: euclidean_pearson value: 84.64895196926211 - type: euclidean_spearman value: 84.87521088458564 - type: manhattan_pearson value: 84.5177988181249 - type: manhattan_spearman value: 84.75674790631112 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) config: en-en split: test revision: faeb762787bd10488a50c8b5be4a3b82e411949c metrics: - type: cos_sim_pearson value: 88.2908976539434 - type: cos_sim_spearman value: 88.69212374652548 - type: euclidean_pearson value: 89.05936753064138 - type: euclidean_spearman value: 88.69212374652548 - type: manhattan_pearson value: 89.01731329350842 - type: manhattan_spearman value: 88.59540957427528 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) config: en split: test revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 metrics: - type: cos_sim_pearson value: 64.635158432127 - type: cos_sim_spearman value: 64.27073239884248 - type: euclidean_pearson value: 66.21751368412394 - type: euclidean_spearman value: 64.27073239884248 - type: manhattan_pearson value: 66.15088837552513 - type: manhattan_spearman value: 64.15068735594215 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark config: default split: test revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 metrics: - type: cos_sim_pearson value: 86.42604775914748 - type: cos_sim_spearman value: 87.44031003802738 - type: euclidean_pearson value: 87.43316615586107 - type: euclidean_spearman value: 87.44031555024793 - type: manhattan_pearson value: 87.31365950205998 - type: manhattan_spearman value: 87.2941204713115 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR config: default split: test revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab metrics: - type: map value: 81.48265276238037 - type: mrr value: 95.01133216819491 - type: nAUC_map_diff1 value: 7.09429277125843 - type: nAUC_map_max value: 49.94029950841183 - type: nAUC_mrr_diff1 value: 50.34339361427615 - type: nAUC_mrr_max value: 79.45634760563627 - task: type: Retrieval dataset: type: mteb/scifact name: MTEB SciFact config: default split: test revision: 0228b52cf27578f30900b9e5271d331663a030d7 metrics: - type: map_at_1 value: 54.678000000000004 - type: map_at_10 value: 63.80200000000001 - type: map_at_100 value: 64.41 - type: map_at_1000 value: 64.444 - type: map_at_20 value: 64.164 - type: map_at_3 value: 60.925 - type: map_at_5 value: 62.572 - type: mrr_at_1 value: 57.333333333333336 - type: mrr_at_10 value: 65.36190476190477 - type: mrr_at_100 value: 65.81385754624793 - type: mrr_at_1000 value: 65.84158115043161 - type: mrr_at_20 value: 65.65317979234389 - type: mrr_at_3 value: 63.0 - type: mrr_at_5 value: 64.41666666666667 - type: nauc_map_at_1000_diff1 value: 69.67809035584142 - type: nauc_map_at_1000_max value: 48.96413821482873 - type: nauc_map_at_100_diff1 value: 69.65364392825721 - type: nauc_map_at_100_max value: 48.94627564310418 - type: nauc_map_at_10_diff1 value: 69.57800581171497 - type: nauc_map_at_10_max value: 48.67897603770599 - type: nauc_map_at_1_diff1 value: 73.01790551734226 - type: nauc_map_at_1_max value: 45.104875261851916 - type: nauc_map_at_20_diff1 value: 69.61362614678308 - type: nauc_map_at_20_max value: 48.84035952302365 - type: nauc_map_at_3_diff1 value: 70.38590108900951 - type: nauc_map_at_3_max value: 46.43843764953461 - type: nauc_map_at_5_diff1 value: 70.52260971372544 - type: nauc_map_at_5_max value: 48.476797316953416 - type: nauc_mrr_at_1000_diff1 value: 69.5164716438929 - type: nauc_mrr_at_1000_max value: 51.01945033286556 - type: nauc_mrr_at_100_diff1 value: 69.49106498529696 - type: nauc_mrr_at_100_max value: 50.99948856957515 - type: nauc_mrr_at_10_diff1 value: 69.36739032618048 - type: nauc_mrr_at_10_max value: 51.12574202596153 - type: nauc_mrr_at_1_diff1 value: 72.36743551825465 - type: nauc_mrr_at_1_max value: 47.803395494440075 - type: nauc_mrr_at_20_diff1 value: 69.45736580905654 - type: nauc_mrr_at_20_max value: 51.125870226973 - type: nauc_mrr_at_3_diff1 value: 70.18553354726865 - type: nauc_mrr_at_3_max value: 50.0088700597719 - type: nauc_mrr_at_5_diff1 value: 70.13590171518094 - type: nauc_mrr_at_5_max value: 51.43303825871863 - type: nauc_ndcg_at_1000_diff1 value: 68.59647494694465 - type: nauc_ndcg_at_1000_max value: 50.84423309342368 - type: nauc_ndcg_at_100_diff1 value: 67.82685442401551 - type: nauc_ndcg_at_100_max value: 50.69511102823468 - type: nauc_ndcg_at_10_diff1 value: 67.42699840483867 - type: nauc_ndcg_at_10_max value: 50.367163593062216 - type: nauc_ndcg_at_1_diff1 value: 72.36743551825465 - type: nauc_ndcg_at_1_max value: 47.803395494440075 - type: nauc_ndcg_at_20_diff1 value: 67.52338557685287 - type: nauc_ndcg_at_20_max value: 50.79192152642611 - type: nauc_ndcg_at_3_diff1 value: 69.49850475650591 - type: nauc_ndcg_at_3_max value: 47.69895483323495 - type: nauc_ndcg_at_5_diff1 value: 69.63182188059407 - type: nauc_ndcg_at_5_max value: 50.692751514480086 - type: nauc_precision_at_1000_diff1 value: -22.076090806418165 - type: nauc_precision_at_1000_max value: 15.457406924757958 - type: nauc_precision_at_100_diff1 value: -8.958717111709591 - type: nauc_precision_at_100_max value: 23.39568973722963 - type: nauc_precision_at_10_diff1 value: 18.145183858510112 - type: nauc_precision_at_10_max value: 38.39226201682134 - type: nauc_precision_at_1_diff1 value: 72.36743551825465 - type: nauc_precision_at_1_max value: 47.803395494440075 - type: nauc_precision_at_20_diff1 value: 8.777082697589234 - type: nauc_precision_at_20_max value: 35.78403592135664 - type: nauc_precision_at_3_diff1 value: 51.376349362119726 - type: nauc_precision_at_3_max value: 42.810598626104 - type: nauc_precision_at_5_diff1 value: 40.9296274632404 - type: nauc_precision_at_5_max value: 45.61709594788111 - type: nauc_recall_at_1000_diff1 value: 77.7077497665738 - type: nauc_recall_at_1000_max value: 100.0 - type: nauc_recall_at_100_diff1 value: 46.35143397460506 - type: nauc_recall_at_100_max value: 53.648092062544094 - type: nauc_recall_at_10_diff1 value: 55.95034026811939 - type: nauc_recall_at_10_max value: 52.01290145176738 - type: nauc_recall_at_1_diff1 value: 73.01790551734226 - type: nauc_recall_at_1_max value: 45.104875261851916 - type: nauc_recall_at_20_diff1 value: 53.20691212906598 - type: nauc_recall_at_20_max value: 54.72055923613812 - type: nauc_recall_at_3_diff1 value: 66.24729517787732 - type: nauc_recall_at_3_max value: 46.4286391286208 - type: nauc_recall_at_5_diff1 value: 65.39243758829612 - type: nauc_recall_at_5_max value: 54.04465496293424 - type: ndcg_at_1 value: 57.333 - type: ndcg_at_10 value: 68.471 - type: ndcg_at_100 value: 71.16900000000001 - type: ndcg_at_1000 value: 71.934 - type: ndcg_at_20 value: 69.706 - type: ndcg_at_3 value: 63.404 - type: ndcg_at_5 value: 65.93900000000001 - type: precision_at_1 value: 57.333 - type: precision_at_10 value: 9.133 - type: precision_at_100 value: 1.06 - type: precision_at_1000 value: 0.11199999999999999 - type: precision_at_20 value: 4.833 - type: precision_at_3 value: 24.444 - type: precision_at_5 value: 16.267 - type: recall_at_1 value: 54.678000000000004 - type: recall_at_10 value: 81.244 - type: recall_at_100 value: 93.43299999999999 - type: recall_at_1000 value: 99.333 - type: recall_at_20 value: 86.02199999999999 - type: recall_at_3 value: 67.878 - type: recall_at_5 value: 74.06700000000001 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions config: default split: test revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 metrics: - type: cos_sim_accuracy value: 99.79702970297029 - type: cos_sim_ap value: 94.75976312657727 - type: cos_sim_f1 value: 89.7029702970297 - type: cos_sim_precision value: 88.8235294117647 - type: cos_sim_recall value: 90.60000000000001 - type: dot_accuracy value: 99.79702970297029 - type: dot_ap value: 94.75976312657727 - type: dot_f1 value: 89.7029702970297 - type: dot_precision value: 88.8235294117647 - type: dot_recall value: 90.60000000000001 - type: euclidean_accuracy value: 99.79702970297029 - type: euclidean_ap value: 94.75976312657727 - type: euclidean_f1 value: 89.7029702970297 - type: euclidean_precision value: 88.8235294117647 - type: euclidean_recall value: 90.60000000000001 - type: manhattan_accuracy value: 99.79207920792079 - type: manhattan_ap value: 94.55264782714548 - type: manhattan_f1 value: 89.30753564154786 - type: manhattan_precision value: 90.97510373443983 - type: manhattan_recall value: 87.7 - type: max_accuracy value: 99.79702970297029 - type: max_ap value: 94.75976312657727 - type: max_f1 value: 89.7029702970297 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering config: default split: test revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 metrics: - type: v_measure value: 61.56750444120404 - type: v_measures value: [0.6143066231410037, 0.6484543949297663, 0.5142449933097217, 0.6062244515066045, 0.5739832317328382] - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P config: default split: test revision: 815ca46b2622cec33ccafc3735d572c266efdb44 metrics: - type: v_measure value: 33.59242218010789 - type: v_measures value: [0.32392220242444014, 0.32718501590648996, 0.32976751674384125, 0.3203167039651582, 0.32297852686667516] - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions config: default split: test revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 metrics: - type: map value: 49.200337844678295 - type: mrr value: 49.95621923011629 - type: nAUC_map_diff1 value: 35.98209315270176 - type: nAUC_map_max value: 12.780142559584018 - type: nAUC_mrr_diff1 value: 36.332794804706545 - type: nAUC_mrr_max value: 13.650142919994185 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval config: default split: test revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c metrics: - type: cos_sim_pearson value: 30.273860160997156 - type: cos_sim_spearman value: 30.256535662300372 - type: dot_pearson value: 30.273861391746525 - type: dot_spearman value: 30.25651496178948 - task: type: Retrieval dataset: type: mteb/trec-covid name: MTEB TRECCOVID config: default split: test revision: bb9466bac8153a0349341eb1b22e06409e78ef4e metrics: - type: map_at_1 value: 0.161 - type: map_at_10 value: 1.0410000000000001 - type: map_at_100 value: 5.558 - type: map_at_1000 value: 15.296000000000001 - type: map_at_20 value: 1.755 - type: map_at_3 value: 0.437 - type: map_at_5 value: 0.628 - type: mrr_at_1 value: 64.0 - type: mrr_at_10 value: 73.77222222222223 - type: mrr_at_100 value: 74.10160298638559 - type: mrr_at_1000 value: 74.10160298638559 - type: mrr_at_20 value: 73.9540404040404 - type: mrr_at_3 value: 70.0 - type: mrr_at_5 value: 73.3 - type: nauc_map_at_1000_diff1 value: -1.9604698462344767 - type: nauc_map_at_1000_max value: 53.358985327864715 - type: nauc_map_at_100_diff1 value: -0.5161702872199433 - type: nauc_map_at_100_max value: 46.53235662200936 - type: nauc_map_at_10_diff1 value: 3.9256229008973924 - type: nauc_map_at_10_max value: 26.82251128561187 - type: nauc_map_at_1_diff1 value: 4.127739790921559 - type: nauc_map_at_1_max value: 10.596981259216367 - type: nauc_map_at_20_diff1 value: 1.6849110098963962 - type: nauc_map_at_20_max value: 32.645496648968034 - type: nauc_map_at_3_diff1 value: 6.0165331028800715 - type: nauc_map_at_3_max value: 16.690807762030964 - type: nauc_map_at_5_diff1 value: 3.151304647408004 - type: nauc_map_at_5_max value: 22.316738900184266 - type: nauc_mrr_at_1000_diff1 value: 8.665368838521031 - type: nauc_mrr_at_1000_max value: 48.23582325840371 - type: nauc_mrr_at_100_diff1 value: 8.665368838521031 - type: nauc_mrr_at_100_max value: 48.23582325840371 - type: nauc_mrr_at_10_diff1 value: 9.331739591001785 - type: nauc_mrr_at_10_max value: 48.89453679791346 - type: nauc_mrr_at_1_diff1 value: 0.5428534934218375 - type: nauc_mrr_at_1_max value: 35.480265678886184 - type: nauc_mrr_at_20_diff1 value: 8.583612874582078 - type: nauc_mrr_at_20_max value: 48.472852713493815 - type: nauc_mrr_at_3_diff1 value: 9.854859452507785 - type: nauc_mrr_at_3_max value: 50.846959397391124 - type: nauc_mrr_at_5_diff1 value: 9.223998156393836 - type: nauc_mrr_at_5_max value: 49.03025489502146 - type: nauc_ndcg_at_1000_diff1 value: 1.8220536687131508 - type: nauc_ndcg_at_1000_max value: 49.93338657126098 - type: nauc_ndcg_at_100_diff1 value: 3.0321366749721466 - type: nauc_ndcg_at_100_max value: 51.5400551444691 - type: nauc_ndcg_at_10_diff1 value: 8.579231691222738 - type: nauc_ndcg_at_10_max value: 46.76810123148509 - type: nauc_ndcg_at_1_diff1 value: -3.008490760055262 - type: nauc_ndcg_at_1_max value: 26.51379381261564 - type: nauc_ndcg_at_20_diff1 value: 7.688195422853383 - type: nauc_ndcg_at_20_max value: 49.01894979259541 - type: nauc_ndcg_at_3_diff1 value: 4.482353844424605 - type: nauc_ndcg_at_3_max value: 41.5568352350729 - type: nauc_ndcg_at_5_diff1 value: 5.0965815841199005 - type: nauc_ndcg_at_5_max value: 48.0173572974474 - type: nauc_precision_at_1000_diff1 value: -1.522279672648178 - type: nauc_precision_at_1000_max value: 48.14464502949045 - type: nauc_precision_at_100_diff1 value: 0.3803220198283984 - type: nauc_precision_at_100_max value: 54.606530270656286 - type: nauc_precision_at_10_diff1 value: 10.17060237919451 - type: nauc_precision_at_10_max value: 51.07174301856844 - type: nauc_precision_at_1_diff1 value: 0.5428534934218375 - type: nauc_precision_at_1_max value: 35.480265678886184 - type: nauc_precision_at_20_diff1 value: 5.7555901777882825 - type: nauc_precision_at_20_max value: 51.63129458224357 - type: nauc_precision_at_3_diff1 value: 6.697227309781205 - type: nauc_precision_at_3_max value: 48.75128365287913 - type: nauc_precision_at_5_diff1 value: 3.675984060933098 - type: nauc_precision_at_5_max value: 54.034024908467984 - type: nauc_recall_at_1000_diff1 value: 4.99265973475678 - type: nauc_recall_at_1000_max value: 47.39125173916412 - type: nauc_recall_at_100_diff1 value: 0.6537761781214193 - type: nauc_recall_at_100_max value: 37.699187570549284 - type: nauc_recall_at_10_diff1 value: 8.17628273209583 - type: nauc_recall_at_10_max value: 20.08017524188776 - type: nauc_recall_at_1_diff1 value: 4.127739790921559 - type: nauc_recall_at_1_max value: 10.596981259216367 - type: nauc_recall_at_20_diff1 value: 4.210774786757633 - type: nauc_recall_at_20_max value: 24.62795703117278 - type: nauc_recall_at_3_diff1 value: 8.110601816749067 - type: nauc_recall_at_3_max value: 15.51743399119835 - type: nauc_recall_at_5_diff1 value: 5.312647840696286 - type: nauc_recall_at_5_max value: 18.88262264074873 - type: ndcg_at_1 value: 57.99999999999999 - type: ndcg_at_10 value: 49.443 - type: ndcg_at_100 value: 37.479 - type: ndcg_at_1000 value: 38.735 - type: ndcg_at_20 value: 46.115 - type: ndcg_at_3 value: 54.458 - type: ndcg_at_5 value: 53.601 - type: precision_at_1 value: 64.0 - type: precision_at_10 value: 52.2 - type: precision_at_100 value: 38.64 - type: precision_at_1000 value: 17.98 - type: precision_at_20 value: 48.5 - type: precision_at_3 value: 59.333000000000006 - type: precision_at_5 value: 57.99999999999999 - type: recall_at_1 value: 0.161 - type: recall_at_10 value: 1.31 - type: recall_at_100 value: 9.105 - type: recall_at_1000 value: 38.330999999999996 - type: recall_at_20 value: 2.382 - type: recall_at_3 value: 0.47400000000000003 - type: recall_at_5 value: 0.735 - task: type: Retrieval dataset: type: mteb/touche2020 name: MTEB Touche2020 config: default split: test revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f metrics: - type: map_at_1 value: 1.773 - type: map_at_10 value: 5.819 - type: map_at_100 value: 9.791 - type: map_at_1000 value: 11.152 - type: map_at_20 value: 7.800999999999999 - type: map_at_3 value: 3.778 - type: map_at_5 value: 4.5760000000000005 - type: mrr_at_1 value: 26.53061224489796 - type: mrr_at_10 value: 36.790573372206026 - type: mrr_at_100 value: 38.9989669597618 - type: mrr_at_1000 value: 38.9989669597618 - type: mrr_at_20 value: 38.626122355982716 - type: mrr_at_3 value: 32.6530612244898 - type: mrr_at_5 value: 35.61224489795919 - type: nauc_map_at_1000_diff1 value: 5.982723549423614 - type: nauc_map_at_1000_max value: -38.778150862184184 - type: nauc_map_at_100_diff1 value: 8.35542066448559 - type: nauc_map_at_100_max value: -36.95988939092301 - type: nauc_map_at_10_diff1 value: 16.727112997939685 - type: nauc_map_at_10_max value: -34.864329426120634 - type: nauc_map_at_1_diff1 value: 15.68698509866931 - type: nauc_map_at_1_max value: -38.49856837383727 - type: nauc_map_at_20_diff1 value: 12.131431984811561 - type: nauc_map_at_20_max value: -35.58027185761546 - type: nauc_map_at_3_diff1 value: 26.51714323506847 - type: nauc_map_at_3_max value: -42.90178838356341 - type: nauc_map_at_5_diff1 value: 19.97087359992273 - type: nauc_map_at_5_max value: -39.34647766735783 - type: nauc_mrr_at_1000_diff1 value: 1.9748555750391668 - type: nauc_mrr_at_1000_max value: -38.23327494222865 - type: nauc_mrr_at_100_diff1 value: 1.9748555750391668 - type: nauc_mrr_at_100_max value: -38.23327494222865 - type: nauc_mrr_at_10_diff1 value: 3.026383371657329 - type: nauc_mrr_at_10_max value: -37.357911445290675 - type: nauc_mrr_at_1_diff1 value: -3.6234515245372143 - type: nauc_mrr_at_1_max value: -34.918413083959635 - type: nauc_mrr_at_20_diff1 value: 2.785643581147567 - type: nauc_mrr_at_20_max value: -39.027829254611596 - type: nauc_mrr_at_3_diff1 value: -0.652257776491799 - type: nauc_mrr_at_3_max value: -35.777046370826746 - type: nauc_mrr_at_5_diff1 value: -1.0723788161779884 - type: nauc_mrr_at_5_max value: -37.74663638254447 - type: nauc_ndcg_at_1000_diff1 value: -5.603421065562765 - type: nauc_ndcg_at_1000_max value: -48.302585883872254 - type: nauc_ndcg_at_100_diff1 value: -0.4142249503459385 - type: nauc_ndcg_at_100_max value: -44.266171155232605 - type: nauc_ndcg_at_10_diff1 value: 9.09113299295375 - type: nauc_ndcg_at_10_max value: -33.95506408586246 - type: nauc_ndcg_at_1_diff1 value: -2.3509910942232137 - type: nauc_ndcg_at_1_max value: -31.9181129505804 - type: nauc_ndcg_at_20_diff1 value: 6.711479298611486 - type: nauc_ndcg_at_20_max value: -41.17797709135335 - type: nauc_ndcg_at_3_diff1 value: 7.181833876703895 - type: nauc_ndcg_at_3_max value: -36.76673572406525 - type: nauc_ndcg_at_5_diff1 value: 4.220139919243461 - type: nauc_ndcg_at_5_max value: -36.01626374596527 - type: nauc_precision_at_1000_diff1 value: -21.500096998480743 - type: nauc_precision_at_1000_max value: 17.423538290188787 - type: nauc_precision_at_100_diff1 value: -20.576518815311264 - type: nauc_precision_at_100_max value: -33.43443540744943 - type: nauc_precision_at_10_diff1 value: 5.916347999274714 - type: nauc_precision_at_10_max value: -31.375195302655644 - type: nauc_precision_at_1_diff1 value: -3.6234515245372143 - type: nauc_precision_at_1_max value: -34.918413083959635 - type: nauc_precision_at_20_diff1 value: -5.5440042532199145 - type: nauc_precision_at_20_max value: -41.606107555682485 - type: nauc_precision_at_3_diff1 value: 9.493481238903758 - type: nauc_precision_at_3_max value: -39.77246958608447 - type: nauc_precision_at_5_diff1 value: 0.9616292047999846 - type: nauc_precision_at_5_max value: -36.360179658412726 - type: nauc_recall_at_1000_diff1 value: -18.950139772725883 - type: nauc_recall_at_1000_max value: -49.50001953592577 - type: nauc_recall_at_100_diff1 value: -4.101410227998355 - type: nauc_recall_at_100_max value: -43.41533454443838 - type: nauc_recall_at_10_diff1 value: 15.471367681837625 - type: nauc_recall_at_10_max value: -30.21854343397064 - type: nauc_recall_at_1_diff1 value: 15.68698509866931 - type: nauc_recall_at_1_max value: -38.49856837383727 - type: nauc_recall_at_20_diff1 value: 6.295179926244505 - type: nauc_recall_at_20_max value: -39.20825139905824 - type: nauc_recall_at_3_diff1 value: 26.54887286634497 - type: nauc_recall_at_3_max value: -39.46308906643022 - type: nauc_recall_at_5_diff1 value: 15.955004561636251 - type: nauc_recall_at_5_max value: -36.491415148404585 - type: ndcg_at_1 value: 23.469 - type: ndcg_at_10 value: 16.07 - type: ndcg_at_100 value: 25.61 - type: ndcg_at_1000 value: 38.092999999999996 - type: ndcg_at_20 value: 17.980999999999998 - type: ndcg_at_3 value: 18.332 - type: ndcg_at_5 value: 17.302 - type: precision_at_1 value: 26.531 - type: precision_at_10 value: 14.285999999999998 - type: precision_at_100 value: 5.469 - type: precision_at_1000 value: 1.331 - type: precision_at_20 value: 12.449 - type: precision_at_3 value: 19.048000000000002 - type: precision_at_5 value: 17.551 - type: recall_at_1 value: 1.773 - type: recall_at_10 value: 10.698 - type: recall_at_100 value: 35.684 - type: recall_at_1000 value: 72.932 - type: recall_at_20 value: 18.723 - type: recall_at_3 value: 4.788 - type: recall_at_5 value: 6.715 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification config: default split: test revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de metrics: - type: accuracy value: 63.38378906249999 - type: ap value: 11.003129023708196 - type: ap_weighted value: 11.003129023708196 - type: f1 value: 48.57435688911943 - type: f1_weighted value: 71.50552650067205 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification config: default split: test revision: d604517c81ca91fe16a244d1248fc021f9ecee7a metrics: - type: accuracy value: 62.467458970005666 - type: f1 value: 62.66021402025028 - type: f1_weighted value: 61.71229183506532 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering config: default split: test revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 metrics: - type: v_measure value: 44.96874447839597 - type: v_measures value: [0.44089700334011683, 0.4808871718296333, 0.4655762216311635, 0.4108889263207817, 0.45449139982441744] - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 config: default split: test revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 metrics: - type: cos_sim_accuracy value: 85.83179352685224 - type: cos_sim_ap value: 73.75870454252158 - type: cos_sim_f1 value: 67.61786600496278 - type: cos_sim_precision value: 63.81733021077284 - type: cos_sim_recall value: 71.89973614775725 - type: dot_accuracy value: 85.83179352685224 - type: dot_ap value: 73.75870536224079 - type: dot_f1 value: 67.61786600496278 - type: dot_precision value: 63.81733021077284 - type: dot_recall value: 71.89973614775725 - type: euclidean_accuracy value: 85.83179352685224 - type: euclidean_ap value: 73.7587242895193 - type: euclidean_f1 value: 67.61786600496278 - type: euclidean_precision value: 63.81733021077284 - type: euclidean_recall value: 71.89973614775725 - type: manhattan_accuracy value: 85.91524110389224 - type: manhattan_ap value: 73.77139111004601 - type: manhattan_f1 value: 67.52419453632244 - type: manhattan_precision value: 63.023096272581746 - type: manhattan_recall value: 72.71767810026385 - type: max_accuracy value: 85.91524110389224 - type: max_ap value: 73.77139111004601 - type: max_f1 value: 67.61786600496278 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus config: default split: test revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf metrics: - type: cos_sim_accuracy value: 89.2711607870532 - type: cos_sim_ap value: 86.48986874793765 - type: cos_sim_f1 value: 79.10280373831775 - type: cos_sim_precision value: 76.87836070338614 - type: cos_sim_recall value: 81.45980905451185 - type: dot_accuracy value: 89.2711607870532 - type: dot_ap value: 86.48986996838993 - type: dot_f1 value: 79.10280373831775 - type: dot_precision value: 76.87836070338614 - type: dot_recall value: 81.45980905451185 - type: euclidean_accuracy value: 89.2711607870532 - type: euclidean_ap value: 86.4898691356683 - type: euclidean_f1 value: 79.10280373831775 - type: euclidean_precision value: 76.87836070338614 - type: euclidean_recall value: 81.45980905451185 - type: manhattan_accuracy value: 89.2711607870532 - type: manhattan_ap value: 86.46475884590569 - type: manhattan_f1 value: 78.9534579927593 - type: manhattan_precision value: 76.61716769286491 - type: manhattan_recall value: 81.43671080997844 - type: max_accuracy value: 89.2711607870532 - type: max_ap value: 86.48986996838993 - type: max_f1 value: 79.10280373831775 license: apache-2.0 --- # [bilingual-embedding-base](https://huggingface.co/Lajavaness/bilingual-embedding-base) Bilingual-embedding is the Embedding Model for bilingual language: french and english. This model is a specialized sentence-embedding trained specifically for the bilingual language, leveraging the robust capabilities of [XLM-RoBERTa](https://huggingface.co/FacebookAI/xlm-roberta-base), a pre-trained language model based on the [XLM-RoBERTa](https://huggingface.co/FacebookAI/xlm-roberta-base) architecture. The model utilizes xlm-roberta to encode english-french sentences into a 1024-dimensional vector space, facilitating a wide range of applications from semantic search to text clustering. The embeddings capture the nuanced meanings of english-french sentences, reflecting both the lexical and contextual layers of the language. ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BilingualModel (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Training and Fine-tuning process #### Stage 1: NLI Training - Dataset: [(SNLI+XNLI) for english+french] - Method: Training using Multi-Negative Ranking Loss. This stage focused on improving the model's ability to discern and rank nuanced differences in sentence semantics. ### Stage 3: Continued Fine-tuning for Semantic Textual Similarity on STS Benchmark - Dataset: [STSB-fr and en] - Method: Fine-tuning specifically for the semantic textual similarity benchmark using Siamese BERT-Networks configured with the 'sentence-transformers' library. ### Stage 4: Advanced Augmentation Fine-tuning - Dataset: STSB with generate [silver sample from gold sample](https://www.sbert.net/examples/training/data_augmentation/README.html) - Method: Employed an advanced strategy using [Augmented SBERT](https://arxiv.org/abs/2010.08240) with Pair Sampling Strategies, integrating both Cross-Encoder and Bi-Encoder models. This stage further refined the embeddings by enriching the training data dynamically, enhancing the model's robustness and accuracy. ## Usage: Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = ["Paris est une capitale de la France", "Paris is a capital of France"] model = SentenceTransformer('Lajavaness/bilingual-embedding-base', trust_remote_code=True) print(embeddings) ``` ## Evaluation TODO ## Citation @article{conneau2019unsupervised, title={Unsupervised cross-lingual representation learning at scale}, author={Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin}, journal={arXiv preprint arXiv:1911.02116}, year={2019} } @article{reimers2019sentence, title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks}, author={Nils Reimers, Iryna Gurevych}, journal={https://arxiv.org/abs/1908.10084}, year={2019} } @article{thakur2020augmented, title={Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks}, author={Thakur, Nandan and Reimers, Nils and Daxenberger, Johannes and Gurevych, Iryna}, journal={arXiv e-prints}, pages={arXiv--2010}, year={2020}