|
--- |
|
tags: |
|
- mteb |
|
- feature-extraction |
|
- sentence-similarity |
|
model-index: |
|
- name: v2 |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 76.56716417910448 |
|
- type: ap |
|
value: 39.864746145463656 |
|
- type: f1 |
|
value: 70.60275403114987 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 93.46427500000001 |
|
- type: ap |
|
value: 90.36283359936121 |
|
- type: f1 |
|
value: 93.45329322673612 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 48.77199999999999 |
|
- type: f1 |
|
value: 48.16695258838576 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 40.184999999999995 |
|
- type: map_at_10 |
|
value: 56.114 |
|
- type: map_at_100 |
|
value: 56.676 |
|
- type: map_at_1000 |
|
value: 56.68 |
|
- type: map_at_3 |
|
value: 51.968 |
|
- type: map_at_5 |
|
value: 54.642 |
|
- type: mrr_at_1 |
|
value: 40.896 |
|
- type: mrr_at_10 |
|
value: 56.388000000000005 |
|
- type: mrr_at_100 |
|
value: 56.95099999999999 |
|
- type: mrr_at_1000 |
|
value: 56.95400000000001 |
|
- type: mrr_at_3 |
|
value: 52.251999999999995 |
|
- type: mrr_at_5 |
|
value: 54.879999999999995 |
|
- type: ndcg_at_1 |
|
value: 40.184999999999995 |
|
- type: ndcg_at_10 |
|
value: 64.253 |
|
- type: ndcg_at_100 |
|
value: 66.47 |
|
- type: ndcg_at_1000 |
|
value: 66.549 |
|
- type: ndcg_at_3 |
|
value: 55.945 |
|
- type: ndcg_at_5 |
|
value: 60.742 |
|
- type: precision_at_1 |
|
value: 40.184999999999995 |
|
- type: precision_at_10 |
|
value: 8.982999999999999 |
|
- type: precision_at_100 |
|
value: 0.991 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 22.499 |
|
- type: precision_at_5 |
|
value: 15.817999999999998 |
|
- type: recall_at_1 |
|
value: 40.184999999999995 |
|
- type: recall_at_10 |
|
value: 89.82900000000001 |
|
- type: recall_at_100 |
|
value: 99.075 |
|
- type: recall_at_1000 |
|
value: 99.644 |
|
- type: recall_at_3 |
|
value: 67.496 |
|
- type: recall_at_5 |
|
value: 79.09 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 49.64684811204023 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 43.6640710523389 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 63.71316367624821 |
|
- type: mrr |
|
value: 77.02534845886647 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 88.96786300506704 |
|
- type: cos_sim_spearman |
|
value: 88.08212749295554 |
|
- type: euclidean_pearson |
|
value: 87.1561534920524 |
|
- type: euclidean_spearman |
|
value: 88.016463346151 |
|
- type: manhattan_pearson |
|
value: 87.19359910450564 |
|
- type: manhattan_spearman |
|
value: 88.10803169765825 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 87.46103896103897 |
|
- type: f1 |
|
value: 87.4315144014101 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 41.03554871732576 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 37.974813344124264 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 34.174 |
|
- type: map_at_10 |
|
value: 45.728 |
|
- type: map_at_100 |
|
value: 47.266999999999996 |
|
- type: map_at_1000 |
|
value: 47.39 |
|
- type: map_at_3 |
|
value: 41.667 |
|
- type: map_at_5 |
|
value: 44.028 |
|
- type: mrr_at_1 |
|
value: 41.202 |
|
- type: mrr_at_10 |
|
value: 51.49 |
|
- type: mrr_at_100 |
|
value: 52.159 |
|
- type: mrr_at_1000 |
|
value: 52.197 |
|
- type: mrr_at_3 |
|
value: 48.379 |
|
- type: mrr_at_5 |
|
value: 50.331 |
|
- type: ndcg_at_1 |
|
value: 41.202 |
|
- type: ndcg_at_10 |
|
value: 52.38699999999999 |
|
- type: ndcg_at_100 |
|
value: 57.611999999999995 |
|
- type: ndcg_at_1000 |
|
value: 59.318000000000005 |
|
- type: ndcg_at_3 |
|
value: 46.516000000000005 |
|
- type: ndcg_at_5 |
|
value: 49.519000000000005 |
|
- type: precision_at_1 |
|
value: 41.202 |
|
- type: precision_at_10 |
|
value: 9.971 |
|
- type: precision_at_100 |
|
value: 1.5879999999999999 |
|
- type: precision_at_1000 |
|
value: 0.20500000000000002 |
|
- type: precision_at_3 |
|
value: 22.031 |
|
- type: precision_at_5 |
|
value: 16.309 |
|
- type: recall_at_1 |
|
value: 34.174 |
|
- type: recall_at_10 |
|
value: 65.32900000000001 |
|
- type: recall_at_100 |
|
value: 86.64 |
|
- type: recall_at_1000 |
|
value: 97.069 |
|
- type: recall_at_3 |
|
value: 48.607 |
|
- type: recall_at_5 |
|
value: 56.615 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 34.73 |
|
- type: map_at_10 |
|
value: 45.617999999999995 |
|
- type: map_at_100 |
|
value: 46.888000000000005 |
|
- type: map_at_1000 |
|
value: 47.016999999999996 |
|
- type: map_at_3 |
|
value: 42.425000000000004 |
|
- type: map_at_5 |
|
value: 44.214999999999996 |
|
- type: mrr_at_1 |
|
value: 43.631 |
|
- type: mrr_at_10 |
|
value: 52.014 |
|
- type: mrr_at_100 |
|
value: 52.6 |
|
- type: mrr_at_1000 |
|
value: 52.637 |
|
- type: mrr_at_3 |
|
value: 50.021 |
|
- type: mrr_at_5 |
|
value: 51.23799999999999 |
|
- type: ndcg_at_1 |
|
value: 43.631 |
|
- type: ndcg_at_10 |
|
value: 51.458000000000006 |
|
- type: ndcg_at_100 |
|
value: 55.61000000000001 |
|
- type: ndcg_at_1000 |
|
value: 57.462 |
|
- type: ndcg_at_3 |
|
value: 47.461 |
|
- type: ndcg_at_5 |
|
value: 49.312 |
|
- type: precision_at_1 |
|
value: 43.631 |
|
- type: precision_at_10 |
|
value: 9.661999999999999 |
|
- type: precision_at_100 |
|
value: 1.5270000000000001 |
|
- type: precision_at_1000 |
|
value: 0.198 |
|
- type: precision_at_3 |
|
value: 22.823999999999998 |
|
- type: precision_at_5 |
|
value: 16.075999999999997 |
|
- type: recall_at_1 |
|
value: 34.73 |
|
- type: recall_at_10 |
|
value: 61.041999999999994 |
|
- type: recall_at_100 |
|
value: 78.658 |
|
- type: recall_at_1000 |
|
value: 90.215 |
|
- type: recall_at_3 |
|
value: 48.952 |
|
- type: recall_at_5 |
|
value: 54.422000000000004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 42.047000000000004 |
|
- type: map_at_10 |
|
value: 55.669999999999995 |
|
- type: map_at_100 |
|
value: 56.676 |
|
- type: map_at_1000 |
|
value: 56.728 |
|
- type: map_at_3 |
|
value: 52.275000000000006 |
|
- type: map_at_5 |
|
value: 54.254000000000005 |
|
- type: mrr_at_1 |
|
value: 48.15 |
|
- type: mrr_at_10 |
|
value: 59.036 |
|
- type: mrr_at_100 |
|
value: 59.650999999999996 |
|
- type: mrr_at_1000 |
|
value: 59.675 |
|
- type: mrr_at_3 |
|
value: 56.760999999999996 |
|
- type: mrr_at_5 |
|
value: 58.087 |
|
- type: ndcg_at_1 |
|
value: 48.15 |
|
- type: ndcg_at_10 |
|
value: 61.709 |
|
- type: ndcg_at_100 |
|
value: 65.446 |
|
- type: ndcg_at_1000 |
|
value: 66.388 |
|
- type: ndcg_at_3 |
|
value: 56.333 |
|
- type: ndcg_at_5 |
|
value: 59.028000000000006 |
|
- type: precision_at_1 |
|
value: 48.15 |
|
- type: precision_at_10 |
|
value: 9.893 |
|
- type: precision_at_100 |
|
value: 1.265 |
|
- type: precision_at_1000 |
|
value: 0.13799999999999998 |
|
- type: precision_at_3 |
|
value: 25.266 |
|
- type: precision_at_5 |
|
value: 17.204 |
|
- type: recall_at_1 |
|
value: 42.047000000000004 |
|
- type: recall_at_10 |
|
value: 76.004 |
|
- type: recall_at_100 |
|
value: 91.727 |
|
- type: recall_at_1000 |
|
value: 98.213 |
|
- type: recall_at_3 |
|
value: 61.82 |
|
- type: recall_at_5 |
|
value: 68.42200000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 29.985 |
|
- type: map_at_10 |
|
value: 38.763999999999996 |
|
- type: map_at_100 |
|
value: 39.835 |
|
- type: map_at_1000 |
|
value: 39.900000000000006 |
|
- type: map_at_3 |
|
value: 35.826 |
|
- type: map_at_5 |
|
value: 37.403 |
|
- type: mrr_at_1 |
|
value: 32.202999999999996 |
|
- type: mrr_at_10 |
|
value: 40.94 |
|
- type: mrr_at_100 |
|
value: 41.861 |
|
- type: mrr_at_1000 |
|
value: 41.909 |
|
- type: mrr_at_3 |
|
value: 38.267 |
|
- type: mrr_at_5 |
|
value: 39.748 |
|
- type: ndcg_at_1 |
|
value: 32.202999999999996 |
|
- type: ndcg_at_10 |
|
value: 43.909 |
|
- type: ndcg_at_100 |
|
value: 49.028 |
|
- type: ndcg_at_1000 |
|
value: 50.714999999999996 |
|
- type: ndcg_at_3 |
|
value: 38.239000000000004 |
|
- type: ndcg_at_5 |
|
value: 40.854 |
|
- type: precision_at_1 |
|
value: 32.202999999999996 |
|
- type: precision_at_10 |
|
value: 6.621 |
|
- type: precision_at_100 |
|
value: 0.964 |
|
- type: precision_at_1000 |
|
value: 0.11399999999999999 |
|
- type: precision_at_3 |
|
value: 15.781999999999998 |
|
- type: precision_at_5 |
|
value: 10.96 |
|
- type: recall_at_1 |
|
value: 29.985 |
|
- type: recall_at_10 |
|
value: 57.727 |
|
- type: recall_at_100 |
|
value: 80.833 |
|
- type: recall_at_1000 |
|
value: 93.625 |
|
- type: recall_at_3 |
|
value: 42.396 |
|
- type: recall_at_5 |
|
value: 48.624 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 19.249 |
|
- type: map_at_10 |
|
value: 28.565 |
|
- type: map_at_100 |
|
value: 29.753 |
|
- type: map_at_1000 |
|
value: 29.881 |
|
- type: map_at_3 |
|
value: 25.778000000000002 |
|
- type: map_at_5 |
|
value: 27.21 |
|
- type: mrr_at_1 |
|
value: 23.632 |
|
- type: mrr_at_10 |
|
value: 33.51 |
|
- type: mrr_at_100 |
|
value: 34.372 |
|
- type: mrr_at_1000 |
|
value: 34.443 |
|
- type: mrr_at_3 |
|
value: 30.784 |
|
- type: mrr_at_5 |
|
value: 32.301 |
|
- type: ndcg_at_1 |
|
value: 23.632 |
|
- type: ndcg_at_10 |
|
value: 34.42 |
|
- type: ndcg_at_100 |
|
value: 39.823 |
|
- type: ndcg_at_1000 |
|
value: 42.558 |
|
- type: ndcg_at_3 |
|
value: 29.237000000000002 |
|
- type: ndcg_at_5 |
|
value: 31.465 |
|
- type: precision_at_1 |
|
value: 23.632 |
|
- type: precision_at_10 |
|
value: 6.331 |
|
- type: precision_at_100 |
|
value: 1.042 |
|
- type: precision_at_1000 |
|
value: 0.14100000000000001 |
|
- type: precision_at_3 |
|
value: 14.179 |
|
- type: precision_at_5 |
|
value: 10.299 |
|
- type: recall_at_1 |
|
value: 19.249 |
|
- type: recall_at_10 |
|
value: 47.539 |
|
- type: recall_at_100 |
|
value: 70.612 |
|
- type: recall_at_1000 |
|
value: 89.633 |
|
- type: recall_at_3 |
|
value: 33.082 |
|
- type: recall_at_5 |
|
value: 38.622 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 31.599 |
|
- type: map_at_10 |
|
value: 42.948 |
|
- type: map_at_100 |
|
value: 44.244 |
|
- type: map_at_1000 |
|
value: 44.352000000000004 |
|
- type: map_at_3 |
|
value: 39.352 |
|
- type: map_at_5 |
|
value: 41.397 |
|
- type: mrr_at_1 |
|
value: 38.21 |
|
- type: mrr_at_10 |
|
value: 48.347 |
|
- type: mrr_at_100 |
|
value: 49.132999999999996 |
|
- type: mrr_at_1000 |
|
value: 49.171 |
|
- type: mrr_at_3 |
|
value: 45.653 |
|
- type: mrr_at_5 |
|
value: 47.323 |
|
- type: ndcg_at_1 |
|
value: 38.21 |
|
- type: ndcg_at_10 |
|
value: 49.225 |
|
- type: ndcg_at_100 |
|
value: 54.422000000000004 |
|
- type: ndcg_at_1000 |
|
value: 56.27799999999999 |
|
- type: ndcg_at_3 |
|
value: 43.482 |
|
- type: ndcg_at_5 |
|
value: 46.321 |
|
- type: precision_at_1 |
|
value: 38.21 |
|
- type: precision_at_10 |
|
value: 8.921999999999999 |
|
- type: precision_at_100 |
|
value: 1.333 |
|
- type: precision_at_1000 |
|
value: 0.169 |
|
- type: precision_at_3 |
|
value: 20.372 |
|
- type: precision_at_5 |
|
value: 14.629 |
|
- type: recall_at_1 |
|
value: 31.599 |
|
- type: recall_at_10 |
|
value: 62.364 |
|
- type: recall_at_100 |
|
value: 83.91199999999999 |
|
- type: recall_at_1000 |
|
value: 95.743 |
|
- type: recall_at_3 |
|
value: 46.671 |
|
- type: recall_at_5 |
|
value: 53.772 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.641 |
|
- type: map_at_10 |
|
value: 37.604 |
|
- type: map_at_100 |
|
value: 38.897 |
|
- type: map_at_1000 |
|
value: 39.001000000000005 |
|
- type: map_at_3 |
|
value: 34.04 |
|
- type: map_at_5 |
|
value: 35.684 |
|
- type: mrr_at_1 |
|
value: 32.991 |
|
- type: mrr_at_10 |
|
value: 43.029 |
|
- type: mrr_at_100 |
|
value: 43.782 |
|
- type: mrr_at_1000 |
|
value: 43.830999999999996 |
|
- type: mrr_at_3 |
|
value: 40.164 |
|
- type: mrr_at_5 |
|
value: 41.619 |
|
- type: ndcg_at_1 |
|
value: 32.991 |
|
- type: ndcg_at_10 |
|
value: 44.217 |
|
- type: ndcg_at_100 |
|
value: 49.497 |
|
- type: ndcg_at_1000 |
|
value: 51.598 |
|
- type: ndcg_at_3 |
|
value: 38.208999999999996 |
|
- type: ndcg_at_5 |
|
value: 40.444 |
|
- type: precision_at_1 |
|
value: 32.991 |
|
- type: precision_at_10 |
|
value: 8.436 |
|
- type: precision_at_100 |
|
value: 1.279 |
|
- type: precision_at_1000 |
|
value: 0.163 |
|
- type: precision_at_3 |
|
value: 18.379 |
|
- type: precision_at_5 |
|
value: 13.196 |
|
- type: recall_at_1 |
|
value: 26.641 |
|
- type: recall_at_10 |
|
value: 58.50300000000001 |
|
- type: recall_at_100 |
|
value: 81.228 |
|
- type: recall_at_1000 |
|
value: 95.345 |
|
- type: recall_at_3 |
|
value: 41.6 |
|
- type: recall_at_5 |
|
value: 47.425 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 28.678083333333333 |
|
- type: map_at_10 |
|
value: 38.63366666666666 |
|
- type: map_at_100 |
|
value: 39.84708333333333 |
|
- type: map_at_1000 |
|
value: 39.959583333333335 |
|
- type: map_at_3 |
|
value: 35.49 |
|
- type: map_at_5 |
|
value: 37.23125 |
|
- type: mrr_at_1 |
|
value: 33.813916666666664 |
|
- type: mrr_at_10 |
|
value: 42.955500000000015 |
|
- type: mrr_at_100 |
|
value: 43.75541666666667 |
|
- type: mrr_at_1000 |
|
value: 43.80616666666666 |
|
- type: mrr_at_3 |
|
value: 40.40191666666667 |
|
- type: mrr_at_5 |
|
value: 41.88358333333333 |
|
- type: ndcg_at_1 |
|
value: 33.813916666666664 |
|
- type: ndcg_at_10 |
|
value: 44.361666666666665 |
|
- type: ndcg_at_100 |
|
value: 49.37991666666667 |
|
- type: ndcg_at_1000 |
|
value: 51.432583333333326 |
|
- type: ndcg_at_3 |
|
value: 39.12949999999999 |
|
- type: ndcg_at_5 |
|
value: 41.60183333333333 |
|
- type: precision_at_1 |
|
value: 33.813916666666664 |
|
- type: precision_at_10 |
|
value: 7.759250000000002 |
|
- type: precision_at_100 |
|
value: 1.2108333333333332 |
|
- type: precision_at_1000 |
|
value: 0.158 |
|
- type: precision_at_3 |
|
value: 17.90716666666667 |
|
- type: precision_at_5 |
|
value: 12.765333333333334 |
|
- type: recall_at_1 |
|
value: 28.678083333333333 |
|
- type: recall_at_10 |
|
value: 56.92716666666667 |
|
- type: recall_at_100 |
|
value: 78.74991666666668 |
|
- type: recall_at_1000 |
|
value: 92.73875000000001 |
|
- type: recall_at_3 |
|
value: 42.459916666666665 |
|
- type: recall_at_5 |
|
value: 48.76258333333333 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 27.282 |
|
- type: map_at_10 |
|
value: 34.458 |
|
- type: map_at_100 |
|
value: 35.44 |
|
- type: map_at_1000 |
|
value: 35.536 |
|
- type: map_at_3 |
|
value: 31.912000000000003 |
|
- type: map_at_5 |
|
value: 33.495000000000005 |
|
- type: mrr_at_1 |
|
value: 30.675 |
|
- type: mrr_at_10 |
|
value: 37.563 |
|
- type: mrr_at_100 |
|
value: 38.374 |
|
- type: mrr_at_1000 |
|
value: 38.444 |
|
- type: mrr_at_3 |
|
value: 35.276 |
|
- type: mrr_at_5 |
|
value: 36.718 |
|
- type: ndcg_at_1 |
|
value: 30.675 |
|
- type: ndcg_at_10 |
|
value: 38.838 |
|
- type: ndcg_at_100 |
|
value: 43.527 |
|
- type: ndcg_at_1000 |
|
value: 45.891 |
|
- type: ndcg_at_3 |
|
value: 34.314 |
|
- type: ndcg_at_5 |
|
value: 36.789 |
|
- type: precision_at_1 |
|
value: 30.675 |
|
- type: precision_at_10 |
|
value: 6.012 |
|
- type: precision_at_100 |
|
value: 0.903 |
|
- type: precision_at_1000 |
|
value: 0.117 |
|
- type: precision_at_3 |
|
value: 14.571000000000002 |
|
- type: precision_at_5 |
|
value: 10.306999999999999 |
|
- type: recall_at_1 |
|
value: 27.282 |
|
- type: recall_at_10 |
|
value: 49.198 |
|
- type: recall_at_100 |
|
value: 70.489 |
|
- type: recall_at_1000 |
|
value: 87.902 |
|
- type: recall_at_3 |
|
value: 36.966 |
|
- type: recall_at_5 |
|
value: 43.079 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 19.839000000000002 |
|
- type: map_at_10 |
|
value: 27.68 |
|
- type: map_at_100 |
|
value: 28.851 |
|
- type: map_at_1000 |
|
value: 28.977999999999998 |
|
- type: map_at_3 |
|
value: 25.062 |
|
- type: map_at_5 |
|
value: 26.389000000000003 |
|
- type: mrr_at_1 |
|
value: 23.813000000000002 |
|
- type: mrr_at_10 |
|
value: 31.628 |
|
- type: mrr_at_100 |
|
value: 32.58 |
|
- type: mrr_at_1000 |
|
value: 32.655 |
|
- type: mrr_at_3 |
|
value: 29.29 |
|
- type: mrr_at_5 |
|
value: 30.551000000000002 |
|
- type: ndcg_at_1 |
|
value: 23.813000000000002 |
|
- type: ndcg_at_10 |
|
value: 32.751000000000005 |
|
- type: ndcg_at_100 |
|
value: 38.218 |
|
- type: ndcg_at_1000 |
|
value: 40.979 |
|
- type: ndcg_at_3 |
|
value: 28.043000000000003 |
|
- type: ndcg_at_5 |
|
value: 30.043 |
|
- type: precision_at_1 |
|
value: 23.813000000000002 |
|
- type: precision_at_10 |
|
value: 5.936 |
|
- type: precision_at_100 |
|
value: 1.016 |
|
- type: precision_at_1000 |
|
value: 0.14400000000000002 |
|
- type: precision_at_3 |
|
value: 13.145000000000001 |
|
- type: precision_at_5 |
|
value: 9.443 |
|
- type: recall_at_1 |
|
value: 19.839000000000002 |
|
- type: recall_at_10 |
|
value: 44.072 |
|
- type: recall_at_100 |
|
value: 68.406 |
|
- type: recall_at_1000 |
|
value: 87.749 |
|
- type: recall_at_3 |
|
value: 30.906 |
|
- type: recall_at_5 |
|
value: 36.081 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 28.195999999999998 |
|
- type: map_at_10 |
|
value: 38.345 |
|
- type: map_at_100 |
|
value: 39.561 |
|
- type: map_at_1000 |
|
value: 39.65 |
|
- type: map_at_3 |
|
value: 35.382999999999996 |
|
- type: map_at_5 |
|
value: 37.023 |
|
- type: mrr_at_1 |
|
value: 33.022 |
|
- type: mrr_at_10 |
|
value: 42.504 |
|
- type: mrr_at_100 |
|
value: 43.376 |
|
- type: mrr_at_1000 |
|
value: 43.427 |
|
- type: mrr_at_3 |
|
value: 40.050000000000004 |
|
- type: mrr_at_5 |
|
value: 41.421 |
|
- type: ndcg_at_1 |
|
value: 33.022 |
|
- type: ndcg_at_10 |
|
value: 43.997 |
|
- type: ndcg_at_100 |
|
value: 49.370000000000005 |
|
- type: ndcg_at_1000 |
|
value: 51.38399999999999 |
|
- type: ndcg_at_3 |
|
value: 38.802 |
|
- type: ndcg_at_5 |
|
value: 41.209 |
|
- type: precision_at_1 |
|
value: 33.022 |
|
- type: precision_at_10 |
|
value: 7.351000000000001 |
|
- type: precision_at_100 |
|
value: 1.1440000000000001 |
|
- type: precision_at_1000 |
|
value: 0.14200000000000002 |
|
- type: precision_at_3 |
|
value: 17.724 |
|
- type: precision_at_5 |
|
value: 12.443999999999999 |
|
- type: recall_at_1 |
|
value: 28.195999999999998 |
|
- type: recall_at_10 |
|
value: 57.011 |
|
- type: recall_at_100 |
|
value: 79.922 |
|
- type: recall_at_1000 |
|
value: 93.952 |
|
- type: recall_at_3 |
|
value: 42.857 |
|
- type: recall_at_5 |
|
value: 48.916 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.768 |
|
- type: map_at_10 |
|
value: 35.118 |
|
- type: map_at_100 |
|
value: 36.817 |
|
- type: map_at_1000 |
|
value: 37.037 |
|
- type: map_at_3 |
|
value: 31.997999999999998 |
|
- type: map_at_5 |
|
value: 33.697 |
|
- type: mrr_at_1 |
|
value: 31.621 |
|
- type: mrr_at_10 |
|
value: 40.228 |
|
- type: mrr_at_100 |
|
value: 41.239 |
|
- type: mrr_at_1000 |
|
value: 41.277 |
|
- type: mrr_at_3 |
|
value: 37.614999999999995 |
|
- type: mrr_at_5 |
|
value: 39.058 |
|
- type: ndcg_at_1 |
|
value: 31.621 |
|
- type: ndcg_at_10 |
|
value: 41.347 |
|
- type: ndcg_at_100 |
|
value: 47.620000000000005 |
|
- type: ndcg_at_1000 |
|
value: 49.759 |
|
- type: ndcg_at_3 |
|
value: 36.361 |
|
- type: ndcg_at_5 |
|
value: 38.635000000000005 |
|
- type: precision_at_1 |
|
value: 31.621 |
|
- type: precision_at_10 |
|
value: 8.024000000000001 |
|
- type: precision_at_100 |
|
value: 1.595 |
|
- type: precision_at_1000 |
|
value: 0.244 |
|
- type: precision_at_3 |
|
value: 16.996 |
|
- type: precision_at_5 |
|
value: 12.372 |
|
- type: recall_at_1 |
|
value: 25.768 |
|
- type: recall_at_10 |
|
value: 53.02 |
|
- type: recall_at_100 |
|
value: 81.329 |
|
- type: recall_at_1000 |
|
value: 94.025 |
|
- type: recall_at_3 |
|
value: 38.884 |
|
- type: recall_at_5 |
|
value: 45.057 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.627 |
|
- type: map_at_10 |
|
value: 33.106 |
|
- type: map_at_100 |
|
value: 33.936 |
|
- type: map_at_1000 |
|
value: 34.044999999999995 |
|
- type: map_at_3 |
|
value: 30.162 |
|
- type: map_at_5 |
|
value: 31.979999999999997 |
|
- type: mrr_at_1 |
|
value: 26.617 |
|
- type: mrr_at_10 |
|
value: 35.177 |
|
- type: mrr_at_100 |
|
value: 35.937999999999995 |
|
- type: mrr_at_1000 |
|
value: 36.008 |
|
- type: mrr_at_3 |
|
value: 32.562999999999995 |
|
- type: mrr_at_5 |
|
value: 34.208 |
|
- type: ndcg_at_1 |
|
value: 26.617 |
|
- type: ndcg_at_10 |
|
value: 38.082 |
|
- type: ndcg_at_100 |
|
value: 42.386 |
|
- type: ndcg_at_1000 |
|
value: 44.861000000000004 |
|
- type: ndcg_at_3 |
|
value: 32.557 |
|
- type: ndcg_at_5 |
|
value: 35.603 |
|
- type: precision_at_1 |
|
value: 26.617 |
|
- type: precision_at_10 |
|
value: 5.952 |
|
- type: precision_at_100 |
|
value: 0.874 |
|
- type: precision_at_1000 |
|
value: 0.121 |
|
- type: precision_at_3 |
|
value: 13.617 |
|
- type: precision_at_5 |
|
value: 9.945 |
|
- type: recall_at_1 |
|
value: 24.627 |
|
- type: recall_at_10 |
|
value: 51.317 |
|
- type: recall_at_100 |
|
value: 71.243 |
|
- type: recall_at_1000 |
|
value: 89.39399999999999 |
|
- type: recall_at_3 |
|
value: 36.778 |
|
- type: recall_at_5 |
|
value: 44.116 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.631 |
|
- type: map_at_10 |
|
value: 28.069 |
|
- type: map_at_100 |
|
value: 30.130000000000003 |
|
- type: map_at_1000 |
|
value: 30.318 |
|
- type: map_at_3 |
|
value: 23.430999999999997 |
|
- type: map_at_5 |
|
value: 25.929000000000002 |
|
- type: mrr_at_1 |
|
value: 37.264 |
|
- type: mrr_at_10 |
|
value: 49.608999999999995 |
|
- type: mrr_at_100 |
|
value: 50.349 |
|
- type: mrr_at_1000 |
|
value: 50.373000000000005 |
|
- type: mrr_at_3 |
|
value: 46.515 |
|
- type: mrr_at_5 |
|
value: 48.41 |
|
- type: ndcg_at_1 |
|
value: 37.264 |
|
- type: ndcg_at_10 |
|
value: 37.688 |
|
- type: ndcg_at_100 |
|
value: 45.101 |
|
- type: ndcg_at_1000 |
|
value: 48.19 |
|
- type: ndcg_at_3 |
|
value: 31.471 |
|
- type: ndcg_at_5 |
|
value: 33.719 |
|
- type: precision_at_1 |
|
value: 37.264 |
|
- type: precision_at_10 |
|
value: 11.616 |
|
- type: precision_at_100 |
|
value: 1.9619999999999997 |
|
- type: precision_at_1000 |
|
value: 0.255 |
|
- type: precision_at_3 |
|
value: 23.214000000000002 |
|
- type: precision_at_5 |
|
value: 17.824 |
|
- type: recall_at_1 |
|
value: 16.631 |
|
- type: recall_at_10 |
|
value: 43.516 |
|
- type: recall_at_100 |
|
value: 68.681 |
|
- type: recall_at_1000 |
|
value: 85.751 |
|
- type: recall_at_3 |
|
value: 28.199 |
|
- type: recall_at_5 |
|
value: 34.826 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 9.971 |
|
- type: map_at_10 |
|
value: 22.274 |
|
- type: map_at_100 |
|
value: 32.61 |
|
- type: map_at_1000 |
|
value: 34.422000000000004 |
|
- type: map_at_3 |
|
value: 15.473999999999998 |
|
- type: map_at_5 |
|
value: 18.412 |
|
- type: mrr_at_1 |
|
value: 72.25 |
|
- type: mrr_at_10 |
|
value: 79.945 |
|
- type: mrr_at_100 |
|
value: 80.192 |
|
- type: mrr_at_1000 |
|
value: 80.199 |
|
- type: mrr_at_3 |
|
value: 78.667 |
|
- type: mrr_at_5 |
|
value: 79.49199999999999 |
|
- type: ndcg_at_1 |
|
value: 59.75 |
|
- type: ndcg_at_10 |
|
value: 45.689 |
|
- type: ndcg_at_100 |
|
value: 51.687000000000005 |
|
- type: ndcg_at_1000 |
|
value: 58.904999999999994 |
|
- type: ndcg_at_3 |
|
value: 49.675999999999995 |
|
- type: ndcg_at_5 |
|
value: 47.419 |
|
- type: precision_at_1 |
|
value: 72.25 |
|
- type: precision_at_10 |
|
value: 37.05 |
|
- type: precision_at_100 |
|
value: 12.183 |
|
- type: precision_at_1000 |
|
value: 2.2929999999999997 |
|
- type: precision_at_3 |
|
value: 53.417 |
|
- type: precision_at_5 |
|
value: 46.150000000000006 |
|
- type: recall_at_1 |
|
value: 9.971 |
|
- type: recall_at_10 |
|
value: 27.932000000000002 |
|
- type: recall_at_100 |
|
value: 58.85399999999999 |
|
- type: recall_at_1000 |
|
value: 81.728 |
|
- type: recall_at_3 |
|
value: 16.619999999999997 |
|
- type: recall_at_5 |
|
value: 21.082 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 52.83499999999999 |
|
- type: f1 |
|
value: 47.754076079187044 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fever |
|
name: MTEB FEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 79.783 |
|
- type: map_at_10 |
|
value: 87.224 |
|
- type: map_at_100 |
|
value: 87.401 |
|
- type: map_at_1000 |
|
value: 87.413 |
|
- type: map_at_3 |
|
value: 86.29 |
|
- type: map_at_5 |
|
value: 86.896 |
|
- type: mrr_at_1 |
|
value: 86.09400000000001 |
|
- type: mrr_at_10 |
|
value: 91.789 |
|
- type: mrr_at_100 |
|
value: 91.814 |
|
- type: mrr_at_1000 |
|
value: 91.815 |
|
- type: mrr_at_3 |
|
value: 91.39399999999999 |
|
- type: mrr_at_5 |
|
value: 91.684 |
|
- type: ndcg_at_1 |
|
value: 86.09400000000001 |
|
- type: ndcg_at_10 |
|
value: 90.36999999999999 |
|
- type: ndcg_at_100 |
|
value: 90.95299999999999 |
|
- type: ndcg_at_1000 |
|
value: 91.13799999999999 |
|
- type: ndcg_at_3 |
|
value: 89.13799999999999 |
|
- type: ndcg_at_5 |
|
value: 89.845 |
|
- type: precision_at_1 |
|
value: 86.09400000000001 |
|
- type: precision_at_10 |
|
value: 10.671 |
|
- type: precision_at_100 |
|
value: 1.123 |
|
- type: precision_at_1000 |
|
value: 0.11499999999999999 |
|
- type: precision_at_3 |
|
value: 33.698 |
|
- type: precision_at_5 |
|
value: 20.788999999999998 |
|
- type: recall_at_1 |
|
value: 79.783 |
|
- type: recall_at_10 |
|
value: 95.50999999999999 |
|
- type: recall_at_100 |
|
value: 97.68900000000001 |
|
- type: recall_at_1000 |
|
value: 98.79400000000001 |
|
- type: recall_at_3 |
|
value: 92.14099999999999 |
|
- type: recall_at_5 |
|
value: 94.0 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa |
|
name: MTEB FiQA2018 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.526 |
|
- type: map_at_10 |
|
value: 38.135999999999996 |
|
- type: map_at_100 |
|
value: 40.221000000000004 |
|
- type: map_at_1000 |
|
value: 40.394000000000005 |
|
- type: map_at_3 |
|
value: 33.548 |
|
- type: map_at_5 |
|
value: 35.975 |
|
- type: mrr_at_1 |
|
value: 47.068 |
|
- type: mrr_at_10 |
|
value: 55.224 |
|
- type: mrr_at_100 |
|
value: 56.038 |
|
- type: mrr_at_1000 |
|
value: 56.066 |
|
- type: mrr_at_3 |
|
value: 53.00899999999999 |
|
- type: mrr_at_5 |
|
value: 54.306 |
|
- type: ndcg_at_1 |
|
value: 47.068 |
|
- type: ndcg_at_10 |
|
value: 46.399 |
|
- type: ndcg_at_100 |
|
value: 53.312000000000005 |
|
- type: ndcg_at_1000 |
|
value: 55.946 |
|
- type: ndcg_at_3 |
|
value: 42.954 |
|
- type: ndcg_at_5 |
|
value: 43.765 |
|
- type: precision_at_1 |
|
value: 47.068 |
|
- type: precision_at_10 |
|
value: 12.824 |
|
- type: precision_at_100 |
|
value: 1.986 |
|
- type: precision_at_1000 |
|
value: 0.246 |
|
- type: precision_at_3 |
|
value: 28.807 |
|
- type: precision_at_5 |
|
value: 20.772 |
|
- type: recall_at_1 |
|
value: 23.526 |
|
- type: recall_at_10 |
|
value: 53.242999999999995 |
|
- type: recall_at_100 |
|
value: 78.309 |
|
- type: recall_at_1000 |
|
value: 93.92099999999999 |
|
- type: recall_at_3 |
|
value: 38.716 |
|
- type: recall_at_5 |
|
value: 44.921 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa |
|
name: MTEB HotpotQA |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 41.641 |
|
- type: map_at_10 |
|
value: 67.24 |
|
- type: map_at_100 |
|
value: 68.108 |
|
- type: map_at_1000 |
|
value: 68.157 |
|
- type: map_at_3 |
|
value: 63.834999999999994 |
|
- type: map_at_5 |
|
value: 65.995 |
|
- type: mrr_at_1 |
|
value: 83.282 |
|
- type: mrr_at_10 |
|
value: 88.22 |
|
- type: mrr_at_100 |
|
value: 88.35499999999999 |
|
- type: mrr_at_1000 |
|
value: 88.358 |
|
- type: mrr_at_3 |
|
value: 87.571 |
|
- type: mrr_at_5 |
|
value: 88.01299999999999 |
|
- type: ndcg_at_1 |
|
value: 83.282 |
|
- type: ndcg_at_10 |
|
value: 75.066 |
|
- type: ndcg_at_100 |
|
value: 77.952 |
|
- type: ndcg_at_1000 |
|
value: 78.878 |
|
- type: ndcg_at_3 |
|
value: 70.482 |
|
- type: ndcg_at_5 |
|
value: 73.098 |
|
- type: precision_at_1 |
|
value: 83.282 |
|
- type: precision_at_10 |
|
value: 15.608 |
|
- type: precision_at_100 |
|
value: 1.7840000000000003 |
|
- type: precision_at_1000 |
|
value: 0.191 |
|
- type: precision_at_3 |
|
value: 45.324999999999996 |
|
- type: precision_at_5 |
|
value: 29.256 |
|
- type: recall_at_1 |
|
value: 41.641 |
|
- type: recall_at_10 |
|
value: 78.042 |
|
- type: recall_at_100 |
|
value: 89.223 |
|
- type: recall_at_1000 |
|
value: 95.341 |
|
- type: recall_at_3 |
|
value: 67.988 |
|
- type: recall_at_5 |
|
value: 73.14 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 93.50520000000002 |
|
- type: ap |
|
value: 90.36560251927821 |
|
- type: f1 |
|
value: 93.50064413170799 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco |
|
name: MTEB MSMARCO |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.355999999999998 |
|
- type: map_at_10 |
|
value: 36.082 |
|
- type: map_at_100 |
|
value: 37.239 |
|
- type: map_at_1000 |
|
value: 37.285000000000004 |
|
- type: map_at_3 |
|
value: 32.16 |
|
- type: map_at_5 |
|
value: 34.469 |
|
- type: mrr_at_1 |
|
value: 23.968 |
|
- type: mrr_at_10 |
|
value: 36.708 |
|
- type: mrr_at_100 |
|
value: 37.795 |
|
- type: mrr_at_1000 |
|
value: 37.836 |
|
- type: mrr_at_3 |
|
value: 32.865 |
|
- type: mrr_at_5 |
|
value: 35.154 |
|
- type: ndcg_at_1 |
|
value: 23.968 |
|
- type: ndcg_at_10 |
|
value: 43.152 |
|
- type: ndcg_at_100 |
|
value: 48.615 |
|
- type: ndcg_at_1000 |
|
value: 49.714000000000006 |
|
- type: ndcg_at_3 |
|
value: 35.208 |
|
- type: ndcg_at_5 |
|
value: 39.342 |
|
- type: precision_at_1 |
|
value: 23.968 |
|
- type: precision_at_10 |
|
value: 6.784 |
|
- type: precision_at_100 |
|
value: 0.951 |
|
- type: precision_at_1000 |
|
value: 0.104 |
|
- type: precision_at_3 |
|
value: 14.995 |
|
- type: precision_at_5 |
|
value: 11.092 |
|
- type: recall_at_1 |
|
value: 23.355999999999998 |
|
- type: recall_at_10 |
|
value: 64.828 |
|
- type: recall_at_100 |
|
value: 89.888 |
|
- type: recall_at_1000 |
|
value: 98.181 |
|
- type: recall_at_3 |
|
value: 43.336000000000006 |
|
- type: recall_at_5 |
|
value: 53.274 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 94.97948016415869 |
|
- type: f1 |
|
value: 94.77285510790911 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 78.49749202006383 |
|
- type: f1 |
|
value: 59.36772995632707 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 77.64290517821117 |
|
- type: f1 |
|
value: 75.33296771580456 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 80.76664425016811 |
|
- type: f1 |
|
value: 80.79147962348141 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 35.158637354708034 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 32.39319499403552 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 32.19460802526735 |
|
- type: mrr |
|
value: 33.39458959690712 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus |
|
name: MTEB NFCorpus |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 7.225 |
|
- type: map_at_10 |
|
value: 15.609 |
|
- type: map_at_100 |
|
value: 20.067 |
|
- type: map_at_1000 |
|
value: 21.709999999999997 |
|
- type: map_at_3 |
|
value: 11.518 |
|
- type: map_at_5 |
|
value: 13.469999999999999 |
|
- type: mrr_at_1 |
|
value: 50.15500000000001 |
|
- type: mrr_at_10 |
|
value: 58.711 |
|
- type: mrr_at_100 |
|
value: 59.333000000000006 |
|
- type: mrr_at_1000 |
|
value: 59.362 |
|
- type: mrr_at_3 |
|
value: 56.65599999999999 |
|
- type: mrr_at_5 |
|
value: 57.972 |
|
- type: ndcg_at_1 |
|
value: 48.452 |
|
- type: ndcg_at_10 |
|
value: 38.845 |
|
- type: ndcg_at_100 |
|
value: 36.597 |
|
- type: ndcg_at_1000 |
|
value: 45.472 |
|
- type: ndcg_at_3 |
|
value: 43.947 |
|
- type: ndcg_at_5 |
|
value: 42.097 |
|
- type: precision_at_1 |
|
value: 49.845 |
|
- type: precision_at_10 |
|
value: 28.638 |
|
- type: precision_at_100 |
|
value: 9.229 |
|
- type: precision_at_1000 |
|
value: 2.234 |
|
- type: precision_at_3 |
|
value: 40.867 |
|
- type: precision_at_5 |
|
value: 36.285000000000004 |
|
- type: recall_at_1 |
|
value: 7.225 |
|
- type: recall_at_10 |
|
value: 19.272 |
|
- type: recall_at_100 |
|
value: 37.299 |
|
- type: recall_at_1000 |
|
value: 68.757 |
|
- type: recall_at_3 |
|
value: 12.350999999999999 |
|
- type: recall_at_5 |
|
value: 15.369 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq |
|
name: MTEB NQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 34.453 |
|
- type: map_at_10 |
|
value: 50.748000000000005 |
|
- type: map_at_100 |
|
value: 51.666000000000004 |
|
- type: map_at_1000 |
|
value: 51.687000000000005 |
|
- type: map_at_3 |
|
value: 46.300000000000004 |
|
- type: map_at_5 |
|
value: 49.032 |
|
- type: mrr_at_1 |
|
value: 38.673 |
|
- type: mrr_at_10 |
|
value: 53.11 |
|
- type: mrr_at_100 |
|
value: 53.772 |
|
- type: mrr_at_1000 |
|
value: 53.784 |
|
- type: mrr_at_3 |
|
value: 49.483 |
|
- type: mrr_at_5 |
|
value: 51.751999999999995 |
|
- type: ndcg_at_1 |
|
value: 38.673 |
|
- type: ndcg_at_10 |
|
value: 58.60300000000001 |
|
- type: ndcg_at_100 |
|
value: 62.302 |
|
- type: ndcg_at_1000 |
|
value: 62.763999999999996 |
|
- type: ndcg_at_3 |
|
value: 50.366 |
|
- type: ndcg_at_5 |
|
value: 54.888999999999996 |
|
- type: precision_at_1 |
|
value: 38.673 |
|
- type: precision_at_10 |
|
value: 9.522 |
|
- type: precision_at_100 |
|
value: 1.162 |
|
- type: precision_at_1000 |
|
value: 0.121 |
|
- type: precision_at_3 |
|
value: 22.779 |
|
- type: precision_at_5 |
|
value: 16.256999999999998 |
|
- type: recall_at_1 |
|
value: 34.453 |
|
- type: recall_at_10 |
|
value: 80.074 |
|
- type: recall_at_100 |
|
value: 95.749 |
|
- type: recall_at_1000 |
|
value: 99.165 |
|
- type: recall_at_3 |
|
value: 58.897999999999996 |
|
- type: recall_at_5 |
|
value: 69.349 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora |
|
name: MTEB QuoraRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 71.80499999999999 |
|
- type: map_at_10 |
|
value: 85.773 |
|
- type: map_at_100 |
|
value: 86.4 |
|
- type: map_at_1000 |
|
value: 86.414 |
|
- type: map_at_3 |
|
value: 82.919 |
|
- type: map_at_5 |
|
value: 84.70299999999999 |
|
- type: mrr_at_1 |
|
value: 82.69999999999999 |
|
- type: mrr_at_10 |
|
value: 88.592 |
|
- type: mrr_at_100 |
|
value: 88.682 |
|
- type: mrr_at_1000 |
|
value: 88.683 |
|
- type: mrr_at_3 |
|
value: 87.705 |
|
- type: mrr_at_5 |
|
value: 88.30799999999999 |
|
- type: ndcg_at_1 |
|
value: 82.69 |
|
- type: ndcg_at_10 |
|
value: 89.316 |
|
- type: ndcg_at_100 |
|
value: 90.45100000000001 |
|
- type: ndcg_at_1000 |
|
value: 90.525 |
|
- type: ndcg_at_3 |
|
value: 86.68 |
|
- type: ndcg_at_5 |
|
value: 88.113 |
|
- type: precision_at_1 |
|
value: 82.69 |
|
- type: precision_at_10 |
|
value: 13.507 |
|
- type: precision_at_100 |
|
value: 1.5350000000000001 |
|
- type: precision_at_1000 |
|
value: 0.157 |
|
- type: precision_at_3 |
|
value: 37.927 |
|
- type: precision_at_5 |
|
value: 24.823999999999998 |
|
- type: recall_at_1 |
|
value: 71.80499999999999 |
|
- type: recall_at_10 |
|
value: 95.965 |
|
- type: recall_at_100 |
|
value: 99.70400000000001 |
|
- type: recall_at_1000 |
|
value: 99.992 |
|
- type: recall_at_3 |
|
value: 88.268 |
|
- type: recall_at_5 |
|
value: 92.45 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 60.24178219867024 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 64.99552099515469 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs |
|
name: MTEB SCIDOCS |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.4879999999999995 |
|
- type: map_at_10 |
|
value: 14.774999999999999 |
|
- type: map_at_100 |
|
value: 17.285 |
|
- type: map_at_1000 |
|
value: 17.648 |
|
- type: map_at_3 |
|
value: 10.4 |
|
- type: map_at_5 |
|
value: 12.552 |
|
- type: mrr_at_1 |
|
value: 27.1 |
|
- type: mrr_at_10 |
|
value: 39.251000000000005 |
|
- type: mrr_at_100 |
|
value: 40.335 |
|
- type: mrr_at_1000 |
|
value: 40.367 |
|
- type: mrr_at_3 |
|
value: 35.683 |
|
- type: mrr_at_5 |
|
value: 37.733 |
|
- type: ndcg_at_1 |
|
value: 27.1 |
|
- type: ndcg_at_10 |
|
value: 23.974 |
|
- type: ndcg_at_100 |
|
value: 33.161 |
|
- type: ndcg_at_1000 |
|
value: 38.853 |
|
- type: ndcg_at_3 |
|
value: 22.695999999999998 |
|
- type: ndcg_at_5 |
|
value: 19.881 |
|
- type: precision_at_1 |
|
value: 27.1 |
|
- type: precision_at_10 |
|
value: 12.479999999999999 |
|
- type: precision_at_100 |
|
value: 2.571 |
|
- type: precision_at_1000 |
|
value: 0.393 |
|
- type: precision_at_3 |
|
value: 21.367 |
|
- type: precision_at_5 |
|
value: 17.560000000000002 |
|
- type: recall_at_1 |
|
value: 5.4879999999999995 |
|
- type: recall_at_10 |
|
value: 25.290000000000003 |
|
- type: recall_at_100 |
|
value: 52.222 |
|
- type: recall_at_1000 |
|
value: 79.77300000000001 |
|
- type: recall_at_3 |
|
value: 13.001999999999999 |
|
- type: recall_at_5 |
|
value: 17.812 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.3407705934785 |
|
- type: cos_sim_spearman |
|
value: 81.28145766913589 |
|
- type: euclidean_pearson |
|
value: 82.69277819943873 |
|
- type: euclidean_spearman |
|
value: 81.26097565088551 |
|
- type: manhattan_pearson |
|
value: 82.73440374725746 |
|
- type: manhattan_spearman |
|
value: 81.25376873901254 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 87.23415639286914 |
|
- type: cos_sim_spearman |
|
value: 79.80147936079915 |
|
- type: euclidean_pearson |
|
value: 84.324220218071 |
|
- type: euclidean_spearman |
|
value: 79.71794784987208 |
|
- type: manhattan_pearson |
|
value: 84.27523842345964 |
|
- type: manhattan_spearman |
|
value: 79.58070329781553 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.90966234413125 |
|
- type: cos_sim_spearman |
|
value: 87.10742652814713 |
|
- type: euclidean_pearson |
|
value: 86.28297063322286 |
|
- type: euclidean_spearman |
|
value: 87.09425001932226 |
|
- type: manhattan_pearson |
|
value: 86.19204338411774 |
|
- type: manhattan_spearman |
|
value: 87.02046826723424 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.12351399124411 |
|
- type: cos_sim_spearman |
|
value: 83.32955357808568 |
|
- type: euclidean_pearson |
|
value: 83.81222384305896 |
|
- type: euclidean_spearman |
|
value: 83.1836394454507 |
|
- type: manhattan_pearson |
|
value: 83.79162945392092 |
|
- type: manhattan_spearman |
|
value: 83.14306058903364 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 86.9345194840047 |
|
- type: cos_sim_spearman |
|
value: 88.47286320653176 |
|
- type: euclidean_pearson |
|
value: 87.72825182191445 |
|
- type: euclidean_spearman |
|
value: 88.33484195475864 |
|
- type: manhattan_pearson |
|
value: 87.75121043906692 |
|
- type: manhattan_spearman |
|
value: 88.36695329548576 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.80215370816441 |
|
- type: cos_sim_spearman |
|
value: 86.44917331470305 |
|
- type: euclidean_pearson |
|
value: 85.3458573021962 |
|
- type: euclidean_spearman |
|
value: 86.24853627058414 |
|
- type: manhattan_pearson |
|
value: 85.38477148579328 |
|
- type: manhattan_spearman |
|
value: 86.28201585857053 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 87.20498617189688 |
|
- type: cos_sim_spearman |
|
value: 87.61389142076317 |
|
- type: euclidean_pearson |
|
value: 88.15430699740293 |
|
- type: euclidean_spearman |
|
value: 87.35065666258774 |
|
- type: manhattan_pearson |
|
value: 88.2994571119992 |
|
- type: manhattan_spearman |
|
value: 87.60920178284005 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 68.27672577392406 |
|
- type: cos_sim_spearman |
|
value: 68.31250175566586 |
|
- type: euclidean_pearson |
|
value: 69.45016222616813 |
|
- type: euclidean_spearman |
|
value: 67.93461301528046 |
|
- type: manhattan_pearson |
|
value: 69.39774219739259 |
|
- type: manhattan_spearman |
|
value: 67.78124856615536 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.31916148698113 |
|
- type: cos_sim_spearman |
|
value: 87.45541524487057 |
|
- type: euclidean_pearson |
|
value: 86.5845909408775 |
|
- type: euclidean_spearman |
|
value: 87.2373331768082 |
|
- type: manhattan_pearson |
|
value: 86.64467698948668 |
|
- type: manhattan_spearman |
|
value: 87.26707857525533 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 88.0007930269447 |
|
- type: mrr |
|
value: 96.52852594029063 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact |
|
name: MTEB SciFact |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 60.99400000000001 |
|
- type: map_at_10 |
|
value: 70.923 |
|
- type: map_at_100 |
|
value: 71.299 |
|
- type: map_at_1000 |
|
value: 71.318 |
|
- type: map_at_3 |
|
value: 67.991 |
|
- type: map_at_5 |
|
value: 69.292 |
|
- type: mrr_at_1 |
|
value: 64.333 |
|
- type: mrr_at_10 |
|
value: 71.98400000000001 |
|
- type: mrr_at_100 |
|
value: 72.306 |
|
- type: mrr_at_1000 |
|
value: 72.32499999999999 |
|
- type: mrr_at_3 |
|
value: 69.833 |
|
- type: mrr_at_5 |
|
value: 70.783 |
|
- type: ndcg_at_1 |
|
value: 64.333 |
|
- type: ndcg_at_10 |
|
value: 75.729 |
|
- type: ndcg_at_100 |
|
value: 77.38199999999999 |
|
- type: ndcg_at_1000 |
|
value: 77.788 |
|
- type: ndcg_at_3 |
|
value: 70.774 |
|
- type: ndcg_at_5 |
|
value: 72.478 |
|
- type: precision_at_1 |
|
value: 64.333 |
|
- type: precision_at_10 |
|
value: 10.167 |
|
- type: precision_at_100 |
|
value: 1.0999999999999999 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 27.778000000000002 |
|
- type: precision_at_5 |
|
value: 17.867 |
|
- type: recall_at_1 |
|
value: 60.99400000000001 |
|
- type: recall_at_10 |
|
value: 89.48899999999999 |
|
- type: recall_at_100 |
|
value: 97.0 |
|
- type: recall_at_1000 |
|
value: 100.0 |
|
- type: recall_at_3 |
|
value: 75.85 |
|
- type: recall_at_5 |
|
value: 80.328 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.86435643564356 |
|
- type: cos_sim_ap |
|
value: 96.78001342960285 |
|
- type: cos_sim_f1 |
|
value: 93.07030854830552 |
|
- type: cos_sim_precision |
|
value: 94.16581371545547 |
|
- type: cos_sim_recall |
|
value: 92.0 |
|
- type: dot_accuracy |
|
value: 99.74653465346535 |
|
- type: dot_ap |
|
value: 92.80391251199522 |
|
- type: dot_f1 |
|
value: 87.36426456071075 |
|
- type: dot_precision |
|
value: 86.25730994152046 |
|
- type: dot_recall |
|
value: 88.5 |
|
- type: euclidean_accuracy |
|
value: 99.86138613861387 |
|
- type: euclidean_ap |
|
value: 96.77007810699926 |
|
- type: euclidean_f1 |
|
value: 92.95065458207452 |
|
- type: euclidean_precision |
|
value: 93.6105476673428 |
|
- type: euclidean_recall |
|
value: 92.30000000000001 |
|
- type: manhattan_accuracy |
|
value: 99.86336633663366 |
|
- type: manhattan_ap |
|
value: 96.78913160708261 |
|
- type: manhattan_f1 |
|
value: 93.03030303030305 |
|
- type: manhattan_precision |
|
value: 93.9795918367347 |
|
- type: manhattan_recall |
|
value: 92.10000000000001 |
|
- type: max_accuracy |
|
value: 99.86435643564356 |
|
- type: max_ap |
|
value: 96.78913160708261 |
|
- type: max_f1 |
|
value: 93.07030854830552 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 67.80798406371026 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 35.69251193913337 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 55.04250964616215 |
|
- type: mrr |
|
value: 55.92283125371361 |
|
- task: |
|
type: Summarization |
|
dataset: |
|
type: mteb/summeval |
|
name: MTEB SummEval |
|
config: default |
|
split: test |
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 31.05492162235311 |
|
- type: cos_sim_spearman |
|
value: 30.90473006515039 |
|
- type: dot_pearson |
|
value: 26.85480454105073 |
|
- type: dot_spearman |
|
value: 27.02880537417923 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid |
|
name: MTEB TRECCOVID |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.246 |
|
- type: map_at_10 |
|
value: 2.125 |
|
- type: map_at_100 |
|
value: 12.892999999999999 |
|
- type: map_at_1000 |
|
value: 31.513999999999996 |
|
- type: map_at_3 |
|
value: 0.695 |
|
- type: map_at_5 |
|
value: 1.133 |
|
- type: mrr_at_1 |
|
value: 92.0 |
|
- type: mrr_at_10 |
|
value: 95.667 |
|
- type: mrr_at_100 |
|
value: 95.667 |
|
- type: mrr_at_1000 |
|
value: 95.667 |
|
- type: mrr_at_3 |
|
value: 95.667 |
|
- type: mrr_at_5 |
|
value: 95.667 |
|
- type: ndcg_at_1 |
|
value: 88.0 |
|
- type: ndcg_at_10 |
|
value: 82.464 |
|
- type: ndcg_at_100 |
|
value: 63.351 |
|
- type: ndcg_at_1000 |
|
value: 57.129 |
|
- type: ndcg_at_3 |
|
value: 85.87700000000001 |
|
- type: ndcg_at_5 |
|
value: 86.042 |
|
- type: precision_at_1 |
|
value: 92.0 |
|
- type: precision_at_10 |
|
value: 86.2 |
|
- type: precision_at_100 |
|
value: 65.10000000000001 |
|
- type: precision_at_1000 |
|
value: 25.044 |
|
- type: precision_at_3 |
|
value: 89.333 |
|
- type: precision_at_5 |
|
value: 89.60000000000001 |
|
- type: recall_at_1 |
|
value: 0.246 |
|
- type: recall_at_10 |
|
value: 2.2880000000000003 |
|
- type: recall_at_100 |
|
value: 15.853 |
|
- type: recall_at_1000 |
|
value: 54.05 |
|
- type: recall_at_3 |
|
value: 0.72 |
|
- type: recall_at_5 |
|
value: 1.196 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: webis-touche2020 |
|
name: MTEB Touche2020 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.322 |
|
- type: map_at_10 |
|
value: 11.673 |
|
- type: map_at_100 |
|
value: 18.655 |
|
- type: map_at_1000 |
|
value: 20.058999999999997 |
|
- type: map_at_3 |
|
value: 6.265 |
|
- type: map_at_5 |
|
value: 8.549 |
|
- type: mrr_at_1 |
|
value: 42.857 |
|
- type: mrr_at_10 |
|
value: 55.352999999999994 |
|
- type: mrr_at_100 |
|
value: 55.928999999999995 |
|
- type: mrr_at_1000 |
|
value: 55.928999999999995 |
|
- type: mrr_at_3 |
|
value: 50.0 |
|
- type: mrr_at_5 |
|
value: 53.571000000000005 |
|
- type: ndcg_at_1 |
|
value: 39.796 |
|
- type: ndcg_at_10 |
|
value: 28.225 |
|
- type: ndcg_at_100 |
|
value: 40.452 |
|
- type: ndcg_at_1000 |
|
value: 51.332 |
|
- type: ndcg_at_3 |
|
value: 32.308 |
|
- type: ndcg_at_5 |
|
value: 30.942999999999998 |
|
- type: precision_at_1 |
|
value: 42.857 |
|
- type: precision_at_10 |
|
value: 24.490000000000002 |
|
- type: precision_at_100 |
|
value: 8.366999999999999 |
|
- type: precision_at_1000 |
|
value: 1.5709999999999997 |
|
- type: precision_at_3 |
|
value: 32.653 |
|
- type: precision_at_5 |
|
value: 30.203999999999997 |
|
- type: recall_at_1 |
|
value: 3.322 |
|
- type: recall_at_10 |
|
value: 17.857 |
|
- type: recall_at_100 |
|
value: 51.169 |
|
- type: recall_at_1000 |
|
value: 85.382 |
|
- type: recall_at_3 |
|
value: 7.126 |
|
- type: recall_at_5 |
|
value: 11.186 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 72.1046 |
|
- type: ap |
|
value: 14.84774372187047 |
|
- type: f1 |
|
value: 55.52709376912111 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 60.18958687040181 |
|
- type: f1 |
|
value: 60.53154943862625 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 54.61440440799667 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 87.34577099600644 |
|
- type: cos_sim_ap |
|
value: 78.19613471607386 |
|
- type: cos_sim_f1 |
|
value: 71.30501144746884 |
|
- type: cos_sim_precision |
|
value: 68.83595284872298 |
|
- type: cos_sim_recall |
|
value: 73.95778364116094 |
|
- type: dot_accuracy |
|
value: 82.89324670680098 |
|
- type: dot_ap |
|
value: 63.02362697550343 |
|
- type: dot_f1 |
|
value: 59.69837587006961 |
|
- type: dot_precision |
|
value: 53.2712215320911 |
|
- type: dot_recall |
|
value: 67.8891820580475 |
|
- type: euclidean_accuracy |
|
value: 87.24444179531503 |
|
- type: euclidean_ap |
|
value: 78.38356749852895 |
|
- type: euclidean_f1 |
|
value: 71.42133265771471 |
|
- type: euclidean_precision |
|
value: 68.68908382066277 |
|
- type: euclidean_recall |
|
value: 74.37994722955145 |
|
- type: manhattan_accuracy |
|
value: 87.24444179531503 |
|
- type: manhattan_ap |
|
value: 78.27660966609476 |
|
- type: manhattan_f1 |
|
value: 71.42165173165415 |
|
- type: manhattan_precision |
|
value: 66.00268576544315 |
|
- type: manhattan_recall |
|
value: 77.81002638522428 |
|
- type: max_accuracy |
|
value: 87.34577099600644 |
|
- type: max_ap |
|
value: 78.38356749852895 |
|
- type: max_f1 |
|
value: 71.42165173165415 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.90829355377032 |
|
- type: cos_sim_ap |
|
value: 85.79696678631824 |
|
- type: cos_sim_f1 |
|
value: 77.8494623655914 |
|
- type: cos_sim_precision |
|
value: 76.32610786417105 |
|
- type: cos_sim_recall |
|
value: 79.43486295041576 |
|
- type: dot_accuracy |
|
value: 86.17223580548765 |
|
- type: dot_ap |
|
value: 79.05804163697516 |
|
- type: dot_f1 |
|
value: 72.38855622089154 |
|
- type: dot_precision |
|
value: 69.61467368121713 |
|
- type: dot_recall |
|
value: 75.39267015706807 |
|
- type: euclidean_accuracy |
|
value: 88.94128148406877 |
|
- type: euclidean_ap |
|
value: 85.86615739743813 |
|
- type: euclidean_f1 |
|
value: 77.97001153402537 |
|
- type: euclidean_precision |
|
value: 75.44099647202822 |
|
- type: euclidean_recall |
|
value: 80.67446874037573 |
|
- type: manhattan_accuracy |
|
value: 88.9781503473435 |
|
- type: manhattan_ap |
|
value: 85.91093266751166 |
|
- type: manhattan_f1 |
|
value: 77.96835723791216 |
|
- type: manhattan_precision |
|
value: 74.98577929465301 |
|
- type: manhattan_recall |
|
value: 81.19802894979982 |
|
- type: max_accuracy |
|
value: 88.9781503473435 |
|
- type: max_ap |
|
value: 85.91093266751166 |
|
- type: max_f1 |
|
value: 77.97001153402537 |
|
--- |
|
|
|
# Sionic AI Embedding API v2 |
|
|
|
## About Sionic AI |
|
|
|
Sionic AI delivers more accessible and cost-effective AI technology addressing the various needs to boost productivity and drive innovation. |
|
|
|
The Large Language Model (LLM) is not for research and experimentation. |
|
We offer solutions that leverage LLM to add value to your business. |
|
Anyone can easily train and control AI. |
|
|
|
## How to get embeddings |
|
|
|
Currently, we open the beta version of embedding APIs. |
|
To get embeddings, you should call API endpoint to send your text. |
|
You can send either a single sentence or multiple sentences. |
|
The embeddings that correspond to the inputs will be returned. |
|
|
|
API Endpoint : https://api.sionic.ai/v2/embedding |
|
|
|
### Command line Example |
|
Request: |
|
```shell |
|
curl https://api.sionic.ai/v2/embedding \ |
|
-H "Content-Type: application/json" \ |
|
-d '{ |
|
"inputs": ["first query", "second query", "third query"] |
|
}' |
|
``` |
|
|
|
Response: |
|
```shell |
|
{ |
|
"embedding": [ |
|
[ |
|
0.5567971, |
|
-1.1578958, |
|
-0.7148851, |
|
-0.2326297, |
|
0.4394867, |
|
... |
|
], |
|
[ |
|
0.5049863, |
|
-0.8253384, |
|
-1.0041373, |
|
-0.6503708, |
|
0.5007141, |
|
... |
|
], |
|
[ |
|
0.6059823, |
|
-1.0369557, |
|
-0.6705063, |
|
-0.4467056, |
|
0.8618057, |
|
... |
|
] |
|
] |
|
} |
|
``` |
|
|
|
### Python code Example |
|
Get embeddings by directly calling embedding API. |
|
|
|
```python |
|
from typing import List |
|
import numpy as np |
|
import requests |
|
|
|
def get_embedding(queries: List[str], url): |
|
response = requests.post(url=url, json={'inputs': queries}) |
|
return np.asarray(response.json()['embedding'], dtype=np.float32) |
|
|
|
url = "https://api.sionic.ai/v2/embedding" |
|
inputs1 = ["first query", "second query"] |
|
inputs2 = ["third query", "fourth query"] |
|
embedding1 = get_embedding(inputs1, url=url) |
|
embedding2 = get_embedding(inputs2, url=url) |
|
cos_similarity = (embedding1 / np.linalg.norm(embedding1)) @ (embedding2 / np.linalg.norm(embedding1)).T |
|
print(cos_similarity) |
|
``` |
|
|
|
Using pre-defined [SionicEmbeddingModel](https://huggingface.co/sionic-ai/sionic-ai-v2/blob/main/model_api.py) to obtain embeddings. |
|
|
|
```python |
|
from model_api import SionicEmbeddingModel |
|
import numpy as np |
|
|
|
inputs1 = ["first query", "second query"] |
|
inputs2 = ["third query", "fourth query"] |
|
model = SionicEmbeddingModel(url="https://api.sionic.ai/v2/embedding", |
|
dimension=3072) |
|
embedding1 = model.encode(inputs1) |
|
embedding2 = model.encode(inputs2) |
|
cos_similarity = (embedding1 / np.linalg.norm(embedding1)) @ (embedding2 / np.linalg.norm(embedding1)).T |
|
print(cos_similarity) |
|
``` |
|
We apply the instruction to encode short queries for retrieval tasks. |
|
By using `encode_queries()`, you can use the instruction to encode queries which is prefixed to each query as the following example. |
|
The recommended instruction for both v1 and v2 models is `"query: "`. |
|
|
|
```python |
|
from model_api import SionicEmbeddingModel |
|
import numpy as np |
|
|
|
query = ["first query", "second query"] |
|
passage = ["This is a passage related to the first query", "This is a passage related to the second query"] |
|
model = SionicEmbeddingModel(url="https://api.sionic.ai/v2/embedding", |
|
instruction="query: ", |
|
dimension=3072) |
|
query_embedding = model.encode_queries(query) |
|
passage_embedding = model.encode_corpus(passage) |
|
cos_similarity = (query_embedding / np.linalg.norm(query_embedding)) @ (passage_embedding / np.linalg.norm(passage_embedding)).T |
|
print(cos_similarity) |
|
``` |
|
|
|
## Massive Text Embedding Benchmark (MTEB) Evaluation |
|
|
|
Both versions of Sionic AI's embedding show the state-of-the-art performances on the MTEB! |
|
You can find a code to evaluate MTEB datasets using Sionic embedding APIs [here](https://huggingface.co/sionic-ai/sionic-ai-v2/blob/main/mteb_evaluate.py). |
|
|
|
| Model Name | Dimension | Sequence Length | Average (56) | |
|
|:-----------------------------------------------------------------------:|:---------:|:---------------:|:------------:| |
|
| [sionic-ai/sionic-ai-v2](https://huggingface.co/sionic-ai/sionic-ai-v2) | 3072 | 512 | **65.23** | |
|
| [sionic-ai/sionic-ai-v1](https://huggingface.co/sionic-ai/sionic-ai-v1) | 2048 | 512 | 64.92 | |
|
| [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | 1024 | 512 | 64.23 | |
|
| [gte-large-en](https://huggingface.co/barisaydin/gte-large) | 1024 | 512 | 63.13 | |
|
| [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings/types-of-embedding-models) | 1536 | 8191 | 60.99 | |
|
|
|
|
