metadata
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
model-index:
- name: SGPT-5.8B-weightedmean-nli-bitfit
results:
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en)
config: en
split: test
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
metrics:
- type: accuracy
value: 74.07462686567165
- type: ap
value: 37.44692407529112
- type: f1
value: 68.28971003916419
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (de)
config: de
split: test
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
metrics:
- type: accuracy
value: 66.63811563169165
- type: ap
value: 78.57252079915924
- type: f1
value: 64.5543087846584
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en-ext)
config: en-ext
split: test
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
metrics:
- type: accuracy
value: 77.21889055472263
- type: ap
value: 25.663426367826712
- type: f1
value: 64.26265688503176
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (ja)
config: ja
split: test
revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
metrics:
- type: accuracy
value: 58.06209850107067
- type: ap
value: 14.028219107023915
- type: f1
value: 48.10387189660778
- task:
type: Classification
dataset:
type: mteb/amazon_polarity
name: MTEB AmazonPolarityClassification
config: default
split: test
revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1
metrics:
- type: accuracy
value: 82.30920000000002
- type: ap
value: 76.88786578621213
- type: f1
value: 82.15455656065011
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (en)
config: en
split: test
revision: c379a6705fec24a2493fa68e011692605f44e119
metrics:
- type: accuracy
value: 41.584
- type: f1
value: 41.203137944390114
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (de)
config: de
split: test
revision: c379a6705fec24a2493fa68e011692605f44e119
metrics:
- type: accuracy
value: 35.288000000000004
- type: f1
value: 34.672995558518096
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (es)
config: es
split: test
revision: c379a6705fec24a2493fa68e011692605f44e119
metrics:
- type: accuracy
value: 38.34
- type: f1
value: 37.608755629529455
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (fr)
config: fr
split: test
revision: c379a6705fec24a2493fa68e011692605f44e119
metrics:
- type: accuracy
value: 37.839999999999996
- type: f1
value: 36.86898201563507
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (ja)
config: ja
split: test
revision: c379a6705fec24a2493fa68e011692605f44e119
metrics:
- type: accuracy
value: 30.936000000000003
- type: f1
value: 30.49401738527071
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (zh)
config: zh
split: test
revision: c379a6705fec24a2493fa68e011692605f44e119
metrics:
- type: accuracy
value: 33.75
- type: f1
value: 33.38338946025617
- task:
type: Retrieval
dataset:
type: arguana
name: MTEB ArguAna
config: default
split: test
revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3
metrics:
- type: map_at_1
value: 13.727
- type: map_at_10
value: 26.740000000000002
- type: map_at_100
value: 28.218
- type: map_at_1000
value: 28.246
- type: map_at_3
value: 21.728
- type: map_at_5
value: 24.371000000000002
- type: ndcg_at_1
value: 13.727
- type: ndcg_at_10
value: 35.07
- type: ndcg_at_100
value: 41.947
- type: ndcg_at_1000
value: 42.649
- type: ndcg_at_3
value: 24.484
- type: ndcg_at_5
value: 29.282999999999998
- type: precision_at_1
value: 13.727
- type: precision_at_10
value: 6.223
- type: precision_at_100
value: 0.9369999999999999
- type: precision_at_1000
value: 0.099
- type: precision_at_3
value: 10.835
- type: precision_at_5
value: 8.848
- type: recall_at_1
value: 13.727
- type: recall_at_10
value: 62.233000000000004
- type: recall_at_100
value: 93.67
- type: recall_at_1000
value: 99.14699999999999
- type: recall_at_3
value: 32.504
- type: recall_at_5
value: 44.239
- task:
type: Clustering
dataset:
type: mteb/arxiv-clustering-p2p
name: MTEB ArxivClusteringP2P
config: default
split: test
revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8
metrics:
- type: v_measure
value: 40.553923271901695
- task:
type: Clustering
dataset:
type: mteb/arxiv-clustering-s2s
name: MTEB ArxivClusteringS2S
config: default
split: test
revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3
metrics:
- type: v_measure
value: 32.49323183712211
- task:
type: Reranking
dataset:
type: mteb/askubuntudupquestions-reranking
name: MTEB AskUbuntuDupQuestions
config: default
split: test
revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c
metrics:
- type: map
value: 55.89811361443445
- type: mrr
value: 70.16235764850724
- task:
type: STS
dataset:
type: mteb/biosses-sts
name: MTEB BIOSSES
config: default
split: test
revision: 9ee918f184421b6bd48b78f6c714d86546106103
metrics:
- type: cos_sim_pearson
value: 82.50506557805856
- type: cos_sim_spearman
value: 79.50000423261176
- type: euclidean_pearson
value: 75.76190885392926
- type: euclidean_spearman
value: 76.7330737163434
- type: manhattan_pearson
value: 75.825318036112
- type: manhattan_spearman
value: 76.7415076434559
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (de-en)
config: de-en
split: test
revision: d51519689f32196a32af33b075a01d0e7c51e252
metrics:
- type: accuracy
value: 75.49060542797494
- type: f1
value: 75.15379262352123
- type: precision
value: 74.99391092553932
- type: recall
value: 75.49060542797494
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (fr-en)
config: fr-en
split: test
revision: d51519689f32196a32af33b075a01d0e7c51e252
metrics:
- type: accuracy
value: 0.4182258419546555
- type: f1
value: 0.4182258419546555
- type: precision
value: 0.4182258419546555
- type: recall
value: 0.4182258419546555
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (ru-en)
config: ru-en
split: test
revision: d51519689f32196a32af33b075a01d0e7c51e252
metrics:
- type: accuracy
value: 0.013855213023900243
- type: f1
value: 0.0115460108532502
- type: precision
value: 0.010391409767925183
- type: recall
value: 0.013855213023900243
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (zh-en)
config: zh-en
split: test
revision: d51519689f32196a32af33b075a01d0e7c51e252
metrics:
- type: accuracy
value: 0.315955766192733
- type: f1
value: 0.315955766192733
- type: precision
value: 0.315955766192733
- type: recall
value: 0.315955766192733
- task:
type: Classification
dataset:
type: mteb/banking77
name: MTEB Banking77Classification
config: default
split: test
revision: 44fa15921b4c889113cc5df03dd4901b49161ab7
metrics:
- type: accuracy
value: 81.74025974025973
- type: f1
value: 81.66568824876
- task:
type: Clustering
dataset:
type: mteb/biorxiv-clustering-p2p
name: MTEB BiorxivClusteringP2P
config: default
split: test
revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55
metrics:
- type: v_measure
value: 33.59451202614059
- task:
type: Clustering
dataset:
type: mteb/biorxiv-clustering-s2s
name: MTEB BiorxivClusteringS2S
config: default
split: test
revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1
metrics:
- type: v_measure
value: 29.128241446157165
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackAndroidRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 26.715
- type: map_at_10
value: 35.007
- type: map_at_100
value: 36.352000000000004
- type: map_at_1000
value: 36.51
- type: map_at_3
value: 32.257999999999996
- type: map_at_5
value: 33.595000000000006
- type: ndcg_at_1
value: 33.906
- type: ndcg_at_10
value: 40.353
- type: ndcg_at_100
value: 45.562999999999995
- type: ndcg_at_1000
value: 48.454
- type: ndcg_at_3
value: 36.349
- type: ndcg_at_5
value: 37.856
- type: precision_at_1
value: 33.906
- type: precision_at_10
value: 7.854
- type: precision_at_100
value: 1.29
- type: precision_at_1000
value: 0.188
- type: precision_at_3
value: 17.549
- type: precision_at_5
value: 12.561
- type: recall_at_1
value: 26.715
- type: recall_at_10
value: 49.508
- type: recall_at_100
value: 71.76599999999999
- type: recall_at_1000
value: 91.118
- type: recall_at_3
value: 37.356
- type: recall_at_5
value: 41.836
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackEnglishRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 19.663
- type: map_at_10
value: 27.086
- type: map_at_100
value: 28.066999999999997
- type: map_at_1000
value: 28.18
- type: map_at_3
value: 24.819
- type: map_at_5
value: 26.332
- type: ndcg_at_1
value: 25.732
- type: ndcg_at_10
value: 31.613999999999997
- type: ndcg_at_100
value: 35.757
- type: ndcg_at_1000
value: 38.21
- type: ndcg_at_3
value: 28.332
- type: ndcg_at_5
value: 30.264000000000003
- type: precision_at_1
value: 25.732
- type: precision_at_10
value: 6.038
- type: precision_at_100
value: 1.034
- type: precision_at_1000
value: 0.149
- type: precision_at_3
value: 13.864
- type: precision_at_5
value: 10.241999999999999
- type: recall_at_1
value: 19.663
- type: recall_at_10
value: 39.585
- type: recall_at_100
value: 57.718
- type: recall_at_1000
value: 74.26700000000001
- type: recall_at_3
value: 29.845
- type: recall_at_5
value: 35.105
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackGamingRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 30.125
- type: map_at_10
value: 39.824
- type: map_at_100
value: 40.935
- type: map_at_1000
value: 41.019
- type: map_at_3
value: 37.144
- type: map_at_5
value: 38.647999999999996
- type: ndcg_at_1
value: 34.922
- type: ndcg_at_10
value: 45.072
- type: ndcg_at_100
value: 50.046
- type: ndcg_at_1000
value: 51.895
- type: ndcg_at_3
value: 40.251
- type: ndcg_at_5
value: 42.581
- type: precision_at_1
value: 34.922
- type: precision_at_10
value: 7.303999999999999
- type: precision_at_100
value: 1.0739999999999998
- type: precision_at_1000
value: 0.13
- type: precision_at_3
value: 17.994
- type: precision_at_5
value: 12.475999999999999
- type: recall_at_1
value: 30.125
- type: recall_at_10
value: 57.253
- type: recall_at_100
value: 79.35799999999999
- type: recall_at_1000
value: 92.523
- type: recall_at_3
value: 44.088
- type: recall_at_5
value: 49.893
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackGisRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 16.298000000000002
- type: map_at_10
value: 21.479
- type: map_at_100
value: 22.387
- type: map_at_1000
value: 22.483
- type: map_at_3
value: 19.743
- type: map_at_5
value: 20.444000000000003
- type: ndcg_at_1
value: 17.740000000000002
- type: ndcg_at_10
value: 24.887
- type: ndcg_at_100
value: 29.544999999999998
- type: ndcg_at_1000
value: 32.417
- type: ndcg_at_3
value: 21.274
- type: ndcg_at_5
value: 22.399
- type: precision_at_1
value: 17.740000000000002
- type: precision_at_10
value: 3.932
- type: precision_at_100
value: 0.666
- type: precision_at_1000
value: 0.094
- type: precision_at_3
value: 8.927
- type: precision_at_5
value: 6.056
- type: recall_at_1
value: 16.298000000000002
- type: recall_at_10
value: 34.031
- type: recall_at_100
value: 55.769000000000005
- type: recall_at_1000
value: 78.19500000000001
- type: recall_at_3
value: 23.799999999999997
- type: recall_at_5
value: 26.562
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackMathematicaRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 10.958
- type: map_at_10
value: 16.999
- type: map_at_100
value: 17.979
- type: map_at_1000
value: 18.112000000000002
- type: map_at_3
value: 15.010000000000002
- type: map_at_5
value: 16.256999999999998
- type: ndcg_at_1
value: 14.179
- type: ndcg_at_10
value: 20.985
- type: ndcg_at_100
value: 26.216
- type: ndcg_at_1000
value: 29.675
- type: ndcg_at_3
value: 17.28
- type: ndcg_at_5
value: 19.301
- type: precision_at_1
value: 14.179
- type: precision_at_10
value: 3.968
- type: precision_at_100
value: 0.784
- type: precision_at_1000
value: 0.121
- type: precision_at_3
value: 8.541
- type: precision_at_5
value: 6.468
- type: recall_at_1
value: 10.958
- type: recall_at_10
value: 29.903000000000002
- type: recall_at_100
value: 53.413
- type: recall_at_1000
value: 78.74799999999999
- type: recall_at_3
value: 19.717000000000002
- type: recall_at_5
value: 24.817
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackPhysicsRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 21.217
- type: map_at_10
value: 29.677
- type: map_at_100
value: 30.928
- type: map_at_1000
value: 31.063000000000002
- type: map_at_3
value: 26.611
- type: map_at_5
value: 28.463
- type: ndcg_at_1
value: 26.083000000000002
- type: ndcg_at_10
value: 35.217
- type: ndcg_at_100
value: 40.715
- type: ndcg_at_1000
value: 43.559
- type: ndcg_at_3
value: 30.080000000000002
- type: ndcg_at_5
value: 32.701
- type: precision_at_1
value: 26.083000000000002
- type: precision_at_10
value: 6.622
- type: precision_at_100
value: 1.115
- type: precision_at_1000
value: 0.156
- type: precision_at_3
value: 14.629
- type: precision_at_5
value: 10.837
- type: recall_at_1
value: 21.217
- type: recall_at_10
value: 47.031
- type: recall_at_100
value: 70.378
- type: recall_at_1000
value: 89.704
- type: recall_at_3
value: 32.427
- type: recall_at_5
value: 39.31
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackProgrammersRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 19.274
- type: map_at_10
value: 26.398
- type: map_at_100
value: 27.711000000000002
- type: map_at_1000
value: 27.833000000000002
- type: map_at_3
value: 24.294
- type: map_at_5
value: 25.385
- type: ndcg_at_1
value: 24.886
- type: ndcg_at_10
value: 30.909
- type: ndcg_at_100
value: 36.941
- type: ndcg_at_1000
value: 39.838
- type: ndcg_at_3
value: 27.455000000000002
- type: ndcg_at_5
value: 28.828
- type: precision_at_1
value: 24.886
- type: precision_at_10
value: 5.6739999999999995
- type: precision_at_100
value: 1.0290000000000001
- type: precision_at_1000
value: 0.146
- type: precision_at_3
value: 13.242
- type: precision_at_5
value: 9.292
- type: recall_at_1
value: 19.274
- type: recall_at_10
value: 39.643
- type: recall_at_100
value: 66.091
- type: recall_at_1000
value: 86.547
- type: recall_at_3
value: 29.602
- type: recall_at_5
value: 33.561
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 18.653666666666666
- type: map_at_10
value: 25.606666666666666
- type: map_at_100
value: 26.669333333333334
- type: map_at_1000
value: 26.795833333333334
- type: map_at_3
value: 23.43433333333333
- type: map_at_5
value: 24.609666666666666
- type: ndcg_at_1
value: 22.742083333333333
- type: ndcg_at_10
value: 29.978333333333335
- type: ndcg_at_100
value: 34.89808333333333
- type: ndcg_at_1000
value: 37.806583333333336
- type: ndcg_at_3
value: 26.223666666666674
- type: ndcg_at_5
value: 27.91033333333333
- type: precision_at_1
value: 22.742083333333333
- type: precision_at_10
value: 5.397083333333334
- type: precision_at_100
value: 0.9340000000000002
- type: precision_at_1000
value: 0.13691666666666663
- type: precision_at_3
value: 12.331083333333332
- type: precision_at_5
value: 8.805499999999999
- type: recall_at_1
value: 18.653666666666666
- type: recall_at_10
value: 39.22625000000001
- type: recall_at_100
value: 61.31049999999999
- type: recall_at_1000
value: 82.19058333333334
- type: recall_at_3
value: 28.517333333333333
- type: recall_at_5
value: 32.9565
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackStatsRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 16.07
- type: map_at_10
value: 21.509
- type: map_at_100
value: 22.335
- type: map_at_1000
value: 22.437
- type: map_at_3
value: 19.717000000000002
- type: map_at_5
value: 20.574
- type: ndcg_at_1
value: 18.865000000000002
- type: ndcg_at_10
value: 25.135999999999996
- type: ndcg_at_100
value: 29.483999999999998
- type: ndcg_at_1000
value: 32.303
- type: ndcg_at_3
value: 21.719
- type: ndcg_at_5
value: 23.039
- type: precision_at_1
value: 18.865000000000002
- type: precision_at_10
value: 4.263999999999999
- type: precision_at_100
value: 0.696
- type: precision_at_1000
value: 0.1
- type: precision_at_3
value: 9.866999999999999
- type: precision_at_5
value: 6.902
- type: recall_at_1
value: 16.07
- type: recall_at_10
value: 33.661
- type: recall_at_100
value: 54.001999999999995
- type: recall_at_1000
value: 75.564
- type: recall_at_3
value: 23.956
- type: recall_at_5
value: 27.264
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackTexRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 10.847
- type: map_at_10
value: 15.518
- type: map_at_100
value: 16.384
- type: map_at_1000
value: 16.506
- type: map_at_3
value: 14.093
- type: map_at_5
value: 14.868
- type: ndcg_at_1
value: 13.764999999999999
- type: ndcg_at_10
value: 18.766
- type: ndcg_at_100
value: 23.076
- type: ndcg_at_1000
value: 26.344
- type: ndcg_at_3
value: 16.150000000000002
- type: ndcg_at_5
value: 17.373
- type: precision_at_1
value: 13.764999999999999
- type: precision_at_10
value: 3.572
- type: precision_at_100
value: 0.6779999999999999
- type: precision_at_1000
value: 0.11199999999999999
- type: precision_at_3
value: 7.88
- type: precision_at_5
value: 5.712
- type: recall_at_1
value: 10.847
- type: recall_at_10
value: 25.141999999999996
- type: recall_at_100
value: 44.847
- type: recall_at_1000
value: 68.92099999999999
- type: recall_at_3
value: 17.721999999999998
- type: recall_at_5
value: 20.968999999999998
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackUnixRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 18.377
- type: map_at_10
value: 26.005
- type: map_at_100
value: 26.996
- type: map_at_1000
value: 27.116
- type: map_at_3
value: 23.712
- type: map_at_5
value: 24.859
- type: ndcg_at_1
value: 22.201
- type: ndcg_at_10
value: 30.635
- type: ndcg_at_100
value: 35.623
- type: ndcg_at_1000
value: 38.551
- type: ndcg_at_3
value: 26.565
- type: ndcg_at_5
value: 28.28
- type: precision_at_1
value: 22.201
- type: precision_at_10
value: 5.41
- type: precision_at_100
value: 0.88
- type: precision_at_1000
value: 0.125
- type: precision_at_3
value: 12.531
- type: precision_at_5
value: 8.806
- type: recall_at_1
value: 18.377
- type: recall_at_10
value: 40.908
- type: recall_at_100
value: 63.563
- type: recall_at_1000
value: 84.503
- type: recall_at_3
value: 29.793999999999997
- type: recall_at_5
value: 34.144999999999996
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackWebmastersRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 20.246
- type: map_at_10
value: 27.528000000000002
- type: map_at_100
value: 28.78
- type: map_at_1000
value: 29.002
- type: map_at_3
value: 25.226
- type: map_at_5
value: 26.355
- type: ndcg_at_1
value: 25.099
- type: ndcg_at_10
value: 32.421
- type: ndcg_at_100
value: 37.2
- type: ndcg_at_1000
value: 40.693
- type: ndcg_at_3
value: 28.768
- type: ndcg_at_5
value: 30.23
- type: precision_at_1
value: 25.099
- type: precision_at_10
value: 6.245
- type: precision_at_100
value: 1.269
- type: precision_at_1000
value: 0.218
- type: precision_at_3
value: 13.767999999999999
- type: precision_at_5
value: 9.881
- type: recall_at_1
value: 20.246
- type: recall_at_10
value: 41.336
- type: recall_at_100
value: 63.098
- type: recall_at_1000
value: 86.473
- type: recall_at_3
value: 30.069000000000003
- type: recall_at_5
value: 34.262
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackWordpressRetrieval
config: default
split: test
revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
metrics:
- type: map_at_1
value: 14.054
- type: map_at_10
value: 20.25
- type: map_at_100
value: 21.178
- type: map_at_1000
value: 21.288999999999998
- type: map_at_3
value: 18.584999999999997
- type: map_at_5
value: 19.536
- type: ndcg_at_1
value: 15.527
- type: ndcg_at_10
value: 23.745
- type: ndcg_at_100
value: 28.610999999999997
- type: ndcg_at_1000
value: 31.740000000000002
- type: ndcg_at_3
value: 20.461
- type: ndcg_at_5
value: 22.072
- type: precision_at_1
value: 15.527
- type: precision_at_10
value: 3.882
- type: precision_at_100
value: 0.6930000000000001
- type: precision_at_1000
value: 0.104
- type: precision_at_3
value: 9.181000000000001
- type: precision_at_5
value: 6.433
- type: recall_at_1
value: 14.054
- type: recall_at_10
value: 32.714
- type: recall_at_100
value: 55.723
- type: recall_at_1000
value: 79.72399999999999
- type: recall_at_3
value: 23.832
- type: recall_at_5
value: 27.754
- task:
type: Retrieval
dataset:
type: climate-fever
name: MTEB ClimateFEVER
config: default
split: test
revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce
metrics:
- type: map_at_1
value: 6.122
- type: map_at_10
value: 11.556
- type: map_at_100
value: 12.998000000000001
- type: map_at_1000
value: 13.202
- type: map_at_3
value: 9.657
- type: map_at_5
value: 10.585
- type: ndcg_at_1
value: 15.049000000000001
- type: ndcg_at_10
value: 17.574
- type: ndcg_at_100
value: 24.465999999999998
- type: ndcg_at_1000
value: 28.511999999999997
- type: ndcg_at_3
value: 13.931
- type: ndcg_at_5
value: 15.112
- type: precision_at_1
value: 15.049000000000001
- type: precision_at_10
value: 5.831
- type: precision_at_100
value: 1.322
- type: precision_at_1000
value: 0.20500000000000002
- type: precision_at_3
value: 10.749
- type: precision_at_5
value: 8.365
- type: recall_at_1
value: 6.122
- type: recall_at_10
value: 22.207
- type: recall_at_100
value: 47.08
- type: recall_at_1000
value: 70.182
- type: recall_at_3
value: 13.416
- type: recall_at_5
value: 16.672
- task:
type: Retrieval
dataset:
type: dbpedia-entity
name: MTEB DBPedia
config: default
split: test
revision: f097057d03ed98220bc7309ddb10b71a54d667d6
metrics:
- type: map_at_1
value: 4.672
- type: map_at_10
value: 10.534
- type: map_at_100
value: 14.798
- type: map_at_1000
value: 15.927
- type: map_at_3
value: 7.317
- type: map_at_5
value: 8.726
- type: ndcg_at_1
value: 36.5
- type: ndcg_at_10
value: 26.098
- type: ndcg_at_100
value: 29.215999999999998
- type: ndcg_at_1000
value: 36.254999999999995
- type: ndcg_at_3
value: 29.247
- type: ndcg_at_5
value: 27.692
- type: precision_at_1
value: 47.25
- type: precision_at_10
value: 22.625
- type: precision_at_100
value: 7.042
- type: precision_at_1000
value: 1.6129999999999998
- type: precision_at_3
value: 34.083000000000006
- type: precision_at_5
value: 29.5
- type: recall_at_1
value: 4.672
- type: recall_at_10
value: 15.638
- type: recall_at_100
value: 36.228
- type: recall_at_1000
value: 58.831
- type: recall_at_3
value: 8.578
- type: recall_at_5
value: 11.18
- task:
type: Classification
dataset:
type: mteb/emotion
name: MTEB EmotionClassification
config: default
split: test
revision: 829147f8f75a25f005913200eb5ed41fae320aa1
metrics:
- type: accuracy
value: 49.919999999999995
- type: f1
value: 45.37973678791632
- task:
type: Retrieval
dataset:
type: fever
name: MTEB FEVER
config: default
split: test
revision: 1429cf27e393599b8b359b9b72c666f96b2525f9
metrics:
- type: map_at_1
value: 25.801000000000002
- type: map_at_10
value: 33.941
- type: map_at_100
value: 34.73
- type: map_at_1000
value: 34.793
- type: map_at_3
value: 31.705
- type: map_at_5
value: 33.047
- type: ndcg_at_1
value: 27.933000000000003
- type: ndcg_at_10
value: 38.644
- type: ndcg_at_100
value: 42.594
- type: ndcg_at_1000
value: 44.352000000000004
- type: ndcg_at_3
value: 34.199
- type: ndcg_at_5
value: 36.573
- type: precision_at_1
value: 27.933000000000003
- type: precision_at_10
value: 5.603000000000001
- type: precision_at_100
value: 0.773
- type: precision_at_1000
value: 0.094
- type: precision_at_3
value: 14.171
- type: precision_at_5
value: 9.786999999999999
- type: recall_at_1
value: 25.801000000000002
- type: recall_at_10
value: 50.876
- type: recall_at_100
value: 69.253
- type: recall_at_1000
value: 82.907
- type: recall_at_3
value: 38.879000000000005
- type: recall_at_5
value: 44.651999999999994
- task:
type: Retrieval
dataset:
type: fiqa
name: MTEB FiQA2018
config: default
split: test
revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be
metrics:
- type: map_at_1
value: 9.142
- type: map_at_10
value: 13.841999999999999
- type: map_at_100
value: 14.960999999999999
- type: map_at_1000
value: 15.187000000000001
- type: map_at_3
value: 11.966000000000001
- type: map_at_5
value: 12.921
- type: ndcg_at_1
value: 18.364
- type: ndcg_at_10
value: 18.590999999999998
- type: ndcg_at_100
value: 24.153
- type: ndcg_at_1000
value: 29.104000000000003
- type: ndcg_at_3
value: 16.323
- type: ndcg_at_5
value: 17.000999999999998
- type: precision_at_1
value: 18.364
- type: precision_at_10
value: 5.216
- type: precision_at_100
value: 1.09
- type: precision_at_1000
value: 0.193
- type: precision_at_3
value: 10.751
- type: precision_at_5
value: 7.932
- type: recall_at_1
value: 9.142
- type: recall_at_10
value: 22.747
- type: recall_at_100
value: 44.585
- type: recall_at_1000
value: 75.481
- type: recall_at_3
value: 14.602
- type: recall_at_5
value: 17.957
- task:
type: Retrieval
dataset:
type: hotpotqa
name: MTEB HotpotQA
config: default
split: test
revision: 766870b35a1b9ca65e67a0d1913899973551fc6c
metrics:
- type: map_at_1
value: 18.677
- type: map_at_10
value: 26.616
- type: map_at_100
value: 27.605
- type: map_at_1000
value: 27.711999999999996
- type: map_at_3
value: 24.396
- type: map_at_5
value: 25.627
- type: ndcg_at_1
value: 37.352999999999994
- type: ndcg_at_10
value: 33.995
- type: ndcg_at_100
value: 38.423
- type: ndcg_at_1000
value: 40.947
- type: ndcg_at_3
value: 29.885
- type: ndcg_at_5
value: 31.874999999999996
- type: precision_at_1
value: 37.352999999999994
- type: precision_at_10
value: 7.539999999999999
- type: precision_at_100
value: 1.107
- type: precision_at_1000
value: 0.145
- type: precision_at_3
value: 18.938
- type: precision_at_5
value: 12.943
- type: recall_at_1
value: 18.677
- type: recall_at_10
value: 37.698
- type: recall_at_100
value: 55.354000000000006
- type: recall_at_1000
value: 72.255
- type: recall_at_3
value: 28.406
- type: recall_at_5
value: 32.357
- task:
type: Classification
dataset:
type: mteb/imdb
name: MTEB ImdbClassification
config: default
split: test
revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4
metrics:
- type: accuracy
value: 74.3292
- type: ap
value: 68.30186110189658
- type: f1
value: 74.20709636944783
- task:
type: Retrieval
dataset:
type: msmarco
name: MTEB MSMARCO
config: default
split: validation
revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849
metrics:
- type: map_at_1
value: 6.889000000000001
- type: map_at_10
value: 12.321
- type: map_at_100
value: 13.416
- type: map_at_1000
value: 13.525
- type: map_at_3
value: 10.205
- type: map_at_5
value: 11.342
- type: ndcg_at_1
value: 7.092
- type: ndcg_at_10
value: 15.827
- type: ndcg_at_100
value: 21.72
- type: ndcg_at_1000
value: 24.836
- type: ndcg_at_3
value: 11.393
- type: ndcg_at_5
value: 13.462
- type: precision_at_1
value: 7.092
- type: precision_at_10
value: 2.7969999999999997
- type: precision_at_100
value: 0.583
- type: precision_at_1000
value: 0.08499999999999999
- type: precision_at_3
value: 5.019
- type: precision_at_5
value: 4.06
- type: recall_at_1
value: 6.889000000000001
- type: recall_at_10
value: 26.791999999999998
- type: recall_at_100
value: 55.371
- type: recall_at_1000
value: 80.12899999999999
- type: recall_at_3
value: 14.573
- type: recall_at_5
value: 19.557
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (en)
config: en
split: test
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
metrics:
- type: accuracy
value: 89.6374829001368
- type: f1
value: 89.20878379358307
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (de)
config: de
split: test
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
metrics:
- type: accuracy
value: 84.54212454212454
- type: f1
value: 82.81080100037023
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (es)
config: es
split: test
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
metrics:
- type: accuracy
value: 86.46430953969313
- type: f1
value: 86.00019824223267
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (fr)
config: fr
split: test
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
metrics:
- type: accuracy
value: 81.31850923896022
- type: f1
value: 81.07860454762863
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (hi)
config: hi
split: test
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
metrics:
- type: accuracy
value: 58.23234134098243
- type: f1
value: 56.63845098081841
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (th)
config: th
split: test
revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
metrics:
- type: accuracy
value: 72.28571428571429
- type: f1
value: 70.95796714592039
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (en)
config: en
split: test
revision: 6299947a7777084cc2d4b64235bf7190381ce755
metrics:
- type: accuracy
value: 70.68171454628363
- type: f1
value: 52.57188062729139
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (de)
config: de
split: test
revision: 6299947a7777084cc2d4b64235bf7190381ce755
metrics:
- type: accuracy
value: 60.521273598196665
- type: f1
value: 42.70492970339204
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (es)
config: es
split: test
revision: 6299947a7777084cc2d4b64235bf7190381ce755
metrics:
- type: accuracy
value: 64.32288192128087
- type: f1
value: 45.97360620220273
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (fr)
config: fr
split: test
revision: 6299947a7777084cc2d4b64235bf7190381ce755
metrics:
- type: accuracy
value: 58.67209520826808
- type: f1
value: 42.82844991304579
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (hi)
config: hi
split: test
revision: 6299947a7777084cc2d4b64235bf7190381ce755
metrics:
- type: accuracy
value: 41.95769092864826
- type: f1
value: 28.914127631431263
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (th)
config: th
split: test
revision: 6299947a7777084cc2d4b64235bf7190381ce755
metrics:
- type: accuracy
value: 55.28390596745027
- type: f1
value: 38.33899250561289
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (en)
config: en
split: test
revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
metrics:
- type: accuracy
value: 70.00336247478144
- type: f1
value: 68.72041942191649
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (en)
config: en
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 75.0268997982515
- type: f1
value: 75.29844481506652
- task:
type: Clustering
dataset:
type: mteb/medrxiv-clustering-p2p
name: MTEB MedrxivClusteringP2P
config: default
split: test
revision: dcefc037ef84348e49b0d29109e891c01067226b
metrics:
- type: v_measure
value: 30.327566856300813
- task:
type: Clustering
dataset:
type: mteb/medrxiv-clustering-s2s
name: MTEB MedrxivClusteringS2S
config: default
split: test
revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc
metrics:
- type: v_measure
value: 28.01650210863619
- task:
type: Reranking
dataset:
type: mteb/mind_small
name: MTEB MindSmallReranking
config: default
split: test
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
metrics:
- type: map
value: 31.11041256752524
- type: mrr
value: 32.14172939750204
- task:
type: Retrieval
dataset:
type: nfcorpus
name: MTEB NFCorpus
config: default
split: test
revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610
metrics:
- type: map_at_1
value: 3.527
- type: map_at_10
value: 9.283
- type: map_at_100
value: 11.995000000000001
- type: map_at_1000
value: 13.33
- type: map_at_3
value: 6.223
- type: map_at_5
value: 7.68
- type: ndcg_at_1
value: 36.223
- type: ndcg_at_10
value: 28.255999999999997
- type: ndcg_at_100
value: 26.355
- type: ndcg_at_1000
value: 35.536
- type: ndcg_at_3
value: 31.962000000000003
- type: ndcg_at_5
value: 30.61
- type: precision_at_1
value: 37.771
- type: precision_at_10
value: 21.889
- type: precision_at_100
value: 7.1080000000000005
- type: precision_at_1000
value: 1.989
- type: precision_at_3
value: 30.857
- type: precision_at_5
value: 27.307
- type: recall_at_1
value: 3.527
- type: recall_at_10
value: 14.015
- type: recall_at_100
value: 28.402
- type: recall_at_1000
value: 59.795
- type: recall_at_3
value: 7.5969999999999995
- type: recall_at_5
value: 10.641
- task:
type: Retrieval
dataset:
type: nq
name: MTEB NQ
config: default
split: test
revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c
metrics:
- type: map_at_1
value: 11.631
- type: map_at_10
value: 19.532
- type: map_at_100
value: 20.821
- type: map_at_1000
value: 20.910999999999998
- type: map_at_3
value: 16.597
- type: map_at_5
value: 18.197
- type: ndcg_at_1
value: 13.413
- type: ndcg_at_10
value: 24.628
- type: ndcg_at_100
value: 30.883
- type: ndcg_at_1000
value: 33.216
- type: ndcg_at_3
value: 18.697
- type: ndcg_at_5
value: 21.501
- type: precision_at_1
value: 13.413
- type: precision_at_10
value: 4.571
- type: precision_at_100
value: 0.812
- type: precision_at_1000
value: 0.10300000000000001
- type: precision_at_3
value: 8.845
- type: precision_at_5
value: 6.889000000000001
- type: recall_at_1
value: 11.631
- type: recall_at_10
value: 38.429
- type: recall_at_100
value: 67.009
- type: recall_at_1000
value: 84.796
- type: recall_at_3
value: 22.74
- type: recall_at_5
value: 29.266
- task:
type: Retrieval
dataset:
type: quora
name: MTEB QuoraRetrieval
config: default
split: test
revision: 6205996560df11e3a3da9ab4f926788fc30a7db4
metrics:
- type: map_at_1
value: 66.64
- type: map_at_10
value: 80.394
- type: map_at_100
value: 81.099
- type: map_at_1000
value: 81.122
- type: map_at_3
value: 77.289
- type: map_at_5
value: 79.25999999999999
- type: ndcg_at_1
value: 76.85
- type: ndcg_at_10
value: 84.68
- type: ndcg_at_100
value: 86.311
- type: ndcg_at_1000
value: 86.49900000000001
- type: ndcg_at_3
value: 81.295
- type: ndcg_at_5
value: 83.199
- type: precision_at_1
value: 76.85
- type: precision_at_10
value: 12.928999999999998
- type: precision_at_100
value: 1.51
- type: precision_at_1000
value: 0.156
- type: precision_at_3
value: 35.557
- type: precision_at_5
value: 23.576
- type: recall_at_1
value: 66.64
- type: recall_at_10
value: 93.059
- type: recall_at_100
value: 98.922
- type: recall_at_1000
value: 99.883
- type: recall_at_3
value: 83.49499999999999
- type: recall_at_5
value: 88.729
- task:
type: Clustering
dataset:
type: mteb/reddit-clustering
name: MTEB RedditClustering
config: default
split: test
revision: b2805658ae38990172679479369a78b86de8c390
metrics:
- type: v_measure
value: 42.17131361041068
- task:
type: Clustering
dataset:
type: mteb/reddit-clustering-p2p
name: MTEB RedditClusteringP2P
config: default
split: test
revision: 385e3cb46b4cfa89021f56c4380204149d0efe33
metrics:
- type: v_measure
value: 48.01815621479994
- task:
type: Retrieval
dataset:
type: scidocs
name: MTEB SCIDOCS
config: default
split: test
revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5
metrics:
- type: map_at_1
value: 3.198
- type: map_at_10
value: 7.550999999999999
- type: map_at_100
value: 9.232
- type: map_at_1000
value: 9.51
- type: map_at_3
value: 5.2940000000000005
- type: map_at_5
value: 6.343999999999999
- type: ndcg_at_1
value: 15.8
- type: ndcg_at_10
value: 13.553999999999998
- type: ndcg_at_100
value: 20.776
- type: ndcg_at_1000
value: 26.204
- type: ndcg_at_3
value: 12.306000000000001
- type: ndcg_at_5
value: 10.952
- type: precision_at_1
value: 15.8
- type: precision_at_10
value: 7.180000000000001
- type: precision_at_100
value: 1.762
- type: precision_at_1000
value: 0.307
- type: precision_at_3
value: 11.333
- type: precision_at_5
value: 9.62
- type: recall_at_1
value: 3.198
- type: recall_at_10
value: 14.575
- type: recall_at_100
value: 35.758
- type: recall_at_1000
value: 62.317
- type: recall_at_3
value: 6.922000000000001
- type: recall_at_5
value: 9.767000000000001
- task:
type: STS
dataset:
type: mteb/sickr-sts
name: MTEB SICK-R
config: default
split: test
revision: 20a6d6f312dd54037fe07a32d58e5e168867909d
metrics:
- type: cos_sim_pearson
value: 84.5217161312271
- type: cos_sim_spearman
value: 79.58562467776268
- type: euclidean_pearson
value: 76.69364353942403
- type: euclidean_spearman
value: 74.68959282070473
- type: manhattan_pearson
value: 76.81159265133732
- type: manhattan_spearman
value: 74.7519444048176
- task:
type: STS
dataset:
type: mteb/sts12-sts
name: MTEB STS12
config: default
split: test
revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f
metrics:
- type: cos_sim_pearson
value: 83.70403706922605
- type: cos_sim_spearman
value: 74.28502198729447
- type: euclidean_pearson
value: 83.32719404608066
- type: euclidean_spearman
value: 75.92189433460788
- type: manhattan_pearson
value: 83.35841543005293
- type: manhattan_spearman
value: 75.94458615451978
- task:
type: STS
dataset:
type: mteb/sts13-sts
name: MTEB STS13
config: default
split: test
revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9
metrics:
- type: cos_sim_pearson
value: 84.94127878986795
- type: cos_sim_spearman
value: 85.35148434923192
- type: euclidean_pearson
value: 81.71127467071571
- type: euclidean_spearman
value: 82.88240481546771
- type: manhattan_pearson
value: 81.72826221967252
- type: manhattan_spearman
value: 82.90725064625128
- task:
type: STS
dataset:
type: mteb/sts14-sts
name: MTEB STS14
config: default
split: test
revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b
metrics:
- type: cos_sim_pearson
value: 83.1474704168523
- type: cos_sim_spearman
value: 79.20612995350827
- type: euclidean_pearson
value: 78.85993329596555
- type: euclidean_spearman
value: 78.91956572744715
- type: manhattan_pearson
value: 78.89999720522347
- type: manhattan_spearman
value: 78.93956842550107
- task:
type: STS
dataset:
type: mteb/sts15-sts
name: MTEB STS15
config: default
split: test
revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6
metrics:
- type: cos_sim_pearson
value: 84.81255514055894
- type: cos_sim_spearman
value: 85.5217140762934
- type: euclidean_pearson
value: 82.15024353784499
- type: euclidean_spearman
value: 83.04155334389833
- type: manhattan_pearson
value: 82.18598945053624
- type: manhattan_spearman
value: 83.07248357693301
- task:
type: STS
dataset:
type: mteb/sts16-sts
name: MTEB STS16
config: default
split: test
revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd
metrics:
- type: cos_sim_pearson
value: 80.63248465157822
- type: cos_sim_spearman
value: 82.53853238521991
- type: euclidean_pearson
value: 78.33936863828221
- type: euclidean_spearman
value: 79.16305579487414
- type: manhattan_pearson
value: 78.3888359870894
- type: manhattan_spearman
value: 79.18504473136467
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-en)
config: en-en
split: test
revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
metrics:
- type: cos_sim_pearson
value: 90.09066290639687
- type: cos_sim_spearman
value: 90.43893699357069
- type: euclidean_pearson
value: 82.39520777222396
- type: euclidean_spearman
value: 81.23948185395952
- type: manhattan_pearson
value: 82.35529784653383
- type: manhattan_spearman
value: 81.12681522483975
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (en)
config: en
split: test
revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
metrics:
- type: cos_sim_pearson
value: 63.52752323046846
- type: cos_sim_spearman
value: 63.19719780439462
- type: euclidean_pearson
value: 58.29085490641428
- type: euclidean_spearman
value: 58.975178656335046
- type: manhattan_pearson
value: 58.183542772416985
- type: manhattan_spearman
value: 59.190630462178994
- task:
type: STS
dataset:
type: mteb/stsbenchmark-sts
name: MTEB STSBenchmark
config: default
split: test
revision: 8913289635987208e6e7c72789e4be2fe94b6abd
metrics:
- type: cos_sim_pearson
value: 85.45100366635687
- type: cos_sim_spearman
value: 85.66816193002651
- type: euclidean_pearson
value: 81.87976731329091
- type: euclidean_spearman
value: 82.01382867690964
- type: manhattan_pearson
value: 81.88260155706726
- type: manhattan_spearman
value: 82.05258597906492
- task:
type: Reranking
dataset:
type: mteb/scidocs-reranking
name: MTEB SciDocsRR
config: default
split: test
revision: 56a6d0140cf6356659e2a7c1413286a774468d44
metrics:
- type: map
value: 77.53549990038017
- type: mrr
value: 93.37474163454556
- task:
type: Retrieval
dataset:
type: scifact
name: MTEB SciFact
config: default
split: test
revision: a75ae049398addde9b70f6b268875f5cbce99089
metrics:
- type: map_at_1
value: 31.167
- type: map_at_10
value: 40.778
- type: map_at_100
value: 42.063
- type: map_at_1000
value: 42.103
- type: map_at_3
value: 37.12
- type: map_at_5
value: 39.205
- type: ndcg_at_1
value: 33.667
- type: ndcg_at_10
value: 46.662
- type: ndcg_at_100
value: 51.995999999999995
- type: ndcg_at_1000
value: 53.254999999999995
- type: ndcg_at_3
value: 39.397999999999996
- type: ndcg_at_5
value: 42.934
- type: precision_at_1
value: 33.667
- type: precision_at_10
value: 7.1
- type: precision_at_100
value: 0.993
- type: precision_at_1000
value: 0.11
- type: precision_at_3
value: 16.111
- type: precision_at_5
value: 11.600000000000001
- type: recall_at_1
value: 31.167
- type: recall_at_10
value: 63.744
- type: recall_at_100
value: 87.156
- type: recall_at_1000
value: 97.556
- type: recall_at_3
value: 44
- type: recall_at_5
value: 52.556000000000004
- task:
type: PairClassification
dataset:
type: mteb/sprintduplicatequestions-pairclassification
name: MTEB SprintDuplicateQuestions
config: default
split: test
revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea
metrics:
- type: cos_sim_accuracy
value: 99.55148514851486
- type: cos_sim_ap
value: 80.535236573428
- type: cos_sim_f1
value: 75.01331912626532
- type: cos_sim_precision
value: 80.27366020524515
- type: cos_sim_recall
value: 70.39999999999999
- type: dot_accuracy
value: 99.04851485148515
- type: dot_ap
value: 28.505358821499726
- type: dot_f1
value: 36.36363636363637
- type: dot_precision
value: 37.160751565762006
- type: dot_recall
value: 35.6
- type: euclidean_accuracy
value: 99.4990099009901
- type: euclidean_ap
value: 74.95819047075476
- type: euclidean_f1
value: 71.15489874110564
- type: euclidean_precision
value: 78.59733978234583
- type: euclidean_recall
value: 65
- type: manhattan_accuracy
value: 99.50198019801981
- type: manhattan_ap
value: 75.02070096015086
- type: manhattan_f1
value: 71.20535714285712
- type: manhattan_precision
value: 80.55555555555556
- type: manhattan_recall
value: 63.800000000000004
- type: max_accuracy
value: 99.55148514851486
- type: max_ap
value: 80.535236573428
- type: max_f1
value: 75.01331912626532
- task:
type: Clustering
dataset:
type: mteb/stackexchange-clustering
name: MTEB StackExchangeClustering
config: default
split: test
revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235
metrics:
- type: v_measure
value: 54.13314692311623
- task:
type: Clustering
dataset:
type: mteb/stackexchange-clustering-p2p
name: MTEB StackExchangeClusteringP2P
config: default
split: test
revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0
metrics:
- type: v_measure
value: 31.115181648287145
- task:
type: Reranking
dataset:
type: mteb/stackoverflowdupquestions-reranking
name: MTEB StackOverflowDupQuestions
config: default
split: test
revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9
metrics:
- type: map
value: 44.771112666694336
- type: mrr
value: 45.30415764790765
- task:
type: Summarization
dataset:
type: mteb/summeval
name: MTEB SummEval
config: default
split: test
revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122
metrics:
- type: cos_sim_pearson
value: 30.849429597669374
- type: cos_sim_spearman
value: 30.384175038360194
- type: dot_pearson
value: 29.030383429536823
- type: dot_spearman
value: 28.03273624951732
- task:
type: Retrieval
dataset:
type: trec-covid
name: MTEB TRECCOVID
config: default
split: test
revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217
metrics:
- type: map_at_1
value: 0.19499999999999998
- type: map_at_10
value: 1.0959999999999999
- type: map_at_100
value: 5.726
- type: map_at_1000
value: 13.611999999999998
- type: map_at_3
value: 0.45399999999999996
- type: map_at_5
value: 0.67
- type: ndcg_at_1
value: 71
- type: ndcg_at_10
value: 55.352999999999994
- type: ndcg_at_100
value: 40.797
- type: ndcg_at_1000
value: 35.955999999999996
- type: ndcg_at_3
value: 63.263000000000005
- type: ndcg_at_5
value: 60.14000000000001
- type: precision_at_1
value: 78
- type: precision_at_10
value: 56.99999999999999
- type: precision_at_100
value: 41.199999999999996
- type: precision_at_1000
value: 16.154
- type: precision_at_3
value: 66.667
- type: precision_at_5
value: 62.8
- type: recall_at_1
value: 0.19499999999999998
- type: recall_at_10
value: 1.3639999999999999
- type: recall_at_100
value: 9.317
- type: recall_at_1000
value: 33.629999999999995
- type: recall_at_3
value: 0.49300000000000005
- type: recall_at_5
value: 0.756
- task:
type: Retrieval
dataset:
type: webis-touche2020
name: MTEB Touche2020
config: default
split: test
revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b
metrics:
- type: map_at_1
value: 1.335
- type: map_at_10
value: 6.293
- type: map_at_100
value: 10.928
- type: map_at_1000
value: 12.359
- type: map_at_3
value: 3.472
- type: map_at_5
value: 4.935
- type: ndcg_at_1
value: 19.387999999999998
- type: ndcg_at_10
value: 16.178
- type: ndcg_at_100
value: 28.149
- type: ndcg_at_1000
value: 39.845000000000006
- type: ndcg_at_3
value: 19.171
- type: ndcg_at_5
value: 17.864
- type: precision_at_1
value: 20.408
- type: precision_at_10
value: 14.49
- type: precision_at_100
value: 6.306000000000001
- type: precision_at_1000
value: 1.3860000000000001
- type: precision_at_3
value: 21.088
- type: precision_at_5
value: 18.367
- type: recall_at_1
value: 1.335
- type: recall_at_10
value: 10.825999999999999
- type: recall_at_100
value: 39.251000000000005
- type: recall_at_1000
value: 74.952
- type: recall_at_3
value: 4.9110000000000005
- type: recall_at_5
value: 7.312
- task:
type: Classification
dataset:
type: mteb/toxic_conversations_50k
name: MTEB ToxicConversationsClassification
config: default
split: test
revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de
metrics:
- type: accuracy
value: 69.93339999999999
- type: ap
value: 13.87476602492533
- type: f1
value: 53.867357615848555
- task:
type: Classification
dataset:
type: mteb/tweet_sentiment_extraction
name: MTEB TweetSentimentExtractionClassification
config: default
split: test
revision: 62146448f05be9e52a36b8ee9936447ea787eede
metrics:
- type: accuracy
value: 62.43916242218449
- type: f1
value: 62.870386304954685
- task:
type: Clustering
dataset:
type: mteb/twentynewsgroups-clustering
name: MTEB TwentyNewsgroupsClustering
config: default
split: test
revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4
metrics:
- type: v_measure
value: 37.202082549859796
- task:
type: PairClassification
dataset:
type: mteb/twittersemeval2015-pairclassification
name: MTEB TwitterSemEval2015
config: default
split: test
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
metrics:
- type: cos_sim_accuracy
value: 83.65023544137807
- type: cos_sim_ap
value: 65.99787692764193
- type: cos_sim_f1
value: 62.10650887573965
- type: cos_sim_precision
value: 56.30901287553648
- type: cos_sim_recall
value: 69.23482849604221
- type: dot_accuracy
value: 79.10830303391549
- type: dot_ap
value: 48.80109642320246
- type: dot_f1
value: 51.418744625967314
- type: dot_precision
value: 40.30253107683091
- type: dot_recall
value: 71.00263852242745
- type: euclidean_accuracy
value: 82.45812719794957
- type: euclidean_ap
value: 60.09969493259607
- type: euclidean_f1
value: 57.658573789246226
- type: euclidean_precision
value: 55.62913907284768
- type: euclidean_recall
value: 59.84168865435356
- type: manhattan_accuracy
value: 82.46408773916671
- type: manhattan_ap
value: 60.116199786815116
- type: manhattan_f1
value: 57.683903860160235
- type: manhattan_precision
value: 53.41726618705036
- type: manhattan_recall
value: 62.69129287598945
- type: max_accuracy
value: 83.65023544137807
- type: max_ap
value: 65.99787692764193
- type: max_f1
value: 62.10650887573965
- task:
type: PairClassification
dataset:
type: mteb/twitterurlcorpus-pairclassification
name: MTEB TwitterURLCorpus
config: default
split: test
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
metrics:
- type: cos_sim_accuracy
value: 88.34943920518494
- type: cos_sim_ap
value: 84.5428891020442
- type: cos_sim_f1
value: 77.09709933923172
- type: cos_sim_precision
value: 74.83150952967607
- type: cos_sim_recall
value: 79.50415768401602
- type: dot_accuracy
value: 84.53448208949432
- type: dot_ap
value: 73.96328242371995
- type: dot_f1
value: 70.00553786515299
- type: dot_precision
value: 63.58777665995976
- type: dot_recall
value: 77.86418232214352
- type: euclidean_accuracy
value: 86.87662514068381
- type: euclidean_ap
value: 81.45499631520235
- type: euclidean_f1
value: 73.46567109816063
- type: euclidean_precision
value: 69.71037533697381
- type: euclidean_recall
value: 77.6485987064983
- type: manhattan_accuracy
value: 86.88244654014825
- type: manhattan_ap
value: 81.47180273946366
- type: manhattan_f1
value: 73.44624393136418
- type: manhattan_precision
value: 70.80385852090032
- type: manhattan_recall
value: 76.29350169387126
- type: max_accuracy
value: 88.34943920518494
- type: max_ap
value: 84.5428891020442
- type: max_f1
value: 77.09709933923172
SGPT-5.8B-weightedmean-msmarco-specb-bitfit
Usage
For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt
Evaluation Results
For eval results, refer to our paper: https://arxiv.org/abs/2202.08904
Training
The model was trained with the parameters:
DataLoader:
torch.utils.data.dataloader.DataLoader of length 249592 with parameters:
{'batch_size': 2, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
Loss:
sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:
{'scale': 20.0, 'similarity_fct': 'cos_sim'}
Parameters of the fit()-Method:
{
"epochs": 10,
"evaluation_steps": 0,
"evaluator": "NoneType",
"max_grad_norm": 1,
"optimizer_class": "<class 'transformers.optimization.AdamW'>",
"optimizer_params": {
"lr": 5e-05
},
"scheduler": "WarmupLinear",
"steps_per_epoch": null,
"warmup_steps": 1000,
"weight_decay": 0.01
}
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTJModel
(1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)
Citing & Authors
@article{muennighoff2022sgpt,
title={SGPT: GPT Sentence Embeddings for Semantic Search},
author={Muennighoff, Niklas},
journal={arXiv preprint arXiv:2202.08904},
year={2022}
}