|
--- |
|
pipeline_tag: sentence-similarity |
|
language: multilingual |
|
license: apache-2.0 |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
- mteb |
|
model-index: |
|
- name: distiluse-base-multilingual-cased-v2 |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 71.80597014925372 |
|
- type: ap |
|
value: 33.70263085714158 |
|
- type: f1 |
|
value: 65.44989712268762 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (de) |
|
config: de |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 68.13704496788009 |
|
- type: ap |
|
value: 80.6706553308835 |
|
- type: f1 |
|
value: 66.6468090116337 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en-ext) |
|
config: en-ext |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 72.96101949025487 |
|
- type: ap |
|
value: 22.209148737301962 |
|
- type: f1 |
|
value: 60.428775420466906 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (ja) |
|
config: ja |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 65.38543897216275 |
|
- type: ap |
|
value: 16.13590032328447 |
|
- type: f1 |
|
value: 53.20720298606364 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 67.9988 |
|
- type: ap |
|
value: 62.59891275364823 |
|
- type: f1 |
|
value: 67.73408963897285 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 35.454 |
|
- type: f1 |
|
value: 35.01958914240701 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (de) |
|
config: de |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 35.032000000000004 |
|
- type: f1 |
|
value: 33.93976447064354 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (es) |
|
config: es |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 36.242000000000004 |
|
- type: f1 |
|
value: 34.98879083946539 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (fr) |
|
config: fr |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 35.699999999999996 |
|
- type: f1 |
|
value: 34.74911268048424 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (ja) |
|
config: ja |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 31.075999999999997 |
|
- type: f1 |
|
value: 30.525865114811996 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (zh) |
|
config: zh |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 33.894000000000005 |
|
- type: f1 |
|
value: 32.63851365829613 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 33.59372253035037 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 53.752292029725815 |
|
- type: mrr |
|
value: 68.26968737633557 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.26094784825986 |
|
- type: cos_sim_spearman |
|
value: 78.34033925464169 |
|
- type: euclidean_pearson |
|
value: 77.43607353262966 |
|
- type: euclidean_spearman |
|
value: 76.77765304536669 |
|
- type: manhattan_pearson |
|
value: 77.43287991423313 |
|
- type: manhattan_spearman |
|
value: 76.849341425823 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 71.48051948051949 |
|
- type: f1 |
|
value: 70.45713884617551 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 40.045 |
|
- type: f1 |
|
value: 36.59544493168501 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 61.516799999999996 |
|
- type: ap |
|
value: 57.302114956239514 |
|
- type: f1 |
|
value: 61.24392423075582 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 91.59142726858185 |
|
- type: f1 |
|
value: 91.16731589297895 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (de) |
|
config: de |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 86.19047619047619 |
|
- type: f1 |
|
value: 84.42185095665184 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (es) |
|
config: es |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 87.74516344229485 |
|
- type: f1 |
|
value: 86.89629934160831 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (fr) |
|
config: fr |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 84.61321641089883 |
|
- type: f1 |
|
value: 83.86194715158408 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (hi) |
|
config: hi |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 76.4144854786662 |
|
- type: f1 |
|
value: 74.66143814759417 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (th) |
|
config: th |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 73.61663652802893 |
|
- type: f1 |
|
value: 71.59773512640322 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 66.40218878248974 |
|
- type: f1 |
|
value: 44.0157655128108 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (de) |
|
config: de |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 59.208227669766124 |
|
- type: f1 |
|
value: 36.59415374962454 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (es) |
|
config: es |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 57.21147431621081 |
|
- type: f1 |
|
value: 38.46167201793877 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (fr) |
|
config: fr |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 53.40745380519887 |
|
- type: f1 |
|
value: 36.87813951228687 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (hi) |
|
config: hi |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 45.54320544998208 |
|
- type: f1 |
|
value: 28.091086881484788 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (th) |
|
config: th |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 47.732368896925855 |
|
- type: f1 |
|
value: 29.87429451601028 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (af) |
|
config: af |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 40.02017484868864 |
|
- type: f1 |
|
value: 35.75859698769357 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (am) |
|
config: am |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 2.347007397444519 |
|
- type: f1 |
|
value: 0.7465390699534603 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ar) |
|
config: ar |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 43.143913920645595 |
|
- type: f1 |
|
value: 38.85558637592047 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (az) |
|
config: az |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 25.601882985877605 |
|
- type: f1 |
|
value: 25.205774742990254 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (bn) |
|
config: bn |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 4.84196368527236 |
|
- type: f1 |
|
value: 1.7486302624639154 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (cy) |
|
config: cy |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 15.43375924680565 |
|
- type: f1 |
|
value: 14.212012285498213 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (da) |
|
config: da |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 52.33355749831876 |
|
- type: f1 |
|
value: 48.18484932318873 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (de) |
|
config: de |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 51.573638197713514 |
|
- type: f1 |
|
value: 45.55934579164648 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (el) |
|
config: el |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 49.65366509751178 |
|
- type: f1 |
|
value: 45.64683808611846 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 66.71149966375253 |
|
- type: f1 |
|
value: 63.78255507050109 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (es) |
|
config: es |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 56.573638197713514 |
|
- type: f1 |
|
value: 54.98029542986489 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (fa) |
|
config: fa |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 55.35642232683256 |
|
- type: f1 |
|
value: 50.20214626269123 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (fi) |
|
config: fi |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 45.71620712844654 |
|
- type: f1 |
|
value: 42.200836560817535 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (fr) |
|
config: fr |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.02084734364491 |
|
- type: f1 |
|
value: 53.910650671151814 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (he) |
|
config: he |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 46.7350369872226 |
|
- type: f1 |
|
value: 42.509857120773866 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (hi) |
|
config: hi |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 48.55077336919973 |
|
- type: f1 |
|
value: 43.993275443482936 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (hu) |
|
config: hu |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 50.64559515803631 |
|
- type: f1 |
|
value: 45.28464736653043 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (hy) |
|
config: hy |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 40.79354404841964 |
|
- type: f1 |
|
value: 36.90100598587695 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (id) |
|
config: id |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 55.99529253530599 |
|
- type: f1 |
|
value: 52.44999289764702 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (is) |
|
config: is |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 16.079354404841965 |
|
- type: f1 |
|
value: 14.926428149458182 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (it) |
|
config: it |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.64626765299259 |
|
- type: f1 |
|
value: 53.7737970315679 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ja) |
|
config: ja |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 55.329522528581045 |
|
- type: f1 |
|
value: 50.89055472943818 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (jv) |
|
config: jv |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 28.164088769334228 |
|
- type: f1 |
|
value: 25.896264320477325 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ka) |
|
config: ka |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 29.411566913248148 |
|
- type: f1 |
|
value: 26.845594782986996 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (km) |
|
config: km |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 4.791526563550773 |
|
- type: f1 |
|
value: 1.4491239093711443 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (kn) |
|
config: kn |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 3.365837256220579 |
|
- type: f1 |
|
value: 1.3064783225018712 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ko) |
|
config: ko |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 49.9663752521856 |
|
- type: f1 |
|
value: 46.28463081207797 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (lv) |
|
config: lv |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 44.31405514458642 |
|
- type: f1 |
|
value: 41.59880687298492 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ml) |
|
config: ml |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 3.2447881640887695 |
|
- type: f1 |
|
value: 1.1130430676330432 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (mn) |
|
config: mn |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 40.36650975117687 |
|
- type: f1 |
|
value: 36.405182949755556 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ms) |
|
config: ms |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 47.969065232010756 |
|
- type: f1 |
|
value: 43.564724873023735 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (my) |
|
config: my |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 38.483523873570945 |
|
- type: f1 |
|
value: 33.325537301233815 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (nb) |
|
config: nb |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 46.008742434431745 |
|
- type: f1 |
|
value: 43.1074675107609 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (nl) |
|
config: nl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 58.29186281102891 |
|
- type: f1 |
|
value: 53.383269502572276 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 53.10020174848689 |
|
- type: f1 |
|
value: 48.491009241597 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pt) |
|
config: pt |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 58.62811028917283 |
|
- type: f1 |
|
value: 56.39037901287144 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ro) |
|
config: ro |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 50.632145258910555 |
|
- type: f1 |
|
value: 47.52272047301657 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ru) |
|
config: ru |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 57.95897780766644 |
|
- type: f1 |
|
value: 53.79707075942384 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sl) |
|
config: sl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 50.65904505716207 |
|
- type: f1 |
|
value: 48.69839976207718 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sq) |
|
config: sq |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 50.25218560860794 |
|
- type: f1 |
|
value: 46.925456055473525 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sv) |
|
config: sv |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 52.410894418291875 |
|
- type: f1 |
|
value: 47.64228703598475 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (sw) |
|
config: sw |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 19.293880295897782 |
|
- type: f1 |
|
value: 17.66502971829105 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ta) |
|
config: ta |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 3.7861466039004705 |
|
- type: f1 |
|
value: 1.2869466371674323 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (te) |
|
config: te |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 3.3591123066577 |
|
- type: f1 |
|
value: 1.3191646312270082 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (th) |
|
config: th |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 45.279085406859444 |
|
- type: f1 |
|
value: 42.5424265903176 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (tl) |
|
config: tl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 28.43981170141224 |
|
- type: f1 |
|
value: 25.283226291015392 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (tr) |
|
config: tr |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 50.474108944182916 |
|
- type: f1 |
|
value: 47.186574797430794 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (ur) |
|
config: ur |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 46.02891728312038 |
|
- type: f1 |
|
value: 41.42008348263186 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (vi) |
|
config: vi |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 45.252185608607945 |
|
- type: f1 |
|
value: 41.69045540062304 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 59.21990585070611 |
|
- type: f1 |
|
value: 56.214011316092495 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (zh-TW) |
|
config: zh-TW |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 54.96301277740416 |
|
- type: f1 |
|
value: 53.020268356293045 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (af) |
|
config: af |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 53.665097511768664 |
|
- type: f1 |
|
value: 48.81662825721646 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (am) |
|
config: am |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 7.720242098184263 |
|
- type: f1 |
|
value: 3.0172360162047553 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ar) |
|
config: ar |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 52.188971082716876 |
|
- type: f1 |
|
value: 52.360668734058116 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (az) |
|
config: az |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 34.74781439139206 |
|
- type: f1 |
|
value: 32.55953852645334 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (bn) |
|
config: bn |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 10.652320107599191 |
|
- type: f1 |
|
value: 6.439785272600618 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (cy) |
|
config: cy |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 21.237390719569603 |
|
- type: f1 |
|
value: 18.428497244325158 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (da) |
|
config: da |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 62.54875588433087 |
|
- type: f1 |
|
value: 60.69001958508912 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (de) |
|
config: de |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 61.40215198386013 |
|
- type: f1 |
|
value: 58.07492599013545 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (el) |
|
config: el |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 60.67585743106927 |
|
- type: f1 |
|
value: 58.055827627792056 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 74.00470746469401 |
|
- type: f1 |
|
value: 72.22931856264793 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (es) |
|
config: es |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.6133154001345 |
|
- type: f1 |
|
value: 63.345907502958184 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (fa) |
|
config: fa |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 59.2434431741762 |
|
- type: f1 |
|
value: 57.40580117369346 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (fi) |
|
config: fi |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 54.660390047074635 |
|
- type: f1 |
|
value: 51.45432689446743 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (fr) |
|
config: fr |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.19502353732346 |
|
- type: f1 |
|
value: 63.50200684075783 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (he) |
|
config: he |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 54.744451916610636 |
|
- type: f1 |
|
value: 52.621508448089294 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (hi) |
|
config: hi |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 55.985205110961665 |
|
- type: f1 |
|
value: 53.70079438430524 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (hu) |
|
config: hu |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 61.20040349697378 |
|
- type: f1 |
|
value: 58.5060672562612 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (hy) |
|
config: hy |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 49.63349024882314 |
|
- type: f1 |
|
value: 47.39478501763526 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (id) |
|
config: id |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.25218560860793 |
|
- type: f1 |
|
value: 63.45266636240826 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (is) |
|
config: is |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 22.599193006052456 |
|
- type: f1 |
|
value: 21.93829297740852 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (it) |
|
config: it |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.63349024882314 |
|
- type: f1 |
|
value: 63.15345402734339 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ja) |
|
config: ja |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 62.32010759919301 |
|
- type: f1 |
|
value: 60.02914271738089 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (jv) |
|
config: jv |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 35.76664425016812 |
|
- type: f1 |
|
value: 33.52830525064859 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ka) |
|
config: ka |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 39.08204438466712 |
|
- type: f1 |
|
value: 37.312566552928736 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (km) |
|
config: km |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 9.236718224613314 |
|
- type: f1 |
|
value: 3.41684484979606 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (kn) |
|
config: kn |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 8.278412911903162 |
|
- type: f1 |
|
value: 3.9418094806677426 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ko) |
|
config: ko |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 57.595830531271005 |
|
- type: f1 |
|
value: 56.42188880877947 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (lv) |
|
config: lv |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 51.72158708809683 |
|
- type: f1 |
|
value: 49.903136843275256 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ml) |
|
config: ml |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 8.254875588433089 |
|
- type: f1 |
|
value: 4.06813409809564 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (mn) |
|
config: mn |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 47.20914593140552 |
|
- type: f1 |
|
value: 44.780121017940225 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ms) |
|
config: ms |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 55.64559515803632 |
|
- type: f1 |
|
value: 53.10457083056076 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (my) |
|
config: my |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 43.308675184936114 |
|
- type: f1 |
|
value: 40.40654924373442 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (nb) |
|
config: nb |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 54.983187626092814 |
|
- type: f1 |
|
value: 54.22408282419106 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (nl) |
|
config: nl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 67.4915938130464 |
|
- type: f1 |
|
value: 64.66608521628295 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 61.28782784129119 |
|
- type: f1 |
|
value: 59.364955179296544 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pt) |
|
config: pt |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.26361802286483 |
|
- type: f1 |
|
value: 63.01306314842478 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ro) |
|
config: ro |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 58.02622730329523 |
|
- type: f1 |
|
value: 55.8928740774695 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ru) |
|
config: ru |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 65.41358439811701 |
|
- type: f1 |
|
value: 64.15512608670188 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sl) |
|
config: sl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 59.357767316745125 |
|
- type: f1 |
|
value: 58.284479078165106 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sq) |
|
config: sq |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 62.686617350369865 |
|
- type: f1 |
|
value: 59.49767603465277 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sv) |
|
config: sv |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 64.35104236718225 |
|
- type: f1 |
|
value: 61.62298238070601 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (sw) |
|
config: sw |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 25.12104909213181 |
|
- type: f1 |
|
value: 22.063961287382483 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ta) |
|
config: ta |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 8.671822461331539 |
|
- type: f1 |
|
value: 4.160922973001201 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (te) |
|
config: te |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 7.821116341627439 |
|
- type: f1 |
|
value: 3.59600077788794 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (th) |
|
config: th |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 54.64694014794888 |
|
- type: f1 |
|
value: 51.591586777977504 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (tl) |
|
config: tl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 36.08607935440485 |
|
- type: f1 |
|
value: 32.46731674317254 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (tr) |
|
config: tr |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 60.89441829186282 |
|
- type: f1 |
|
value: 60.11999627480401 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (ur) |
|
config: ur |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 54.707464694014796 |
|
- type: f1 |
|
value: 52.46709289947395 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (vi) |
|
config: vi |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 55.1546738399462 |
|
- type: f1 |
|
value: 54.110902262235584 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 66.4357767316745 |
|
- type: f1 |
|
value: 64.94684758602547 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (zh-TW) |
|
config: zh-TW |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 62.88836583725623 |
|
- type: f1 |
|
value: 61.7106895387137 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 30.389522323606887 |
|
- type: mrr |
|
value: 31.507198662637208 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.18466748223793 |
|
- type: cos_sim_spearman |
|
value: 75.24738784985722 |
|
- type: euclidean_pearson |
|
value: 78.51159752223624 |
|
- type: euclidean_spearman |
|
value: 75.46087065937311 |
|
- type: manhattan_pearson |
|
value: 77.16743820738003 |
|
- type: manhattan_spearman |
|
value: 73.49433694282183 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.35237266605724 |
|
- type: cos_sim_spearman |
|
value: 72.95904349793416 |
|
- type: euclidean_pearson |
|
value: 73.07895490202789 |
|
- type: euclidean_spearman |
|
value: 71.66451640969629 |
|
- type: manhattan_pearson |
|
value: 73.08359981539324 |
|
- type: manhattan_spearman |
|
value: 71.91126963073746 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 68.26126180159085 |
|
- type: cos_sim_spearman |
|
value: 70.5821267642011 |
|
- type: euclidean_pearson |
|
value: 69.32005598610408 |
|
- type: euclidean_spearman |
|
value: 69.91767420734864 |
|
- type: manhattan_pearson |
|
value: 69.65574245013867 |
|
- type: manhattan_spearman |
|
value: 70.22188522513176 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 73.8304467062826 |
|
- type: cos_sim_spearman |
|
value: 70.28565248557119 |
|
- type: euclidean_pearson |
|
value: 72.80361711138981 |
|
- type: euclidean_spearman |
|
value: 70.63777081958187 |
|
- type: manhattan_pearson |
|
value: 72.88892597106383 |
|
- type: manhattan_spearman |
|
value: 70.86449280993048 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.41478503988436 |
|
- type: cos_sim_spearman |
|
value: 81.94087130039843 |
|
- type: euclidean_pearson |
|
value: 81.23351470401855 |
|
- type: euclidean_spearman |
|
value: 81.43266713211875 |
|
- type: manhattan_pearson |
|
value: 81.16667353510842 |
|
- type: manhattan_spearman |
|
value: 81.24163241523068 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 75.08475719822 |
|
- type: cos_sim_spearman |
|
value: 76.80438358515593 |
|
- type: euclidean_pearson |
|
value: 75.90649123881406 |
|
- type: euclidean_spearman |
|
value: 75.9482319164023 |
|
- type: manhattan_pearson |
|
value: 75.64396465387331 |
|
- type: manhattan_spearman |
|
value: 75.56185817375638 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (ko-ko) |
|
config: ko-ko |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 76.57756740555968 |
|
- type: cos_sim_spearman |
|
value: 76.39843364267264 |
|
- type: euclidean_pearson |
|
value: 75.40424583472578 |
|
- type: euclidean_spearman |
|
value: 75.31307938562327 |
|
- type: manhattan_pearson |
|
value: 74.73109587053861 |
|
- type: manhattan_spearman |
|
value: 74.54667368714956 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (ar-ar) |
|
config: ar-ar |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 76.54105158056127 |
|
- type: cos_sim_spearman |
|
value: 77.34104635434048 |
|
- type: euclidean_pearson |
|
value: 75.28125389103582 |
|
- type: euclidean_spearman |
|
value: 75.42418151345 |
|
- type: manhattan_pearson |
|
value: 74.2691880967768 |
|
- type: manhattan_spearman |
|
value: 74.14253657856801 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-ar) |
|
config: en-ar |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 77.02928931510961 |
|
- type: cos_sim_spearman |
|
value: 77.45907270306685 |
|
- type: euclidean_pearson |
|
value: 77.47937379735676 |
|
- type: euclidean_spearman |
|
value: 77.21301895586583 |
|
- type: manhattan_pearson |
|
value: 76.6676288138473 |
|
- type: manhattan_spearman |
|
value: 76.7187203876331 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-de) |
|
config: en-de |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.85147526701459 |
|
- type: cos_sim_spearman |
|
value: 80.24439450219447 |
|
- type: euclidean_pearson |
|
value: 80.16905693851314 |
|
- type: euclidean_spearman |
|
value: 79.30869641757035 |
|
- type: manhattan_pearson |
|
value: 79.4830024429918 |
|
- type: manhattan_spearman |
|
value: 78.64845690144578 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.23328074603815 |
|
- type: cos_sim_spearman |
|
value: 86.18847213007086 |
|
- type: euclidean_pearson |
|
value: 85.91331577309407 |
|
- type: euclidean_spearman |
|
value: 85.89967500124904 |
|
- type: manhattan_pearson |
|
value: 85.13857617716477 |
|
- type: manhattan_spearman |
|
value: 84.82259586513993 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-tr) |
|
config: en-tr |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 75.38182956463326 |
|
- type: cos_sim_spearman |
|
value: 74.34143229429068 |
|
- type: euclidean_pearson |
|
value: 76.66151217728661 |
|
- type: euclidean_spearman |
|
value: 75.68846427284615 |
|
- type: manhattan_pearson |
|
value: 75.55942040372382 |
|
- type: manhattan_spearman |
|
value: 74.67284614447757 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (es-en) |
|
config: es-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 76.94108940753875 |
|
- type: cos_sim_spearman |
|
value: 77.39619379750977 |
|
- type: euclidean_pearson |
|
value: 76.7736720732895 |
|
- type: euclidean_spearman |
|
value: 76.29160645031078 |
|
- type: manhattan_pearson |
|
value: 74.69337188827635 |
|
- type: manhattan_spearman |
|
value: 74.47874230344613 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (es-es) |
|
config: es-es |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 83.99450399002905 |
|
- type: cos_sim_spearman |
|
value: 83.71182297187157 |
|
- type: euclidean_pearson |
|
value: 85.14304799861979 |
|
- type: euclidean_spearman |
|
value: 83.69127569618827 |
|
- type: manhattan_pearson |
|
value: 84.90116866712872 |
|
- type: manhattan_spearman |
|
value: 83.31690582990805 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (fr-en) |
|
config: fr-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.12525161262887 |
|
- type: cos_sim_spearman |
|
value: 79.27905944348255 |
|
- type: euclidean_pearson |
|
value: 80.37847361563627 |
|
- type: euclidean_spearman |
|
value: 79.45430583111714 |
|
- type: manhattan_pearson |
|
value: 79.39311209355259 |
|
- type: manhattan_spearman |
|
value: 78.35224091918822 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (it-en) |
|
config: it-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 80.35229136945712 |
|
- type: cos_sim_spearman |
|
value: 80.82110464777067 |
|
- type: euclidean_pearson |
|
value: 80.8820546236635 |
|
- type: euclidean_spearman |
|
value: 80.52608029482144 |
|
- type: manhattan_pearson |
|
value: 79.87881836256757 |
|
- type: manhattan_spearman |
|
value: 79.21409642635105 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (nl-en) |
|
config: nl-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 80.08711291606406 |
|
- type: cos_sim_spearman |
|
value: 80.50747550174945 |
|
- type: euclidean_pearson |
|
value: 80.19128295947303 |
|
- type: euclidean_spearman |
|
value: 79.80068556328985 |
|
- type: manhattan_pearson |
|
value: 79.2805531467 |
|
- type: manhattan_spearman |
|
value: 78.67459586691882 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 63.749476793187654 |
|
- type: cos_sim_spearman |
|
value: 62.87618960301087 |
|
- type: euclidean_pearson |
|
value: 62.00259194547161 |
|
- type: euclidean_spearman |
|
value: 60.14134804263504 |
|
- type: manhattan_pearson |
|
value: 61.85663435862556 |
|
- type: manhattan_spearman |
|
value: 60.49194043559385 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de) |
|
config: de |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 30.728588031668387 |
|
- type: cos_sim_spearman |
|
value: 35.72910641917946 |
|
- type: euclidean_pearson |
|
value: 27.727483814940634 |
|
- type: euclidean_spearman |
|
value: 36.697908777201874 |
|
- type: manhattan_pearson |
|
value: 26.887457740598375 |
|
- type: manhattan_spearman |
|
value: 35.65193589164902 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (es) |
|
config: es |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 58.515732517017895 |
|
- type: cos_sim_spearman |
|
value: 59.34352724163223 |
|
- type: euclidean_pearson |
|
value: 59.37822334487575 |
|
- type: euclidean_spearman |
|
value: 59.952966536792296 |
|
- type: manhattan_pearson |
|
value: 59.34905346132589 |
|
- type: manhattan_spearman |
|
value: 59.58363163864109 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl) |
|
config: pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 26.73251862968695 |
|
- type: cos_sim_spearman |
|
value: 34.57702083368428 |
|
- type: euclidean_pearson |
|
value: 11.555722679629111 |
|
- type: euclidean_spearman |
|
value: 33.83302978677857 |
|
- type: manhattan_pearson |
|
value: 11.30958607896797 |
|
- type: manhattan_spearman |
|
value: 33.45113736058396 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (tr) |
|
config: tr |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 50.59069907623683 |
|
- type: cos_sim_spearman |
|
value: 54.07437321160808 |
|
- type: euclidean_pearson |
|
value: 55.31327716542195 |
|
- type: euclidean_spearman |
|
value: 55.862881519289 |
|
- type: manhattan_pearson |
|
value: 55.76874086920313 |
|
- type: manhattan_spearman |
|
value: 56.389207939925434 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (ar) |
|
config: ar |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 43.19525519197726 |
|
- type: cos_sim_spearman |
|
value: 49.04013064287781 |
|
- type: euclidean_pearson |
|
value: 41.51101650799975 |
|
- type: euclidean_spearman |
|
value: 45.69491981920255 |
|
- type: manhattan_pearson |
|
value: 41.798306097489686 |
|
- type: manhattan_spearman |
|
value: 45.88969916327865 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (ru) |
|
config: ru |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 46.72887212606245 |
|
- type: cos_sim_spearman |
|
value: 52.40251410115027 |
|
- type: euclidean_pearson |
|
value: 42.61087105318375 |
|
- type: euclidean_spearman |
|
value: 49.31647979068464 |
|
- type: manhattan_pearson |
|
value: 41.971488569524226 |
|
- type: manhattan_spearman |
|
value: 48.603948080104416 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (zh) |
|
config: zh |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 50.282899703556204 |
|
- type: cos_sim_spearman |
|
value: 54.31518993723914 |
|
- type: euclidean_pearson |
|
value: 46.92686134587321 |
|
- type: euclidean_spearman |
|
value: 50.4258942374202 |
|
- type: manhattan_pearson |
|
value: 47.119373335384516 |
|
- type: manhattan_spearman |
|
value: 50.290545214030644 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (fr) |
|
config: fr |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 76.6695578258507 |
|
- type: cos_sim_spearman |
|
value: 76.41254265129491 |
|
- type: euclidean_pearson |
|
value: 68.10573760855496 |
|
- type: euclidean_spearman |
|
value: 71.53756176277794 |
|
- type: manhattan_pearson |
|
value: 67.71247571269289 |
|
- type: manhattan_spearman |
|
value: 71.52537846395397 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de-en) |
|
config: de-en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 52.39033873441029 |
|
- type: cos_sim_spearman |
|
value: 47.50888019756861 |
|
- type: euclidean_pearson |
|
value: 54.09329593694967 |
|
- type: euclidean_spearman |
|
value: 46.745911343795036 |
|
- type: manhattan_pearson |
|
value: 55.071517962875795 |
|
- type: manhattan_spearman |
|
value: 47.82505012490346 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (es-en) |
|
config: es-en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 66.22856680218524 |
|
- type: cos_sim_spearman |
|
value: 68.9583551854743 |
|
- type: euclidean_pearson |
|
value: 69.45990476537347 |
|
- type: euclidean_spearman |
|
value: 69.51326488176926 |
|
- type: manhattan_pearson |
|
value: 69.2654378415376 |
|
- type: manhattan_spearman |
|
value: 69.25549968332008 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (it) |
|
config: it |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 63.86370050619784 |
|
- type: cos_sim_spearman |
|
value: 65.10152541505573 |
|
- type: euclidean_pearson |
|
value: 61.23738658178195 |
|
- type: euclidean_spearman |
|
value: 62.77231926242124 |
|
- type: manhattan_pearson |
|
value: 61.20141239111747 |
|
- type: manhattan_spearman |
|
value: 62.58683030963466 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl-en) |
|
config: pl-en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 70.24310112698741 |
|
- type: cos_sim_spearman |
|
value: 71.32608389737901 |
|
- type: euclidean_pearson |
|
value: 69.53167907457565 |
|
- type: euclidean_spearman |
|
value: 69.24756304760876 |
|
- type: manhattan_pearson |
|
value: 69.4432001214127 |
|
- type: manhattan_spearman |
|
value: 69.92998467998946 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (zh-en) |
|
config: zh-en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 62.96457033320131 |
|
- type: cos_sim_spearman |
|
value: 61.750627475845285 |
|
- type: euclidean_pearson |
|
value: 59.58377101704754 |
|
- type: euclidean_spearman |
|
value: 55.91175172327044 |
|
- type: manhattan_pearson |
|
value: 59.64672089274813 |
|
- type: manhattan_spearman |
|
value: 55.93114256617111 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (es-it) |
|
config: es-it |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 60.54093974085284 |
|
- type: cos_sim_spearman |
|
value: 63.277246213501634 |
|
- type: euclidean_pearson |
|
value: 59.21790717375445 |
|
- type: euclidean_spearman |
|
value: 60.77632900198518 |
|
- type: manhattan_pearson |
|
value: 59.572573245502824 |
|
- type: manhattan_spearman |
|
value: 60.86391917522135 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de-fr) |
|
config: de-fr |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 56.2735220514599 |
|
- type: cos_sim_spearman |
|
value: 60.76242915296164 |
|
- type: euclidean_pearson |
|
value: 54.73358313453174 |
|
- type: euclidean_spearman |
|
value: 59.01153256838316 |
|
- type: manhattan_pearson |
|
value: 53.30971466711619 |
|
- type: manhattan_spearman |
|
value: 57.427602926148516 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (de-pl) |
|
config: de-pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 33.210422466959244 |
|
- type: cos_sim_spearman |
|
value: 36.09068930156353 |
|
- type: euclidean_pearson |
|
value: 36.72425141682268 |
|
- type: euclidean_spearman |
|
value: 33.3808081935963 |
|
- type: manhattan_pearson |
|
value: 35.47249118003641 |
|
- type: manhattan_spearman |
|
value: 31.964279432613434 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (fr-pl) |
|
config: fr-pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 62.721710627517034 |
|
- type: cos_sim_spearman |
|
value: 61.97797868009122 |
|
- type: euclidean_pearson |
|
value: 63.59898515445168 |
|
- type: euclidean_spearman |
|
value: 84.51542547285167 |
|
- type: manhattan_pearson |
|
value: 62.15380605376377 |
|
- type: manhattan_spearman |
|
value: 73.24670207647144 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.6839488629375 |
|
- type: cos_sim_spearman |
|
value: 80.75478754676419 |
|
- type: euclidean_pearson |
|
value: 80.67588249670365 |
|
- type: euclidean_spearman |
|
value: 80.2296669116562 |
|
- type: manhattan_pearson |
|
value: 79.79275882752755 |
|
- type: manhattan_spearman |
|
value: 79.41562131296504 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 69.21586199162861 |
|
- type: mrr |
|
value: 88.86282290694054 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.62079207920792 |
|
- type: cos_sim_ap |
|
value: 87.14976457350163 |
|
- type: cos_sim_f1 |
|
value: 81.07317073170732 |
|
- type: cos_sim_precision |
|
value: 79.14285714285715 |
|
- type: cos_sim_recall |
|
value: 83.1 |
|
- type: dot_accuracy |
|
value: 99.57722772277228 |
|
- type: dot_ap |
|
value: 84.07833605976549 |
|
- type: dot_f1 |
|
value: 77.88461538461539 |
|
- type: dot_precision |
|
value: 75.0 |
|
- type: dot_recall |
|
value: 81.0 |
|
- type: euclidean_accuracy |
|
value: 99.61287128712871 |
|
- type: euclidean_ap |
|
value: 86.94165408325189 |
|
- type: euclidean_f1 |
|
value: 80.33596837944663 |
|
- type: euclidean_precision |
|
value: 79.39453125 |
|
- type: euclidean_recall |
|
value: 81.3 |
|
- type: manhattan_accuracy |
|
value: 99.64653465346535 |
|
- type: manhattan_ap |
|
value: 88.43495903247096 |
|
- type: manhattan_f1 |
|
value: 81.7193675889328 |
|
- type: manhattan_precision |
|
value: 80.76171875 |
|
- type: manhattan_recall |
|
value: 82.69999999999999 |
|
- type: max_accuracy |
|
value: 99.64653465346535 |
|
- type: max_ap |
|
value: 88.43495903247096 |
|
- type: max_f1 |
|
value: 81.7193675889328 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 41.92031499253617 |
|
- type: mrr |
|
value: 42.11711389101095 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 69.0936 |
|
- type: ap |
|
value: 13.464419132094955 |
|
- type: f1 |
|
value: 53.17756829624628 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 59.968873797396704 |
|
- type: f1 |
|
value: 60.23697658216021 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 82.7978780473267 |
|
- type: cos_sim_ap |
|
value: 61.669291081213906 |
|
- type: cos_sim_f1 |
|
value: 57.68693665100927 |
|
- type: cos_sim_precision |
|
value: 55.59089796917054 |
|
- type: cos_sim_recall |
|
value: 59.94722955145119 |
|
- type: dot_accuracy |
|
value: 81.68921738093819 |
|
- type: dot_ap |
|
value: 57.39705387908134 |
|
- type: dot_f1 |
|
value: 54.72479298587434 |
|
- type: dot_precision |
|
value: 50.814111261872455 |
|
- type: dot_recall |
|
value: 59.287598944591025 |
|
- type: euclidean_accuracy |
|
value: 82.85152291828098 |
|
- type: euclidean_ap |
|
value: 62.456817170822255 |
|
- type: euclidean_f1 |
|
value: 58.32305795314425 |
|
- type: euclidean_precision |
|
value: 54.745370370370374 |
|
- type: euclidean_recall |
|
value: 62.401055408970976 |
|
- type: manhattan_accuracy |
|
value: 82.76807534124099 |
|
- type: manhattan_ap |
|
value: 61.85267667234618 |
|
- type: manhattan_f1 |
|
value: 57.62629336579428 |
|
- type: manhattan_precision |
|
value: 53.49152542372882 |
|
- type: manhattan_recall |
|
value: 62.45382585751978 |
|
- type: max_accuracy |
|
value: 82.85152291828098 |
|
- type: max_ap |
|
value: 62.456817170822255 |
|
- type: max_f1 |
|
value: 58.32305795314425 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.03896456708192 |
|
- type: cos_sim_ap |
|
value: 84.0249558879327 |
|
- type: cos_sim_f1 |
|
value: 76.26290458870642 |
|
- type: cos_sim_precision |
|
value: 72.93233082706767 |
|
- type: cos_sim_recall |
|
value: 79.91222667077302 |
|
- type: dot_accuracy |
|
value: 87.87402491558971 |
|
- type: dot_ap |
|
value: 83.20076543059169 |
|
- type: dot_f1 |
|
value: 76.02826329490517 |
|
- type: dot_precision |
|
value: 73.52898863472882 |
|
- type: dot_recall |
|
value: 78.70341854019095 |
|
- type: euclidean_accuracy |
|
value: 87.96328637404433 |
|
- type: euclidean_ap |
|
value: 83.78378095020464 |
|
- type: euclidean_f1 |
|
value: 75.94917787742901 |
|
- type: euclidean_precision |
|
value: 73.78739471391229 |
|
- type: euclidean_recall |
|
value: 78.24145364952264 |
|
- type: manhattan_accuracy |
|
value: 87.99239337136648 |
|
- type: manhattan_ap |
|
value: 83.72045889779073 |
|
- type: manhattan_f1 |
|
value: 75.93527315914488 |
|
- type: manhattan_precision |
|
value: 73.30180567497851 |
|
- type: manhattan_recall |
|
value: 78.76501385894672 |
|
- type: max_accuracy |
|
value: 88.03896456708192 |
|
- type: max_ap |
|
value: 84.0249558879327 |
|
- type: max_f1 |
|
value: 76.26290458870642 |
|
--- |
|
|
|
# sentence-transformers/distiluse-base-multilingual-cased-v2 |
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 512 dimensional dense vector space and can be used for tasks like clustering or semantic search. |
|
|
|
|
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can use the model like this: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
sentences = ["This is an example sentence", "Each sentence is converted"] |
|
|
|
model = SentenceTransformer('sentence-transformers/distiluse-base-multilingual-cased-v2') |
|
embeddings = model.encode(sentences) |
|
print(embeddings) |
|
``` |
|
|
|
|
|
|
|
## Evaluation Results |
|
|
|
|
|
|
|
For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/distiluse-base-multilingual-cased-v2) |
|
|
|
|
|
|
|
## Full Model Architecture |
|
``` |
|
SentenceTransformer( |
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: DistilBertModel |
|
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) |
|
(2): Dense({'in_features': 768, 'out_features': 512, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'}) |
|
) |
|
``` |
|
|
|
## Citing & Authors |
|
|
|
This model was trained by [sentence-transformers](https://www.sbert.net/). |
|
|
|
If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084): |
|
```bibtex |
|
@inproceedings{reimers-2019-sentence-bert, |
|
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
|
author = "Reimers, Nils and Gurevych, Iryna", |
|
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
|
month = "11", |
|
year = "2019", |
|
publisher = "Association for Computational Linguistics", |
|
url = "http://arxiv.org/abs/1908.10084", |
|
} |
|
``` |
