---
base_model: Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2
datasets:
- Omartificial-Intelligence-Space/Arabic-stsb
- Omartificial-Intelligence-Space/Arabic-NLi-Pair-Class
language:
- ar
library_name: sentence-transformers
metrics:
- pearson_cosine
- spearman_cosine
- pearson_manhattan
- spearman_manhattan
- pearson_euclidean
- spearman_euclidean
- pearson_dot
- spearman_dot
- pearson_max
- spearman_max
pipeline_tag: sentence-similarity
tags:
- mteb
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:947818
- loss:SoftmaxLoss
- loss:CosineSimilarityLoss
- transformers
model-index:
- name: Omartificial-Intelligence-Space/GATE-AraBert-v1
results:
- dataset:
config: ar
name: MTEB MIRACLRetrieval (ar)
revision: main
split: dev
type: miracl/mmteb-miracl
metrics:
- type: ndcg_at_1
value: 48.998999999999995
- type: ndcg_at_3
value: 50.329
- type: ndcg_at_5
value: 52.806
- type: ndcg_at_10
value: 56.114
- type: ndcg_at_20
value: 58.504999999999995
- type: ndcg_at_100
value: 61.126000000000005
- type: ndcg_at_1000
value: 62.388
- type: map_at_1
value: 32.54
- type: map_at_3
value: 43.749
- type: map_at_5
value: 46.345
- type: map_at_10
value: 48.379
- type: map_at_20
value: 49.346000000000004
- type: map_at_100
value: 49.913000000000004
- type: map_at_1000
value: 49.992
- type: recall_at_1
value: 32.54
- type: recall_at_3
value: 51.413
- type: recall_at_5
value: 58.653999999999996
- type: recall_at_10
value: 67.123
- type: recall_at_20
value: 74.655
- type: recall_at_100
value: 85.68299999999999
- type: recall_at_1000
value: 93.67
- type: precision_at_1
value: 48.998999999999995
- type: precision_at_3
value: 28.522
- type: precision_at_5
value: 20.283
- type: precision_at_10
value: 12.151
- type: precision_at_20
value: 6.946
- type: precision_at_100
value: 1.652
- type: precision_at_1000
value: 0.184
- type: mrr_at_1
value: 48.998599999999996
- type: mrr_at_3
value: 57.1881
- type: mrr_at_5
value: 58.6107
- type: mrr_at_10
value: 59.49529999999999
- type: mrr_at_20
value: 59.8446
- type: mrr_at_100
value: 60.0413
- type: mrr_at_1000
value: 60.0566
- type: nauc_ndcg_at_1_max
value: 29.901699999999998
- type: nauc_ndcg_at_1_std
value: -14.327599999999999
- type: nauc_ndcg_at_1_diff1
value: 44.8786
- type: nauc_ndcg_at_3_max
value: 32.1094
- type: nauc_ndcg_at_3_std
value: -16.5335
- type: nauc_ndcg_at_3_diff1
value: 39.5919
- type: nauc_ndcg_at_5_max
value: 33.5012
- type: nauc_ndcg_at_5_std
value: -16.3301
- type: nauc_ndcg_at_5_diff1
value: 40.2422
- type: nauc_ndcg_at_10_max
value: 34.6286
- type: nauc_ndcg_at_10_std
value: -13.6744
- type: nauc_ndcg_at_10_diff1
value: 39.9697
- type: nauc_ndcg_at_20_max
value: 36.2012
- type: nauc_ndcg_at_20_std
value: -12.052999999999999
- type: nauc_ndcg_at_20_diff1
value: 40.4644
- type: nauc_ndcg_at_100_max
value: 36.3365
- type: nauc_ndcg_at_100_std
value: -10.5412
- type: nauc_ndcg_at_100_diff1
value: 40.509
- type: nauc_ndcg_at_1000_max
value: 36.0541
- type: nauc_ndcg_at_1000_std
value: -10.0865
- type: nauc_ndcg_at_1000_diff1
value: 40.5604
- type: nauc_map_at_1_max
value: 21.8058
- type: nauc_map_at_1_std
value: -19.9526
- type: nauc_map_at_1_diff1
value: 44.4087
- type: nauc_map_at_3_max
value: 28.890500000000003
- type: nauc_map_at_3_std
value: -19.290499999999998
- type: nauc_map_at_3_diff1
value: 40.854
- type: nauc_map_at_5_max
value: 30.636799999999997
- type: nauc_map_at_5_std
value: -18.2761
- type: nauc_map_at_5_diff1
value: 40.6916
- type: nauc_map_at_10_max
value: 31.7128
- type: nauc_map_at_10_std
value: -16.7294
- type: nauc_map_at_10_diff1
value: 40.4383
- type: nauc_map_at_20_max
value: 32.3947
- type: nauc_map_at_20_std
value: -16.003500000000003
- type: nauc_map_at_20_diff1
value: 40.5663
- type: nauc_map_at_100_max
value: 32.4921
- type: nauc_map_at_100_std
value: -15.667700000000002
- type: nauc_map_at_100_diff1
value: 40.5719
- type: nauc_map_at_1000_max
value: 32.487700000000004
- type: nauc_map_at_1000_std
value: -15.621099999999998
- type: nauc_map_at_1000_diff1
value: 40.5743
- type: nauc_recall_at_1_max
value: 21.8058
- type: nauc_recall_at_1_std
value: -19.9526
- type: nauc_recall_at_1_diff1
value: 44.4087
- type: nauc_recall_at_3_max
value: 29.251500000000004
- type: nauc_recall_at_3_std
value: -18.343300000000003
- type: nauc_recall_at_3_diff1
value: 34.902699999999996
- type: nauc_recall_at_5_max
value: 32.901599999999995
- type: nauc_recall_at_5_std
value: -16.3291
- type: nauc_recall_at_5_diff1
value: 34.0112
- type: nauc_recall_at_10_max
value: 35.2032
- type: nauc_recall_at_10_std
value: -9.4354
- type: nauc_recall_at_10_diff1
value: 32.3218
- type: nauc_recall_at_20_max
value: 41.2983
- type: nauc_recall_at_20_std
value: -3.343
- type: nauc_recall_at_20_diff1
value: 33.3028
- type: nauc_recall_at_100_max
value: 46.009699999999995
- type: nauc_recall_at_100_std
value: 10.536
- type: nauc_recall_at_100_diff1
value: 32.4706
- type: nauc_recall_at_1000_max
value: 53.3901
- type: nauc_recall_at_1000_std
value: 38.3829
- type: nauc_recall_at_1000_diff1
value: 30.8604
- type: nauc_precision_at_1_max
value: 29.901699999999998
- type: nauc_precision_at_1_std
value: -14.327599999999999
- type: nauc_precision_at_1_diff1
value: 44.8786
- type: nauc_precision_at_3_max
value: 31.6568
- type: nauc_precision_at_3_std
value: -2.5404
- type: nauc_precision_at_3_diff1
value: 19.357499999999998
- type: nauc_precision_at_5_max
value: 31.013800000000003
- type: nauc_precision_at_5_std
value: 3.505
- type: nauc_precision_at_5_diff1
value: 13.6183
- type: nauc_precision_at_10_max
value: 27.8315
- type: nauc_precision_at_10_std
value: 12.5719
- type: nauc_precision_at_10_diff1
value: 6.3679
- type: nauc_precision_at_20_max
value: 25.851200000000002
- type: nauc_precision_at_20_std
value: 18.206
- type: nauc_precision_at_20_diff1
value: 2.226
- type: nauc_precision_at_100_max
value: 17.8541
- type: nauc_precision_at_100_std
value: 24.4343
- type: nauc_precision_at_100_diff1
value: -5.285
- type: nauc_precision_at_1000_max
value: 11.475399999999999
- type: nauc_precision_at_1000_std
value: 26.5099
- type: nauc_precision_at_1000_diff1
value: -9.5944
- type: nauc_mrr_at_1_max
value: 29.901699999999998
- type: nauc_mrr_at_1_std
value: -14.327599999999999
- type: nauc_mrr_at_1_diff1
value: 44.8786
- type: nauc_mrr_at_3_max
value: 34.1117
- type: nauc_mrr_at_3_std
value: -12.4902
- type: nauc_mrr_at_3_diff1
value: 42.321799999999996
- type: nauc_mrr_at_5_max
value: 34.767700000000005
- type: nauc_mrr_at_5_std
value: -12.034799999999999
- type: nauc_mrr_at_5_diff1
value: 42.5597
- type: nauc_mrr_at_10_max
value: 34.7657
- type: nauc_mrr_at_10_std
value: -11.275300000000001
- type: nauc_mrr_at_10_diff1
value: 42.5102
- type: nauc_mrr_at_20_max
value: 34.8375
- type: nauc_mrr_at_20_std
value: -11.2196
- type: nauc_mrr_at_20_diff1
value: 42.6012
- type: nauc_mrr_at_100_max
value: 34.7628
- type: nauc_mrr_at_100_std
value: -11.2163
- type: nauc_mrr_at_100_diff1
value: 42.5973
- type: nauc_mrr_at_1000_max
value: 34.7522
- type: nauc_mrr_at_1000_std
value: -11.2227
- type: nauc_mrr_at_1000_diff1
value: 42.5959
- type: main_score
value: 56.114
task:
type: Retrieval
- dataset:
config: ar
name: MTEB MIRACLRetrievalHardNegatives (ar)
revision: 95c8db7d4a6e9c1d8a60601afd63d553ae20a2eb
split: dev
type: mteb/miracl-hard-negatives
metrics:
- type: ndcg_at_1
value: 48.699999999999996
- type: ndcg_at_3
value: 51.161
- type: ndcg_at_5
value: 53.923
- type: ndcg_at_10
value: 57.737
- type: ndcg_at_20
value: 60.475
- type: ndcg_at_100
value: 63.096
- type: ndcg_at_1000
value: 64.203
- type: map_at_1
value: 32.108
- type: map_at_3
value: 44.405
- type: map_at_5
value: 47.164
- type: map_at_10
value: 49.477
- type: map_at_20
value: 50.583999999999996
- type: map_at_100
value: 51.212999999999994
- type: map_at_1000
value: 51.281
- type: recall_at_1
value: 32.108
- type: recall_at_3
value: 52.675000000000004
- type: recall_at_5
value: 60.709
- type: recall_at_10
value: 70.61
- type: recall_at_20
value: 79.208
- type: recall_at_100
value: 89.805
- type: recall_at_1000
value: 96.935
- type: precision_at_1
value: 48.699999999999996
- type: precision_at_3
value: 29.7
- type: precision_at_5
value: 21.34
- type: precision_at_10
value: 12.98
- type: precision_at_20
value: 7.485
- type: precision_at_100
value: 1.772
- type: precision_at_1000
value: 0.193
- type: mrr_at_1
value: 48.699999999999996
- type: mrr_at_3
value: 57.5333
- type: mrr_at_5
value: 59.1333
- type: mrr_at_10
value: 60.1163
- type: mrr_at_20
value: 60.5298
- type: mrr_at_100
value: 60.691700000000004
- type: mrr_at_1000
value: 60.707699999999996
- type: nauc_ndcg_at_1_max
value: 28.267999999999997
- type: nauc_ndcg_at_1_std
value: -14.380299999999998
- type: nauc_ndcg_at_1_diff1
value: 46.6563
- type: nauc_ndcg_at_3_max
value: 30.7733
- type: nauc_ndcg_at_3_std
value: -18.2892
- type: nauc_ndcg_at_3_diff1
value: 42.4678
- type: nauc_ndcg_at_5_max
value: 31.014799999999997
- type: nauc_ndcg_at_5_std
value: -19.1886
- type: nauc_ndcg_at_5_diff1
value: 42.4011
- type: nauc_ndcg_at_10_max
value: 31.895400000000002
- type: nauc_ndcg_at_10_std
value: -17.1444
- type: nauc_ndcg_at_10_diff1
value: 41.7072
- type: nauc_ndcg_at_20_max
value: 33.373799999999996
- type: nauc_ndcg_at_20_std
value: -15.8958
- type: nauc_ndcg_at_20_diff1
value: 42.174800000000005
- type: nauc_ndcg_at_100_max
value: 33.3595
- type: nauc_ndcg_at_100_std
value: -13.684
- type: nauc_ndcg_at_100_diff1
value: 42.5765
- type: nauc_ndcg_at_1000_max
value: 33.306799999999996
- type: nauc_ndcg_at_1000_std
value: -13.309099999999999
- type: nauc_ndcg_at_1000_diff1
value: 42.7722
- type: nauc_map_at_1_max
value: 20.7316
- type: nauc_map_at_1_std
value: -19.7081
- type: nauc_map_at_1_diff1
value: 45.9673
- type: nauc_map_at_3_max
value: 27.112399999999997
- type: nauc_map_at_3_std
value: -21.3088
- type: nauc_map_at_3_diff1
value: 43.3808
- type: nauc_map_at_5_max
value: 28.2607
- type: nauc_map_at_5_std
value: -20.7577
- type: nauc_map_at_5_diff1
value: 42.7665
- type: nauc_map_at_10_max
value: 29.418899999999997
- type: nauc_map_at_10_std
value: -19.5259
- type: nauc_map_at_10_diff1
value: 42.277100000000004
- type: nauc_map_at_20_max
value: 30.0292
- type: nauc_map_at_20_std
value: -19.0805
- type: nauc_map_at_20_diff1
value: 42.5082
- type: nauc_map_at_100_max
value: 30.098599999999998
- type: nauc_map_at_100_std
value: -18.570600000000002
- type: nauc_map_at_100_diff1
value: 42.58
- type: nauc_map_at_1000_max
value: 30.0989
- type: nauc_map_at_1000_std
value: -18.543499999999998
- type: nauc_map_at_1000_diff1
value: 42.5925
- type: nauc_recall_at_1_max
value: 20.7316
- type: nauc_recall_at_1_std
value: -19.7081
- type: nauc_recall_at_1_diff1
value: 45.9673
- type: nauc_recall_at_3_max
value: 27.832600000000003
- type: nauc_recall_at_3_std
value: -20.2727
- type: nauc_recall_at_3_diff1
value: 37.7177
- type: nauc_recall_at_5_max
value: 29.0228
- type: nauc_recall_at_5_std
value: -21.0861
- type: nauc_recall_at_5_diff1
value: 35.5742
- type: nauc_recall_at_10_max
value: 29.876
- type: nauc_recall_at_10_std
value: -15.8439
- type: nauc_recall_at_10_diff1
value: 31.639499999999998
- type: nauc_recall_at_20_max
value: 36.5508
- type: nauc_recall_at_20_std
value: -10.3007
- type: nauc_recall_at_20_diff1
value: 31.490800000000004
- type: nauc_recall_at_100_max
value: 41.573
- type: nauc_recall_at_100_std
value: 10.3383
- type: nauc_recall_at_100_diff1
value: 32.034400000000005
- type: nauc_recall_at_1000_max
value: 61.4362
- type: nauc_recall_at_1000_std
value: 69.59440000000001
- type: nauc_recall_at_1000_diff1
value: 29.5155
- type: nauc_precision_at_1_max
value: 28.267999999999997
- type: nauc_precision_at_1_std
value: -14.380299999999998
- type: nauc_precision_at_1_diff1
value: 46.6563
- type: nauc_precision_at_3_max
value: 28.3698
- type: nauc_precision_at_3_std
value: -5.6800999999999995
- type: nauc_precision_at_3_diff1
value: 20.9306
- type: nauc_precision_at_5_max
value: 27.0061
- type: nauc_precision_at_5_std
value: -0.2876
- type: nauc_precision_at_5_diff1
value: 12.3067
- type: nauc_precision_at_10_max
value: 24.1662
- type: nauc_precision_at_10_std
value: 8.8292
- type: nauc_precision_at_10_diff1
value: 4.0011
- type: nauc_precision_at_20_max
value: 21.4451
- type: nauc_precision_at_20_std
value: 13.3651
- type: nauc_precision_at_20_diff1
value: -0.7437
- type: nauc_precision_at_100_max
value: 13.9599
- type: nauc_precision_at_100_std
value: 22.5858
- type: nauc_precision_at_100_diff1
value: -6.833
- type: nauc_precision_at_1000_max
value: 10.3447
- type: nauc_precision_at_1000_std
value: 25.1309
- type: nauc_precision_at_1000_diff1
value: -10.5
- type: nauc_mrr_at_1_max
value: 28.267999999999997
- type: nauc_mrr_at_1_std
value: -14.380299999999998
- type: nauc_mrr_at_1_diff1
value: 46.6563
- type: nauc_mrr_at_3_max
value: 33.0318
- type: nauc_mrr_at_3_std
value: -12.7141
- type: nauc_mrr_at_3_diff1
value: 44.562200000000004
- type: nauc_mrr_at_5_max
value: 33.292699999999996
- type: nauc_mrr_at_5_std
value: -13.118599999999999
- type: nauc_mrr_at_5_diff1
value: 44.8335
- type: nauc_mrr_at_10_max
value: 33.2046
- type: nauc_mrr_at_10_std
value: -12.4948
- type: nauc_mrr_at_10_diff1
value: 44.7418
- type: nauc_mrr_at_20_max
value: 33.2849
- type: nauc_mrr_at_20_std
value: -12.2826
- type: nauc_mrr_at_20_diff1
value: 44.7205
- type: nauc_mrr_at_100_max
value: 33.196799999999996
- type: nauc_mrr_at_100_std
value: -12.2722
- type: nauc_mrr_at_100_diff1
value: 44.804300000000005
- type: nauc_mrr_at_1000_max
value: 33.1847
- type: nauc_mrr_at_1000_std
value: -12.2764
- type: nauc_mrr_at_1000_diff1
value: 44.806400000000004
- type: main_score
value: 57.737
task:
type: Retrieval
- dataset:
config: ara-ara
name: MTEB MLQARetrieval (ara-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 47.776
- type: ndcg_at_3
value: 57.977999999999994
- type: ndcg_at_5
value: 60.275999999999996
- type: ndcg_at_10
value: 62.580000000000005
- type: ndcg_at_20
value: 63.857
- type: ndcg_at_100
value: 66.053
- type: ndcg_at_1000
value: 66.77
- type: map_at_1
value: 47.776
- type: map_at_3
value: 55.545
- type: map_at_5
value: 56.812
- type: map_at_10
value: 57.75599999999999
- type: map_at_20
value: 58.10999999999999
- type: map_at_100
value: 58.404999999999994
- type: map_at_1000
value: 58.440000000000005
- type: recall_at_1
value: 47.776
- type: recall_at_3
value: 64.99000000000001
- type: recall_at_5
value: 70.6
- type: recall_at_10
value: 77.756
- type: recall_at_20
value: 82.785
- type: recall_at_100
value: 94.77799999999999
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 47.776
- type: precision_at_3
value: 21.663
- type: precision_at_5
value: 14.12
- type: precision_at_10
value: 7.776
- type: precision_at_20
value: 4.139
- type: precision_at_100
value: 0.9480000000000001
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 47.775600000000004
- type: mrr_at_3
value: 55.5448
- type: mrr_at_5
value: 56.8117
- type: mrr_at_10
value: 57.7562
- type: mrr_at_20
value: 58.10999999999999
- type: mrr_at_100
value: 58.404900000000005
- type: mrr_at_1000
value: 58.4397
- type: nauc_ndcg_at_1_max
value: 68.0477
- type: nauc_ndcg_at_1_std
value: 19.1911
- type: nauc_ndcg_at_1_diff1
value: 73.80640000000001
- type: nauc_ndcg_at_3_max
value: 70.82159999999999
- type: nauc_ndcg_at_3_std
value: 19.7998
- type: nauc_ndcg_at_3_diff1
value: 68.7149
- type: nauc_ndcg_at_5_max
value: 70.812
- type: nauc_ndcg_at_5_std
value: 19.3612
- type: nauc_ndcg_at_5_diff1
value: 67.281
- type: nauc_ndcg_at_10_max
value: 71.02380000000001
- type: nauc_ndcg_at_10_std
value: 20.4318
- type: nauc_ndcg_at_10_diff1
value: 67.0146
- type: nauc_ndcg_at_20_max
value: 70.2041
- type: nauc_ndcg_at_20_std
value: 19.5577
- type: nauc_ndcg_at_20_diff1
value: 66.4613
- type: nauc_ndcg_at_100_max
value: 70.2647
- type: nauc_ndcg_at_100_std
value: 20.4363
- type: nauc_ndcg_at_100_diff1
value: 67.5021
- type: nauc_ndcg_at_1000_max
value: 70.1325
- type: nauc_ndcg_at_1000_std
value: 19.697
- type: nauc_ndcg_at_1000_diff1
value: 68.05290000000001
- type: nauc_map_at_1_max
value: 68.0477
- type: nauc_map_at_1_std
value: 19.1911
- type: nauc_map_at_1_diff1
value: 73.80640000000001
- type: nauc_map_at_3_max
value: 70.0727
- type: nauc_map_at_3_std
value: 19.4799
- type: nauc_map_at_3_diff1
value: 69.9485
- type: nauc_map_at_5_max
value: 70.015
- type: nauc_map_at_5_std
value: 19.2605
- type: nauc_map_at_5_diff1
value: 69.199
- type: nauc_map_at_10_max
value: 70.0897
- type: nauc_map_at_10_std
value: 19.6128
- type: nauc_map_at_10_diff1
value: 69.1356
- type: nauc_map_at_20_max
value: 69.8725
- type: nauc_map_at_20_std
value: 19.3681
- type: nauc_map_at_20_diff1
value: 69.0124
- type: nauc_map_at_100_max
value: 69.8656
- type: nauc_map_at_100_std
value: 19.4493
- type: nauc_map_at_100_diff1
value: 69.1296
- type: nauc_map_at_1000_max
value: 69.8608
- type: nauc_map_at_1000_std
value: 19.4191
- type: nauc_map_at_1000_diff1
value: 69.1547
- type: nauc_recall_at_1_max
value: 68.0477
- type: nauc_recall_at_1_std
value: 19.1911
- type: nauc_recall_at_1_diff1
value: 73.80640000000001
- type: nauc_recall_at_3_max
value: 73.2764
- type: nauc_recall_at_3_std
value: 20.9191
- type: nauc_recall_at_3_diff1
value: 64.7349
- type: nauc_recall_at_5_max
value: 73.7417
- type: nauc_recall_at_5_std
value: 19.6995
- type: nauc_recall_at_5_diff1
value: 60.2181
- type: nauc_recall_at_10_max
value: 75.31
- type: nauc_recall_at_10_std
value: 24.833199999999998
- type: nauc_recall_at_10_diff1
value: 57.29729999999999
- type: nauc_recall_at_20_max
value: 70.9915
- type: nauc_recall_at_20_std
value: 20.3983
- type: nauc_recall_at_20_diff1
value: 51.2804
- type: nauc_recall_at_100_max
value: 75.0448
- type: nauc_recall_at_100_std
value: 46.0233
- type: nauc_recall_at_100_diff1
value: 48.8265
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 68.0477
- type: nauc_precision_at_1_std
value: 19.1911
- type: nauc_precision_at_1_diff1
value: 73.80640000000001
- type: nauc_precision_at_3_max
value: 73.2764
- type: nauc_precision_at_3_std
value: 20.9191
- type: nauc_precision_at_3_diff1
value: 64.7349
- type: nauc_precision_at_5_max
value: 73.7417
- type: nauc_precision_at_5_std
value: 19.6995
- type: nauc_precision_at_5_diff1
value: 60.2181
- type: nauc_precision_at_10_max
value: 75.31
- type: nauc_precision_at_10_std
value: 24.833199999999998
- type: nauc_precision_at_10_diff1
value: 57.29729999999999
- type: nauc_precision_at_20_max
value: 70.9915
- type: nauc_precision_at_20_std
value: 20.3983
- type: nauc_precision_at_20_diff1
value: 51.2804
- type: nauc_precision_at_100_max
value: 75.0448
- type: nauc_precision_at_100_std
value: 46.0233
- type: nauc_precision_at_100_diff1
value: 48.8265
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 68.0477
- type: nauc_mrr_at_1_std
value: 19.1911
- type: nauc_mrr_at_1_diff1
value: 73.80640000000001
- type: nauc_mrr_at_3_max
value: 70.0727
- type: nauc_mrr_at_3_std
value: 19.4799
- type: nauc_mrr_at_3_diff1
value: 69.9485
- type: nauc_mrr_at_5_max
value: 70.015
- type: nauc_mrr_at_5_std
value: 19.2605
- type: nauc_mrr_at_5_diff1
value: 69.199
- type: nauc_mrr_at_10_max
value: 70.0897
- type: nauc_mrr_at_10_std
value: 19.6128
- type: nauc_mrr_at_10_diff1
value: 69.1356
- type: nauc_mrr_at_20_max
value: 69.8725
- type: nauc_mrr_at_20_std
value: 19.3681
- type: nauc_mrr_at_20_diff1
value: 69.0124
- type: nauc_mrr_at_100_max
value: 69.8656
- type: nauc_mrr_at_100_std
value: 19.4493
- type: nauc_mrr_at_100_diff1
value: 69.1296
- type: nauc_mrr_at_1000_max
value: 69.8608
- type: nauc_mrr_at_1000_std
value: 19.4191
- type: nauc_mrr_at_1000_diff1
value: 69.1547
- type: main_score
value: 62.580000000000005
task:
type: Retrieval
- dataset:
config: ara-deu
name: MTEB MLQARetrieval (ara-deu)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 5.314
- type: ndcg_at_3
value: 8.046000000000001
- type: ndcg_at_5
value: 9.417
- type: ndcg_at_10
value: 11.569
- type: ndcg_at_20
value: 13.972999999999999
- type: ndcg_at_100
value: 22.402
- type: ndcg_at_1000
value: 25.611
- type: map_at_1
value: 5.314
- type: map_at_3
value: 7.327
- type: map_at_5
value: 8.076
- type: map_at_10
value: 8.943
- type: map_at_20
value: 9.58
- type: map_at_100
value: 10.629
- type: map_at_1000
value: 10.799
- type: recall_at_1
value: 5.314
- type: recall_at_3
value: 10.145
- type: recall_at_5
value: 13.527000000000001
- type: recall_at_10
value: 20.29
- type: recall_at_20
value: 29.952
- type: recall_at_100
value: 77.295
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 5.314
- type: precision_at_3
value: 3.382
- type: precision_at_5
value: 2.705
- type: precision_at_10
value: 2.029
- type: precision_at_20
value: 1.498
- type: precision_at_100
value: 0.773
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 5.314
- type: mrr_at_3
value: 7.3269
- type: mrr_at_5
value: 8.0757
- type: mrr_at_10
value: 8.9428
- type: mrr_at_20
value: 9.5802
- type: mrr_at_100
value: 10.629
- type: mrr_at_1000
value: 10.7994
- type: nauc_ndcg_at_1_max
value: -5.6322
- type: nauc_ndcg_at_1_std
value: 14.3928
- type: nauc_ndcg_at_1_diff1
value: 17.617
- type: nauc_ndcg_at_3_max
value: -0.6146
- type: nauc_ndcg_at_3_std
value: 14.7604
- type: nauc_ndcg_at_3_diff1
value: 12.4519
- type: nauc_ndcg_at_5_max
value: -5.6473
- type: nauc_ndcg_at_5_std
value: 15.792700000000002
- type: nauc_ndcg_at_5_diff1
value: 9.0822
- type: nauc_ndcg_at_10_max
value: -4.7285
- type: nauc_ndcg_at_10_std
value: 13.0276
- type: nauc_ndcg_at_10_diff1
value: 5.8493
- type: nauc_ndcg_at_20_max
value: -4.6022
- type: nauc_ndcg_at_20_std
value: 12.251900000000001
- type: nauc_ndcg_at_20_diff1
value: 6.0303
- type: nauc_ndcg_at_100_max
value: -5.0885
- type: nauc_ndcg_at_100_std
value: 11.4246
- type: nauc_ndcg_at_100_diff1
value: 9.207799999999999
- type: nauc_ndcg_at_1000_max
value: -5.2688
- type: nauc_ndcg_at_1000_std
value: 13.525699999999999
- type: nauc_ndcg_at_1000_diff1
value: 7.7085
- type: nauc_map_at_1_max
value: -5.6322
- type: nauc_map_at_1_std
value: 14.3928
- type: nauc_map_at_1_diff1
value: 17.617
- type: nauc_map_at_3_max
value: -2.0108
- type: nauc_map_at_3_std
value: 14.7763
- type: nauc_map_at_3_diff1
value: 13.675399999999998
- type: nauc_map_at_5_max
value: -5.1197
- type: nauc_map_at_5_std
value: 15.4899
- type: nauc_map_at_5_diff1
value: 11.6334
- type: nauc_map_at_10_max
value: -4.654599999999999
- type: nauc_map_at_10_std
value: 14.1773
- type: nauc_map_at_10_diff1
value: 10.0021
- type: nauc_map_at_20_max
value: -4.7692
- type: nauc_map_at_20_std
value: 13.8542
- type: nauc_map_at_20_diff1
value: 9.8728
- type: nauc_map_at_100_max
value: -5.025799999999999
- type: nauc_map_at_100_std
value: 13.9741
- type: nauc_map_at_100_diff1
value: 9.909600000000001
- type: nauc_map_at_1000_max
value: -5.037599999999999
- type: nauc_map_at_1000_std
value: 14.094999999999999
- type: nauc_map_at_1000_diff1
value: 9.8368
- type: nauc_recall_at_1_max
value: -5.6322
- type: nauc_recall_at_1_std
value: 14.3928
- type: nauc_recall_at_1_diff1
value: 17.617
- type: nauc_recall_at_3_max
value: 2.4502
- type: nauc_recall_at_3_std
value: 14.7034
- type: nauc_recall_at_3_diff1
value: 9.8065
- type: nauc_recall_at_5_max
value: -6.9356
- type: nauc_recall_at_5_std
value: 16.371
- type: nauc_recall_at_5_diff1
value: 3.9127
- type: nauc_recall_at_10_max
value: -4.9281
- type: nauc_recall_at_10_std
value: 10.7352
- type: nauc_recall_at_10_diff1
value: -1.4319
- type: nauc_recall_at_20_max
value: -4.0954
- type: nauc_recall_at_20_std
value: 9.451
- type: nauc_recall_at_20_diff1
value: 0.42310000000000003
- type: nauc_recall_at_100_max
value: -4.2747
- type: nauc_recall_at_100_std
value: 0.8066
- type: nauc_recall_at_100_diff1
value: 16.4755
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: -5.6322
- type: nauc_precision_at_1_std
value: 14.3928
- type: nauc_precision_at_1_diff1
value: 17.617
- type: nauc_precision_at_3_max
value: 2.4502
- type: nauc_precision_at_3_std
value: 14.7034
- type: nauc_precision_at_3_diff1
value: 9.8065
- type: nauc_precision_at_5_max
value: -6.9356
- type: nauc_precision_at_5_std
value: 16.371
- type: nauc_precision_at_5_diff1
value: 3.9127
- type: nauc_precision_at_10_max
value: -4.9281
- type: nauc_precision_at_10_std
value: 10.7352
- type: nauc_precision_at_10_diff1
value: -1.4319
- type: nauc_precision_at_20_max
value: -4.0954
- type: nauc_precision_at_20_std
value: 9.451
- type: nauc_precision_at_20_diff1
value: 0.42310000000000003
- type: nauc_precision_at_100_max
value: -4.2747
- type: nauc_precision_at_100_std
value: 0.8066
- type: nauc_precision_at_100_diff1
value: 16.4755
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: -5.6322
- type: nauc_mrr_at_1_std
value: 14.3928
- type: nauc_mrr_at_1_diff1
value: 17.617
- type: nauc_mrr_at_3_max
value: -2.0108
- type: nauc_mrr_at_3_std
value: 14.7763
- type: nauc_mrr_at_3_diff1
value: 13.675399999999998
- type: nauc_mrr_at_5_max
value: -5.1197
- type: nauc_mrr_at_5_std
value: 15.4899
- type: nauc_mrr_at_5_diff1
value: 11.6334
- type: nauc_mrr_at_10_max
value: -4.654599999999999
- type: nauc_mrr_at_10_std
value: 14.1773
- type: nauc_mrr_at_10_diff1
value: 10.0021
- type: nauc_mrr_at_20_max
value: -4.7692
- type: nauc_mrr_at_20_std
value: 13.8542
- type: nauc_mrr_at_20_diff1
value: 9.8728
- type: nauc_mrr_at_100_max
value: -5.025799999999999
- type: nauc_mrr_at_100_std
value: 13.9741
- type: nauc_mrr_at_100_diff1
value: 9.909600000000001
- type: nauc_mrr_at_1000_max
value: -5.037599999999999
- type: nauc_mrr_at_1000_std
value: 14.094999999999999
- type: nauc_mrr_at_1000_diff1
value: 9.8368
- type: main_score
value: 11.569
task:
type: Retrieval
- dataset:
config: ara-eng
name: MTEB MLQARetrieval (ara-eng)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 8.124
- type: ndcg_at_3
value: 11.898
- type: ndcg_at_5
value: 13.645999999999999
- type: ndcg_at_10
value: 15.777
- type: ndcg_at_20
value: 17.777
- type: ndcg_at_100
value: 22.487
- type: ndcg_at_1000
value: 27.674
- type: map_at_1
value: 8.124
- type: map_at_3
value: 10.993
- type: map_at_5
value: 11.959999999999999
- type: map_at_10
value: 12.842
- type: map_at_20
value: 13.363
- type: map_at_100
value: 13.938999999999998
- type: map_at_1000
value: 14.141
- type: recall_at_1
value: 8.124
- type: recall_at_3
value: 14.507
- type: recall_at_5
value: 18.762
- type: recall_at_10
value: 25.338
- type: recall_at_20
value: 33.461999999999996
- type: recall_at_100
value: 59.961
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 8.124
- type: precision_at_3
value: 4.836
- type: precision_at_5
value: 3.752
- type: precision_at_10
value: 2.5340000000000003
- type: precision_at_20
value: 1.6729999999999998
- type: precision_at_100
value: 0.6
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 8.123800000000001
- type: mrr_at_3
value: 10.9929
- type: mrr_at_5
value: 11.959999999999999
- type: mrr_at_10
value: 12.8419
- type: mrr_at_20
value: 13.362499999999999
- type: mrr_at_100
value: 13.939099999999998
- type: mrr_at_1000
value: 14.1407
- type: nauc_ndcg_at_1_max
value: 39.6207
- type: nauc_ndcg_at_1_std
value: 10.587100000000001
- type: nauc_ndcg_at_1_diff1
value: 46.9816
- type: nauc_ndcg_at_3_max
value: 38.9332
- type: nauc_ndcg_at_3_std
value: 8.0092
- type: nauc_ndcg_at_3_diff1
value: 36.0903
- type: nauc_ndcg_at_5_max
value: 36.3763
- type: nauc_ndcg_at_5_std
value: 10.8615
- type: nauc_ndcg_at_5_diff1
value: 31.7054
- type: nauc_ndcg_at_10_max
value: 32.4079
- type: nauc_ndcg_at_10_std
value: 8.9961
- type: nauc_ndcg_at_10_diff1
value: 30.3114
- type: nauc_ndcg_at_20_max
value: 29.9964
- type: nauc_ndcg_at_20_std
value: 8.7158
- type: nauc_ndcg_at_20_diff1
value: 27.788400000000003
- type: nauc_ndcg_at_100_max
value: 30.5211
- type: nauc_ndcg_at_100_std
value: 10.2928
- type: nauc_ndcg_at_100_diff1
value: 27.854200000000002
- type: nauc_ndcg_at_1000_max
value: 32.3928
- type: nauc_ndcg_at_1000_std
value: 9.5312
- type: nauc_ndcg_at_1000_diff1
value: 30.3352
- type: nauc_map_at_1_max
value: 39.6207
- type: nauc_map_at_1_std
value: 10.587100000000001
- type: nauc_map_at_1_diff1
value: 46.9816
- type: nauc_map_at_3_max
value: 39.0509
- type: nauc_map_at_3_std
value: 8.551599999999999
- type: nauc_map_at_3_diff1
value: 38.0734
- type: nauc_map_at_5_max
value: 37.4992
- type: nauc_map_at_5_std
value: 10.2897
- type: nauc_map_at_5_diff1
value: 35.309200000000004
- type: nauc_map_at_10_max
value: 35.4801
- type: nauc_map_at_10_std
value: 9.2756
- type: nauc_map_at_10_diff1
value: 34.4628
- type: nauc_map_at_20_max
value: 34.6145
- type: nauc_map_at_20_std
value: 9.085
- type: nauc_map_at_20_diff1
value: 33.5934
- type: nauc_map_at_100_max
value: 34.635
- type: nauc_map_at_100_std
value: 9.4218
- type: nauc_map_at_100_diff1
value: 33.452799999999996
- type: nauc_map_at_1000_max
value: 34.7368
- type: nauc_map_at_1000_std
value: 9.374
- type: nauc_map_at_1000_diff1
value: 33.604099999999995
- type: nauc_recall_at_1_max
value: 39.6207
- type: nauc_recall_at_1_std
value: 10.587100000000001
- type: nauc_recall_at_1_diff1
value: 46.9816
- type: nauc_recall_at_3_max
value: 38.667
- type: nauc_recall_at_3_std
value: 6.7456000000000005
- type: nauc_recall_at_3_diff1
value: 31.561899999999998
- type: nauc_recall_at_5_max
value: 33.8474
- type: nauc_recall_at_5_std
value: 12.266499999999999
- type: nauc_recall_at_5_diff1
value: 24.0205
- type: nauc_recall_at_10_max
value: 25.8202
- type: nauc_recall_at_10_std
value: 8.4748
- type: nauc_recall_at_10_diff1
value: 22.2481
- type: nauc_recall_at_20_max
value: 20.2769
- type: nauc_recall_at_20_std
value: 8.2121
- type: nauc_recall_at_20_diff1
value: 16.342599999999997
- type: nauc_recall_at_100_max
value: 22.013199999999998
- type: nauc_recall_at_100_std
value: 13.5687
- type: nauc_recall_at_100_diff1
value: 16.8797
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 39.6207
- type: nauc_precision_at_1_std
value: 10.587100000000001
- type: nauc_precision_at_1_diff1
value: 46.9816
- type: nauc_precision_at_3_max
value: 38.667
- type: nauc_precision_at_3_std
value: 6.7456000000000005
- type: nauc_precision_at_3_diff1
value: 31.561899999999998
- type: nauc_precision_at_5_max
value: 33.8474
- type: nauc_precision_at_5_std
value: 12.266499999999999
- type: nauc_precision_at_5_diff1
value: 24.0205
- type: nauc_precision_at_10_max
value: 25.8202
- type: nauc_precision_at_10_std
value: 8.4748
- type: nauc_precision_at_10_diff1
value: 22.2481
- type: nauc_precision_at_20_max
value: 20.2769
- type: nauc_precision_at_20_std
value: 8.2121
- type: nauc_precision_at_20_diff1
value: 16.342599999999997
- type: nauc_precision_at_100_max
value: 22.013199999999998
- type: nauc_precision_at_100_std
value: 13.5687
- type: nauc_precision_at_100_diff1
value: 16.8797
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 39.6207
- type: nauc_mrr_at_1_std
value: 10.587100000000001
- type: nauc_mrr_at_1_diff1
value: 46.9816
- type: nauc_mrr_at_3_max
value: 39.0509
- type: nauc_mrr_at_3_std
value: 8.551599999999999
- type: nauc_mrr_at_3_diff1
value: 38.0734
- type: nauc_mrr_at_5_max
value: 37.4992
- type: nauc_mrr_at_5_std
value: 10.2897
- type: nauc_mrr_at_5_diff1
value: 35.309200000000004
- type: nauc_mrr_at_10_max
value: 35.4801
- type: nauc_mrr_at_10_std
value: 9.2756
- type: nauc_mrr_at_10_diff1
value: 34.4628
- type: nauc_mrr_at_20_max
value: 34.6145
- type: nauc_mrr_at_20_std
value: 9.085
- type: nauc_mrr_at_20_diff1
value: 33.5934
- type: nauc_mrr_at_100_max
value: 34.635
- type: nauc_mrr_at_100_std
value: 9.4218
- type: nauc_mrr_at_100_diff1
value: 33.452799999999996
- type: nauc_mrr_at_1000_max
value: 34.7368
- type: nauc_mrr_at_1000_std
value: 9.374
- type: nauc_mrr_at_1000_diff1
value: 33.604099999999995
- type: main_score
value: 15.777
task:
type: Retrieval
- dataset:
config: ara-spa
name: MTEB MLQARetrieval (ara-spa)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 4.348
- type: ndcg_at_3
value: 6.3740000000000006
- type: ndcg_at_5
value: 9.421
- type: ndcg_at_10
value: 11.355
- type: ndcg_at_20
value: 12.812000000000001
- type: ndcg_at_100
value: 22.061
- type: ndcg_at_1000
value: 25.048
- type: map_at_1
value: 4.348
- type: map_at_3
value: 5.797
- type: map_at_5
value: 7.474
- type: map_at_10
value: 8.229000000000001
- type: map_at_20
value: 8.652
- type: map_at_100
value: 9.780999999999999
- type: map_at_1000
value: 9.959999999999999
- type: recall_at_1
value: 4.348
- type: recall_at_3
value: 8.075000000000001
- type: recall_at_5
value: 15.528
- type: recall_at_10
value: 21.739
- type: recall_at_20
value: 27.328999999999997
- type: recall_at_100
value: 79.503
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 4.348
- type: precision_at_3
value: 2.692
- type: precision_at_5
value: 3.106
- type: precision_at_10
value: 2.174
- type: precision_at_20
value: 1.366
- type: precision_at_100
value: 0.795
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 4.3478
- type: mrr_at_3
value: 5.7971
- type: mrr_at_5
value: 7.4741
- type: mrr_at_10
value: 8.2286
- type: mrr_at_20
value: 8.652
- type: mrr_at_100
value: 9.7813
- type: mrr_at_1000
value: 9.9596
- type: nauc_ndcg_at_1_max
value: 23.7695
- type: nauc_ndcg_at_1_std
value: 22.4188
- type: nauc_ndcg_at_1_diff1
value: 2.3605
- type: nauc_ndcg_at_3_max
value: 13.736699999999999
- type: nauc_ndcg_at_3_std
value: 15.007200000000001
- type: nauc_ndcg_at_3_diff1
value: 15.437100000000001
- type: nauc_ndcg_at_5_max
value: 17.2831
- type: nauc_ndcg_at_5_std
value: 23.1015
- type: nauc_ndcg_at_5_diff1
value: 11.3163
- type: nauc_ndcg_at_10_max
value: 10.5633
- type: nauc_ndcg_at_10_std
value: 16.951
- type: nauc_ndcg_at_10_diff1
value: 6.934800000000001
- type: nauc_ndcg_at_20_max
value: 8.6129
- type: nauc_ndcg_at_20_std
value: 19.6507
- type: nauc_ndcg_at_20_diff1
value: 2.3595
- type: nauc_ndcg_at_100_max
value: 2.0564999999999998
- type: nauc_ndcg_at_100_std
value: 16.188
- type: nauc_ndcg_at_100_diff1
value: -4.991099999999999
- type: nauc_ndcg_at_1000_max
value: 7.854500000000001
- type: nauc_ndcg_at_1000_std
value: 18.9095
- type: nauc_ndcg_at_1000_diff1
value: 2.1982999999999997
- type: nauc_map_at_1_max
value: 23.7695
- type: nauc_map_at_1_std
value: 22.4188
- type: nauc_map_at_1_diff1
value: 2.3605
- type: nauc_map_at_3_max
value: 15.101999999999999
- type: nauc_map_at_3_std
value: 16.057299999999998
- type: nauc_map_at_3_diff1
value: 12.2628
- type: nauc_map_at_5_max
value: 16.9873
- type: nauc_map_at_5_std
value: 21.330299999999998
- type: nauc_map_at_5_diff1
value: 10.061
- type: nauc_map_at_10_max
value: 13.5878
- type: nauc_map_at_10_std
value: 18.4412
- type: nauc_map_at_10_diff1
value: 7.7329
- type: nauc_map_at_20_max
value: 12.5743
- type: nauc_map_at_20_std
value: 19.3729
- type: nauc_map_at_20_diff1
value: 5.8843
- type: nauc_map_at_100_max
value: 10.9648
- type: nauc_map_at_100_std
value: 19.0021
- type: nauc_map_at_100_diff1
value: 4.3651
- type: nauc_map_at_1000_max
value: 11.4204
- type: nauc_map_at_1000_std
value: 19.2076
- type: nauc_map_at_1000_diff1
value: 4.925800000000001
- type: nauc_recall_at_1_max
value: 23.7695
- type: nauc_recall_at_1_std
value: 22.4188
- type: nauc_recall_at_1_diff1
value: 2.3605
- type: nauc_recall_at_3_max
value: 11.0412
- type: nauc_recall_at_3_std
value: 12.9179
- type: nauc_recall_at_3_diff1
value: 22.253600000000002
- type: nauc_recall_at_5_max
value: 18.215700000000002
- type: nauc_recall_at_5_std
value: 26.349600000000002
- type: nauc_recall_at_5_diff1
value: 13.0495
- type: nauc_recall_at_10_max
value: 5.7437000000000005
- type: nauc_recall_at_10_std
value: 14.1158
- type: nauc_recall_at_10_diff1
value: 5.1089
- type: nauc_recall_at_20_max
value: 2.763
- type: nauc_recall_at_20_std
value: 20.291600000000003
- type: nauc_recall_at_20_diff1
value: -4.0575
- type: nauc_recall_at_100_max
value: -25.1645
- type: nauc_recall_at_100_std
value: 3.5965
- type: nauc_recall_at_100_diff1
value: -38.3844
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 23.7695
- type: nauc_precision_at_1_std
value: 22.4188
- type: nauc_precision_at_1_diff1
value: 2.3605
- type: nauc_precision_at_3_max
value: 11.0412
- type: nauc_precision_at_3_std
value: 12.9179
- type: nauc_precision_at_3_diff1
value: 22.253600000000002
- type: nauc_precision_at_5_max
value: 18.215700000000002
- type: nauc_precision_at_5_std
value: 26.349600000000002
- type: nauc_precision_at_5_diff1
value: 13.0495
- type: nauc_precision_at_10_max
value: 5.7437000000000005
- type: nauc_precision_at_10_std
value: 14.1158
- type: nauc_precision_at_10_diff1
value: 5.1089
- type: nauc_precision_at_20_max
value: 2.763
- type: nauc_precision_at_20_std
value: 20.291600000000003
- type: nauc_precision_at_20_diff1
value: -4.0575
- type: nauc_precision_at_100_max
value: -25.1645
- type: nauc_precision_at_100_std
value: 3.5965
- type: nauc_precision_at_100_diff1
value: -38.3844
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 23.7695
- type: nauc_mrr_at_1_std
value: 22.4188
- type: nauc_mrr_at_1_diff1
value: 2.3605
- type: nauc_mrr_at_3_max
value: 15.101999999999999
- type: nauc_mrr_at_3_std
value: 16.057299999999998
- type: nauc_mrr_at_3_diff1
value: 12.2628
- type: nauc_mrr_at_5_max
value: 16.9873
- type: nauc_mrr_at_5_std
value: 21.330299999999998
- type: nauc_mrr_at_5_diff1
value: 10.061
- type: nauc_mrr_at_10_max
value: 13.5878
- type: nauc_mrr_at_10_std
value: 18.4412
- type: nauc_mrr_at_10_diff1
value: 7.7329
- type: nauc_mrr_at_20_max
value: 12.5743
- type: nauc_mrr_at_20_std
value: 19.3729
- type: nauc_mrr_at_20_diff1
value: 5.8843
- type: nauc_mrr_at_100_max
value: 10.9648
- type: nauc_mrr_at_100_std
value: 19.0021
- type: nauc_mrr_at_100_diff1
value: 4.3651
- type: nauc_mrr_at_1000_max
value: 11.4204
- type: nauc_mrr_at_1000_std
value: 19.2076
- type: nauc_mrr_at_1000_diff1
value: 4.925800000000001
- type: main_score
value: 11.355
task:
type: Retrieval
- dataset:
config: ara-hin
name: MTEB MLQARetrieval (ara-hin)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 2.1510000000000002
- type: ndcg_at_3
value: 3.7060000000000004
- type: ndcg_at_5
value: 4.353
- type: ndcg_at_10
value: 5.380999999999999
- type: ndcg_at_20
value: 7.364
- type: ndcg_at_100
value: 16.3
- type: ndcg_at_1000
value: 20.613999999999997
- type: map_at_1
value: 2.1510000000000002
- type: map_at_3
value: 3.315
- type: map_at_5
value: 3.665
- type: map_at_10
value: 4.079
- type: map_at_20
value: 4.596
- type: map_at_100
value: 5.575
- type: map_at_1000
value: 5.817
- type: recall_at_1
value: 2.1510000000000002
- type: recall_at_3
value: 4.839
- type: recall_at_5
value: 6.451999999999999
- type: recall_at_10
value: 9.677
- type: recall_at_20
value: 17.742
- type: recall_at_100
value: 69.892
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 2.1510000000000002
- type: precision_at_3
value: 1.6129999999999998
- type: precision_at_5
value: 1.29
- type: precision_at_10
value: 0.968
- type: precision_at_20
value: 0.8869999999999999
- type: precision_at_100
value: 0.699
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 2.1505
- type: mrr_at_3
value: 3.3154000000000003
- type: mrr_at_5
value: 3.6649000000000003
- type: mrr_at_10
value: 4.0792
- type: mrr_at_20
value: 4.5958
- type: mrr_at_100
value: 5.5755
- type: mrr_at_1000
value: 5.8172
- type: nauc_ndcg_at_1_max
value: 10.692599999999999
- type: nauc_ndcg_at_1_std
value: 52.3128
- type: nauc_ndcg_at_1_diff1
value: 12.790099999999999
- type: nauc_ndcg_at_3_max
value: 1.9623000000000002
- type: nauc_ndcg_at_3_std
value: 45.4097
- type: nauc_ndcg_at_3_diff1
value: -7.286
- type: nauc_ndcg_at_5_max
value: -1.4385000000000001
- type: nauc_ndcg_at_5_std
value: 34.5298
- type: nauc_ndcg_at_5_diff1
value: -8.7098
- type: nauc_ndcg_at_10_max
value: -5.5568
- type: nauc_ndcg_at_10_std
value: 30.694
- type: nauc_ndcg_at_10_diff1
value: -13.6821
- type: nauc_ndcg_at_20_max
value: -1.7629
- type: nauc_ndcg_at_20_std
value: 26.1943
- type: nauc_ndcg_at_20_diff1
value: -10.215100000000001
- type: nauc_ndcg_at_100_max
value: -1.482
- type: nauc_ndcg_at_100_std
value: 19.0141
- type: nauc_ndcg_at_100_diff1
value: -6.4361
- type: nauc_ndcg_at_1000_max
value: 0.0408
- type: nauc_ndcg_at_1000_std
value: 28.2621
- type: nauc_ndcg_at_1000_diff1
value: -6.6709000000000005
- type: nauc_map_at_1_max
value: 10.692599999999999
- type: nauc_map_at_1_std
value: 52.3128
- type: nauc_map_at_1_diff1
value: 12.790099999999999
- type: nauc_map_at_3_max
value: 2.7468
- type: nauc_map_at_3_std
value: 46.3179
- type: nauc_map_at_3_diff1
value: -3.8164
- type: nauc_map_at_5_max
value: 0.37429999999999997
- type: nauc_map_at_5_std
value: 39.135799999999996
- type: nauc_map_at_5_diff1
value: -5.2242
- type: nauc_map_at_10_max
value: -1.9640000000000002
- type: nauc_map_at_10_std
value: 36.955
- type: nauc_map_at_10_diff1
value: -8.151
- type: nauc_map_at_20_max
value: -0.1559
- type: nauc_map_at_20_std
value: 35.2312
- type: nauc_map_at_20_diff1
value: -6.7158999999999995
- type: nauc_map_at_100_max
value: 0.19499999999999998
- type: nauc_map_at_100_std
value: 33.1808
- type: nauc_map_at_100_diff1
value: -5.7646999999999995
- type: nauc_map_at_1000_max
value: 0.3463
- type: nauc_map_at_1000_std
value: 34.1502
- type: nauc_map_at_1000_diff1
value: -5.75
- type: nauc_recall_at_1_max
value: 10.692599999999999
- type: nauc_recall_at_1_std
value: 52.3128
- type: nauc_recall_at_1_diff1
value: 12.790099999999999
- type: nauc_recall_at_3_max
value: 0.5755
- type: nauc_recall_at_3_std
value: 43.6627
- type: nauc_recall_at_3_diff1
value: -14.2396
- type: nauc_recall_at_5_max
value: -4.5171
- type: nauc_recall_at_5_std
value: 26.0963
- type: nauc_recall_at_5_diff1
value: -14.508
- type: nauc_recall_at_10_max
value: -10.7825
- type: nauc_recall_at_10_std
value: 21.4919
- type: nauc_recall_at_10_diff1
value: -21.434
- type: nauc_recall_at_20_max
value: -3.3461
- type: nauc_recall_at_20_std
value: 15.4951
- type: nauc_recall_at_20_diff1
value: -13.439699999999998
- type: nauc_recall_at_100_max
value: -4.7297
- type: nauc_recall_at_100_std
value: -1.4209
- type: nauc_recall_at_100_diff1
value: -5.7624
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 10.692599999999999
- type: nauc_precision_at_1_std
value: 52.3128
- type: nauc_precision_at_1_diff1
value: 12.790099999999999
- type: nauc_precision_at_3_max
value: 0.5755
- type: nauc_precision_at_3_std
value: 43.6627
- type: nauc_precision_at_3_diff1
value: -14.2396
- type: nauc_precision_at_5_max
value: -4.5171
- type: nauc_precision_at_5_std
value: 26.0963
- type: nauc_precision_at_5_diff1
value: -14.508
- type: nauc_precision_at_10_max
value: -10.7825
- type: nauc_precision_at_10_std
value: 21.4919
- type: nauc_precision_at_10_diff1
value: -21.434
- type: nauc_precision_at_20_max
value: -3.3461
- type: nauc_precision_at_20_std
value: 15.4951
- type: nauc_precision_at_20_diff1
value: -13.439699999999998
- type: nauc_precision_at_100_max
value: -4.7297
- type: nauc_precision_at_100_std
value: -1.4209
- type: nauc_precision_at_100_diff1
value: -5.7624
- type: nauc_precision_at_1000_max
value: .nan
- type: nauc_precision_at_1000_std
value: .nan
- type: nauc_precision_at_1000_diff1
value: .nan
- type: nauc_mrr_at_1_max
value: 10.692599999999999
- type: nauc_mrr_at_1_std
value: 52.3128
- type: nauc_mrr_at_1_diff1
value: 12.790099999999999
- type: nauc_mrr_at_3_max
value: 2.7468
- type: nauc_mrr_at_3_std
value: 46.3179
- type: nauc_mrr_at_3_diff1
value: -3.8164
- type: nauc_mrr_at_5_max
value: 0.37429999999999997
- type: nauc_mrr_at_5_std
value: 39.135799999999996
- type: nauc_mrr_at_5_diff1
value: -5.2242
- type: nauc_mrr_at_10_max
value: -1.9640000000000002
- type: nauc_mrr_at_10_std
value: 36.955
- type: nauc_mrr_at_10_diff1
value: -8.151
- type: nauc_mrr_at_20_max
value: -0.1559
- type: nauc_mrr_at_20_std
value: 35.2312
- type: nauc_mrr_at_20_diff1
value: -6.7158999999999995
- type: nauc_mrr_at_100_max
value: 0.19499999999999998
- type: nauc_mrr_at_100_std
value: 33.1808
- type: nauc_mrr_at_100_diff1
value: -5.7646999999999995
- type: nauc_mrr_at_1000_max
value: 0.3463
- type: nauc_mrr_at_1000_std
value: 34.1502
- type: nauc_mrr_at_1000_diff1
value: -5.75
- type: main_score
value: 5.380999999999999
task:
type: Retrieval
- dataset:
config: ara-vie
name: MTEB MLQARetrieval (ara-vie)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 7.9750000000000005
- type: ndcg_at_3
value: 12.445
- type: ndcg_at_5
value: 13.475000000000001
- type: ndcg_at_10
value: 15.805
- type: ndcg_at_20
value: 18.428
- type: ndcg_at_100
value: 25.127
- type: ndcg_at_1000
value: 28.828
- type: map_at_1
value: 7.9750000000000005
- type: map_at_3
value: 11.247
- type: map_at_5
value: 11.83
- type: map_at_10
value: 12.763
- type: map_at_20
value: 13.475000000000001
- type: map_at_100
value: 14.280000000000001
- type: map_at_1000
value: 14.488999999999999
- type: recall_at_1
value: 7.9750000000000005
- type: recall_at_3
value: 15.951
- type: recall_at_5
value: 18.404999999999998
- type: recall_at_10
value: 25.767
- type: recall_at_20
value: 36.196
- type: recall_at_100
value: 74.233
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 7.9750000000000005
- type: precision_at_3
value: 5.317
- type: precision_at_5
value: 3.681
- type: precision_at_10
value: 2.577
- type: precision_at_20
value: 1.81
- type: precision_at_100
value: 0.742
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 7.9755
- type: mrr_at_3
value: 11.2474
- type: mrr_at_5
value: 11.830300000000001
- type: mrr_at_10
value: 12.762899999999998
- type: mrr_at_20
value: 13.4747
- type: mrr_at_100
value: 14.2797
- type: mrr_at_1000
value: 14.4888
- type: nauc_ndcg_at_1_max
value: 40.468399999999995
- type: nauc_ndcg_at_1_std
value: 38.8093
- type: nauc_ndcg_at_1_diff1
value: 32.3963
- type: nauc_ndcg_at_3_max
value: 42.1969
- type: nauc_ndcg_at_3_std
value: 36.5489
- type: nauc_ndcg_at_3_diff1
value: 13.6605
- type: nauc_ndcg_at_5_max
value: 42.2313
- type: nauc_ndcg_at_5_std
value: 36.6997
- type: nauc_ndcg_at_5_diff1
value: 15.2503
- type: nauc_ndcg_at_10_max
value: 38.022099999999995
- type: nauc_ndcg_at_10_std
value: 36.700700000000005
- type: nauc_ndcg_at_10_diff1
value: 10.354099999999999
- type: nauc_ndcg_at_20_max
value: 38.0047
- type: nauc_ndcg_at_20_std
value: 36.4105
- type: nauc_ndcg_at_20_diff1
value: 13.1905
- type: nauc_ndcg_at_100_max
value: 36.5648
- type: nauc_ndcg_at_100_std
value: 36.289100000000005
- type: nauc_ndcg_at_100_diff1
value: 13.1811
- type: nauc_ndcg_at_1000_max
value: 39.0477
- type: nauc_ndcg_at_1000_std
value: 36.8313
- type: nauc_ndcg_at_1000_diff1
value: 14.5539
- type: nauc_map_at_1_max
value: 40.468399999999995
- type: nauc_map_at_1_std
value: 38.8093
- type: nauc_map_at_1_diff1
value: 32.3963
- type: nauc_map_at_3_max
value: 41.5572
- type: nauc_map_at_3_std
value: 36.7928
- type: nauc_map_at_3_diff1
value: 17.2202
- type: nauc_map_at_5_max
value: 41.6267
- type: nauc_map_at_5_std
value: 36.9838
- type: nauc_map_at_5_diff1
value: 17.9896
- type: nauc_map_at_10_max
value: 39.7772
- type: nauc_map_at_10_std
value: 36.9807
- type: nauc_map_at_10_diff1
value: 15.532099999999998
- type: nauc_map_at_20_max
value: 39.717400000000005
- type: nauc_map_at_20_std
value: 36.7401
- type: nauc_map_at_20_diff1
value: 16.4453
- type: nauc_map_at_100_max
value: 39.683
- type: nauc_map_at_100_std
value: 36.8802
- type: nauc_map_at_100_diff1
value: 16.4826
- type: nauc_map_at_1000_max
value: 39.8521
- type: nauc_map_at_1000_std
value: 36.9222
- type: nauc_map_at_1000_diff1
value: 16.5798
- type: nauc_recall_at_1_max
value: 40.468399999999995
- type: nauc_recall_at_1_std
value: 38.8093
- type: nauc_recall_at_1_diff1
value: 32.3963
- type: nauc_recall_at_3_max
value: 43.689
- type: nauc_recall_at_3_std
value: 36.0721
- type: nauc_recall_at_3_diff1
value: 5.8803
- type: nauc_recall_at_5_max
value: 43.5236
- type: nauc_recall_at_5_std
value: 36.095
- type: nauc_recall_at_5_diff1
value: 9.9217
- type: nauc_recall_at_10_max
value: 33.7418
- type: nauc_recall_at_10_std
value: 36.149100000000004
- type: nauc_recall_at_10_diff1
value: 0.2461
- type: nauc_recall_at_20_max
value: 34.2507
- type: nauc_recall_at_20_std
value: 36.0445
- type: nauc_recall_at_20_diff1
value: 8.3467
- type: nauc_recall_at_100_max
value: 23.0814
- type: nauc_recall_at_100_std
value: 33.501
- type: nauc_recall_at_100_diff1
value: 5.4292
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 40.468399999999995
- type: nauc_precision_at_1_std
value: 38.8093
- type: nauc_precision_at_1_diff1
value: 32.3963
- type: nauc_precision_at_3_max
value: 43.689
- type: nauc_precision_at_3_std
value: 36.0721
- type: nauc_precision_at_3_diff1
value: 5.8803
- type: nauc_precision_at_5_max
value: 43.5236
- type: nauc_precision_at_5_std
value: 36.095
- type: nauc_precision_at_5_diff1
value: 9.9217
- type: nauc_precision_at_10_max
value: 33.7418
- type: nauc_precision_at_10_std
value: 36.149100000000004
- type: nauc_precision_at_10_diff1
value: 0.2461
- type: nauc_precision_at_20_max
value: 34.2507
- type: nauc_precision_at_20_std
value: 36.0445
- type: nauc_precision_at_20_diff1
value: 8.3467
- type: nauc_precision_at_100_max
value: 23.0814
- type: nauc_precision_at_100_std
value: 33.501
- type: nauc_precision_at_100_diff1
value: 5.4292
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 40.468399999999995
- type: nauc_mrr_at_1_std
value: 38.8093
- type: nauc_mrr_at_1_diff1
value: 32.3963
- type: nauc_mrr_at_3_max
value: 41.5572
- type: nauc_mrr_at_3_std
value: 36.7928
- type: nauc_mrr_at_3_diff1
value: 17.2202
- type: nauc_mrr_at_5_max
value: 41.6267
- type: nauc_mrr_at_5_std
value: 36.9838
- type: nauc_mrr_at_5_diff1
value: 17.9896
- type: nauc_mrr_at_10_max
value: 39.7772
- type: nauc_mrr_at_10_std
value: 36.9807
- type: nauc_mrr_at_10_diff1
value: 15.532099999999998
- type: nauc_mrr_at_20_max
value: 39.717400000000005
- type: nauc_mrr_at_20_std
value: 36.7401
- type: nauc_mrr_at_20_diff1
value: 16.4453
- type: nauc_mrr_at_100_max
value: 39.683
- type: nauc_mrr_at_100_std
value: 36.8802
- type: nauc_mrr_at_100_diff1
value: 16.4826
- type: nauc_mrr_at_1000_max
value: 39.8521
- type: nauc_mrr_at_1000_std
value: 36.9222
- type: nauc_mrr_at_1000_diff1
value: 16.5798
- type: main_score
value: 15.805
task:
type: Retrieval
- dataset:
config: ara-zho
name: MTEB MLQARetrieval (ara-zho)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 3.1910000000000003
- type: ndcg_at_3
value: 4.3950000000000005
- type: ndcg_at_5
value: 4.829
- type: ndcg_at_10
value: 6.345000000000001
- type: ndcg_at_20
value: 8.478
- type: ndcg_at_100
value: 16.824
- type: ndcg_at_1000
value: 21.404999999999998
- type: map_at_1
value: 3.1910000000000003
- type: map_at_3
value: 4.077999999999999
- type: map_at_5
value: 4.317
- type: map_at_10
value: 4.9239999999999995
- type: map_at_20
value: 5.5
- type: map_at_100
value: 6.428000000000001
- type: map_at_1000
value: 6.686
- type: recall_at_1
value: 3.1910000000000003
- type: recall_at_3
value: 5.319
- type: recall_at_5
value: 6.383
- type: recall_at_10
value: 11.17
- type: recall_at_20
value: 19.681
- type: recall_at_100
value: 68.085
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 3.1910000000000003
- type: precision_at_3
value: 1.773
- type: precision_at_5
value: 1.277
- type: precision_at_10
value: 1.117
- type: precision_at_20
value: 0.984
- type: precision_at_100
value: 0.681
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 3.1915
- type: mrr_at_3
value: 4.077999999999999
- type: mrr_at_5
value: 4.3174
- type: mrr_at_10
value: 4.9244
- type: mrr_at_20
value: 5.5003
- type: mrr_at_100
value: 6.4279
- type: mrr_at_1000
value: 6.686100000000001
- type: nauc_ndcg_at_1_max
value: 52.1235
- type: nauc_ndcg_at_1_std
value: 74.9644
- type: nauc_ndcg_at_1_diff1
value: 15.582299999999998
- type: nauc_ndcg_at_3_max
value: 49.012
- type: nauc_ndcg_at_3_std
value: 59.7897
- type: nauc_ndcg_at_3_diff1
value: 16.4879
- type: nauc_ndcg_at_5_max
value: 45.9082
- type: nauc_ndcg_at_5_std
value: 52.541700000000006
- type: nauc_ndcg_at_5_diff1
value: 17.7455
- type: nauc_ndcg_at_10_max
value: 35.829
- type: nauc_ndcg_at_10_std
value: 44.317
- type: nauc_ndcg_at_10_diff1
value: 9.4463
- type: nauc_ndcg_at_20_max
value: 34.4779
- type: nauc_ndcg_at_20_std
value: 42.0456
- type: nauc_ndcg_at_20_diff1
value: 6.1467
- type: nauc_ndcg_at_100_max
value: 28.580699999999997
- type: nauc_ndcg_at_100_std
value: 34.7968
- type: nauc_ndcg_at_100_diff1
value: 5.092300000000001
- type: nauc_ndcg_at_1000_max
value: 35.0548
- type: nauc_ndcg_at_1000_std
value: 45.0736
- type: nauc_ndcg_at_1000_diff1
value: 7.4399999999999995
- type: nauc_map_at_1_max
value: 52.1235
- type: nauc_map_at_1_std
value: 74.9644
- type: nauc_map_at_1_diff1
value: 15.582299999999998
- type: nauc_map_at_3_max
value: 49.9469
- type: nauc_map_at_3_std
value: 62.677499999999995
- type: nauc_map_at_3_diff1
value: 16.5883
- type: nauc_map_at_5_max
value: 47.8272
- type: nauc_map_at_5_std
value: 58.0546
- type: nauc_map_at_5_diff1
value: 17.1466
- type: nauc_map_at_10_max
value: 42.429899999999996
- type: nauc_map_at_10_std
value: 53.3032
- type: nauc_map_at_10_diff1
value: 12.8557
- type: nauc_map_at_20_max
value: 41.4536
- type: nauc_map_at_20_std
value: 51.7925
- type: nauc_map_at_20_diff1
value: 11.2539
- type: nauc_map_at_100_max
value: 39.7072
- type: nauc_map_at_100_std
value: 50.4909
- type: nauc_map_at_100_diff1
value: 10.4435
- type: nauc_map_at_1000_max
value: 40.3365
- type: nauc_map_at_1000_std
value: 51.482099999999996
- type: nauc_map_at_1000_diff1
value: 10.6958
- type: nauc_recall_at_1_max
value: 52.1235
- type: nauc_recall_at_1_std
value: 74.9644
- type: nauc_recall_at_1_diff1
value: 15.582299999999998
- type: nauc_recall_at_3_max
value: 46.8334
- type: nauc_recall_at_3_std
value: 53.393100000000004
- type: nauc_recall_at_3_diff1
value: 16.1799
- type: nauc_recall_at_5_max
value: 41.891
- type: nauc_recall_at_5_std
value: 41.0433
- type: nauc_recall_at_5_diff1
value: 19.148699999999998
- type: nauc_recall_at_10_max
value: 25.2085
- type: nauc_recall_at_10_std
value: 30.531000000000002
- type: nauc_recall_at_10_diff1
value: 3.6998
- type: nauc_recall_at_20_max
value: 26.248500000000003
- type: nauc_recall_at_20_std
value: 30.9738
- type: nauc_recall_at_20_diff1
value: -0.3174
- type: nauc_recall_at_100_max
value: 12.562599999999998
- type: nauc_recall_at_100_std
value: 9.8009
- type: nauc_recall_at_100_diff1
value: -0.7971
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 52.1235
- type: nauc_precision_at_1_std
value: 74.9644
- type: nauc_precision_at_1_diff1
value: 15.582299999999998
- type: nauc_precision_at_3_max
value: 46.8334
- type: nauc_precision_at_3_std
value: 53.393100000000004
- type: nauc_precision_at_3_diff1
value: 16.1799
- type: nauc_precision_at_5_max
value: 41.891
- type: nauc_precision_at_5_std
value: 41.0433
- type: nauc_precision_at_5_diff1
value: 19.148699999999998
- type: nauc_precision_at_10_max
value: 25.2085
- type: nauc_precision_at_10_std
value: 30.531000000000002
- type: nauc_precision_at_10_diff1
value: 3.6998
- type: nauc_precision_at_20_max
value: 26.248500000000003
- type: nauc_precision_at_20_std
value: 30.9738
- type: nauc_precision_at_20_diff1
value: -0.3174
- type: nauc_precision_at_100_max
value: 12.562599999999998
- type: nauc_precision_at_100_std
value: 9.8009
- type: nauc_precision_at_100_diff1
value: -0.7971
- type: nauc_precision_at_1000_max
value: .nan
- type: nauc_precision_at_1000_std
value: .nan
- type: nauc_precision_at_1000_diff1
value: .nan
- type: nauc_mrr_at_1_max
value: 52.1235
- type: nauc_mrr_at_1_std
value: 74.9644
- type: nauc_mrr_at_1_diff1
value: 15.582299999999998
- type: nauc_mrr_at_3_max
value: 49.9469
- type: nauc_mrr_at_3_std
value: 62.677499999999995
- type: nauc_mrr_at_3_diff1
value: 16.5883
- type: nauc_mrr_at_5_max
value: 47.8272
- type: nauc_mrr_at_5_std
value: 58.0546
- type: nauc_mrr_at_5_diff1
value: 17.1466
- type: nauc_mrr_at_10_max
value: 42.429899999999996
- type: nauc_mrr_at_10_std
value: 53.3032
- type: nauc_mrr_at_10_diff1
value: 12.8557
- type: nauc_mrr_at_20_max
value: 41.4536
- type: nauc_mrr_at_20_std
value: 51.7925
- type: nauc_mrr_at_20_diff1
value: 11.2539
- type: nauc_mrr_at_100_max
value: 39.7072
- type: nauc_mrr_at_100_std
value: 50.4909
- type: nauc_mrr_at_100_diff1
value: 10.4435
- type: nauc_mrr_at_1000_max
value: 40.3365
- type: nauc_mrr_at_1000_std
value: 51.482099999999996
- type: nauc_mrr_at_1000_diff1
value: 10.6958
- type: main_score
value: 6.345000000000001
task:
type: Retrieval
- dataset:
config: deu-ara
name: MTEB MLQARetrieval (deu-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 14.01
- type: ndcg_at_3
value: 16.5
- type: ndcg_at_5
value: 18.683
- type: ndcg_at_10
value: 21.281
- type: ndcg_at_20
value: 24.446
- type: ndcg_at_100
value: 30.941999999999997
- type: ndcg_at_1000
value: 33.587
- type: map_at_1
value: 14.01
- type: map_at_3
value: 15.862000000000002
- type: map_at_5
value: 17.069000000000003
- type: map_at_10
value: 18.107
- type: map_at_20
value: 18.970000000000002
- type: map_at_100
value: 19.749
- type: map_at_1000
value: 19.886
- type: recall_at_1
value: 14.01
- type: recall_at_3
value: 18.357
- type: recall_at_5
value: 23.671
- type: recall_at_10
value: 31.884
- type: recall_at_20
value: 44.444
- type: recall_at_100
value: 81.159
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 14.01
- type: precision_at_3
value: 6.119
- type: precision_at_5
value: 4.734
- type: precision_at_10
value: 3.1879999999999997
- type: precision_at_20
value: 2.222
- type: precision_at_100
value: 0.812
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 14.0097
- type: mrr_at_3
value: 15.861500000000001
- type: mrr_at_5
value: 17.069200000000002
- type: mrr_at_10
value: 18.107100000000003
- type: mrr_at_20
value: 18.970200000000002
- type: mrr_at_100
value: 19.7494
- type: mrr_at_1000
value: 19.886200000000002
- type: nauc_ndcg_at_1_max
value: 43.0702
- type: nauc_ndcg_at_1_std
value: 46.6844
- type: nauc_ndcg_at_1_diff1
value: 49.4384
- type: nauc_ndcg_at_3_max
value: 44.3084
- type: nauc_ndcg_at_3_std
value: 46.3781
- type: nauc_ndcg_at_3_diff1
value: 44.414
- type: nauc_ndcg_at_5_max
value: 38.6313
- type: nauc_ndcg_at_5_std
value: 43.126799999999996
- type: nauc_ndcg_at_5_diff1
value: 40.2109
- type: nauc_ndcg_at_10_max
value: 41.1366
- type: nauc_ndcg_at_10_std
value: 46.014500000000005
- type: nauc_ndcg_at_10_diff1
value: 37.3214
- type: nauc_ndcg_at_20_max
value: 38.9123
- type: nauc_ndcg_at_20_std
value: 44.7471
- type: nauc_ndcg_at_20_diff1
value: 35.4861
- type: nauc_ndcg_at_100_max
value: 38.722
- type: nauc_ndcg_at_100_std
value: 44.4238
- type: nauc_ndcg_at_100_diff1
value: 37.1051
- type: nauc_ndcg_at_1000_max
value: 39.6408
- type: nauc_ndcg_at_1000_std
value: 44.6039
- type: nauc_ndcg_at_1000_diff1
value: 38.3912
- type: nauc_map_at_1_max
value: 43.0702
- type: nauc_map_at_1_std
value: 46.6844
- type: nauc_map_at_1_diff1
value: 49.4384
- type: nauc_map_at_3_max
value: 44.0233
- type: nauc_map_at_3_std
value: 46.4105
- type: nauc_map_at_3_diff1
value: 45.6174
- type: nauc_map_at_5_max
value: 40.692099999999996
- type: nauc_map_at_5_std
value: 44.4447
- type: nauc_map_at_5_diff1
value: 43.0774
- type: nauc_map_at_10_max
value: 41.6757
- type: nauc_map_at_10_std
value: 45.6757
- type: nauc_map_at_10_diff1
value: 41.674499999999995
- type: nauc_map_at_20_max
value: 40.9095
- type: nauc_map_at_20_std
value: 45.2125
- type: nauc_map_at_20_diff1
value: 41.0551
- type: nauc_map_at_100_max
value: 40.7693
- type: nauc_map_at_100_std
value: 45.0677
- type: nauc_map_at_100_diff1
value: 41.1131
- type: nauc_map_at_1000_max
value: 40.828399999999995
- type: nauc_map_at_1000_std
value: 45.082100000000004
- type: nauc_map_at_1000_diff1
value: 41.1869
- type: nauc_recall_at_1_max
value: 43.0702
- type: nauc_recall_at_1_std
value: 46.6844
- type: nauc_recall_at_1_diff1
value: 49.4384
- type: nauc_recall_at_3_max
value: 45.048300000000005
- type: nauc_recall_at_3_std
value: 46.307500000000005
- type: nauc_recall_at_3_diff1
value: 41.274699999999996
- type: nauc_recall_at_5_max
value: 33.1668
- type: nauc_recall_at_5_std
value: 39.7359
- type: nauc_recall_at_5_diff1
value: 33.0151
- type: nauc_recall_at_10_max
value: 40.348099999999995
- type: nauc_recall_at_10_std
value: 47.396100000000004
- type: nauc_recall_at_10_diff1
value: 26.8858
- type: nauc_recall_at_20_max
value: 34.015699999999995
- type: nauc_recall_at_20_std
value: 43.903
- type: nauc_recall_at_20_diff1
value: 21.773500000000002
- type: nauc_recall_at_100_max
value: 30.7371
- type: nauc_recall_at_100_std
value: 42.9545
- type: nauc_recall_at_100_diff1
value: 25.157
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 43.0702
- type: nauc_precision_at_1_std
value: 46.6844
- type: nauc_precision_at_1_diff1
value: 49.4384
- type: nauc_precision_at_3_max
value: 45.048300000000005
- type: nauc_precision_at_3_std
value: 46.307500000000005
- type: nauc_precision_at_3_diff1
value: 41.274699999999996
- type: nauc_precision_at_5_max
value: 33.1668
- type: nauc_precision_at_5_std
value: 39.7359
- type: nauc_precision_at_5_diff1
value: 33.0151
- type: nauc_precision_at_10_max
value: 40.348099999999995
- type: nauc_precision_at_10_std
value: 47.396100000000004
- type: nauc_precision_at_10_diff1
value: 26.8858
- type: nauc_precision_at_20_max
value: 34.015699999999995
- type: nauc_precision_at_20_std
value: 43.903
- type: nauc_precision_at_20_diff1
value: 21.773500000000002
- type: nauc_precision_at_100_max
value: 30.7371
- type: nauc_precision_at_100_std
value: 42.9545
- type: nauc_precision_at_100_diff1
value: 25.157
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 43.0702
- type: nauc_mrr_at_1_std
value: 46.6844
- type: nauc_mrr_at_1_diff1
value: 49.4384
- type: nauc_mrr_at_3_max
value: 44.0233
- type: nauc_mrr_at_3_std
value: 46.4105
- type: nauc_mrr_at_3_diff1
value: 45.6174
- type: nauc_mrr_at_5_max
value: 40.692099999999996
- type: nauc_mrr_at_5_std
value: 44.4447
- type: nauc_mrr_at_5_diff1
value: 43.0774
- type: nauc_mrr_at_10_max
value: 41.6757
- type: nauc_mrr_at_10_std
value: 45.6757
- type: nauc_mrr_at_10_diff1
value: 41.674499999999995
- type: nauc_mrr_at_20_max
value: 40.9095
- type: nauc_mrr_at_20_std
value: 45.2125
- type: nauc_mrr_at_20_diff1
value: 41.0551
- type: nauc_mrr_at_100_max
value: 40.7693
- type: nauc_mrr_at_100_std
value: 45.0677
- type: nauc_mrr_at_100_diff1
value: 41.1131
- type: nauc_mrr_at_1000_max
value: 40.828399999999995
- type: nauc_mrr_at_1000_std
value: 45.082100000000004
- type: nauc_mrr_at_1000_diff1
value: 41.1869
- type: main_score
value: 21.281
task:
type: Retrieval
- dataset:
config: eng-ara
name: MTEB MLQARetrieval (eng-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 19.536
- type: ndcg_at_3
value: 24.058
- type: ndcg_at_5
value: 25.980999999999998
- type: ndcg_at_10
value: 28.753
- type: ndcg_at_20
value: 30.620000000000005
- type: ndcg_at_100
value: 35.260000000000005
- type: ndcg_at_1000
value: 38.761
- type: map_at_1
value: 19.536
- type: map_at_3
value: 22.888
- type: map_at_5
value: 23.962
- type: map_at_10
value: 25.119000000000003
- type: map_at_20
value: 25.636
- type: map_at_100
value: 26.255
- type: map_at_1000
value: 26.398
- type: recall_at_1
value: 19.536
- type: recall_at_3
value: 27.466
- type: recall_at_5
value: 32.108
- type: recall_at_10
value: 40.619
- type: recall_at_20
value: 47.969
- type: recall_at_100
value: 73.30799999999999
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 19.536
- type: precision_at_3
value: 9.155000000000001
- type: precision_at_5
value: 6.422
- type: precision_at_10
value: 4.062
- type: precision_at_20
value: 2.398
- type: precision_at_100
value: 0.733
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 19.535800000000002
- type: mrr_at_3
value: 22.8885
- type: mrr_at_5
value: 23.962
- type: mrr_at_10
value: 25.118499999999997
- type: mrr_at_20
value: 25.6361
- type: mrr_at_100
value: 26.2549
- type: mrr_at_1000
value: 26.398300000000003
- type: nauc_ndcg_at_1_max
value: 52.850699999999996
- type: nauc_ndcg_at_1_std
value: 41.456700000000005
- type: nauc_ndcg_at_1_diff1
value: 53.9262
- type: nauc_ndcg_at_3_max
value: 51.3258
- type: nauc_ndcg_at_3_std
value: 43.0837
- type: nauc_ndcg_at_3_diff1
value: 49.108000000000004
- type: nauc_ndcg_at_5_max
value: 49.876
- type: nauc_ndcg_at_5_std
value: 41.9895
- type: nauc_ndcg_at_5_diff1
value: 46.8046
- type: nauc_ndcg_at_10_max
value: 48.5261
- type: nauc_ndcg_at_10_std
value: 40.809
- type: nauc_ndcg_at_10_diff1
value: 44.2462
- type: nauc_ndcg_at_20_max
value: 47.919200000000004
- type: nauc_ndcg_at_20_std
value: 40.4803
- type: nauc_ndcg_at_20_diff1
value: 45.0426
- type: nauc_ndcg_at_100_max
value: 47.7577
- type: nauc_ndcg_at_100_std
value: 40.4339
- type: nauc_ndcg_at_100_diff1
value: 44.05
- type: nauc_ndcg_at_1000_max
value: 48.8544
- type: nauc_ndcg_at_1000_std
value: 40.7645
- type: nauc_ndcg_at_1000_diff1
value: 46.1338
- type: nauc_map_at_1_max
value: 52.850699999999996
- type: nauc_map_at_1_std
value: 41.456700000000005
- type: nauc_map_at_1_diff1
value: 53.9262
- type: nauc_map_at_3_max
value: 51.649699999999996
- type: nauc_map_at_3_std
value: 42.5908
- type: nauc_map_at_3_diff1
value: 50.2691
- type: nauc_map_at_5_max
value: 50.7742
- type: nauc_map_at_5_std
value: 41.923700000000004
- type: nauc_map_at_5_diff1
value: 48.8816
- type: nauc_map_at_10_max
value: 50.1526
- type: nauc_map_at_10_std
value: 41.4055
- type: nauc_map_at_10_diff1
value: 47.754200000000004
- type: nauc_map_at_20_max
value: 50.021800000000006
- type: nauc_map_at_20_std
value: 41.330299999999994
- type: nauc_map_at_20_diff1
value: 48.0004
- type: nauc_map_at_100_max
value: 49.999700000000004
- type: nauc_map_at_100_std
value: 41.2853
- type: nauc_map_at_100_diff1
value: 47.896
- type: nauc_map_at_1000_max
value: 50.0375
- type: nauc_map_at_1000_std
value: 41.288399999999996
- type: nauc_map_at_1000_diff1
value: 47.9883
- type: nauc_recall_at_1_max
value: 52.850699999999996
- type: nauc_recall_at_1_std
value: 41.456700000000005
- type: nauc_recall_at_1_diff1
value: 53.9262
- type: nauc_recall_at_3_max
value: 50.47469999999999
- type: nauc_recall_at_3_std
value: 44.4385
- type: nauc_recall_at_3_diff1
value: 46.001
- type: nauc_recall_at_5_max
value: 47.4505
- type: nauc_recall_at_5_std
value: 42.1153
- type: nauc_recall_at_5_diff1
value: 41.2404
- type: nauc_recall_at_10_max
value: 43.9761
- type: nauc_recall_at_10_std
value: 38.9027
- type: nauc_recall_at_10_diff1
value: 34.449200000000005
- type: nauc_recall_at_20_max
value: 41.4318
- type: nauc_recall_at_20_std
value: 37.572100000000006
- type: nauc_recall_at_20_diff1
value: 36.978899999999996
- type: nauc_recall_at_100_max
value: 37.5385
- type: nauc_recall_at_100_std
value: 36.7443
- type: nauc_recall_at_100_diff1
value: 25.948300000000003
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 52.850699999999996
- type: nauc_precision_at_1_std
value: 41.456700000000005
- type: nauc_precision_at_1_diff1
value: 53.9262
- type: nauc_precision_at_3_max
value: 50.47469999999999
- type: nauc_precision_at_3_std
value: 44.4385
- type: nauc_precision_at_3_diff1
value: 46.001
- type: nauc_precision_at_5_max
value: 47.4505
- type: nauc_precision_at_5_std
value: 42.1153
- type: nauc_precision_at_5_diff1
value: 41.2404
- type: nauc_precision_at_10_max
value: 43.9761
- type: nauc_precision_at_10_std
value: 38.9027
- type: nauc_precision_at_10_diff1
value: 34.449200000000005
- type: nauc_precision_at_20_max
value: 41.4318
- type: nauc_precision_at_20_std
value: 37.572100000000006
- type: nauc_precision_at_20_diff1
value: 36.978899999999996
- type: nauc_precision_at_100_max
value: 37.5385
- type: nauc_precision_at_100_std
value: 36.7443
- type: nauc_precision_at_100_diff1
value: 25.948300000000003
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 52.850699999999996
- type: nauc_mrr_at_1_std
value: 41.456700000000005
- type: nauc_mrr_at_1_diff1
value: 53.9262
- type: nauc_mrr_at_3_max
value: 51.649699999999996
- type: nauc_mrr_at_3_std
value: 42.5908
- type: nauc_mrr_at_3_diff1
value: 50.2691
- type: nauc_mrr_at_5_max
value: 50.7742
- type: nauc_mrr_at_5_std
value: 41.923700000000004
- type: nauc_mrr_at_5_diff1
value: 48.8816
- type: nauc_mrr_at_10_max
value: 50.1526
- type: nauc_mrr_at_10_std
value: 41.4055
- type: nauc_mrr_at_10_diff1
value: 47.754200000000004
- type: nauc_mrr_at_20_max
value: 50.021800000000006
- type: nauc_mrr_at_20_std
value: 41.330299999999994
- type: nauc_mrr_at_20_diff1
value: 48.0004
- type: nauc_mrr_at_100_max
value: 49.999700000000004
- type: nauc_mrr_at_100_std
value: 41.2853
- type: nauc_mrr_at_100_diff1
value: 47.896
- type: nauc_mrr_at_1000_max
value: 50.0375
- type: nauc_mrr_at_1000_std
value: 41.288399999999996
- type: nauc_mrr_at_1000_diff1
value: 47.9883
- type: main_score
value: 28.753
task:
type: Retrieval
- dataset:
config: spa-ara
name: MTEB MLQARetrieval (spa-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 13.665
- type: ndcg_at_3
value: 18.123
- type: ndcg_at_5
value: 19.619
- type: ndcg_at_10
value: 20.896
- type: ndcg_at_20
value: 24.048
- type: ndcg_at_100
value: 32.034
- type: ndcg_at_1000
value: 34.071
- type: map_at_1
value: 13.665
- type: map_at_3
value: 17.081
- type: map_at_5
value: 17.888
- type: map_at_10
value: 18.458
- type: map_at_20
value: 19.326999999999998
- type: map_at_100
value: 20.321
- type: map_at_1000
value: 20.433
- type: recall_at_1
value: 13.665
- type: recall_at_3
value: 21.118000000000002
- type: recall_at_5
value: 24.845
- type: recall_at_10
value: 28.571
- type: recall_at_20
value: 40.994
- type: recall_at_100
value: 85.714
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 13.665
- type: precision_at_3
value: 7.039
- type: precision_at_5
value: 4.968999999999999
- type: precision_at_10
value: 2.857
- type: precision_at_20
value: 2.0500000000000003
- type: precision_at_100
value: 0.857
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 13.664599999999998
- type: mrr_at_3
value: 17.0807
- type: mrr_at_5
value: 17.8882
- type: mrr_at_10
value: 18.4576
- type: mrr_at_20
value: 19.3272
- type: mrr_at_100
value: 20.321
- type: mrr_at_1000
value: 20.4326
- type: nauc_ndcg_at_1_max
value: 52.861599999999996
- type: nauc_ndcg_at_1_std
value: 43.0659
- type: nauc_ndcg_at_1_diff1
value: 38.386399999999995
- type: nauc_ndcg_at_3_max
value: 40.7645
- type: nauc_ndcg_at_3_std
value: 36.5724
- type: nauc_ndcg_at_3_diff1
value: 29.4165
- type: nauc_ndcg_at_5_max
value: 39.867200000000004
- type: nauc_ndcg_at_5_std
value: 35.2971
- type: nauc_ndcg_at_5_diff1
value: 30.758200000000002
- type: nauc_ndcg_at_10_max
value: 40.3682
- type: nauc_ndcg_at_10_std
value: 36.240899999999996
- type: nauc_ndcg_at_10_diff1
value: 33.5072
- type: nauc_ndcg_at_20_max
value: 38.9305
- type: nauc_ndcg_at_20_std
value: 34.6001
- type: nauc_ndcg_at_20_diff1
value: 31.099700000000002
- type: nauc_ndcg_at_100_max
value: 36.5928
- type: nauc_ndcg_at_100_std
value: 31.613400000000002
- type: nauc_ndcg_at_100_diff1
value: 32.4518
- type: nauc_ndcg_at_1000_max
value: 40.0662
- type: nauc_ndcg_at_1000_std
value: 35.1148
- type: nauc_ndcg_at_1000_diff1
value: 32.6003
- type: nauc_map_at_1_max
value: 52.861599999999996
- type: nauc_map_at_1_std
value: 43.0659
- type: nauc_map_at_1_diff1
value: 38.386399999999995
- type: nauc_map_at_3_max
value: 42.9409
- type: nauc_map_at_3_std
value: 37.804500000000004
- type: nauc_map_at_3_diff1
value: 30.864200000000004
- type: nauc_map_at_5_max
value: 42.35
- type: nauc_map_at_5_std
value: 36.9876
- type: nauc_map_at_5_diff1
value: 31.6284
- type: nauc_map_at_10_max
value: 42.6802
- type: nauc_map_at_10_std
value: 37.4979
- type: nauc_map_at_10_diff1
value: 33.0231
- type: nauc_map_at_20_max
value: 42.3121
- type: nauc_map_at_20_std
value: 37.001400000000004
- type: nauc_map_at_20_diff1
value: 32.427299999999995
- type: nauc_map_at_100_max
value: 41.9711
- type: nauc_map_at_100_std
value: 36.5756
- type: nauc_map_at_100_diff1
value: 32.7472
- type: nauc_map_at_1000_max
value: 42.163000000000004
- type: nauc_map_at_1000_std
value: 36.7708
- type: nauc_map_at_1000_diff1
value: 32.7495
- type: nauc_recall_at_1_max
value: 52.861599999999996
- type: nauc_recall_at_1_std
value: 43.0659
- type: nauc_recall_at_1_diff1
value: 38.386399999999995
- type: nauc_recall_at_3_max
value: 35.4833
- type: nauc_recall_at_3_std
value: 33.5567
- type: nauc_recall_at_3_diff1
value: 25.971
- type: nauc_recall_at_5_max
value: 34.0439
- type: nauc_recall_at_5_std
value: 31.2459
- type: nauc_recall_at_5_diff1
value: 28.999799999999997
- type: nauc_recall_at_10_max
value: 35.1446
- type: nauc_recall_at_10_std
value: 33.4707
- type: nauc_recall_at_10_diff1
value: 35.409400000000005
- type: nauc_recall_at_20_max
value: 30.5251
- type: nauc_recall_at_20_std
value: 28.4605
- type: nauc_recall_at_20_diff1
value: 27.086399999999998
- type: nauc_recall_at_100_max
value: -4.0771
- type: nauc_recall_at_100_std
value: -9.1631
- type: nauc_recall_at_100_diff1
value: 30.3414
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 52.861599999999996
- type: nauc_precision_at_1_std
value: 43.0659
- type: nauc_precision_at_1_diff1
value: 38.386399999999995
- type: nauc_precision_at_3_max
value: 35.4833
- type: nauc_precision_at_3_std
value: 33.5567
- type: nauc_precision_at_3_diff1
value: 25.971
- type: nauc_precision_at_5_max
value: 34.0439
- type: nauc_precision_at_5_std
value: 31.2459
- type: nauc_precision_at_5_diff1
value: 28.999799999999997
- type: nauc_precision_at_10_max
value: 35.1446
- type: nauc_precision_at_10_std
value: 33.4707
- type: nauc_precision_at_10_diff1
value: 35.409400000000005
- type: nauc_precision_at_20_max
value: 30.5251
- type: nauc_precision_at_20_std
value: 28.4605
- type: nauc_precision_at_20_diff1
value: 27.086399999999998
- type: nauc_precision_at_100_max
value: -4.0771
- type: nauc_precision_at_100_std
value: -9.1631
- type: nauc_precision_at_100_diff1
value: 30.3414
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 52.861599999999996
- type: nauc_mrr_at_1_std
value: 43.0659
- type: nauc_mrr_at_1_diff1
value: 38.386399999999995
- type: nauc_mrr_at_3_max
value: 42.9409
- type: nauc_mrr_at_3_std
value: 37.804500000000004
- type: nauc_mrr_at_3_diff1
value: 30.864200000000004
- type: nauc_mrr_at_5_max
value: 42.35
- type: nauc_mrr_at_5_std
value: 36.9876
- type: nauc_mrr_at_5_diff1
value: 31.6284
- type: nauc_mrr_at_10_max
value: 42.6802
- type: nauc_mrr_at_10_std
value: 37.4979
- type: nauc_mrr_at_10_diff1
value: 33.0231
- type: nauc_mrr_at_20_max
value: 42.3121
- type: nauc_mrr_at_20_std
value: 37.001400000000004
- type: nauc_mrr_at_20_diff1
value: 32.427299999999995
- type: nauc_mrr_at_100_max
value: 41.9711
- type: nauc_mrr_at_100_std
value: 36.5756
- type: nauc_mrr_at_100_diff1
value: 32.7472
- type: nauc_mrr_at_1000_max
value: 42.163000000000004
- type: nauc_mrr_at_1000_std
value: 36.7708
- type: nauc_mrr_at_1000_diff1
value: 32.7495
- type: main_score
value: 20.896
task:
type: Retrieval
- dataset:
config: hin-ara
name: MTEB MLQARetrieval (hin-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 6.451999999999999
- type: ndcg_at_3
value: 8.953999999999999
- type: ndcg_at_5
value: 9.81
- type: ndcg_at_10
value: 11.885
- type: ndcg_at_20
value: 14.363000000000001
- type: ndcg_at_100
value: 21.972
- type: ndcg_at_1000
value: 26.001
- type: map_at_1
value: 6.451999999999999
- type: map_at_3
value: 8.333
- type: map_at_5
value: 8.790000000000001
- type: map_at_10
value: 9.64
- type: map_at_20
value: 10.337
- type: map_at_100
value: 11.264000000000001
- type: map_at_1000
value: 11.494
- type: recall_at_1
value: 6.451999999999999
- type: recall_at_3
value: 10.753
- type: recall_at_5
value: 12.903
- type: recall_at_10
value: 19.355
- type: recall_at_20
value: 29.032000000000004
- type: recall_at_100
value: 72.043
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 6.451999999999999
- type: precision_at_3
value: 3.5839999999999996
- type: precision_at_5
value: 2.581
- type: precision_at_10
value: 1.9349999999999998
- type: precision_at_20
value: 1.452
- type: precision_at_100
value: 0.72
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 6.4516
- type: mrr_at_3
value: 8.333300000000001
- type: mrr_at_5
value: 8.7903
- type: mrr_at_10
value: 9.6397
- type: mrr_at_20
value: 10.337200000000001
- type: mrr_at_100
value: 11.2636
- type: mrr_at_1000
value: 11.4943
- type: nauc_ndcg_at_1_max
value: 65.62599999999999
- type: nauc_ndcg_at_1_std
value: 31.183699999999998
- type: nauc_ndcg_at_1_diff1
value: 68.4738
- type: nauc_ndcg_at_3_max
value: 48.579899999999995
- type: nauc_ndcg_at_3_std
value: 29.991200000000003
- type: nauc_ndcg_at_3_diff1
value: 46.6277
- type: nauc_ndcg_at_5_max
value: 46.9782
- type: nauc_ndcg_at_5_std
value: 28.7305
- type: nauc_ndcg_at_5_diff1
value: 44.6164
- type: nauc_ndcg_at_10_max
value: 41.7329
- type: nauc_ndcg_at_10_std
value: 25.4911
- type: nauc_ndcg_at_10_diff1
value: 38.5108
- type: nauc_ndcg_at_20_max
value: 37.8093
- type: nauc_ndcg_at_20_std
value: 21.448700000000002
- type: nauc_ndcg_at_20_diff1
value: 38.9368
- type: nauc_ndcg_at_100_max
value: 40.6472
- type: nauc_ndcg_at_100_std
value: 20.0442
- type: nauc_ndcg_at_100_diff1
value: 36.172900000000006
- type: nauc_ndcg_at_1000_max
value: 43.277100000000004
- type: nauc_ndcg_at_1000_std
value: 24.933
- type: nauc_ndcg_at_1000_diff1
value: 41.5223
- type: nauc_map_at_1_max
value: 65.62599999999999
- type: nauc_map_at_1_std
value: 31.183699999999998
- type: nauc_map_at_1_diff1
value: 68.4738
- type: nauc_map_at_3_max
value: 51.6864
- type: nauc_map_at_3_std
value: 30.0209
- type: nauc_map_at_3_diff1
value: 50.6979
- type: nauc_map_at_5_max
value: 50.5107
- type: nauc_map_at_5_std
value: 29.2857
- type: nauc_map_at_5_diff1
value: 49.412099999999995
- type: nauc_map_at_10_max
value: 47.7907
- type: nauc_map_at_10_std
value: 27.9057
- type: nauc_map_at_10_diff1
value: 46.1772
- type: nauc_map_at_20_max
value: 46.1477
- type: nauc_map_at_20_std
value: 26.500200000000003
- type: nauc_map_at_20_diff1
value: 46.2022
- type: nauc_map_at_100_max
value: 46.739000000000004
- type: nauc_map_at_100_std
value: 26.590700000000002
- type: nauc_map_at_100_diff1
value: 45.6739
- type: nauc_map_at_1000_max
value: 46.9594
- type: nauc_map_at_1000_std
value: 26.942
- type: nauc_map_at_1000_diff1
value: 46.092
- type: nauc_recall_at_1_max
value: 65.62599999999999
- type: nauc_recall_at_1_std
value: 31.183699999999998
- type: nauc_recall_at_1_diff1
value: 68.4738
- type: nauc_recall_at_3_max
value: 41.5971
- type: nauc_recall_at_3_std
value: 29.9872
- type: nauc_recall_at_3_diff1
value: 37.4489
- type: nauc_recall_at_5_max
value: 39.6359
- type: nauc_recall_at_5_std
value: 27.478
- type: nauc_recall_at_5_diff1
value: 34.4535
- type: nauc_recall_at_10_max
value: 30.612699999999997
- type: nauc_recall_at_10_std
value: 20.4392
- type: nauc_recall_at_10_diff1
value: 24.479300000000002
- type: nauc_recall_at_20_max
value: 23.7768
- type: nauc_recall_at_20_std
value: 11.6351
- type: nauc_recall_at_20_diff1
value: 27.6022
- type: nauc_recall_at_100_max
value: 29.2454
- type: nauc_recall_at_100_std
value: -1.1219
- type: nauc_recall_at_100_diff1
value: 13.123000000000001
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 65.62599999999999
- type: nauc_precision_at_1_std
value: 31.183699999999998
- type: nauc_precision_at_1_diff1
value: 68.4738
- type: nauc_precision_at_3_max
value: 41.5971
- type: nauc_precision_at_3_std
value: 29.9872
- type: nauc_precision_at_3_diff1
value: 37.4489
- type: nauc_precision_at_5_max
value: 39.6359
- type: nauc_precision_at_5_std
value: 27.478
- type: nauc_precision_at_5_diff1
value: 34.4535
- type: nauc_precision_at_10_max
value: 30.612699999999997
- type: nauc_precision_at_10_std
value: 20.4392
- type: nauc_precision_at_10_diff1
value: 24.479300000000002
- type: nauc_precision_at_20_max
value: 23.7768
- type: nauc_precision_at_20_std
value: 11.6351
- type: nauc_precision_at_20_diff1
value: 27.6022
- type: nauc_precision_at_100_max
value: 29.2454
- type: nauc_precision_at_100_std
value: -1.1219
- type: nauc_precision_at_100_diff1
value: 13.123000000000001
- type: nauc_precision_at_1000_max
value: .nan
- type: nauc_precision_at_1000_std
value: .nan
- type: nauc_precision_at_1000_diff1
value: .nan
- type: nauc_mrr_at_1_max
value: 65.62599999999999
- type: nauc_mrr_at_1_std
value: 31.183699999999998
- type: nauc_mrr_at_1_diff1
value: 68.4738
- type: nauc_mrr_at_3_max
value: 51.6864
- type: nauc_mrr_at_3_std
value: 30.0209
- type: nauc_mrr_at_3_diff1
value: 50.6979
- type: nauc_mrr_at_5_max
value: 50.5107
- type: nauc_mrr_at_5_std
value: 29.2857
- type: nauc_mrr_at_5_diff1
value: 49.412099999999995
- type: nauc_mrr_at_10_max
value: 47.7907
- type: nauc_mrr_at_10_std
value: 27.9057
- type: nauc_mrr_at_10_diff1
value: 46.1772
- type: nauc_mrr_at_20_max
value: 46.1477
- type: nauc_mrr_at_20_std
value: 26.500200000000003
- type: nauc_mrr_at_20_diff1
value: 46.2022
- type: nauc_mrr_at_100_max
value: 46.739000000000004
- type: nauc_mrr_at_100_std
value: 26.590700000000002
- type: nauc_mrr_at_100_diff1
value: 45.6739
- type: nauc_mrr_at_1000_max
value: 46.9594
- type: nauc_mrr_at_1000_std
value: 26.942
- type: nauc_mrr_at_1000_diff1
value: 46.092
- type: main_score
value: 11.885
task:
type: Retrieval
- dataset:
config: vie-ara
name: MTEB MLQARetrieval (vie-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 11.043
- type: ndcg_at_3
value: 16.287
- type: ndcg_at_5
value: 17.527
- type: ndcg_at_10
value: 20.605
- type: ndcg_at_20
value: 22.445999999999998
- type: ndcg_at_100
value: 30.470999999999997
- type: ndcg_at_1000
value: 32.505
- type: map_at_1
value: 11.043
- type: map_at_3
value: 14.927999999999999
- type: map_at_5
value: 15.603
- type: map_at_10
value: 16.817
- type: map_at_20
value: 17.311
- type: map_at_100
value: 18.284
- type: map_at_1000
value: 18.401
- type: recall_at_1
value: 11.043
- type: recall_at_3
value: 20.244999999999997
- type: recall_at_5
value: 23.313
- type: recall_at_10
value: 33.129
- type: recall_at_20
value: 40.491
- type: recall_at_100
value: 85.89
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 11.043
- type: precision_at_3
value: 6.748
- type: precision_at_5
value: 4.662999999999999
- type: precision_at_10
value: 3.313
- type: precision_at_20
value: 2.025
- type: precision_at_100
value: 0.859
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 11.0429
- type: mrr_at_3
value: 14.9284
- type: mrr_at_5
value: 15.6033
- type: mrr_at_10
value: 16.8171
- type: mrr_at_20
value: 17.311
- type: mrr_at_100
value: 18.2844
- type: mrr_at_1000
value: 18.4012
- type: nauc_ndcg_at_1_max
value: 54.1021
- type: nauc_ndcg_at_1_std
value: 53.44370000000001
- type: nauc_ndcg_at_1_diff1
value: 47.1684
- type: nauc_ndcg_at_3_max
value: 45.426899999999996
- type: nauc_ndcg_at_3_std
value: 51.2884
- type: nauc_ndcg_at_3_diff1
value: 32.707
- type: nauc_ndcg_at_5_max
value: 43.5548
- type: nauc_ndcg_at_5_std
value: 49.0835
- type: nauc_ndcg_at_5_diff1
value: 31.1453
- type: nauc_ndcg_at_10_max
value: 40.505
- type: nauc_ndcg_at_10_std
value: 45.5248
- type: nauc_ndcg_at_10_diff1
value: 28.603499999999997
- type: nauc_ndcg_at_20_max
value: 43.107800000000005
- type: nauc_ndcg_at_20_std
value: 48.747299999999996
- type: nauc_ndcg_at_20_diff1
value: 28.317999999999998
- type: nauc_ndcg_at_100_max
value: 44.664
- type: nauc_ndcg_at_100_std
value: 49.1295
- type: nauc_ndcg_at_100_diff1
value: 32.799499999999995
- type: nauc_ndcg_at_1000_max
value: 44.534600000000005
- type: nauc_ndcg_at_1000_std
value: 48.8575
- type: nauc_ndcg_at_1000_diff1
value: 32.2291
- type: nauc_map_at_1_max
value: 54.1021
- type: nauc_map_at_1_std
value: 53.44370000000001
- type: nauc_map_at_1_diff1
value: 47.1684
- type: nauc_map_at_3_max
value: 46.8518
- type: nauc_map_at_3_std
value: 51.5093
- type: nauc_map_at_3_diff1
value: 35.4337
- type: nauc_map_at_5_max
value: 45.747
- type: nauc_map_at_5_std
value: 50.20139999999999
- type: nauc_map_at_5_diff1
value: 34.4251
- type: nauc_map_at_10_max
value: 44.3589
- type: nauc_map_at_10_std
value: 48.4788
- type: nauc_map_at_10_diff1
value: 33.143299999999996
- type: nauc_map_at_20_max
value: 45.189299999999996
- type: nauc_map_at_20_std
value: 49.452400000000004
- type: nauc_map_at_20_diff1
value: 33.0928
- type: nauc_map_at_100_max
value: 45.4789
- type: nauc_map_at_100_std
value: 49.4771
- type: nauc_map_at_100_diff1
value: 33.894999999999996
- type: nauc_map_at_1000_max
value: 45.486599999999996
- type: nauc_map_at_1000_std
value: 49.4681
- type: nauc_map_at_1000_diff1
value: 33.881499999999996
- type: nauc_recall_at_1_max
value: 54.1021
- type: nauc_recall_at_1_std
value: 53.44370000000001
- type: nauc_recall_at_1_diff1
value: 47.1684
- type: nauc_recall_at_3_max
value: 42.262
- type: nauc_recall_at_3_std
value: 50.8492
- type: nauc_recall_at_3_diff1
value: 26.5145
- type: nauc_recall_at_5_max
value: 38.5884
- type: nauc_recall_at_5_std
value: 46.433
- type: nauc_recall_at_5_diff1
value: 23.8445
- type: nauc_recall_at_10_max
value: 31.824599999999997
- type: nauc_recall_at_10_std
value: 38.658300000000004
- type: nauc_recall_at_10_diff1
value: 18.9615
- type: nauc_recall_at_20_max
value: 39.5261
- type: nauc_recall_at_20_std
value: 48.375099999999996
- type: nauc_recall_at_20_diff1
value: 18.1526
- type: nauc_recall_at_100_max
value: 46.645900000000005
- type: nauc_recall_at_100_std
value: 52.282
- type: nauc_recall_at_100_diff1
value: 39.5874
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 54.1021
- type: nauc_precision_at_1_std
value: 53.44370000000001
- type: nauc_precision_at_1_diff1
value: 47.1684
- type: nauc_precision_at_3_max
value: 42.262
- type: nauc_precision_at_3_std
value: 50.8492
- type: nauc_precision_at_3_diff1
value: 26.5145
- type: nauc_precision_at_5_max
value: 38.5884
- type: nauc_precision_at_5_std
value: 46.433
- type: nauc_precision_at_5_diff1
value: 23.8445
- type: nauc_precision_at_10_max
value: 31.824599999999997
- type: nauc_precision_at_10_std
value: 38.658300000000004
- type: nauc_precision_at_10_diff1
value: 18.9615
- type: nauc_precision_at_20_max
value: 39.5261
- type: nauc_precision_at_20_std
value: 48.375099999999996
- type: nauc_precision_at_20_diff1
value: 18.1526
- type: nauc_precision_at_100_max
value: 46.645900000000005
- type: nauc_precision_at_100_std
value: 52.282
- type: nauc_precision_at_100_diff1
value: 39.5874
- type: nauc_precision_at_1000_max
value: 100.0
- type: nauc_precision_at_1000_std
value: 100.0
- type: nauc_precision_at_1000_diff1
value: 100.0
- type: nauc_mrr_at_1_max
value: 54.1021
- type: nauc_mrr_at_1_std
value: 53.44370000000001
- type: nauc_mrr_at_1_diff1
value: 47.1684
- type: nauc_mrr_at_3_max
value: 46.8518
- type: nauc_mrr_at_3_std
value: 51.5093
- type: nauc_mrr_at_3_diff1
value: 35.4337
- type: nauc_mrr_at_5_max
value: 45.747
- type: nauc_mrr_at_5_std
value: 50.20139999999999
- type: nauc_mrr_at_5_diff1
value: 34.4251
- type: nauc_mrr_at_10_max
value: 44.3589
- type: nauc_mrr_at_10_std
value: 48.4788
- type: nauc_mrr_at_10_diff1
value: 33.143299999999996
- type: nauc_mrr_at_20_max
value: 45.189299999999996
- type: nauc_mrr_at_20_std
value: 49.452400000000004
- type: nauc_mrr_at_20_diff1
value: 33.0928
- type: nauc_mrr_at_100_max
value: 45.4789
- type: nauc_mrr_at_100_std
value: 49.4771
- type: nauc_mrr_at_100_diff1
value: 33.894999999999996
- type: nauc_mrr_at_1000_max
value: 45.486599999999996
- type: nauc_mrr_at_1000_std
value: 49.4681
- type: nauc_mrr_at_1000_diff1
value: 33.881499999999996
- type: main_score
value: 20.605
task:
type: Retrieval
- dataset:
config: zho-ara
name: MTEB MLQARetrieval (zho-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: validation
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 6.383
- type: ndcg_at_3
value: 10.328
- type: ndcg_at_5
value: 11.838
- type: ndcg_at_10
value: 13.689000000000002
- type: ndcg_at_20
value: 15.722
- type: ndcg_at_100
value: 23.307
- type: ndcg_at_1000
value: 27.029999999999998
- type: map_at_1
value: 6.383
- type: map_at_3
value: 9.309000000000001
- type: map_at_5
value: 10.133000000000001
- type: map_at_10
value: 10.873
- type: map_at_20
value: 11.44
- type: map_at_100
value: 12.350999999999999
- type: map_at_1000
value: 12.556999999999999
- type: recall_at_1
value: 6.383
- type: recall_at_3
value: 13.297999999999998
- type: recall_at_5
value: 17.021
- type: recall_at_10
value: 22.872
- type: recall_at_20
value: 30.851
- type: recall_at_100
value: 73.936
- type: recall_at_1000
value: 100.0
- type: precision_at_1
value: 6.383
- type: precision_at_3
value: 4.433
- type: precision_at_5
value: 3.404
- type: precision_at_10
value: 2.287
- type: precision_at_20
value: 1.543
- type: precision_at_100
value: 0.739
- type: precision_at_1000
value: 0.1
- type: mrr_at_1
value: 6.383
- type: mrr_at_3
value: 9.3085
- type: mrr_at_5
value: 10.133000000000001
- type: mrr_at_10
value: 10.873
- type: mrr_at_20
value: 11.4397
- type: mrr_at_100
value: 12.351099999999999
- type: mrr_at_1000
value: 12.5574
- type: nauc_ndcg_at_1_max
value: 50.0144
- type: nauc_ndcg_at_1_std
value: 32.6946
- type: nauc_ndcg_at_1_diff1
value: 33.5185
- type: nauc_ndcg_at_3_max
value: 44.979
- type: nauc_ndcg_at_3_std
value: 28.504099999999998
- type: nauc_ndcg_at_3_diff1
value: 24.4499
- type: nauc_ndcg_at_5_max
value: 41.5632
- type: nauc_ndcg_at_5_std
value: 26.865899999999996
- type: nauc_ndcg_at_5_diff1
value: 18.3327
- type: nauc_ndcg_at_10_max
value: 40.2654
- type: nauc_ndcg_at_10_std
value: 26.555899999999998
- type: nauc_ndcg_at_10_diff1
value: 15.7278
- type: nauc_ndcg_at_20_max
value: 36.7268
- type: nauc_ndcg_at_20_std
value: 24.6418
- type: nauc_ndcg_at_20_diff1
value: 14.249500000000001
- type: nauc_ndcg_at_100_max
value: 36.518299999999996
- type: nauc_ndcg_at_100_std
value: 22.374
- type: nauc_ndcg_at_100_diff1
value: 13.3634
- type: nauc_ndcg_at_1000_max
value: 39.212799999999994
- type: nauc_ndcg_at_1000_std
value: 24.5445
- type: nauc_ndcg_at_1000_diff1
value: 16.542399999999997
- type: nauc_map_at_1_max
value: 50.0144
- type: nauc_map_at_1_std
value: 32.6946
- type: nauc_map_at_1_diff1
value: 33.5185
- type: nauc_map_at_3_max
value: 45.4219
- type: nauc_map_at_3_std
value: 28.604200000000002
- type: nauc_map_at_3_diff1
value: 26.0065
- type: nauc_map_at_5_max
value: 43.292199999999994
- type: nauc_map_at_5_std
value: 27.6021
- type: nauc_map_at_5_diff1
value: 22.1641
- type: nauc_map_at_10_max
value: 42.696400000000004
- type: nauc_map_at_10_std
value: 27.2362
- type: nauc_map_at_10_diff1
value: 20.849999999999998
- type: nauc_map_at_20_max
value: 41.5271
- type: nauc_map_at_20_std
value: 26.6845
- type: nauc_map_at_20_diff1
value: 20.3821
- type: nauc_map_at_100_max
value: 41.4541
- type: nauc_map_at_100_std
value: 26.062400000000004
- type: nauc_map_at_100_diff1
value: 20.0368
- type: nauc_map_at_1000_max
value: 41.637800000000006
- type: nauc_map_at_1000_std
value: 26.2118
- type: nauc_map_at_1000_diff1
value: 20.2331
- type: nauc_recall_at_1_max
value: 50.0144
- type: nauc_recall_at_1_std
value: 32.6946
- type: nauc_recall_at_1_diff1
value: 33.5185
- type: nauc_recall_at_3_max
value: 44.1713
- type: nauc_recall_at_3_std
value: 28.48
- type: nauc_recall_at_3_diff1
value: 21.162
- type: nauc_recall_at_5_max
value: 38.082100000000004
- type: nauc_recall_at_5_std
value: 25.4647
- type: nauc_recall_at_5_diff1
value: 10.5258
- type: nauc_recall_at_10_max
value: 35.6992
- type: nauc_recall_at_10_std
value: 25.643
- type: nauc_recall_at_10_diff1
value: 6.1484
- type: nauc_recall_at_20_max
value: 27.3641
- type: nauc_recall_at_20_std
value: 20.6726
- type: nauc_recall_at_20_diff1
value: 3.2234
- type: nauc_recall_at_100_max
value: 22.9219
- type: nauc_recall_at_100_std
value: 11.5033
- type: nauc_recall_at_100_diff1
value: -3.3923
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_precision_at_1_max
value: 50.0144
- type: nauc_precision_at_1_std
value: 32.6946
- type: nauc_precision_at_1_diff1
value: 33.5185
- type: nauc_precision_at_3_max
value: 44.1713
- type: nauc_precision_at_3_std
value: 28.48
- type: nauc_precision_at_3_diff1
value: 21.162
- type: nauc_precision_at_5_max
value: 38.082100000000004
- type: nauc_precision_at_5_std
value: 25.4647
- type: nauc_precision_at_5_diff1
value: 10.5258
- type: nauc_precision_at_10_max
value: 35.6992
- type: nauc_precision_at_10_std
value: 25.643
- type: nauc_precision_at_10_diff1
value: 6.1484
- type: nauc_precision_at_20_max
value: 27.3641
- type: nauc_precision_at_20_std
value: 20.6726
- type: nauc_precision_at_20_diff1
value: 3.2234
- type: nauc_precision_at_100_max
value: 22.9219
- type: nauc_precision_at_100_std
value: 11.5033
- type: nauc_precision_at_100_diff1
value: -3.3923
- type: nauc_precision_at_1000_max
value: .nan
- type: nauc_precision_at_1000_std
value: .nan
- type: nauc_precision_at_1000_diff1
value: .nan
- type: nauc_mrr_at_1_max
value: 50.0144
- type: nauc_mrr_at_1_std
value: 32.6946
- type: nauc_mrr_at_1_diff1
value: 33.5185
- type: nauc_mrr_at_3_max
value: 45.4219
- type: nauc_mrr_at_3_std
value: 28.604200000000002
- type: nauc_mrr_at_3_diff1
value: 26.0065
- type: nauc_mrr_at_5_max
value: 43.292199999999994
- type: nauc_mrr_at_5_std
value: 27.6021
- type: nauc_mrr_at_5_diff1
value: 22.1641
- type: nauc_mrr_at_10_max
value: 42.696400000000004
- type: nauc_mrr_at_10_std
value: 27.2362
- type: nauc_mrr_at_10_diff1
value: 20.849999999999998
- type: nauc_mrr_at_20_max
value: 41.5271
- type: nauc_mrr_at_20_std
value: 26.6845
- type: nauc_mrr_at_20_diff1
value: 20.3821
- type: nauc_mrr_at_100_max
value: 41.4541
- type: nauc_mrr_at_100_std
value: 26.062400000000004
- type: nauc_mrr_at_100_diff1
value: 20.0368
- type: nauc_mrr_at_1000_max
value: 41.6376
- type: nauc_mrr_at_1000_std
value: 26.2117
- type: nauc_mrr_at_1000_diff1
value: 20.233
- type: main_score
value: 13.689000000000002
task:
type: Retrieval
- dataset:
config: ara-ara
name: MTEB MLQARetrieval (ara-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 37.409
- type: ndcg_at_3
value: 44.269
- type: ndcg_at_5
value: 46.23
- type: ndcg_at_10
value: 48.076
- type: ndcg_at_20
value: 49.679
- type: ndcg_at_100
value: 52.037
- type: ndcg_at_1000
value: 53.958
- type: map_at_1
value: 37.399
- type: map_at_3
value: 42.577999999999996
- type: map_at_5
value: 43.661
- type: map_at_10
value: 44.42
- type: map_at_20
value: 44.861000000000004
- type: map_at_100
value: 45.179
- type: map_at_1000
value: 45.242
- type: recall_at_1
value: 37.399
- type: recall_at_3
value: 49.156
- type: recall_at_5
value: 53.937999999999995
- type: recall_at_10
value: 59.657000000000004
- type: recall_at_20
value: 65.995
- type: recall_at_100
value: 78.821
- type: recall_at_1000
value: 94.45
- type: precision_at_1
value: 37.409
- type: precision_at_3
value: 16.389
- type: precision_at_5
value: 10.789
- type: precision_at_10
value: 5.9670000000000005
- type: precision_at_20
value: 3.3000000000000003
- type: precision_at_100
value: 0.788
- type: precision_at_1000
value: 0.094
- type: mrr_at_1
value: 37.4086
- type: mrr_at_3
value: 42.587
- type: mrr_at_5
value: 43.6699
- type: mrr_at_10
value: 44.4297
- type: mrr_at_20
value: 44.8704
- type: mrr_at_100
value: 45.1881
- type: mrr_at_1000
value: 45.251000000000005
- type: nauc_ndcg_at_1_max
value: 61.8437
- type: nauc_ndcg_at_1_std
value: 10.782
- type: nauc_ndcg_at_1_diff1
value: 66.1842
- type: nauc_ndcg_at_3_max
value: 63.157399999999996
- type: nauc_ndcg_at_3_std
value: 13.114899999999999
- type: nauc_ndcg_at_3_diff1
value: 60.312
- type: nauc_ndcg_at_5_max
value: 63.027100000000004
- type: nauc_ndcg_at_5_std
value: 13.995099999999999
- type: nauc_ndcg_at_5_diff1
value: 59.272499999999994
- type: nauc_ndcg_at_10_max
value: 63.0273
- type: nauc_ndcg_at_10_std
value: 14.898700000000002
- type: nauc_ndcg_at_10_diff1
value: 58.2739
- type: nauc_ndcg_at_20_max
value: 62.785199999999996
- type: nauc_ndcg_at_20_std
value: 15.259800000000002
- type: nauc_ndcg_at_20_diff1
value: 57.8913
- type: nauc_ndcg_at_100_max
value: 62.641999999999996
- type: nauc_ndcg_at_100_std
value: 15.738299999999999
- type: nauc_ndcg_at_100_diff1
value: 58.2303
- type: nauc_ndcg_at_1000_max
value: 62.7624
- type: nauc_ndcg_at_1000_std
value: 15.1653
- type: nauc_ndcg_at_1000_diff1
value: 58.9359
- type: nauc_map_at_1_max
value: 61.800900000000006
- type: nauc_map_at_1_std
value: 10.7369
- type: nauc_map_at_1_diff1
value: 66.18270000000001
- type: nauc_map_at_3_max
value: 62.8757
- type: nauc_map_at_3_std
value: 12.5061
- type: nauc_map_at_3_diff1
value: 61.767
- type: nauc_map_at_5_max
value: 62.793299999999995
- type: nauc_map_at_5_std
value: 12.964500000000001
- type: nauc_map_at_5_diff1
value: 61.211000000000006
- type: nauc_map_at_10_max
value: 62.8054
- type: nauc_map_at_10_std
value: 13.328000000000001
- type: nauc_map_at_10_diff1
value: 60.833400000000005
- type: nauc_map_at_20_max
value: 62.734199999999994
- type: nauc_map_at_20_std
value: 13.4114
- type: nauc_map_at_20_diff1
value: 60.747099999999996
- type: nauc_map_at_100_max
value: 62.7054
- type: nauc_map_at_100_std
value: 13.4556
- type: nauc_map_at_100_diff1
value: 60.79259999999999
- type: nauc_map_at_1000_max
value: 62.71099999999999
- type: nauc_map_at_1000_std
value: 13.444400000000002
- type: nauc_map_at_1000_diff1
value: 60.815
- type: nauc_recall_at_1_max
value: 61.800900000000006
- type: nauc_recall_at_1_std
value: 10.7369
- type: nauc_recall_at_1_diff1
value: 66.18270000000001
- type: nauc_recall_at_3_max
value: 63.914300000000004
- type: nauc_recall_at_3_std
value: 14.8614
- type: nauc_recall_at_3_diff1
value: 56.044700000000006
- type: nauc_recall_at_5_max
value: 63.6523
- type: nauc_recall_at_5_std
value: 17.2352
- type: nauc_recall_at_5_diff1
value: 53.2316
- type: nauc_recall_at_10_max
value: 63.6138
- type: nauc_recall_at_10_std
value: 20.4315
- type: nauc_recall_at_10_diff1
value: 49.4388
- type: nauc_recall_at_20_max
value: 62.605
- type: nauc_recall_at_20_std
value: 22.8045
- type: nauc_recall_at_20_diff1
value: 46.5945
- type: nauc_recall_at_100_max
value: 61.5178
- type: nauc_recall_at_100_std
value: 30.4825
- type: nauc_recall_at_100_diff1
value: 44.9405
- type: nauc_recall_at_1000_max
value: 63.473
- type: nauc_recall_at_1000_std
value: 39.1421
- type: nauc_recall_at_1000_diff1
value: 43.4873
- type: nauc_precision_at_1_max
value: 61.8437
- type: nauc_precision_at_1_std
value: 10.782
- type: nauc_precision_at_1_diff1
value: 66.1842
- type: nauc_precision_at_3_max
value: 63.962799999999994
- type: nauc_precision_at_3_std
value: 14.908299999999999
- type: nauc_precision_at_3_diff1
value: 56.0511
- type: nauc_precision_at_5_max
value: 63.7072
- type: nauc_precision_at_5_std
value: 17.2854
- type: nauc_precision_at_5_diff1
value: 53.2417
- type: nauc_precision_at_10_max
value: 63.672200000000004
- type: nauc_precision_at_10_std
value: 20.485300000000002
- type: nauc_precision_at_10_diff1
value: 49.4491
- type: nauc_precision_at_20_max
value: 62.674600000000005
- type: nauc_precision_at_20_std
value: 22.8667
- type: nauc_precision_at_20_diff1
value: 46.6088
- type: nauc_precision_at_100_max
value: 61.622600000000006
- type: nauc_precision_at_100_std
value: 30.5766
- type: nauc_precision_at_100_diff1
value: 44.9643
- type: nauc_precision_at_1000_max
value: 63.131400000000006
- type: nauc_precision_at_1000_std
value: 39.6527
- type: nauc_precision_at_1000_diff1
value: 42.9196
- type: nauc_mrr_at_1_max
value: 61.8437
- type: nauc_mrr_at_1_std
value: 10.782
- type: nauc_mrr_at_1_diff1
value: 66.1842
- type: nauc_mrr_at_3_max
value: 62.9188
- type: nauc_mrr_at_3_std
value: 12.5514
- type: nauc_mrr_at_3_diff1
value: 61.768699999999995
- type: nauc_mrr_at_5_max
value: 62.836800000000004
- type: nauc_mrr_at_5_std
value: 13.0102
- type: nauc_mrr_at_5_diff1
value: 61.2128
- type: nauc_mrr_at_10_max
value: 62.8492
- type: nauc_mrr_at_10_std
value: 13.3741
- type: nauc_mrr_at_10_diff1
value: 60.8352
- type: nauc_mrr_at_20_max
value: 62.7783
- type: nauc_mrr_at_20_std
value: 13.4578
- type: nauc_mrr_at_20_diff1
value: 60.74889999999999
- type: nauc_mrr_at_100_max
value: 62.7497
- type: nauc_mrr_at_100_std
value: 13.5022
- type: nauc_mrr_at_100_diff1
value: 60.7944
- type: nauc_mrr_at_1000_max
value: 62.7546
- type: nauc_mrr_at_1000_std
value: 13.490499999999999
- type: nauc_mrr_at_1000_diff1
value: 60.8168
- type: main_score
value: 48.076
task:
type: Retrieval
- dataset:
config: ara-deu
name: MTEB MLQARetrieval (ara-deu)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 2.791
- type: ndcg_at_3
value: 4.3549999999999995
- type: ndcg_at_5
value: 5.07
- type: ndcg_at_10
value: 5.766
- type: ndcg_at_20
value: 6.417000000000001
- type: ndcg_at_100
value: 8.908000000000001
- type: ndcg_at_1000
value: 15.326
- type: map_at_1
value: 2.791
- type: map_at_3
value: 3.9539999999999997
- type: map_at_5
value: 4.346
- type: map_at_10
value: 4.638
- type: map_at_20
value: 4.812
- type: map_at_100
value: 5.124
- type: map_at_1000
value: 5.295
- type: recall_at_1
value: 2.791
- type: recall_at_3
value: 5.522
- type: recall_at_5
value: 7.282
- type: recall_at_10
value: 9.405
- type: recall_at_20
value: 12.015
- type: recall_at_100
value: 25.91
- type: recall_at_1000
value: 80.734
- type: precision_at_1
value: 2.791
- type: precision_at_3
value: 1.841
- type: precision_at_5
value: 1.456
- type: precision_at_10
value: 0.941
- type: precision_at_20
value: 0.601
- type: precision_at_100
value: 0.259
- type: precision_at_1000
value: 0.08099999999999999
- type: mrr_at_1
value: 2.7913
- type: mrr_at_3
value: 3.9543000000000004
- type: mrr_at_5
value: 4.3457
- type: mrr_at_10
value: 4.6375
- type: mrr_at_20
value: 4.8121
- type: mrr_at_100
value: 5.1238
- type: mrr_at_1000
value: 5.2951
- type: nauc_ndcg_at_1_max
value: 34.5978
- type: nauc_ndcg_at_1_std
value: 17.4239
- type: nauc_ndcg_at_1_diff1
value: 35.4068
- type: nauc_ndcg_at_3_max
value: 28.0902
- type: nauc_ndcg_at_3_std
value: 12.8766
- type: nauc_ndcg_at_3_diff1
value: 22.7616
- type: nauc_ndcg_at_5_max
value: 22.8207
- type: nauc_ndcg_at_5_std
value: 9.350200000000001
- type: nauc_ndcg_at_5_diff1
value: 18.7183
- type: nauc_ndcg_at_10_max
value: 20.48
- type: nauc_ndcg_at_10_std
value: 9.2047
- type: nauc_ndcg_at_10_diff1
value: 16.7652
- type: nauc_ndcg_at_20_max
value: 18.709999999999997
- type: nauc_ndcg_at_20_std
value: 9.0791
- type: nauc_ndcg_at_20_diff1
value: 15.6751
- type: nauc_ndcg_at_100_max
value: 13.586899999999998
- type: nauc_ndcg_at_100_std
value: 8.5573
- type: nauc_ndcg_at_100_diff1
value: 12.148100000000001
- type: nauc_ndcg_at_1000_max
value: 15.2638
- type: nauc_ndcg_at_1000_std
value: 7.302599999999999
- type: nauc_ndcg_at_1000_diff1
value: 12.572600000000001
- type: nauc_map_at_1_max
value: 34.5978
- type: nauc_map_at_1_std
value: 17.4239
- type: nauc_map_at_1_diff1
value: 35.4068
- type: nauc_map_at_3_max
value: 28.9982
- type: nauc_map_at_3_std
value: 13.498899999999999
- type: nauc_map_at_3_diff1
value: 24.8336
- type: nauc_map_at_5_max
value: 25.525799999999997
- type: nauc_map_at_5_std
value: 11.149199999999999
- type: nauc_map_at_5_diff1
value: 22.0446
- type: nauc_map_at_10_max
value: 24.0998
- type: nauc_map_at_10_std
value: 10.9335
- type: nauc_map_at_10_diff1
value: 20.7947
- type: nauc_map_at_20_max
value: 23.3003
- type: nauc_map_at_20_std
value: 10.7974
- type: nauc_map_at_20_diff1
value: 20.2792
- type: nauc_map_at_100_max
value: 22.081500000000002
- type: nauc_map_at_100_std
value: 10.7316
- type: nauc_map_at_100_diff1
value: 19.3017
- type: nauc_map_at_1000_max
value: 22.0606
- type: nauc_map_at_1000_std
value: 10.643600000000001
- type: nauc_map_at_1000_diff1
value: 19.2116
- type: nauc_recall_at_1_max
value: 34.5978
- type: nauc_recall_at_1_std
value: 17.4239
- type: nauc_recall_at_1_diff1
value: 35.4068
- type: nauc_recall_at_3_max
value: 26.2729
- type: nauc_recall_at_3_std
value: 11.6354
- type: nauc_recall_at_3_diff1
value: 18.5034
- type: nauc_recall_at_5_max
value: 17.7507
- type: nauc_recall_at_5_std
value: 5.9993
- type: nauc_recall_at_5_diff1
value: 12.6586
- type: nauc_recall_at_10_max
value: 14.6041
- type: nauc_recall_at_10_std
value: 6.549099999999999
- type: nauc_recall_at_10_diff1
value: 10.4291
- type: nauc_recall_at_20_max
value: 11.9476
- type: nauc_recall_at_20_std
value: 6.8790000000000004
- type: nauc_recall_at_20_diff1
value: 9.1605
- type: nauc_recall_at_100_max
value: 3.1727
- type: nauc_recall_at_100_std
value: 6.1497
- type: nauc_recall_at_100_diff1
value: 4.1892
- type: nauc_recall_at_1000_max
value: 6.299200000000001
- type: nauc_recall_at_1000_std
value: -1.9514
- type: nauc_recall_at_1000_diff1
value: 3.0020000000000002
- type: nauc_precision_at_1_max
value: 34.5978
- type: nauc_precision_at_1_std
value: 17.4239
- type: nauc_precision_at_1_diff1
value: 35.4068
- type: nauc_precision_at_3_max
value: 26.2729
- type: nauc_precision_at_3_std
value: 11.6354
- type: nauc_precision_at_3_diff1
value: 18.5034
- type: nauc_precision_at_5_max
value: 17.7507
- type: nauc_precision_at_5_std
value: 5.9993
- type: nauc_precision_at_5_diff1
value: 12.6586
- type: nauc_precision_at_10_max
value: 14.6041
- type: nauc_precision_at_10_std
value: 6.549099999999999
- type: nauc_precision_at_10_diff1
value: 10.4291
- type: nauc_precision_at_20_max
value: 11.9476
- type: nauc_precision_at_20_std
value: 6.8790000000000004
- type: nauc_precision_at_20_diff1
value: 9.1605
- type: nauc_precision_at_100_max
value: 3.1727
- type: nauc_precision_at_100_std
value: 6.1497
- type: nauc_precision_at_100_diff1
value: 4.1892
- type: nauc_precision_at_1000_max
value: 6.287800000000001
- type: nauc_precision_at_1000_std
value: -1.6616
- type: nauc_precision_at_1000_diff1
value: 2.9132000000000002
- type: nauc_mrr_at_1_max
value: 34.5978
- type: nauc_mrr_at_1_std
value: 17.4239
- type: nauc_mrr_at_1_diff1
value: 35.4068
- type: nauc_mrr_at_3_max
value: 28.9982
- type: nauc_mrr_at_3_std
value: 13.498899999999999
- type: nauc_mrr_at_3_diff1
value: 24.8336
- type: nauc_mrr_at_5_max
value: 25.525799999999997
- type: nauc_mrr_at_5_std
value: 11.149199999999999
- type: nauc_mrr_at_5_diff1
value: 22.0446
- type: nauc_mrr_at_10_max
value: 24.0998
- type: nauc_mrr_at_10_std
value: 10.9335
- type: nauc_mrr_at_10_diff1
value: 20.7947
- type: nauc_mrr_at_20_max
value: 23.3003
- type: nauc_mrr_at_20_std
value: 10.7974
- type: nauc_mrr_at_20_diff1
value: 20.2792
- type: nauc_mrr_at_100_max
value: 22.081500000000002
- type: nauc_mrr_at_100_std
value: 10.7316
- type: nauc_mrr_at_100_diff1
value: 19.3017
- type: nauc_mrr_at_1000_max
value: 22.060399999999998
- type: nauc_mrr_at_1000_std
value: 10.646700000000001
- type: nauc_mrr_at_1000_diff1
value: 19.2106
- type: main_score
value: 5.766
task:
type: Retrieval
- dataset:
config: ara-eng
name: MTEB MLQARetrieval (ara-eng)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 4.22
- type: ndcg_at_3
value: 5.6610000000000005
- type: ndcg_at_5
value: 6.326
- type: ndcg_at_10
value: 7.115
- type: ndcg_at_20
value: 7.936999999999999
- type: ndcg_at_100
value: 9.958
- type: ndcg_at_1000
value: 14.051
- type: map_at_1
value: 4.22
- type: map_at_3
value: 5.303999999999999
- type: map_at_5
value: 5.673
- type: map_at_10
value: 5.9959999999999996
- type: map_at_20
value: 6.22
- type: map_at_100
value: 6.47
- type: map_at_1000
value: 6.589
- type: recall_at_1
value: 4.22
- type: recall_at_3
value: 6.694999999999999
- type: recall_at_5
value: 8.308
- type: recall_at_10
value: 10.764999999999999
- type: recall_at_20
value: 14.029
- type: recall_at_100
value: 25.356
- type: recall_at_1000
value: 59.724
- type: precision_at_1
value: 4.22
- type: precision_at_3
value: 2.2319999999999998
- type: precision_at_5
value: 1.662
- type: precision_at_10
value: 1.077
- type: precision_at_20
value: 0.701
- type: precision_at_100
value: 0.254
- type: precision_at_1000
value: 0.06
- type: mrr_at_1
value: 4.2198
- type: mrr_at_3
value: 5.3045
- type: mrr_at_5
value: 5.673
- type: mrr_at_10
value: 5.9957
- type: mrr_at_20
value: 6.2199
- type: mrr_at_100
value: 6.47
- type: mrr_at_1000
value: 6.5886
- type: nauc_ndcg_at_1_max
value: 45.7573
- type: nauc_ndcg_at_1_std
value: 23.5781
- type: nauc_ndcg_at_1_diff1
value: 37.2693
- type: nauc_ndcg_at_3_max
value: 40.7861
- type: nauc_ndcg_at_3_std
value: 23.9256
- type: nauc_ndcg_at_3_diff1
value: 28.6483
- type: nauc_ndcg_at_5_max
value: 37.9242
- type: nauc_ndcg_at_5_std
value: 23.7166
- type: nauc_ndcg_at_5_diff1
value: 24.8838
- type: nauc_ndcg_at_10_max
value: 35.049
- type: nauc_ndcg_at_10_std
value: 23.6796
- type: nauc_ndcg_at_10_diff1
value: 21.646099999999997
- type: nauc_ndcg_at_20_max
value: 33.929500000000004
- type: nauc_ndcg_at_20_std
value: 24.2057
- type: nauc_ndcg_at_20_diff1
value: 19.796
- type: nauc_ndcg_at_100_max
value: 29.94
- type: nauc_ndcg_at_100_std
value: 23.7044
- type: nauc_ndcg_at_100_diff1
value: 16.0009
- type: nauc_ndcg_at_1000_max
value: 29.1111
- type: nauc_ndcg_at_1000_std
value: 23.602500000000003
- type: nauc_ndcg_at_1000_diff1
value: 15.564300000000001
- type: nauc_map_at_1_max
value: 45.7573
- type: nauc_map_at_1_std
value: 23.5781
- type: nauc_map_at_1_diff1
value: 37.2693
- type: nauc_map_at_3_max
value: 41.783
- type: nauc_map_at_3_std
value: 23.843700000000002
- type: nauc_map_at_3_diff1
value: 30.3402
- type: nauc_map_at_5_max
value: 39.9281
- type: nauc_map_at_5_std
value: 23.7139
- type: nauc_map_at_5_diff1
value: 27.895500000000002
- type: nauc_map_at_10_max
value: 38.412600000000005
- type: nauc_map_at_10_std
value: 23.7264
- type: nauc_map_at_10_diff1
value: 26.1631
- type: nauc_map_at_20_max
value: 37.9744
- type: nauc_map_at_20_std
value: 23.9232
- type: nauc_map_at_20_diff1
value: 25.426700000000004
- type: nauc_map_at_100_max
value: 37.1871
- type: nauc_map_at_100_std
value: 23.8582
- type: nauc_map_at_100_diff1
value: 24.6278
- type: nauc_map_at_1000_max
value: 37.1183
- type: nauc_map_at_1000_std
value: 23.8237
- type: nauc_map_at_1000_diff1
value: 24.564700000000002
- type: nauc_recall_at_1_max
value: 45.7573
- type: nauc_recall_at_1_std
value: 23.5781
- type: nauc_recall_at_1_diff1
value: 37.2693
- type: nauc_recall_at_3_max
value: 38.4929
- type: nauc_recall_at_3_std
value: 24.1177
- type: nauc_recall_at_3_diff1
value: 24.7676
- type: nauc_recall_at_5_max
value: 33.7023
- type: nauc_recall_at_5_std
value: 23.710800000000003
- type: nauc_recall_at_5_diff1
value: 18.5779
- type: nauc_recall_at_10_max
value: 28.675600000000003
- type: nauc_recall_at_10_std
value: 23.546
- type: nauc_recall_at_10_diff1
value: 13.2072
- type: nauc_recall_at_20_max
value: 27.0225
- type: nauc_recall_at_20_std
value: 24.744
- type: nauc_recall_at_20_diff1
value: 10.2559
- type: nauc_recall_at_100_max
value: 18.7396
- type: nauc_recall_at_100_std
value: 23.1994
- type: nauc_recall_at_100_diff1
value: 3.3373
- type: nauc_recall_at_1000_max
value: 15.7201
- type: nauc_recall_at_1000_std
value: 23.3749
- type: nauc_recall_at_1000_diff1
value: 2.0107
- type: nauc_precision_at_1_max
value: 45.7573
- type: nauc_precision_at_1_std
value: 23.5781
- type: nauc_precision_at_1_diff1
value: 37.2693
- type: nauc_precision_at_3_max
value: 38.4929
- type: nauc_precision_at_3_std
value: 24.1177
- type: nauc_precision_at_3_diff1
value: 24.7676
- type: nauc_precision_at_5_max
value: 33.7023
- type: nauc_precision_at_5_std
value: 23.710800000000003
- type: nauc_precision_at_5_diff1
value: 18.5779
- type: nauc_precision_at_10_max
value: 28.675600000000003
- type: nauc_precision_at_10_std
value: 23.546
- type: nauc_precision_at_10_diff1
value: 13.2072
- type: nauc_precision_at_20_max
value: 27.0225
- type: nauc_precision_at_20_std
value: 24.744
- type: nauc_precision_at_20_diff1
value: 10.2559
- type: nauc_precision_at_100_max
value: 18.7396
- type: nauc_precision_at_100_std
value: 23.1994
- type: nauc_precision_at_100_diff1
value: 3.3373
- type: nauc_precision_at_1000_max
value: 15.736
- type: nauc_precision_at_1000_std
value: 23.383599999999998
- type: nauc_precision_at_1000_diff1
value: 1.9839
- type: nauc_mrr_at_1_max
value: 45.7573
- type: nauc_mrr_at_1_std
value: 23.5781
- type: nauc_mrr_at_1_diff1
value: 37.2693
- type: nauc_mrr_at_3_max
value: 41.783
- type: nauc_mrr_at_3_std
value: 23.843700000000002
- type: nauc_mrr_at_3_diff1
value: 30.3402
- type: nauc_mrr_at_5_max
value: 39.9281
- type: nauc_mrr_at_5_std
value: 23.7139
- type: nauc_mrr_at_5_diff1
value: 27.895500000000002
- type: nauc_mrr_at_10_max
value: 38.412600000000005
- type: nauc_mrr_at_10_std
value: 23.7264
- type: nauc_mrr_at_10_diff1
value: 26.1631
- type: nauc_mrr_at_20_max
value: 37.9744
- type: nauc_mrr_at_20_std
value: 23.9232
- type: nauc_mrr_at_20_diff1
value: 25.426700000000004
- type: nauc_mrr_at_100_max
value: 37.1871
- type: nauc_mrr_at_100_std
value: 23.8582
- type: nauc_mrr_at_100_diff1
value: 24.6278
- type: nauc_mrr_at_1000_max
value: 37.1187
- type: nauc_mrr_at_1000_std
value: 23.824
- type: nauc_mrr_at_1000_diff1
value: 24.564700000000002
- type: main_score
value: 7.115
task:
type: Retrieval
- dataset:
config: ara-spa
name: MTEB MLQARetrieval (ara-spa)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 2.8819999999999997
- type: ndcg_at_3
value: 3.8920000000000003
- type: ndcg_at_5
value: 4.362
- type: ndcg_at_10
value: 4.874
- type: ndcg_at_20
value: 5.577
- type: ndcg_at_100
value: 7.475
- type: ndcg_at_1000
value: 13.319
- type: map_at_1
value: 2.8819999999999997
- type: map_at_3
value: 3.6319999999999997
- type: map_at_5
value: 3.887
- type: map_at_10
value: 4.092
- type: map_at_20
value: 4.285
- type: map_at_100
value: 4.52
- type: map_at_1000
value: 4.6739999999999995
- type: recall_at_1
value: 2.8819999999999997
- type: recall_at_3
value: 4.651000000000001
- type: recall_at_5
value: 5.814
- type: recall_at_10
value: 7.4319999999999995
- type: recall_at_20
value: 10.212
- type: recall_at_100
value: 20.880000000000003
- type: recall_at_1000
value: 70.829
- type: precision_at_1
value: 2.8819999999999997
- type: precision_at_3
value: 1.55
- type: precision_at_5
value: 1.163
- type: precision_at_10
value: 0.743
- type: precision_at_20
value: 0.511
- type: precision_at_100
value: 0.209
- type: precision_at_1000
value: 0.07100000000000001
- type: mrr_at_1
value: 2.8817
- type: mrr_at_3
value: 3.6316
- type: mrr_at_5
value: 3.8869000000000002
- type: mrr_at_10
value: 4.0919
- type: mrr_at_20
value: 4.285
- type: mrr_at_100
value: 4.5202
- type: mrr_at_1000
value: 4.6743
- type: nauc_ndcg_at_1_max
value: 47.1362
- type: nauc_ndcg_at_1_std
value: 14.2825
- type: nauc_ndcg_at_1_diff1
value: 38.096000000000004
- type: nauc_ndcg_at_3_max
value: 38.689099999999996
- type: nauc_ndcg_at_3_std
value: 12.6034
- type: nauc_ndcg_at_3_diff1
value: 30.3921
- type: nauc_ndcg_at_5_max
value: 34.777
- type: nauc_ndcg_at_5_std
value: 11.8104
- type: nauc_ndcg_at_5_diff1
value: 27.0992
- type: nauc_ndcg_at_10_max
value: 31.8033
- type: nauc_ndcg_at_10_std
value: 9.7654
- type: nauc_ndcg_at_10_diff1
value: 23.2973
- type: nauc_ndcg_at_20_max
value: 26.986900000000002
- type: nauc_ndcg_at_20_std
value: 10.7005
- type: nauc_ndcg_at_20_diff1
value: 18.296599999999998
- type: nauc_ndcg_at_100_max
value: 21.573999999999998
- type: nauc_ndcg_at_100_std
value: 8.9887
- type: nauc_ndcg_at_100_diff1
value: 13.883899999999999
- type: nauc_ndcg_at_1000_max
value: 20.985300000000002
- type: nauc_ndcg_at_1000_std
value: 8.067499999999999
- type: nauc_ndcg_at_1000_diff1
value: 13.703499999999998
- type: nauc_map_at_1_max
value: 47.1362
- type: nauc_map_at_1_std
value: 14.2825
- type: nauc_map_at_1_diff1
value: 38.096000000000004
- type: nauc_map_at_3_max
value: 40.436899999999994
- type: nauc_map_at_3_std
value: 12.9125
- type: nauc_map_at_3_diff1
value: 31.889
- type: nauc_map_at_5_max
value: 37.8897
- type: nauc_map_at_5_std
value: 12.3844
- type: nauc_map_at_5_diff1
value: 29.755100000000002
- type: nauc_map_at_10_max
value: 36.477599999999995
- type: nauc_map_at_10_std
value: 11.43
- type: nauc_map_at_10_diff1
value: 27.807900000000004
- type: nauc_map_at_20_max
value: 34.5317
- type: nauc_map_at_20_std
value: 11.6688
- type: nauc_map_at_20_diff1
value: 25.8067
- type: nauc_map_at_100_max
value: 33.2592
- type: nauc_map_at_100_std
value: 11.3353
- type: nauc_map_at_100_diff1
value: 24.659
- type: nauc_map_at_1000_max
value: 33.1157
- type: nauc_map_at_1000_std
value: 11.3467
- type: nauc_map_at_1000_diff1
value: 24.5554
- type: nauc_recall_at_1_max
value: 47.1362
- type: nauc_recall_at_1_std
value: 14.2825
- type: nauc_recall_at_1_diff1
value: 38.096000000000004
- type: nauc_recall_at_3_max
value: 34.7084
- type: nauc_recall_at_3_std
value: 11.9111
- type: nauc_recall_at_3_diff1
value: 27.0121
- type: nauc_recall_at_5_max
value: 28.368
- type: nauc_recall_at_5_std
value: 10.643600000000001
- type: nauc_recall_at_5_diff1
value: 21.6482
- type: nauc_recall_at_10_max
value: 23.1214
- type: nauc_recall_at_10_std
value: 6.5197
- type: nauc_recall_at_10_diff1
value: 14.9552
- type: nauc_recall_at_20_max
value: 15.0664
- type: nauc_recall_at_20_std
value: 9.4758
- type: nauc_recall_at_20_diff1
value: 6.3974
- type: nauc_recall_at_100_max
value: 7.5886
- type: nauc_recall_at_100_std
value: 6.01
- type: nauc_recall_at_100_diff1
value: 1.4206999999999999
- type: nauc_recall_at_1000_max
value: 4.789899999999999
- type: nauc_recall_at_1000_std
value: 1.011
- type: nauc_recall_at_1000_diff1
value: 0.0832
- type: nauc_precision_at_1_max
value: 47.1362
- type: nauc_precision_at_1_std
value: 14.2825
- type: nauc_precision_at_1_diff1
value: 38.096000000000004
- type: nauc_precision_at_3_max
value: 34.7084
- type: nauc_precision_at_3_std
value: 11.9111
- type: nauc_precision_at_3_diff1
value: 27.0121
- type: nauc_precision_at_5_max
value: 28.368
- type: nauc_precision_at_5_std
value: 10.643600000000001
- type: nauc_precision_at_5_diff1
value: 21.6482
- type: nauc_precision_at_10_max
value: 23.1214
- type: nauc_precision_at_10_std
value: 6.5197
- type: nauc_precision_at_10_diff1
value: 14.9552
- type: nauc_precision_at_20_max
value: 15.0664
- type: nauc_precision_at_20_std
value: 9.4758
- type: nauc_precision_at_20_diff1
value: 6.3974
- type: nauc_precision_at_100_max
value: 7.5886
- type: nauc_precision_at_100_std
value: 6.01
- type: nauc_precision_at_100_diff1
value: 1.4206999999999999
- type: nauc_precision_at_1000_max
value: 4.789899999999999
- type: nauc_precision_at_1000_std
value: 1.011
- type: nauc_precision_at_1000_diff1
value: 0.0832
- type: nauc_mrr_at_1_max
value: 47.1362
- type: nauc_mrr_at_1_std
value: 14.2825
- type: nauc_mrr_at_1_diff1
value: 38.096000000000004
- type: nauc_mrr_at_3_max
value: 40.436899999999994
- type: nauc_mrr_at_3_std
value: 12.9125
- type: nauc_mrr_at_3_diff1
value: 31.889
- type: nauc_mrr_at_5_max
value: 37.8897
- type: nauc_mrr_at_5_std
value: 12.3844
- type: nauc_mrr_at_5_diff1
value: 29.755100000000002
- type: nauc_mrr_at_10_max
value: 36.477599999999995
- type: nauc_mrr_at_10_std
value: 11.43
- type: nauc_mrr_at_10_diff1
value: 27.807900000000004
- type: nauc_mrr_at_20_max
value: 34.5317
- type: nauc_mrr_at_20_std
value: 11.6688
- type: nauc_mrr_at_20_diff1
value: 25.8067
- type: nauc_mrr_at_100_max
value: 33.2592
- type: nauc_mrr_at_100_std
value: 11.3353
- type: nauc_mrr_at_100_diff1
value: 24.659
- type: nauc_mrr_at_1000_max
value: 33.1157
- type: nauc_mrr_at_1000_std
value: 11.3467
- type: nauc_mrr_at_1000_diff1
value: 24.5554
- type: main_score
value: 4.874
task:
type: Retrieval
- dataset:
config: ara-hin
name: MTEB MLQARetrieval (ara-hin)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 0.819
- type: ndcg_at_3
value: 1.121
- type: ndcg_at_5
value: 1.299
- type: ndcg_at_10
value: 1.4569999999999999
- type: ndcg_at_20
value: 1.6650000000000003
- type: ndcg_at_100
value: 2.781
- type: ndcg_at_1000
value: 9.578000000000001
- type: map_at_1
value: 0.819
- type: map_at_3
value: 1.038
- type: map_at_5
value: 1.1360000000000001
- type: map_at_10
value: 1.2
- type: map_at_20
value: 1.258
- type: map_at_100
value: 1.391
- type: map_at_1000
value: 1.548
- type: recall_at_1
value: 0.819
- type: recall_at_3
value: 1.365
- type: recall_at_5
value: 1.802
- type: recall_at_10
value: 2.294
- type: recall_at_20
value: 3.113
- type: recall_at_100
value: 9.447999999999999
- type: recall_at_1000
value: 69.08800000000001
- type: precision_at_1
value: 0.819
- type: precision_at_3
value: 0.455
- type: precision_at_5
value: 0.36
- type: precision_at_10
value: 0.22899999999999998
- type: precision_at_20
value: 0.156
- type: precision_at_100
value: 0.094
- type: precision_at_1000
value: 0.06899999999999999
- type: mrr_at_1
value: 0.8191999999999999
- type: mrr_at_3
value: 1.0377
- type: mrr_at_5
value: 1.1360000000000001
- type: mrr_at_10
value: 1.2004000000000001
- type: mrr_at_20
value: 1.258
- type: mrr_at_100
value: 1.3911
- type: mrr_at_1000
value: 1.5481
- type: nauc_ndcg_at_1_max
value: 28.4629
- type: nauc_ndcg_at_1_std
value: 86.70360000000001
- type: nauc_ndcg_at_1_diff1
value: 43.5697
- type: nauc_ndcg_at_3_max
value: 18.0924
- type: nauc_ndcg_at_3_std
value: 78.1234
- type: nauc_ndcg_at_3_diff1
value: 33.7982
- type: nauc_ndcg_at_5_max
value: 11.5877
- type: nauc_ndcg_at_5_std
value: 74.4088
- type: nauc_ndcg_at_5_diff1
value: 27.729300000000002
- type: nauc_ndcg_at_10_max
value: 9.3156
- type: nauc_ndcg_at_10_std
value: 70.3377
- type: nauc_ndcg_at_10_diff1
value: 25.659
- type: nauc_ndcg_at_20_max
value: 5.0206
- type: nauc_ndcg_at_20_std
value: 63.075
- type: nauc_ndcg_at_20_diff1
value: 20.4666
- type: nauc_ndcg_at_100_max
value: -1.1461999999999999
- type: nauc_ndcg_at_100_std
value: 43.0295
- type: nauc_ndcg_at_100_diff1
value: 10.297
- type: nauc_ndcg_at_1000_max
value: -0.8538
- type: nauc_ndcg_at_1000_std
value: 33.259499999999996
- type: nauc_ndcg_at_1000_diff1
value: 7.0244
- type: nauc_map_at_1_max
value: 28.4629
- type: nauc_map_at_1_std
value: 86.70360000000001
- type: nauc_map_at_1_diff1
value: 43.5697
- type: nauc_map_at_3_max
value: 20.494200000000003
- type: nauc_map_at_3_std
value: 80.1961
- type: nauc_map_at_3_diff1
value: 36.4402
- type: nauc_map_at_5_max
value: 16.0907
- type: nauc_map_at_5_std
value: 77.84609999999999
- type: nauc_map_at_5_diff1
value: 32.2829
- type: nauc_map_at_10_max
value: 14.629900000000001
- type: nauc_map_at_10_std
value: 75.6746
- type: nauc_map_at_10_diff1
value: 30.972499999999997
- type: nauc_map_at_20_max
value: 12.7444
- type: nauc_map_at_20_std
value: 72.7994
- type: nauc_map_at_20_diff1
value: 28.7936
- type: nauc_map_at_100_max
value: 10.5063
- type: nauc_map_at_100_std
value: 66.96220000000001
- type: nauc_map_at_100_diff1
value: 25.623600000000003
- type: nauc_map_at_1000_max
value: 10.1672
- type: nauc_map_at_1000_std
value: 64.77579999999999
- type: nauc_map_at_1000_diff1
value: 24.873700000000003
- type: nauc_recall_at_1_max
value: 28.4629
- type: nauc_recall_at_1_std
value: 86.70360000000001
- type: nauc_recall_at_1_diff1
value: 43.5697
- type: nauc_recall_at_3_max
value: 12.6908
- type: nauc_recall_at_3_std
value: 73.4395
- type: nauc_recall_at_3_diff1
value: 27.758300000000002
- type: nauc_recall_at_5_max
value: 2.7872999999999997
- type: nauc_recall_at_5_std
value: 67.5835
- type: nauc_recall_at_5_diff1
value: 18.7977
- type: nauc_recall_at_10_max
value: 0.4262
- type: nauc_recall_at_10_std
value: 60.8976
- type: nauc_recall_at_10_diff1
value: 16.715
- type: nauc_recall_at_20_max
value: -6.0712
- type: nauc_recall_at_20_std
value: 48.002
- type: nauc_recall_at_20_diff1
value: 8.2361
- type: nauc_recall_at_100_max
value: -8.9148
- type: nauc_recall_at_100_std
value: 24.9211
- type: nauc_recall_at_100_diff1
value: -0.736
- type: nauc_recall_at_1000_max
value: -5.863
- type: nauc_recall_at_1000_std
value: 16.300700000000003
- type: nauc_recall_at_1000_diff1
value: -3.3021000000000003
- type: nauc_precision_at_1_max
value: 28.4629
- type: nauc_precision_at_1_std
value: 86.70360000000001
- type: nauc_precision_at_1_diff1
value: 43.5697
- type: nauc_precision_at_3_max
value: 12.6908
- type: nauc_precision_at_3_std
value: 73.4395
- type: nauc_precision_at_3_diff1
value: 27.758300000000002
- type: nauc_precision_at_5_max
value: 2.7872999999999997
- type: nauc_precision_at_5_std
value: 67.5835
- type: nauc_precision_at_5_diff1
value: 18.7977
- type: nauc_precision_at_10_max
value: 0.4262
- type: nauc_precision_at_10_std
value: 60.8976
- type: nauc_precision_at_10_diff1
value: 16.715
- type: nauc_precision_at_20_max
value: -6.0712
- type: nauc_precision_at_20_std
value: 48.002
- type: nauc_precision_at_20_diff1
value: 8.2361
- type: nauc_precision_at_100_max
value: -8.9148
- type: nauc_precision_at_100_std
value: 24.9211
- type: nauc_precision_at_100_diff1
value: -0.736
- type: nauc_precision_at_1000_max
value: -5.863
- type: nauc_precision_at_1000_std
value: 16.300700000000003
- type: nauc_precision_at_1000_diff1
value: -3.3021000000000003
- type: nauc_mrr_at_1_max
value: 28.4629
- type: nauc_mrr_at_1_std
value: 86.70360000000001
- type: nauc_mrr_at_1_diff1
value: 43.5697
- type: nauc_mrr_at_3_max
value: 20.494200000000003
- type: nauc_mrr_at_3_std
value: 80.1961
- type: nauc_mrr_at_3_diff1
value: 36.4402
- type: nauc_mrr_at_5_max
value: 16.0907
- type: nauc_mrr_at_5_std
value: 77.84609999999999
- type: nauc_mrr_at_5_diff1
value: 32.2829
- type: nauc_mrr_at_10_max
value: 14.629900000000001
- type: nauc_mrr_at_10_std
value: 75.6746
- type: nauc_mrr_at_10_diff1
value: 30.972499999999997
- type: nauc_mrr_at_20_max
value: 12.7444
- type: nauc_mrr_at_20_std
value: 72.7994
- type: nauc_mrr_at_20_diff1
value: 28.7936
- type: nauc_mrr_at_100_max
value: 10.5063
- type: nauc_mrr_at_100_std
value: 66.96220000000001
- type: nauc_mrr_at_100_diff1
value: 25.623600000000003
- type: nauc_mrr_at_1000_max
value: 10.1672
- type: nauc_mrr_at_1000_std
value: 64.77579999999999
- type: nauc_mrr_at_1000_diff1
value: 24.873700000000003
- type: main_score
value: 1.4569999999999999
task:
type: Retrieval
- dataset:
config: ara-vie
name: MTEB MLQARetrieval (ara-vie)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 4.006
- type: ndcg_at_3
value: 5.050000000000001
- type: ndcg_at_5
value: 5.902
- type: ndcg_at_10
value: 6.569
- type: ndcg_at_20
value: 7.140000000000001
- type: ndcg_at_100
value: 9.009
- type: ndcg_at_1000
value: 14.323
- type: map_at_1
value: 4.006
- type: map_at_3
value: 4.787
- type: map_at_5
value: 5.264
- type: map_at_10
value: 5.532
- type: map_at_20
value: 5.684
- type: map_at_100
value: 5.907
- type: map_at_1000
value: 6.0440000000000005
- type: recall_at_1
value: 4.006
- type: recall_at_3
value: 5.813
- type: recall_at_5
value: 7.865
- type: recall_at_10
value: 9.966
- type: recall_at_20
value: 12.262
- type: recall_at_100
value: 22.863
- type: recall_at_1000
value: 68.539
- type: precision_at_1
value: 4.006
- type: precision_at_3
value: 1.9380000000000002
- type: precision_at_5
value: 1.5730000000000002
- type: precision_at_10
value: 0.997
- type: precision_at_20
value: 0.613
- type: precision_at_100
value: 0.22899999999999998
- type: precision_at_1000
value: 0.06899999999999999
- type: mrr_at_1
value: 4.0059
- type: mrr_at_3
value: 4.7875
- type: mrr_at_5
value: 5.2638
- type: mrr_at_10
value: 5.532
- type: mrr_at_20
value: 5.6837
- type: mrr_at_100
value: 5.9072
- type: mrr_at_1000
value: 6.0439
- type: nauc_ndcg_at_1_max
value: 44.577099999999994
- type: nauc_ndcg_at_1_std
value: 10.616100000000001
- type: nauc_ndcg_at_1_diff1
value: 34.3529
- type: nauc_ndcg_at_3_max
value: 39.8485
- type: nauc_ndcg_at_3_std
value: 11.876000000000001
- type: nauc_ndcg_at_3_diff1
value: 27.028299999999998
- type: nauc_ndcg_at_5_max
value: 37.3023
- type: nauc_ndcg_at_5_std
value: 13.7019
- type: nauc_ndcg_at_5_diff1
value: 23.568
- type: nauc_ndcg_at_10_max
value: 35.0011
- type: nauc_ndcg_at_10_std
value: 13.831199999999999
- type: nauc_ndcg_at_10_diff1
value: 19.4633
- type: nauc_ndcg_at_20_max
value: 33.744800000000005
- type: nauc_ndcg_at_20_std
value: 14.3111
- type: nauc_ndcg_at_20_diff1
value: 18.5553
- type: nauc_ndcg_at_100_max
value: 30.2195
- type: nauc_ndcg_at_100_std
value: 12.7726
- type: nauc_ndcg_at_100_diff1
value: 15.2496
- type: nauc_ndcg_at_1000_max
value: 29.773500000000002
- type: nauc_ndcg_at_1000_std
value: 13.373099999999999
- type: nauc_ndcg_at_1000_diff1
value: 13.888900000000001
- type: nauc_map_at_1_max
value: 44.577099999999994
- type: nauc_map_at_1_std
value: 10.616100000000001
- type: nauc_map_at_1_diff1
value: 34.3529
- type: nauc_map_at_3_max
value: 40.868
- type: nauc_map_at_3_std
value: 11.6217
- type: nauc_map_at_3_diff1
value: 28.4768
- type: nauc_map_at_5_max
value: 39.1918
- type: nauc_map_at_5_std
value: 12.8027
- type: nauc_map_at_5_diff1
value: 26.1448
- type: nauc_map_at_10_max
value: 38.0953
- type: nauc_map_at_10_std
value: 12.9042
- type: nauc_map_at_10_diff1
value: 24.0735
- type: nauc_map_at_20_max
value: 37.6658
- type: nauc_map_at_20_std
value: 13.112599999999999
- type: nauc_map_at_20_diff1
value: 23.7032
- type: nauc_map_at_100_max
value: 37.0012
- type: nauc_map_at_100_std
value: 12.947600000000001
- type: nauc_map_at_100_diff1
value: 23.0135
- type: nauc_map_at_1000_max
value: 36.9286
- type: nauc_map_at_1000_std
value: 12.9997
- type: nauc_map_at_1000_diff1
value: 22.9205
- type: nauc_recall_at_1_max
value: 44.577099999999994
- type: nauc_recall_at_1_std
value: 10.616100000000001
- type: nauc_recall_at_1_diff1
value: 34.3529
- type: nauc_recall_at_3_max
value: 37.4035
- type: nauc_recall_at_3_std
value: 12.4802
- type: nauc_recall_at_3_diff1
value: 23.5961
- type: nauc_recall_at_5_max
value: 33.4019
- type: nauc_recall_at_5_std
value: 15.5884
- type: nauc_recall_at_5_diff1
value: 18.3202
- type: nauc_recall_at_10_max
value: 29.1543
- type: nauc_recall_at_10_std
value: 15.4612
- type: nauc_recall_at_10_diff1
value: 10.857700000000001
- type: nauc_recall_at_20_max
value: 26.7422
- type: nauc_recall_at_20_std
value: 16.2892
- type: nauc_recall_at_20_diff1
value: 9.7584
- type: nauc_recall_at_100_max
value: 19.994999999999997
- type: nauc_recall_at_100_std
value: 11.3707
- type: nauc_recall_at_100_diff1
value: 4.3003
- type: nauc_recall_at_1000_max
value: 17.8368
- type: nauc_recall_at_1000_std
value: 13.031699999999999
- type: nauc_recall_at_1000_diff1
value: -2.5641000000000003
- type: nauc_precision_at_1_max
value: 44.577099999999994
- type: nauc_precision_at_1_std
value: 10.616100000000001
- type: nauc_precision_at_1_diff1
value: 34.3529
- type: nauc_precision_at_3_max
value: 37.4035
- type: nauc_precision_at_3_std
value: 12.4802
- type: nauc_precision_at_3_diff1
value: 23.5961
- type: nauc_precision_at_5_max
value: 33.4019
- type: nauc_precision_at_5_std
value: 15.5884
- type: nauc_precision_at_5_diff1
value: 18.3202
- type: nauc_precision_at_10_max
value: 29.1543
- type: nauc_precision_at_10_std
value: 15.4612
- type: nauc_precision_at_10_diff1
value: 10.857700000000001
- type: nauc_precision_at_20_max
value: 26.7422
- type: nauc_precision_at_20_std
value: 16.2892
- type: nauc_precision_at_20_diff1
value: 9.7584
- type: nauc_precision_at_100_max
value: 19.994999999999997
- type: nauc_precision_at_100_std
value: 11.3707
- type: nauc_precision_at_100_diff1
value: 4.3003
- type: nauc_precision_at_1000_max
value: 17.8368
- type: nauc_precision_at_1000_std
value: 13.031699999999999
- type: nauc_precision_at_1000_diff1
value: -2.5641000000000003
- type: nauc_mrr_at_1_max
value: 44.577099999999994
- type: nauc_mrr_at_1_std
value: 10.616100000000001
- type: nauc_mrr_at_1_diff1
value: 34.3529
- type: nauc_mrr_at_3_max
value: 40.868
- type: nauc_mrr_at_3_std
value: 11.6217
- type: nauc_mrr_at_3_diff1
value: 28.4768
- type: nauc_mrr_at_5_max
value: 39.1918
- type: nauc_mrr_at_5_std
value: 12.8027
- type: nauc_mrr_at_5_diff1
value: 26.1448
- type: nauc_mrr_at_10_max
value: 38.0953
- type: nauc_mrr_at_10_std
value: 12.9042
- type: nauc_mrr_at_10_diff1
value: 24.0735
- type: nauc_mrr_at_20_max
value: 37.6658
- type: nauc_mrr_at_20_std
value: 13.112599999999999
- type: nauc_mrr_at_20_diff1
value: 23.7032
- type: nauc_mrr_at_100_max
value: 37.0012
- type: nauc_mrr_at_100_std
value: 12.947600000000001
- type: nauc_mrr_at_100_diff1
value: 23.0135
- type: nauc_mrr_at_1000_max
value: 36.9286
- type: nauc_mrr_at_1000_std
value: 12.9997
- type: nauc_mrr_at_1000_diff1
value: 22.9205
- type: main_score
value: 6.569
task:
type: Retrieval
- dataset:
config: ara-zho
name: MTEB MLQARetrieval (ara-zho)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 1.517
- type: ndcg_at_3
value: 1.8659999999999999
- type: ndcg_at_5
value: 1.976
- type: ndcg_at_10
value: 2.298
- type: ndcg_at_20
value: 2.6
- type: ndcg_at_100
value: 3.7510000000000003
- type: ndcg_at_1000
value: 9.671000000000001
- type: map_at_1
value: 1.517
- type: map_at_3
value: 1.7870000000000001
- type: map_at_5
value: 1.8499999999999999
- type: map_at_10
value: 1.983
- type: map_at_20
value: 2.0650000000000004
- type: map_at_100
value: 2.2030000000000003
- type: map_at_1000
value: 2.335
- type: recall_at_1
value: 1.517
- type: recall_at_3
value: 2.092
- type: recall_at_5
value: 2.3539999999999996
- type: recall_at_10
value: 3.347
- type: recall_at_20
value: 4.55
- type: recall_at_100
value: 11.088000000000001
- type: recall_at_1000
value: 63.285000000000004
- type: precision_at_1
value: 1.517
- type: precision_at_3
value: 0.697
- type: precision_at_5
value: 0.471
- type: precision_at_10
value: 0.335
- type: precision_at_20
value: 0.22799999999999998
- type: precision_at_100
value: 0.11100000000000002
- type: precision_at_1000
value: 0.063
- type: mrr_at_1
value: 1.5167
- type: mrr_at_3
value: 1.7870000000000001
- type: mrr_at_5
value: 1.8497
- type: mrr_at_10
value: 1.9826
- type: mrr_at_20
value: 2.0648
- type: mrr_at_100
value: 2.2030000000000003
- type: mrr_at_1000
value: 2.3349
- type: nauc_ndcg_at_1_max
value: 63.634299999999996
- type: nauc_ndcg_at_1_std
value: 95.4541
- type: nauc_ndcg_at_1_diff1
value: -24.7316
- type: nauc_ndcg_at_3_max
value: 53.355
- type: nauc_ndcg_at_3_std
value: 93.6084
- type: nauc_ndcg_at_3_diff1
value: -26.3293
- type: nauc_ndcg_at_5_max
value: 49.2227
- type: nauc_ndcg_at_5_std
value: 91.1334
- type: nauc_ndcg_at_5_diff1
value: -26.1085
- type: nauc_ndcg_at_10_max
value: 41.6822
- type: nauc_ndcg_at_10_std
value: 87.53
- type: nauc_ndcg_at_10_diff1
value: -28.179
- type: nauc_ndcg_at_20_max
value: 35.585499999999996
- type: nauc_ndcg_at_20_std
value: 80.7905
- type: nauc_ndcg_at_20_diff1
value: -26.804299999999998
- type: nauc_ndcg_at_100_max
value: 20.177300000000002
- type: nauc_ndcg_at_100_std
value: 62.33030000000001
- type: nauc_ndcg_at_100_diff1
value: -23.2323
- type: nauc_ndcg_at_1000_max
value: 12.9937
- type: nauc_ndcg_at_1000_std
value: 48.0397
- type: nauc_ndcg_at_1000_diff1
value: -20.048099999999998
- type: nauc_map_at_1_max
value: 63.634299999999996
- type: nauc_map_at_1_std
value: 95.4541
- type: nauc_map_at_1_diff1
value: -24.7316
- type: nauc_map_at_3_max
value: 55.5442
- type: nauc_map_at_3_std
value: 94.1071
- type: nauc_map_at_3_diff1
value: -26.223200000000002
- type: nauc_map_at_5_max
value: 52.9787
- type: nauc_map_at_5_std
value: 92.66
- type: nauc_map_at_5_diff1
value: -26.0793
- type: nauc_map_at_10_max
value: 48.9187
- type: nauc_map_at_10_std
value: 90.8484
- type: nauc_map_at_10_diff1
value: -27.150000000000002
- type: nauc_map_at_20_max
value: 46.544799999999995
- type: nauc_map_at_20_std
value: 88.45530000000001
- type: nauc_map_at_20_diff1
value: -26.677
- type: nauc_map_at_100_max
value: 42.7356
- type: nauc_map_at_100_std
value: 84.4144
- type: nauc_map_at_100_diff1
value: -25.987
- type: nauc_map_at_1000_max
value: 41.639700000000005
- type: nauc_map_at_1000_std
value: 82.8745
- type: nauc_map_at_1000_diff1
value: -25.5622
- type: nauc_recall_at_1_max
value: 63.634299999999996
- type: nauc_recall_at_1_std
value: 95.4541
- type: nauc_recall_at_1_diff1
value: -24.7316
- type: nauc_recall_at_3_max
value: 47.9288
- type: nauc_recall_at_3_std
value: 92.33670000000001
- type: nauc_recall_at_3_diff1
value: -26.512900000000002
- type: nauc_recall_at_5_max
value: 40.2409
- type: nauc_recall_at_5_std
value: 87.3105
- type: nauc_recall_at_5_diff1
value: -26.096200000000003
- type: nauc_recall_at_10_max
value: 27.5162
- type: nauc_recall_at_10_std
value: 80.73989999999999
- type: nauc_recall_at_10_diff1
value: -30.259000000000004
- type: nauc_recall_at_20_max
value: 17.3798
- type: nauc_recall_at_20_std
value: 67.0634
- type: nauc_recall_at_20_diff1
value: -26.7445
- type: nauc_recall_at_100_max
value: -2.2403
- type: nauc_recall_at_100_std
value: 38.4021
- type: nauc_recall_at_100_diff1
value: -19.6933
- type: nauc_recall_at_1000_max
value: -7.8642
- type: nauc_recall_at_1000_std
value: 17.9166
- type: nauc_recall_at_1000_diff1
value: -15.6436
- type: nauc_precision_at_1_max
value: 63.634299999999996
- type: nauc_precision_at_1_std
value: 95.4541
- type: nauc_precision_at_1_diff1
value: -24.7316
- type: nauc_precision_at_3_max
value: 47.9288
- type: nauc_precision_at_3_std
value: 92.33670000000001
- type: nauc_precision_at_3_diff1
value: -26.512900000000002
- type: nauc_precision_at_5_max
value: 40.2409
- type: nauc_precision_at_5_std
value: 87.3105
- type: nauc_precision_at_5_diff1
value: -26.096200000000003
- type: nauc_precision_at_10_max
value: 27.5162
- type: nauc_precision_at_10_std
value: 80.73989999999999
- type: nauc_precision_at_10_diff1
value: -30.259000000000004
- type: nauc_precision_at_20_max
value: 17.3798
- type: nauc_precision_at_20_std
value: 67.0634
- type: nauc_precision_at_20_diff1
value: -26.7445
- type: nauc_precision_at_100_max
value: -2.2403
- type: nauc_precision_at_100_std
value: 38.4021
- type: nauc_precision_at_100_diff1
value: -19.6933
- type: nauc_precision_at_1000_max
value: -7.8642
- type: nauc_precision_at_1000_std
value: 17.9166
- type: nauc_precision_at_1000_diff1
value: -15.6436
- type: nauc_mrr_at_1_max
value: 63.634299999999996
- type: nauc_mrr_at_1_std
value: 95.4541
- type: nauc_mrr_at_1_diff1
value: -24.7316
- type: nauc_mrr_at_3_max
value: 55.5442
- type: nauc_mrr_at_3_std
value: 94.1071
- type: nauc_mrr_at_3_diff1
value: -26.223200000000002
- type: nauc_mrr_at_5_max
value: 52.9787
- type: nauc_mrr_at_5_std
value: 92.66
- type: nauc_mrr_at_5_diff1
value: -26.0793
- type: nauc_mrr_at_10_max
value: 48.9187
- type: nauc_mrr_at_10_std
value: 90.8484
- type: nauc_mrr_at_10_diff1
value: -27.150000000000002
- type: nauc_mrr_at_20_max
value: 46.544799999999995
- type: nauc_mrr_at_20_std
value: 88.45530000000001
- type: nauc_mrr_at_20_diff1
value: -26.677
- type: nauc_mrr_at_100_max
value: 42.7356
- type: nauc_mrr_at_100_std
value: 84.4144
- type: nauc_mrr_at_100_diff1
value: -25.987
- type: nauc_mrr_at_1000_max
value: 41.639700000000005
- type: nauc_mrr_at_1000_std
value: 82.8745
- type: nauc_mrr_at_1000_diff1
value: -25.5622
- type: main_score
value: 2.298
task:
type: Retrieval
- dataset:
config: deu-ara
name: MTEB MLQARetrieval (deu-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 7.277
- type: ndcg_at_3
value: 9.679
- type: ndcg_at_5
value: 10.331
- type: ndcg_at_10
value: 11.507000000000001
- type: ndcg_at_20
value: 12.631999999999998
- type: ndcg_at_100
value: 15.307
- type: ndcg_at_1000
value: 21.534
- type: map_at_1
value: 7.277
- type: map_at_3
value: 9.066
- type: map_at_5
value: 9.43
- type: map_at_10
value: 9.915000000000001
- type: map_at_20
value: 10.227
- type: map_at_100
value: 10.577
- type: map_at_1000
value: 10.744
- type: recall_at_1
value: 7.277
- type: recall_at_3
value: 11.461
- type: recall_at_5
value: 13.038
- type: recall_at_10
value: 16.677
- type: recall_at_20
value: 21.104
- type: recall_at_100
value: 35.839999999999996
- type: recall_at_1000
value: 88.842
- type: precision_at_1
value: 7.277
- type: precision_at_3
value: 3.82
- type: precision_at_5
value: 2.608
- type: precision_at_10
value: 1.6680000000000001
- type: precision_at_20
value: 1.055
- type: precision_at_100
value: 0.358
- type: precision_at_1000
value: 0.089
- type: mrr_at_1
value: 7.2771
- type: mrr_at_3
value: 9.0661
- type: mrr_at_5
value: 9.43
- type: mrr_at_10
value: 9.9147
- type: mrr_at_20
value: 10.2265
- type: mrr_at_100
value: 10.577
- type: mrr_at_1000
value: 10.7438
- type: nauc_ndcg_at_1_max
value: 47.8688
- type: nauc_ndcg_at_1_std
value: 32.6716
- type: nauc_ndcg_at_1_diff1
value: 46.5387
- type: nauc_ndcg_at_3_max
value: 43.919200000000004
- type: nauc_ndcg_at_3_std
value: 32.7668
- type: nauc_ndcg_at_3_diff1
value: 37.4043
- type: nauc_ndcg_at_5_max
value: 43.068
- type: nauc_ndcg_at_5_std
value: 32.0507
- type: nauc_ndcg_at_5_diff1
value: 36.8883
- type: nauc_ndcg_at_10_max
value: 40.9779
- type: nauc_ndcg_at_10_std
value: 31.059900000000003
- type: nauc_ndcg_at_10_diff1
value: 35.016000000000005
- type: nauc_ndcg_at_20_max
value: 40.4037
- type: nauc_ndcg_at_20_std
value: 30.0859
- type: nauc_ndcg_at_20_diff1
value: 33.9489
- type: nauc_ndcg_at_100_max
value: 40.1081
- type: nauc_ndcg_at_100_std
value: 31.3556
- type: nauc_ndcg_at_100_diff1
value: 30.457
- type: nauc_ndcg_at_1000_max
value: 40.2846
- type: nauc_ndcg_at_1000_std
value: 31.1719
- type: nauc_ndcg_at_1000_diff1
value: 31.5382
- type: nauc_map_at_1_max
value: 47.8688
- type: nauc_map_at_1_std
value: 32.6716
- type: nauc_map_at_1_diff1
value: 46.5387
- type: nauc_map_at_3_max
value: 44.6183
- type: nauc_map_at_3_std
value: 32.5048
- type: nauc_map_at_3_diff1
value: 39.2215
- type: nauc_map_at_5_max
value: 44.122099999999996
- type: nauc_map_at_5_std
value: 32.095
- type: nauc_map_at_5_diff1
value: 38.860299999999995
- type: nauc_map_at_10_max
value: 43.119600000000005
- type: nauc_map_at_10_std
value: 31.6621
- type: nauc_map_at_10_diff1
value: 37.957
- type: nauc_map_at_20_max
value: 42.8588
- type: nauc_map_at_20_std
value: 31.3143
- type: nauc_map_at_20_diff1
value: 37.532
- type: nauc_map_at_100_max
value: 42.7644
- type: nauc_map_at_100_std
value: 31.4857
- type: nauc_map_at_100_diff1
value: 36.9312
- type: nauc_map_at_1000_max
value: 42.7352
- type: nauc_map_at_1000_std
value: 31.4817
- type: nauc_map_at_1000_diff1
value: 36.9232
- type: nauc_recall_at_1_max
value: 47.8688
- type: nauc_recall_at_1_std
value: 32.6716
- type: nauc_recall_at_1_diff1
value: 46.5387
- type: nauc_recall_at_3_max
value: 42.3086
- type: nauc_recall_at_3_std
value: 33.4621
- type: nauc_recall_at_3_diff1
value: 33.1431
- type: nauc_recall_at_5_max
value: 40.651399999999995
- type: nauc_recall_at_5_std
value: 31.935799999999997
- type: nauc_recall_at_5_diff1
value: 32.4983
- type: nauc_recall_at_10_max
value: 36.2724
- type: nauc_recall_at_10_std
value: 29.6023
- type: nauc_recall_at_10_diff1
value: 28.849200000000003
- type: nauc_recall_at_20_max
value: 35.614000000000004
- type: nauc_recall_at_20_std
value: 27.3563
- type: nauc_recall_at_20_diff1
value: 27.0766
- type: nauc_recall_at_100_max
value: 35.6417
- type: nauc_recall_at_100_std
value: 32.0197
- type: nauc_recall_at_100_diff1
value: 17.066100000000002
- type: nauc_recall_at_1000_max
value: 32.5633
- type: nauc_recall_at_1000_std
value: 31.3263
- type: nauc_recall_at_1000_diff1
value: 7.978000000000001
- type: nauc_precision_at_1_max
value: 47.8688
- type: nauc_precision_at_1_std
value: 32.6716
- type: nauc_precision_at_1_diff1
value: 46.5387
- type: nauc_precision_at_3_max
value: 42.3086
- type: nauc_precision_at_3_std
value: 33.4621
- type: nauc_precision_at_3_diff1
value: 33.1431
- type: nauc_precision_at_5_max
value: 40.651399999999995
- type: nauc_precision_at_5_std
value: 31.935799999999997
- type: nauc_precision_at_5_diff1
value: 32.4983
- type: nauc_precision_at_10_max
value: 36.2724
- type: nauc_precision_at_10_std
value: 29.6023
- type: nauc_precision_at_10_diff1
value: 28.849200000000003
- type: nauc_precision_at_20_max
value: 35.614000000000004
- type: nauc_precision_at_20_std
value: 27.3563
- type: nauc_precision_at_20_diff1
value: 27.0766
- type: nauc_precision_at_100_max
value: 35.6417
- type: nauc_precision_at_100_std
value: 32.0197
- type: nauc_precision_at_100_diff1
value: 17.066100000000002
- type: nauc_precision_at_1000_max
value: 32.5633
- type: nauc_precision_at_1000_std
value: 31.3263
- type: nauc_precision_at_1000_diff1
value: 7.978000000000001
- type: nauc_mrr_at_1_max
value: 47.8688
- type: nauc_mrr_at_1_std
value: 32.6716
- type: nauc_mrr_at_1_diff1
value: 46.5387
- type: nauc_mrr_at_3_max
value: 44.6183
- type: nauc_mrr_at_3_std
value: 32.5048
- type: nauc_mrr_at_3_diff1
value: 39.2215
- type: nauc_mrr_at_5_max
value: 44.122099999999996
- type: nauc_mrr_at_5_std
value: 32.095
- type: nauc_mrr_at_5_diff1
value: 38.860299999999995
- type: nauc_mrr_at_10_max
value: 43.119600000000005
- type: nauc_mrr_at_10_std
value: 31.6621
- type: nauc_mrr_at_10_diff1
value: 37.957
- type: nauc_mrr_at_20_max
value: 42.8588
- type: nauc_mrr_at_20_std
value: 31.3143
- type: nauc_mrr_at_20_diff1
value: 37.532
- type: nauc_mrr_at_100_max
value: 42.7644
- type: nauc_mrr_at_100_std
value: 31.4857
- type: nauc_mrr_at_100_diff1
value: 36.9312
- type: nauc_mrr_at_1000_max
value: 42.7352
- type: nauc_mrr_at_1000_std
value: 31.4817
- type: nauc_mrr_at_1000_diff1
value: 36.9232
- type: main_score
value: 11.507000000000001
task:
type: Retrieval
- dataset:
config: eng-ara
name: MTEB MLQARetrieval (eng-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 9.318999999999999
- type: ndcg_at_3
value: 12.491
- type: ndcg_at_5
value: 13.739
- type: ndcg_at_10
value: 15.155
- type: ndcg_at_20
value: 16.68
- type: ndcg_at_100
value: 19.3
- type: ndcg_at_1000
value: 22.942999999999998
- type: map_at_1
value: 9.31
- type: map_at_3
value: 11.684999999999999
- type: map_at_5
value: 12.376
- type: map_at_10
value: 12.959000000000001
- type: map_at_20
value: 13.374
- type: map_at_100
value: 13.71
- type: map_at_1000
value: 13.815
- type: recall_at_1
value: 9.31
- type: recall_at_3
value: 14.823
- type: recall_at_5
value: 17.86
- type: recall_at_10
value: 22.248
- type: recall_at_20
value: 28.305000000000003
- type: recall_at_100
value: 42.828
- type: recall_at_1000
value: 73.411
- type: precision_at_1
value: 9.318999999999999
- type: precision_at_3
value: 4.944
- type: precision_at_5
value: 3.5740000000000003
- type: precision_at_10
value: 2.226
- type: precision_at_20
value: 1.4160000000000001
- type: precision_at_100
value: 0.428
- type: precision_at_1000
value: 0.073
- type: mrr_at_1
value: 9.3193
- type: mrr_at_3
value: 11.6945
- type: mrr_at_5
value: 12.3855
- type: mrr_at_10
value: 12.9683
- type: mrr_at_20
value: 13.3831
- type: mrr_at_100
value: 13.7187
- type: mrr_at_1000
value: 13.8241
- type: nauc_ndcg_at_1_max
value: 45.0893
- type: nauc_ndcg_at_1_std
value: 32.0194
- type: nauc_ndcg_at_1_diff1
value: 42.4658
- type: nauc_ndcg_at_3_max
value: 43.308400000000006
- type: nauc_ndcg_at_3_std
value: 34.098099999999995
- type: nauc_ndcg_at_3_diff1
value: 34.645900000000005
- type: nauc_ndcg_at_5_max
value: 42.0717
- type: nauc_ndcg_at_5_std
value: 33.7493
- type: nauc_ndcg_at_5_diff1
value: 32.1742
- type: nauc_ndcg_at_10_max
value: 41.3293
- type: nauc_ndcg_at_10_std
value: 34.0058
- type: nauc_ndcg_at_10_diff1
value: 30.4406
- type: nauc_ndcg_at_20_max
value: 40.8478
- type: nauc_ndcg_at_20_std
value: 34.4417
- type: nauc_ndcg_at_20_diff1
value: 29.3758
- type: nauc_ndcg_at_100_max
value: 39.7843
- type: nauc_ndcg_at_100_std
value: 35.0737
- type: nauc_ndcg_at_100_diff1
value: 28.2686
- type: nauc_ndcg_at_1000_max
value: 39.9456
- type: nauc_ndcg_at_1000_std
value: 34.777
- type: nauc_ndcg_at_1000_diff1
value: 28.885499999999997
- type: nauc_map_at_1_max
value: 45.176500000000004
- type: nauc_map_at_1_std
value: 32.0417
- type: nauc_map_at_1_diff1
value: 42.5589
- type: nauc_map_at_3_max
value: 43.756099999999996
- type: nauc_map_at_3_std
value: 33.645599999999995
- type: nauc_map_at_3_diff1
value: 36.3289
- type: nauc_map_at_5_max
value: 43.012299999999996
- type: nauc_map_at_5_std
value: 33.4766
- type: nauc_map_at_5_diff1
value: 34.780499999999996
- type: nauc_map_at_10_max
value: 42.636
- type: nauc_map_at_10_std
value: 33.5984
- type: nauc_map_at_10_diff1
value: 33.9015
- type: nauc_map_at_20_max
value: 42.4673
- type: nauc_map_at_20_std
value: 33.7222
- type: nauc_map_at_20_diff1
value: 33.5187
- type: nauc_map_at_100_max
value: 42.2746
- type: nauc_map_at_100_std
value: 33.8046
- type: nauc_map_at_100_diff1
value: 33.3105
- type: nauc_map_at_1000_max
value: 42.2774
- type: nauc_map_at_1000_std
value: 33.7939
- type: nauc_map_at_1000_diff1
value: 33.3391
- type: nauc_recall_at_1_max
value: 45.176500000000004
- type: nauc_recall_at_1_std
value: 32.0417
- type: nauc_recall_at_1_diff1
value: 42.5589
- type: nauc_recall_at_3_max
value: 42.257099999999994
- type: nauc_recall_at_3_std
value: 35.2072
- type: nauc_recall_at_3_diff1
value: 30.6237
- type: nauc_recall_at_5_max
value: 39.8985
- type: nauc_recall_at_5_std
value: 34.300999999999995
- type: nauc_recall_at_5_diff1
value: 26.259300000000003
- type: nauc_recall_at_10_max
value: 38.467
- type: nauc_recall_at_10_std
value: 34.8373
- type: nauc_recall_at_10_diff1
value: 22.9966
- type: nauc_recall_at_20_max
value: 37.434200000000004
- type: nauc_recall_at_20_std
value: 36.0462
- type: nauc_recall_at_20_diff1
value: 20.8388
- type: nauc_recall_at_100_max
value: 33.9476
- type: nauc_recall_at_100_std
value: 38.4241
- type: nauc_recall_at_100_diff1
value: 17.5443
- type: nauc_recall_at_1000_max
value: 32.5266
- type: nauc_recall_at_1000_std
value: 37.7257
- type: nauc_recall_at_1000_diff1
value: 16.796
- type: nauc_precision_at_1_max
value: 45.0893
- type: nauc_precision_at_1_std
value: 32.0194
- type: nauc_precision_at_1_diff1
value: 42.4658
- type: nauc_precision_at_3_max
value: 42.207699999999996
- type: nauc_precision_at_3_std
value: 35.1976
- type: nauc_precision_at_3_diff1
value: 30.573800000000002
- type: nauc_precision_at_5_max
value: 39.856700000000004
- type: nauc_precision_at_5_std
value: 34.293099999999995
- type: nauc_precision_at_5_diff1
value: 26.217800000000004
- type: nauc_precision_at_10_max
value: 38.4379
- type: nauc_precision_at_10_std
value: 34.8365
- type: nauc_precision_at_10_diff1
value: 22.9656
- type: nauc_precision_at_20_max
value: 37.4081
- type: nauc_precision_at_20_std
value: 36.045300000000005
- type: nauc_precision_at_20_diff1
value: 20.8112
- type: nauc_precision_at_100_max
value: 33.8791
- type: nauc_precision_at_100_std
value: 38.398199999999996
- type: nauc_precision_at_100_diff1
value: 17.483
- type: nauc_precision_at_1000_max
value: 32.368900000000004
- type: nauc_precision_at_1000_std
value: 37.6796
- type: nauc_precision_at_1000_diff1
value: 16.6938
- type: nauc_mrr_at_1_max
value: 45.0893
- type: nauc_mrr_at_1_std
value: 32.0194
- type: nauc_mrr_at_1_diff1
value: 42.4658
- type: nauc_mrr_at_3_max
value: 43.6834
- type: nauc_mrr_at_3_std
value: 33.6254
- type: nauc_mrr_at_3_diff1
value: 36.2554
- type: nauc_mrr_at_5_max
value: 42.9432
- type: nauc_mrr_at_5_std
value: 33.457300000000004
- type: nauc_mrr_at_5_diff1
value: 34.7112
- type: nauc_mrr_at_10_max
value: 42.569
- type: nauc_mrr_at_10_std
value: 33.5795
- type: nauc_mrr_at_10_diff1
value: 33.834599999999995
- type: nauc_mrr_at_20_max
value: 42.4011
- type: nauc_mrr_at_20_std
value: 33.7034
- type: nauc_mrr_at_20_diff1
value: 33.4529
- type: nauc_mrr_at_100_max
value: 42.2124
- type: nauc_mrr_at_100_std
value: 33.7868
- type: nauc_mrr_at_100_diff1
value: 33.2487
- type: nauc_mrr_at_1000_max
value: 42.2151
- type: nauc_mrr_at_1000_std
value: 33.7761
- type: nauc_mrr_at_1000_diff1
value: 33.2771
- type: main_score
value: 15.155
task:
type: Retrieval
- dataset:
config: spa-ara
name: MTEB MLQARetrieval (spa-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 4.752
- type: ndcg_at_3
value: 6.393
- type: ndcg_at_5
value: 7.1499999999999995
- type: ndcg_at_10
value: 8.133
- type: ndcg_at_20
value: 9.016
- type: ndcg_at_100
value: 11.258
- type: ndcg_at_1000
value: 16.875
- type: map_at_1
value: 4.752
- type: map_at_3
value: 5.982
- type: map_at_5
value: 6.407
- type: map_at_10
value: 6.814000000000001
- type: map_at_20
value: 7.049999999999999
- type: map_at_100
value: 7.340000000000001
- type: map_at_1000
value: 7.488
- type: recall_at_1
value: 4.752
- type: recall_at_3
value: 7.582999999999999
- type: recall_at_5
value: 9.403
- type: recall_at_10
value: 12.437
- type: recall_at_20
value: 15.976
- type: recall_at_100
value: 28.362
- type: recall_at_1000
value: 76.39
- type: precision_at_1
value: 4.752
- type: precision_at_3
value: 2.528
- type: precision_at_5
value: 1.881
- type: precision_at_10
value: 1.244
- type: precision_at_20
value: 0.799
- type: precision_at_100
value: 0.28400000000000003
- type: precision_at_1000
value: 0.076
- type: mrr_at_1
value: 4.7523
- type: mrr_at_3
value: 5.9825
- type: mrr_at_5
value: 6.407100000000001
- type: mrr_at_10
value: 6.8138000000000005
- type: mrr_at_20
value: 7.0498
- type: mrr_at_100
value: 7.3404
- type: mrr_at_1000
value: 7.4879
- type: nauc_ndcg_at_1_max
value: 56.8283
- type: nauc_ndcg_at_1_std
value: 48.84
- type: nauc_ndcg_at_1_diff1
value: 38.2398
- type: nauc_ndcg_at_3_max
value: 46.0021
- type: nauc_ndcg_at_3_std
value: 40.9246
- type: nauc_ndcg_at_3_diff1
value: 31.483100000000004
- type: nauc_ndcg_at_5_max
value: 43.3732
- type: nauc_ndcg_at_5_std
value: 40.313900000000004
- type: nauc_ndcg_at_5_diff1
value: 27.905400000000004
- type: nauc_ndcg_at_10_max
value: 41.2826
- type: nauc_ndcg_at_10_std
value: 39.362
- type: nauc_ndcg_at_10_diff1
value: 24.7114
- type: nauc_ndcg_at_20_max
value: 39.129799999999996
- type: nauc_ndcg_at_20_std
value: 37.6917
- type: nauc_ndcg_at_20_diff1
value: 22.9635
- type: nauc_ndcg_at_100_max
value: 37.062400000000004
- type: nauc_ndcg_at_100_std
value: 36.4566
- type: nauc_ndcg_at_100_diff1
value: 21.5259
- type: nauc_ndcg_at_1000_max
value: 38.057900000000004
- type: nauc_ndcg_at_1000_std
value: 36.775600000000004
- type: nauc_ndcg_at_1000_diff1
value: 21.6453
- type: nauc_map_at_1_max
value: 56.8283
- type: nauc_map_at_1_std
value: 48.84
- type: nauc_map_at_1_diff1
value: 38.2398
- type: nauc_map_at_3_max
value: 48.106100000000005
- type: nauc_map_at_3_std
value: 42.485800000000005
- type: nauc_map_at_3_diff1
value: 32.7816
- type: nauc_map_at_5_max
value: 46.328399999999995
- type: nauc_map_at_5_std
value: 41.9688
- type: nauc_map_at_5_diff1
value: 30.4734
- type: nauc_map_at_10_max
value: 45.1533
- type: nauc_map_at_10_std
value: 41.43
- type: nauc_map_at_10_diff1
value: 28.749000000000002
- type: nauc_map_at_20_max
value: 44.405699999999996
- type: nauc_map_at_20_std
value: 40.8655
- type: nauc_map_at_20_diff1
value: 28.1439
- type: nauc_map_at_100_max
value: 43.8735
- type: nauc_map_at_100_std
value: 40.5522
- type: nauc_map_at_100_diff1
value: 27.7292
- type: nauc_map_at_1000_max
value: 43.8779
- type: nauc_map_at_1000_std
value: 40.527499999999996
- type: nauc_map_at_1000_diff1
value: 27.6959
- type: nauc_recall_at_1_max
value: 56.8283
- type: nauc_recall_at_1_std
value: 48.84
- type: nauc_recall_at_1_diff1
value: 38.2398
- type: nauc_recall_at_3_max
value: 41.2025
- type: nauc_recall_at_3_std
value: 37.3559
- type: nauc_recall_at_3_diff1
value: 28.525699999999997
- type: nauc_recall_at_5_max
value: 37.3073
- type: nauc_recall_at_5_std
value: 37.0189
- type: nauc_recall_at_5_diff1
value: 22.5119
- type: nauc_recall_at_10_max
value: 34.1699
- type: nauc_recall_at_10_std
value: 35.6651
- type: nauc_recall_at_10_diff1
value: 17.0338
- type: nauc_recall_at_20_max
value: 29.956100000000003
- type: nauc_recall_at_20_std
value: 32.1089
- type: nauc_recall_at_20_diff1
value: 13.839699999999999
- type: nauc_recall_at_100_max
value: 27.210600000000003
- type: nauc_recall_at_100_std
value: 30.466500000000003
- type: nauc_recall_at_100_diff1
value: 12.812000000000001
- type: nauc_recall_at_1000_max
value: 28.7444
- type: nauc_recall_at_1000_std
value: 30.451299999999996
- type: nauc_recall_at_1000_diff1
value: 10.0083
- type: nauc_precision_at_1_max
value: 56.8283
- type: nauc_precision_at_1_std
value: 48.84
- type: nauc_precision_at_1_diff1
value: 38.2398
- type: nauc_precision_at_3_max
value: 41.2025
- type: nauc_precision_at_3_std
value: 37.3559
- type: nauc_precision_at_3_diff1
value: 28.525699999999997
- type: nauc_precision_at_5_max
value: 37.3073
- type: nauc_precision_at_5_std
value: 37.0189
- type: nauc_precision_at_5_diff1
value: 22.5119
- type: nauc_precision_at_10_max
value: 34.1699
- type: nauc_precision_at_10_std
value: 35.6651
- type: nauc_precision_at_10_diff1
value: 17.0338
- type: nauc_precision_at_20_max
value: 29.956100000000003
- type: nauc_precision_at_20_std
value: 32.1089
- type: nauc_precision_at_20_diff1
value: 13.839699999999999
- type: nauc_precision_at_100_max
value: 27.210600000000003
- type: nauc_precision_at_100_std
value: 30.466500000000003
- type: nauc_precision_at_100_diff1
value: 12.812000000000001
- type: nauc_precision_at_1000_max
value: 28.7444
- type: nauc_precision_at_1000_std
value: 30.451299999999996
- type: nauc_precision_at_1000_diff1
value: 10.0083
- type: nauc_mrr_at_1_max
value: 56.8283
- type: nauc_mrr_at_1_std
value: 48.84
- type: nauc_mrr_at_1_diff1
value: 38.2398
- type: nauc_mrr_at_3_max
value: 48.106100000000005
- type: nauc_mrr_at_3_std
value: 42.485800000000005
- type: nauc_mrr_at_3_diff1
value: 32.7816
- type: nauc_mrr_at_5_max
value: 46.328399999999995
- type: nauc_mrr_at_5_std
value: 41.9688
- type: nauc_mrr_at_5_diff1
value: 30.4734
- type: nauc_mrr_at_10_max
value: 45.1533
- type: nauc_mrr_at_10_std
value: 41.43
- type: nauc_mrr_at_10_diff1
value: 28.749000000000002
- type: nauc_mrr_at_20_max
value: 44.405699999999996
- type: nauc_mrr_at_20_std
value: 40.8655
- type: nauc_mrr_at_20_diff1
value: 28.1439
- type: nauc_mrr_at_100_max
value: 43.8735
- type: nauc_mrr_at_100_std
value: 40.5522
- type: nauc_mrr_at_100_diff1
value: 27.7292
- type: nauc_mrr_at_1000_max
value: 43.8779
- type: nauc_mrr_at_1000_std
value: 40.527499999999996
- type: nauc_mrr_at_1000_diff1
value: 27.6959
- type: main_score
value: 8.133
task:
type: Retrieval
- dataset:
config: hin-ara
name: MTEB MLQARetrieval (hin-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 1.966
- type: ndcg_at_3
value: 3.4639999999999995
- type: ndcg_at_5
value: 3.8609999999999998
- type: ndcg_at_10
value: 4.311
- type: ndcg_at_20
value: 4.671
- type: ndcg_at_100
value: 6.401999999999999
- type: ndcg_at_1000
value: 12.772
- type: map_at_1
value: 1.966
- type: map_at_3
value: 3.0949999999999998
- type: map_at_5
value: 3.311
- type: map_at_10
value: 3.4909999999999997
- type: map_at_20
value: 3.5900000000000003
- type: map_at_100
value: 3.81
- type: map_at_1000
value: 3.963
- type: recall_at_1
value: 1.966
- type: recall_at_3
value: 4.533
- type: recall_at_5
value: 5.516
- type: recall_at_10
value: 6.936000000000001
- type: recall_at_20
value: 8.356
- type: recall_at_100
value: 17.968
- type: recall_at_1000
value: 73.457
- type: precision_at_1
value: 1.966
- type: precision_at_3
value: 1.5110000000000001
- type: precision_at_5
value: 1.103
- type: precision_at_10
value: 0.694
- type: precision_at_20
value: 0.418
- type: precision_at_100
value: 0.18
- type: precision_at_1000
value: 0.073
- type: mrr_at_1
value: 1.9661000000000002
- type: mrr_at_3
value: 3.0947999999999998
- type: mrr_at_5
value: 3.3105999999999995
- type: mrr_at_10
value: 3.4908
- type: mrr_at_20
value: 3.5904
- type: mrr_at_100
value: 3.8104
- type: mrr_at_1000
value: 3.9633000000000003
- type: nauc_ndcg_at_1_max
value: 59.740700000000004
- type: nauc_ndcg_at_1_std
value: 31.430000000000003
- type: nauc_ndcg_at_1_diff1
value: 43.9493
- type: nauc_ndcg_at_3_max
value: 49.201899999999995
- type: nauc_ndcg_at_3_std
value: 26.483
- type: nauc_ndcg_at_3_diff1
value: 26.7397
- type: nauc_ndcg_at_5_max
value: 48.6642
- type: nauc_ndcg_at_5_std
value: 27.133000000000003
- type: nauc_ndcg_at_5_diff1
value: 22.6607
- type: nauc_ndcg_at_10_max
value: 45.3352
- type: nauc_ndcg_at_10_std
value: 23.6477
- type: nauc_ndcg_at_10_diff1
value: 20.5931
- type: nauc_ndcg_at_20_max
value: 43.3667
- type: nauc_ndcg_at_20_std
value: 22.3281
- type: nauc_ndcg_at_20_diff1
value: 21.063499999999998
- type: nauc_ndcg_at_100_max
value: 34.1469
- type: nauc_ndcg_at_100_std
value: 18.507199999999997
- type: nauc_ndcg_at_100_diff1
value: 15.1472
- type: nauc_ndcg_at_1000_max
value: 31.802999999999997
- type: nauc_ndcg_at_1000_std
value: 17.680699999999998
- type: nauc_ndcg_at_1000_diff1
value: 14.7831
- type: nauc_map_at_1_max
value: 59.740700000000004
- type: nauc_map_at_1_std
value: 31.430000000000003
- type: nauc_map_at_1_diff1
value: 43.9493
- type: nauc_map_at_3_max
value: 50.89939999999999
- type: nauc_map_at_3_std
value: 27.5359
- type: nauc_map_at_3_diff1
value: 29.3284
- type: nauc_map_at_5_max
value: 50.524800000000006
- type: nauc_map_at_5_std
value: 27.941399999999998
- type: nauc_map_at_5_diff1
value: 26.6
- type: nauc_map_at_10_max
value: 48.7603
- type: nauc_map_at_10_std
value: 26.1898
- type: nauc_map_at_10_diff1
value: 25.306800000000003
- type: nauc_map_at_20_max
value: 48.0022
- type: nauc_map_at_20_std
value: 25.6818
- type: nauc_map_at_20_diff1
value: 25.339499999999997
- type: nauc_map_at_100_max
value: 45.8106
- type: nauc_map_at_100_std
value: 24.7555
- type: nauc_map_at_100_diff1
value: 23.8426
- type: nauc_map_at_1000_max
value: 45.511
- type: nauc_map_at_1000_std
value: 24.6218
- type: nauc_map_at_1000_diff1
value: 23.7391
- type: nauc_recall_at_1_max
value: 59.740700000000004
- type: nauc_recall_at_1_std
value: 31.430000000000003
- type: nauc_recall_at_1_diff1
value: 43.9493
- type: nauc_recall_at_3_max
value: 45.8341
- type: nauc_recall_at_3_std
value: 24.3252
- type: nauc_recall_at_3_diff1
value: 21.6529
- type: nauc_recall_at_5_max
value: 45.3117
- type: nauc_recall_at_5_std
value: 25.6816
- type: nauc_recall_at_5_diff1
value: 15.4277
- type: nauc_recall_at_10_max
value: 39.5271
- type: nauc_recall_at_10_std
value: 19.1111
- type: nauc_recall_at_10_diff1
value: 13.0274
- type: nauc_recall_at_20_max
value: 36.0293
- type: nauc_recall_at_20_std
value: 16.8502
- type: nauc_recall_at_20_diff1
value: 15.3637
- type: nauc_recall_at_100_max
value: 19.9218
- type: nauc_recall_at_100_std
value: 11.107899999999999
- type: nauc_recall_at_100_diff1
value: 5.5936
- type: nauc_recall_at_1000_max
value: 11.8912
- type: nauc_recall_at_1000_std
value: 8.205
- type: nauc_recall_at_1000_diff1
value: 3.8231
- type: nauc_precision_at_1_max
value: 59.740700000000004
- type: nauc_precision_at_1_std
value: 31.430000000000003
- type: nauc_precision_at_1_diff1
value: 43.9493
- type: nauc_precision_at_3_max
value: 45.8341
- type: nauc_precision_at_3_std
value: 24.3252
- type: nauc_precision_at_3_diff1
value: 21.6529
- type: nauc_precision_at_5_max
value: 45.3117
- type: nauc_precision_at_5_std
value: 25.6816
- type: nauc_precision_at_5_diff1
value: 15.4277
- type: nauc_precision_at_10_max
value: 39.5271
- type: nauc_precision_at_10_std
value: 19.1111
- type: nauc_precision_at_10_diff1
value: 13.0274
- type: nauc_precision_at_20_max
value: 36.0293
- type: nauc_precision_at_20_std
value: 16.8502
- type: nauc_precision_at_20_diff1
value: 15.3637
- type: nauc_precision_at_100_max
value: 19.9218
- type: nauc_precision_at_100_std
value: 11.107899999999999
- type: nauc_precision_at_100_diff1
value: 5.5936
- type: nauc_precision_at_1000_max
value: 11.8912
- type: nauc_precision_at_1000_std
value: 8.205
- type: nauc_precision_at_1000_diff1
value: 3.8231
- type: nauc_mrr_at_1_max
value: 59.740700000000004
- type: nauc_mrr_at_1_std
value: 31.430000000000003
- type: nauc_mrr_at_1_diff1
value: 43.9493
- type: nauc_mrr_at_3_max
value: 50.89939999999999
- type: nauc_mrr_at_3_std
value: 27.5359
- type: nauc_mrr_at_3_diff1
value: 29.3284
- type: nauc_mrr_at_5_max
value: 50.524800000000006
- type: nauc_mrr_at_5_std
value: 27.941399999999998
- type: nauc_mrr_at_5_diff1
value: 26.6
- type: nauc_mrr_at_10_max
value: 48.7603
- type: nauc_mrr_at_10_std
value: 26.1898
- type: nauc_mrr_at_10_diff1
value: 25.306800000000003
- type: nauc_mrr_at_20_max
value: 48.0022
- type: nauc_mrr_at_20_std
value: 25.6818
- type: nauc_mrr_at_20_diff1
value: 25.339499999999997
- type: nauc_mrr_at_100_max
value: 45.8106
- type: nauc_mrr_at_100_std
value: 24.7555
- type: nauc_mrr_at_100_diff1
value: 23.8426
- type: nauc_mrr_at_1000_max
value: 45.511
- type: nauc_mrr_at_1000_std
value: 24.6218
- type: nauc_mrr_at_1000_diff1
value: 23.7391
- type: main_score
value: 4.311
task:
type: Retrieval
- dataset:
config: vie-ara
name: MTEB MLQARetrieval (vie-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 4.543
- type: ndcg_at_3
value: 6.595
- type: ndcg_at_5
value: 7.621
- type: ndcg_at_10
value: 8.845
- type: ndcg_at_20
value: 9.67
- type: ndcg_at_100
value: 12.115
- type: ndcg_at_1000
value: 17.675
- type: map_at_1
value: 4.543
- type: map_at_3
value: 6.074
- type: map_at_5
value: 6.643000000000001
- type: map_at_10
value: 7.142999999999999
- type: map_at_20
value: 7.369000000000001
- type: map_at_100
value: 7.675999999999999
- type: map_at_1000
value: 7.826
- type: recall_at_1
value: 4.543
- type: recall_at_3
value: 8.109
- type: recall_at_5
value: 10.600999999999999
- type: recall_at_10
value: 14.411
- type: recall_at_20
value: 17.684
- type: recall_at_100
value: 31.313999999999997
- type: recall_at_1000
value: 78.749
- type: precision_at_1
value: 4.543
- type: precision_at_3
value: 2.703
- type: precision_at_5
value: 2.12
- type: precision_at_10
value: 1.4409999999999998
- type: precision_at_20
value: 0.8840000000000001
- type: precision_at_100
value: 0.313
- type: precision_at_1000
value: 0.079
- type: mrr_at_1
value: 4.5432
- type: mrr_at_3
value: 6.0739
- type: mrr_at_5
value: 6.6431000000000004
- type: mrr_at_10
value: 7.143199999999999
- type: mrr_at_20
value: 7.3685
- type: mrr_at_100
value: 7.6761
- type: mrr_at_1000
value: 7.826099999999999
- type: nauc_ndcg_at_1_max
value: 48.486200000000004
- type: nauc_ndcg_at_1_std
value: 38.5379
- type: nauc_ndcg_at_1_diff1
value: 42.8345
- type: nauc_ndcg_at_3_max
value: 45.327
- type: nauc_ndcg_at_3_std
value: 37.5748
- type: nauc_ndcg_at_3_diff1
value: 33.6995
- type: nauc_ndcg_at_5_max
value: 43.7954
- type: nauc_ndcg_at_5_std
value: 36.5083
- type: nauc_ndcg_at_5_diff1
value: 28.7847
- type: nauc_ndcg_at_10_max
value: 42.754599999999996
- type: nauc_ndcg_at_10_std
value: 36.4811
- type: nauc_ndcg_at_10_diff1
value: 27.0578
- type: nauc_ndcg_at_20_max
value: 41.6852
- type: nauc_ndcg_at_20_std
value: 35.8215
- type: nauc_ndcg_at_20_diff1
value: 25.190600000000003
- type: nauc_ndcg_at_100_max
value: 40.1615
- type: nauc_ndcg_at_100_std
value: 35.6043
- type: nauc_ndcg_at_100_diff1
value: 24.2871
- type: nauc_ndcg_at_1000_max
value: 39.5862
- type: nauc_ndcg_at_1000_std
value: 35.5414
- type: nauc_ndcg_at_1000_diff1
value: 24.151400000000002
- type: nauc_map_at_1_max
value: 48.486200000000004
- type: nauc_map_at_1_std
value: 38.5379
- type: nauc_map_at_1_diff1
value: 42.8345
- type: nauc_map_at_3_max
value: 46.011
- type: nauc_map_at_3_std
value: 37.8819
- type: nauc_map_at_3_diff1
value: 35.3171
- type: nauc_map_at_5_max
value: 45.0173
- type: nauc_map_at_5_std
value: 37.2376
- type: nauc_map_at_5_diff1
value: 32.0669
- type: nauc_map_at_10_max
value: 44.4675
- type: nauc_map_at_10_std
value: 37.2262
- type: nauc_map_at_10_diff1
value: 31.0951
- type: nauc_map_at_20_max
value: 44.0847
- type: nauc_map_at_20_std
value: 36.998799999999996
- type: nauc_map_at_20_diff1
value: 30.331599999999998
- type: nauc_map_at_100_max
value: 43.6615
- type: nauc_map_at_100_std
value: 36.8941
- type: nauc_map_at_100_diff1
value: 29.978700000000003
- type: nauc_map_at_1000_max
value: 43.6041
- type: nauc_map_at_1000_std
value: 36.8918
- type: nauc_map_at_1000_diff1
value: 29.948399999999996
- type: nauc_recall_at_1_max
value: 48.486200000000004
- type: nauc_recall_at_1_std
value: 38.5379
- type: nauc_recall_at_1_diff1
value: 42.8345
- type: nauc_recall_at_3_max
value: 43.8133
- type: nauc_recall_at_3_std
value: 36.869800000000005
- type: nauc_recall_at_3_diff1
value: 30.2123
- type: nauc_recall_at_5_max
value: 41.3657
- type: nauc_recall_at_5_std
value: 34.9987
- type: nauc_recall_at_5_diff1
value: 22.3402
- type: nauc_recall_at_10_max
value: 39.679199999999994
- type: nauc_recall_at_10_std
value: 35.1281
- type: nauc_recall_at_10_diff1
value: 20.0132
- type: nauc_recall_at_20_max
value: 37.547999999999995
- type: nauc_recall_at_20_std
value: 33.728
- type: nauc_recall_at_20_diff1
value: 16.7879
- type: nauc_recall_at_100_max
value: 35.0701
- type: nauc_recall_at_100_std
value: 33.886300000000006
- type: nauc_recall_at_100_diff1
value: 16.956
- type: nauc_recall_at_1000_max
value: 28.922700000000003
- type: nauc_recall_at_1000_std
value: 32.1604
- type: nauc_recall_at_1000_diff1
value: 11.8056
- type: nauc_precision_at_1_max
value: 48.486200000000004
- type: nauc_precision_at_1_std
value: 38.5379
- type: nauc_precision_at_1_diff1
value: 42.8345
- type: nauc_precision_at_3_max
value: 43.8133
- type: nauc_precision_at_3_std
value: 36.869800000000005
- type: nauc_precision_at_3_diff1
value: 30.2123
- type: nauc_precision_at_5_max
value: 41.3657
- type: nauc_precision_at_5_std
value: 34.9987
- type: nauc_precision_at_5_diff1
value: 22.3402
- type: nauc_precision_at_10_max
value: 39.679199999999994
- type: nauc_precision_at_10_std
value: 35.1281
- type: nauc_precision_at_10_diff1
value: 20.0132
- type: nauc_precision_at_20_max
value: 37.547999999999995
- type: nauc_precision_at_20_std
value: 33.728
- type: nauc_precision_at_20_diff1
value: 16.7879
- type: nauc_precision_at_100_max
value: 35.0701
- type: nauc_precision_at_100_std
value: 33.886300000000006
- type: nauc_precision_at_100_diff1
value: 16.956
- type: nauc_precision_at_1000_max
value: 28.922700000000003
- type: nauc_precision_at_1000_std
value: 32.1604
- type: nauc_precision_at_1000_diff1
value: 11.8056
- type: nauc_mrr_at_1_max
value: 48.486200000000004
- type: nauc_mrr_at_1_std
value: 38.5379
- type: nauc_mrr_at_1_diff1
value: 42.8345
- type: nauc_mrr_at_3_max
value: 46.011
- type: nauc_mrr_at_3_std
value: 37.8819
- type: nauc_mrr_at_3_diff1
value: 35.3171
- type: nauc_mrr_at_5_max
value: 45.0173
- type: nauc_mrr_at_5_std
value: 37.2376
- type: nauc_mrr_at_5_diff1
value: 32.0669
- type: nauc_mrr_at_10_max
value: 44.4675
- type: nauc_mrr_at_10_std
value: 37.2262
- type: nauc_mrr_at_10_diff1
value: 31.0951
- type: nauc_mrr_at_20_max
value: 44.0847
- type: nauc_mrr_at_20_std
value: 36.998799999999996
- type: nauc_mrr_at_20_diff1
value: 30.331599999999998
- type: nauc_mrr_at_100_max
value: 43.6615
- type: nauc_mrr_at_100_std
value: 36.8941
- type: nauc_mrr_at_100_diff1
value: 29.978700000000003
- type: nauc_mrr_at_1000_max
value: 43.6041
- type: nauc_mrr_at_1000_std
value: 36.8918
- type: nauc_mrr_at_1000_diff1
value: 29.948399999999996
- type: main_score
value: 8.845
task:
type: Retrieval
- dataset:
config: zho-ara
name: MTEB MLQARetrieval (zho-ara)
revision: 397ed406c1a7902140303e7faf60fff35b58d285
split: test
type: facebook/mlqa
metrics:
- type: ndcg_at_1
value: 3.872
- type: ndcg_at_3
value: 5.29
- type: ndcg_at_5
value: 5.832
- type: ndcg_at_10
value: 6.339
- type: ndcg_at_20
value: 6.916
- type: ndcg_at_100
value: 8.459999999999999
- type: ndcg_at_1000
value: 13.841999999999999
- type: map_at_1
value: 3.872
- type: map_at_3
value: 4.962
- type: map_at_5
value: 5.266
- type: map_at_10
value: 5.473999999999999
- type: map_at_20
value: 5.63
- type: map_at_100
value: 5.816000000000001
- type: map_at_1000
value: 5.9479999999999995
- type: recall_at_1
value: 3.872
- type: recall_at_3
value: 6.227
- type: recall_at_5
value: 7.535
- type: recall_at_10
value: 9.105
- type: recall_at_20
value: 11.408
- type: recall_at_100
value: 20.147000000000002
- type: recall_at_1000
value: 66.876
- type: precision_at_1
value: 3.872
- type: precision_at_3
value: 2.076
- type: precision_at_5
value: 1.5070000000000001
- type: precision_at_10
value: 0.911
- type: precision_at_20
value: 0.5700000000000001
- type: precision_at_100
value: 0.201
- type: precision_at_1000
value: 0.067
- type: mrr_at_1
value: 3.8723
- type: mrr_at_3
value: 4.9625
- type: mrr_at_5
value: 5.266
- type: mrr_at_10
value: 5.4741
- type: mrr_at_20
value: 5.6301
- type: mrr_at_100
value: 5.8158
- type: mrr_at_1000
value: 5.9481
- type: nauc_ndcg_at_1_max
value: 68.4911
- type: nauc_ndcg_at_1_std
value: 37.5175
- type: nauc_ndcg_at_1_diff1
value: 53.80740000000001
- type: nauc_ndcg_at_3_max
value: 58.3947
- type: nauc_ndcg_at_3_std
value: 33.907199999999996
- type: nauc_ndcg_at_3_diff1
value: 36.8774
- type: nauc_ndcg_at_5_max
value: 58.0924
- type: nauc_ndcg_at_5_std
value: 33.6458
- type: nauc_ndcg_at_5_diff1
value: 35.6709
- type: nauc_ndcg_at_10_max
value: 54.031099999999995
- type: nauc_ndcg_at_10_std
value: 33.0146
- type: nauc_ndcg_at_10_diff1
value: 32.1803
- type: nauc_ndcg_at_20_max
value: 52.032199999999996
- type: nauc_ndcg_at_20_std
value: 32.7671
- type: nauc_ndcg_at_20_diff1
value: 30.3153
- type: nauc_ndcg_at_100_max
value: 46.935300000000005
- type: nauc_ndcg_at_100_std
value: 29.4879
- type: nauc_ndcg_at_100_diff1
value: 27.7553
- type: nauc_ndcg_at_1000_max
value: 44.128499999999995
- type: nauc_ndcg_at_1000_std
value: 27.8742
- type: nauc_ndcg_at_1000_diff1
value: 24.7943
- type: nauc_map_at_1_max
value: 68.4911
- type: nauc_map_at_1_std
value: 37.5175
- type: nauc_map_at_1_diff1
value: 53.80740000000001
- type: nauc_map_at_3_max
value: 60.39099999999999
- type: nauc_map_at_3_std
value: 34.7021
- type: nauc_map_at_3_diff1
value: 40.122099999999996
- type: nauc_map_at_5_max
value: 60.1398
- type: nauc_map_at_5_std
value: 34.5078
- type: nauc_map_at_5_diff1
value: 39.207300000000004
- type: nauc_map_at_10_max
value: 58.1008
- type: nauc_map_at_10_std
value: 34.1899
- type: nauc_map_at_10_diff1
value: 37.4103
- type: nauc_map_at_20_max
value: 57.3164
- type: nauc_map_at_20_std
value: 34.0334
- type: nauc_map_at_20_diff1
value: 36.6298
- type: nauc_map_at_100_max
value: 56.3412
- type: nauc_map_at_100_std
value: 33.4595
- type: nauc_map_at_100_diff1
value: 36.0572
- type: nauc_map_at_1000_max
value: 56.1735
- type: nauc_map_at_1000_std
value: 33.3775
- type: nauc_map_at_1000_diff1
value: 35.8636
- type: nauc_recall_at_1_max
value: 68.4911
- type: nauc_recall_at_1_std
value: 37.5175
- type: nauc_recall_at_1_diff1
value: 53.80740000000001
- type: nauc_recall_at_3_max
value: 53.771899999999995
- type: nauc_recall_at_3_std
value: 32.0411
- type: nauc_recall_at_3_diff1
value: 29.396099999999997
- type: nauc_recall_at_5_max
value: 53.8627
- type: nauc_recall_at_5_std
value: 31.838699999999996
- type: nauc_recall_at_5_diff1
value: 28.3882
- type: nauc_recall_at_10_max
value: 45.913
- type: nauc_recall_at_10_std
value: 30.7397
- type: nauc_recall_at_10_diff1
value: 22.115499999999997
- type: nauc_recall_at_20_max
value: 42.4268
- type: nauc_recall_at_20_std
value: 30.6865
- type: nauc_recall_at_20_diff1
value: 19.3518
- type: nauc_recall_at_100_max
value: 32.3071
- type: nauc_recall_at_100_std
value: 22.7085
- type: nauc_recall_at_100_diff1
value: 16.379099999999998
- type: nauc_recall_at_1000_max
value: 20.924699999999998
- type: nauc_recall_at_1000_std
value: 16.1665
- type: nauc_recall_at_1000_diff1
value: 4.9604
- type: nauc_precision_at_1_max
value: 68.4911
- type: nauc_precision_at_1_std
value: 37.5175
- type: nauc_precision_at_1_diff1
value: 53.80740000000001
- type: nauc_precision_at_3_max
value: 53.771899999999995
- type: nauc_precision_at_3_std
value: 32.0411
- type: nauc_precision_at_3_diff1
value: 29.396099999999997
- type: nauc_precision_at_5_max
value: 53.8627
- type: nauc_precision_at_5_std
value: 31.838699999999996
- type: nauc_precision_at_5_diff1
value: 28.3882
- type: nauc_precision_at_10_max
value: 45.913
- type: nauc_precision_at_10_std
value: 30.7397
- type: nauc_precision_at_10_diff1
value: 22.115499999999997
- type: nauc_precision_at_20_max
value: 42.4268
- type: nauc_precision_at_20_std
value: 30.6865
- type: nauc_precision_at_20_diff1
value: 19.3518
- type: nauc_precision_at_100_max
value: 32.3071
- type: nauc_precision_at_100_std
value: 22.7085
- type: nauc_precision_at_100_diff1
value: 16.379099999999998
- type: nauc_precision_at_1000_max
value: 20.7196
- type: nauc_precision_at_1000_std
value: 15.9772
- type: nauc_precision_at_1000_diff1
value: 4.7801
- type: nauc_mrr_at_1_max
value: 68.4911
- type: nauc_mrr_at_1_std
value: 37.5175
- type: nauc_mrr_at_1_diff1
value: 53.80740000000001
- type: nauc_mrr_at_3_max
value: 60.39099999999999
- type: nauc_mrr_at_3_std
value: 34.7021
- type: nauc_mrr_at_3_diff1
value: 40.122099999999996
- type: nauc_mrr_at_5_max
value: 60.1398
- type: nauc_mrr_at_5_std
value: 34.5078
- type: nauc_mrr_at_5_diff1
value: 39.207300000000004
- type: nauc_mrr_at_10_max
value: 58.1008
- type: nauc_mrr_at_10_std
value: 34.1899
- type: nauc_mrr_at_10_diff1
value: 37.4103
- type: nauc_mrr_at_20_max
value: 57.3164
- type: nauc_mrr_at_20_std
value: 34.0334
- type: nauc_mrr_at_20_diff1
value: 36.6298
- type: nauc_mrr_at_100_max
value: 56.3412
- type: nauc_mrr_at_100_std
value: 33.4595
- type: nauc_mrr_at_100_diff1
value: 36.0572
- type: nauc_mrr_at_1000_max
value: 56.1735
- type: nauc_mrr_at_1000_std
value: 33.3776
- type: nauc_mrr_at_1000_diff1
value: 35.8637
- type: main_score
value: 6.339
task:
type: Retrieval
- dataset:
config: ar
name: MTEB MintakaRetrieval (ar)
revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e
split: test
type: jinaai/mintakaqa
metrics:
- type: ndcg_at_1
value: 11.212
- type: ndcg_at_3
value: 16.08
- type: ndcg_at_5
value: 17.543
- type: ndcg_at_10
value: 19.13
- type: ndcg_at_20
value: 20.557
- type: ndcg_at_100
value: 23.247999999999998
- type: ndcg_at_1000
value: 28.098
- type: map_at_1
value: 11.212
- type: map_at_3
value: 14.904
- type: map_at_5
value: 15.719
- type: map_at_10
value: 16.375
- type: map_at_20
value: 16.769000000000002
- type: map_at_100
value: 17.11
- type: map_at_1000
value: 17.244
- type: recall_at_1
value: 11.212
- type: recall_at_3
value: 19.473
- type: recall_at_5
value: 23.014000000000003
- type: recall_at_10
value: 27.916
- type: recall_at_20
value: 33.545
- type: recall_at_100
value: 48.525
- type: recall_at_1000
value: 89.605
- type: precision_at_1
value: 11.212
- type: precision_at_3
value: 6.491
- type: precision_at_5
value: 4.603
- type: precision_at_10
value: 2.792
- type: precision_at_20
value: 1.677
- type: precision_at_100
value: 0.485
- type: precision_at_1000
value: 0.09
- type: mrr_at_1
value: 11.212
- type: mrr_at_3
value: 14.9039
- type: mrr_at_5
value: 15.7187
- type: mrr_at_10
value: 16.3746
- type: mrr_at_20
value: 16.7694
- type: mrr_at_100
value: 17.1099
- type: mrr_at_1000
value: 17.244200000000003
- type: nauc_ndcg_at_1_max
value: 27.026600000000002
- type: nauc_ndcg_at_1_std
value: 12.168700000000001
- type: nauc_ndcg_at_1_diff1
value: 27.619100000000003
- type: nauc_ndcg_at_3_max
value: 27.6494
- type: nauc_ndcg_at_3_std
value: 12.6518
- type: nauc_ndcg_at_3_diff1
value: 23.411
- type: nauc_ndcg_at_5_max
value: 27.441100000000002
- type: nauc_ndcg_at_5_std
value: 12.9306
- type: nauc_ndcg_at_5_diff1
value: 22.6911
- type: nauc_ndcg_at_10_max
value: 27.034200000000002
- type: nauc_ndcg_at_10_std
value: 12.889600000000002
- type: nauc_ndcg_at_10_diff1
value: 21.4738
- type: nauc_ndcg_at_20_max
value: 26.3529
- type: nauc_ndcg_at_20_std
value: 12.751399999999999
- type: nauc_ndcg_at_20_diff1
value: 21.0119
- type: nauc_ndcg_at_100_max
value: 24.846799999999998
- type: nauc_ndcg_at_100_std
value: 11.4812
- type: nauc_ndcg_at_100_diff1
value: 20.7978
- type: nauc_ndcg_at_1000_max
value: 25.3245
- type: nauc_ndcg_at_1000_std
value: 11.5287
- type: nauc_ndcg_at_1000_diff1
value: 21.4932
- type: nauc_map_at_1_max
value: 27.026600000000002
- type: nauc_map_at_1_std
value: 12.168700000000001
- type: nauc_map_at_1_diff1
value: 27.619100000000003
- type: nauc_map_at_3_max
value: 27.6584
- type: nauc_map_at_3_std
value: 12.5275
- type: nauc_map_at_3_diff1
value: 24.3659
- type: nauc_map_at_5_max
value: 27.5215
- type: nauc_map_at_5_std
value: 12.6811
- type: nauc_map_at_5_diff1
value: 23.9118
- type: nauc_map_at_10_max
value: 27.3557
- type: nauc_map_at_10_std
value: 12.6884
- type: nauc_map_at_10_diff1
value: 23.2999
- type: nauc_map_at_20_max
value: 27.132
- type: nauc_map_at_20_std
value: 12.6529
- type: nauc_map_at_20_diff1
value: 23.1266
- type: nauc_map_at_100_max
value: 26.9011
- type: nauc_map_at_100_std
value: 12.471400000000001
- type: nauc_map_at_100_diff1
value: 23.1097
- type: nauc_map_at_1000_max
value: 26.9045
- type: nauc_map_at_1000_std
value: 12.4617
- type: nauc_map_at_1000_diff1
value: 23.128
- type: nauc_recall_at_1_max
value: 27.026600000000002
- type: nauc_recall_at_1_std
value: 12.168700000000001
- type: nauc_recall_at_1_diff1
value: 27.619100000000003
- type: nauc_recall_at_3_max
value: 27.5842
- type: nauc_recall_at_3_std
value: 12.953500000000002
- type: nauc_recall_at_3_diff1
value: 21.125
- type: nauc_recall_at_5_max
value: 27.2095
- type: nauc_recall_at_5_std
value: 13.541400000000001
- type: nauc_recall_at_5_diff1
value: 19.8588
- type: nauc_recall_at_10_max
value: 26.1646
- type: nauc_recall_at_10_std
value: 13.309199999999999
- type: nauc_recall_at_10_diff1
value: 17.2978
- type: nauc_recall_at_20_max
value: 24.232200000000002
- type: nauc_recall_at_20_std
value: 12.8466
- type: nauc_recall_at_20_diff1
value: 16.2314
- type: nauc_recall_at_100_max
value: 18.0113
- type: nauc_recall_at_100_std
value: 7.484399999999999
- type: nauc_recall_at_100_diff1
value: 15.167300000000001
- type: nauc_recall_at_1000_max
value: 13.0555
- type: nauc_recall_at_1000_std
value: 1.2099
- type: nauc_recall_at_1000_diff1
value: 16.101599999999998
- type: nauc_precision_at_1_max
value: 27.026600000000002
- type: nauc_precision_at_1_std
value: 12.168700000000001
- type: nauc_precision_at_1_diff1
value: 27.619100000000003
- type: nauc_precision_at_3_max
value: 27.5842
- type: nauc_precision_at_3_std
value: 12.953500000000002
- type: nauc_precision_at_3_diff1
value: 21.125
- type: nauc_precision_at_5_max
value: 27.2095
- type: nauc_precision_at_5_std
value: 13.541400000000001
- type: nauc_precision_at_5_diff1
value: 19.8588
- type: nauc_precision_at_10_max
value: 26.1646
- type: nauc_precision_at_10_std
value: 13.309199999999999
- type: nauc_precision_at_10_diff1
value: 17.2978
- type: nauc_precision_at_20_max
value: 24.232200000000002
- type: nauc_precision_at_20_std
value: 12.8466
- type: nauc_precision_at_20_diff1
value: 16.2314
- type: nauc_precision_at_100_max
value: 18.0113
- type: nauc_precision_at_100_std
value: 7.484399999999999
- type: nauc_precision_at_100_diff1
value: 15.167300000000001
- type: nauc_precision_at_1000_max
value: 13.0555
- type: nauc_precision_at_1000_std
value: 1.2099
- type: nauc_precision_at_1000_diff1
value: 16.101599999999998
- type: nauc_mrr_at_1_max
value: 27.026600000000002
- type: nauc_mrr_at_1_std
value: 12.168700000000001
- type: nauc_mrr_at_1_diff1
value: 27.619100000000003
- type: nauc_mrr_at_3_max
value: 27.6584
- type: nauc_mrr_at_3_std
value: 12.5275
- type: nauc_mrr_at_3_diff1
value: 24.3659
- type: nauc_mrr_at_5_max
value: 27.5215
- type: nauc_mrr_at_5_std
value: 12.6811
- type: nauc_mrr_at_5_diff1
value: 23.9118
- type: nauc_mrr_at_10_max
value: 27.3557
- type: nauc_mrr_at_10_std
value: 12.6884
- type: nauc_mrr_at_10_diff1
value: 23.2999
- type: nauc_mrr_at_20_max
value: 27.132
- type: nauc_mrr_at_20_std
value: 12.6529
- type: nauc_mrr_at_20_diff1
value: 23.1266
- type: nauc_mrr_at_100_max
value: 26.9011
- type: nauc_mrr_at_100_std
value: 12.471400000000001
- type: nauc_mrr_at_100_diff1
value: 23.1097
- type: nauc_mrr_at_1000_max
value: 26.9045
- type: nauc_mrr_at_1000_std
value: 12.4617
- type: nauc_mrr_at_1000_diff1
value: 23.128
- type: main_score
value: 19.13
task:
type: Retrieval
- dataset:
config: arabic
name: MTEB MrTidyRetrieval (arabic)
revision: fc24a3ce8f09746410daee3d5cd823ff7a0675b7
split: test
type: mteb/mrtidy
metrics:
- type: ndcg_at_1
value: 35.245
- type: ndcg_at_3
value: 46.414
- type: ndcg_at_5
value: 49.985
- type: ndcg_at_10
value: 52.861000000000004
- type: ndcg_at_20
value: 54.37
- type: ndcg_at_100
value: 56.187
- type: ndcg_at_1000
value: 57.133
- type: map_at_1
value: 32.917
- type: map_at_3
value: 42.9
- type: map_at_5
value: 44.949
- type: map_at_10
value: 46.261
- type: map_at_20
value: 46.7
- type: map_at_100
value: 46.977999999999994
- type: map_at_1000
value: 47.016999999999996
- type: recall_at_1
value: 32.917
- type: recall_at_3
value: 54.641
- type: recall_at_5
value: 62.966
- type: recall_at_10
value: 71.307
- type: recall_at_20
value: 76.858
- type: recall_at_100
value: 86.031
- type: recall_at_1000
value: 93.232
- type: precision_at_1
value: 35.245
- type: precision_at_3
value: 19.889000000000003
- type: precision_at_5
value: 13.821
- type: precision_at_10
value: 7.928
- type: precision_at_20
value: 4.329000000000001
- type: precision_at_100
value: 0.987
- type: precision_at_1000
value: 0.108
- type: mrr_at_1
value: 35.2451
- type: mrr_at_3
value: 44.8813
- type: mrr_at_5
value: 46.8794
- type: mrr_at_10
value: 47.9775
- type: mrr_at_20
value: 48.3884
- type: mrr_at_100
value: 48.6026
- type: mrr_at_1000
value: 48.6276
- type: nauc_ndcg_at_1_max
value: 31.467299999999998
- type: nauc_ndcg_at_1_std
value: -3.4500999999999995
- type: nauc_ndcg_at_1_diff1
value: 44.9659
- type: nauc_ndcg_at_3_max
value: 39.3846
- type: nauc_ndcg_at_3_std
value: -0.7613
- type: nauc_ndcg_at_3_diff1
value: 39.8967
- type: nauc_ndcg_at_5_max
value: 40.5875
- type: nauc_ndcg_at_5_std
value: 1.2409999999999999
- type: nauc_ndcg_at_5_diff1
value: 39.3455
- type: nauc_ndcg_at_10_max
value: 41.3447
- type: nauc_ndcg_at_10_std
value: 2.3071
- type: nauc_ndcg_at_10_diff1
value: 39.6731
- type: nauc_ndcg_at_20_max
value: 41.504799999999996
- type: nauc_ndcg_at_20_std
value: 3.1558999999999995
- type: nauc_ndcg_at_20_diff1
value: 39.833800000000004
- type: nauc_ndcg_at_100_max
value: 41.3001
- type: nauc_ndcg_at_100_std
value: 3.8255
- type: nauc_ndcg_at_100_diff1
value: 39.6933
- type: nauc_ndcg_at_1000_max
value: 40.836299999999994
- type: nauc_ndcg_at_1000_std
value: 3.6927000000000003
- type: nauc_ndcg_at_1000_diff1
value: 39.837
- type: nauc_map_at_1_max
value: 29.6563
- type: nauc_map_at_1_std
value: -4.593
- type: nauc_map_at_1_diff1
value: 43.434200000000004
- type: nauc_map_at_3_max
value: 37.2705
- type: nauc_map_at_3_std
value: -2.0496
- type: nauc_map_at_3_diff1
value: 40.612700000000004
- type: nauc_map_at_5_max
value: 38.034
- type: nauc_map_at_5_std
value: -1.0169
- type: nauc_map_at_5_diff1
value: 40.436899999999994
- type: nauc_map_at_10_max
value: 38.311299999999996
- type: nauc_map_at_10_std
value: -0.7025
- type: nauc_map_at_10_diff1
value: 40.5876
- type: nauc_map_at_20_max
value: 38.324799999999996
- type: nauc_map_at_20_std
value: -0.4921
- type: nauc_map_at_20_diff1
value: 40.6363
- type: nauc_map_at_100_max
value: 38.3104
- type: nauc_map_at_100_std
value: -0.395
- type: nauc_map_at_100_diff1
value: 40.603
- type: nauc_map_at_1000_max
value: 38.2962
- type: nauc_map_at_1000_std
value: -0.38430000000000003
- type: nauc_map_at_1000_diff1
value: 40.609
- type: nauc_recall_at_1_max
value: 29.6563
- type: nauc_recall_at_1_std
value: -4.593
- type: nauc_recall_at_1_diff1
value: 43.434200000000004
- type: nauc_recall_at_3_max
value: 44.2481
- type: nauc_recall_at_3_std
value: 1.4938
- type: nauc_recall_at_3_diff1
value: 36.2463
- type: nauc_recall_at_5_max
value: 47.4339
- type: nauc_recall_at_5_std
value: 7.088
- type: nauc_recall_at_5_diff1
value: 34.0247
- type: nauc_recall_at_10_max
value: 51.703900000000004
- type: nauc_recall_at_10_std
value: 12.3656
- type: nauc_recall_at_10_diff1
value: 34.1721
- type: nauc_recall_at_20_max
value: 54.7168
- type: nauc_recall_at_20_std
value: 18.6634
- type: nauc_recall_at_20_diff1
value: 34.294000000000004
- type: nauc_recall_at_100_max
value: 60.014500000000005
- type: nauc_recall_at_100_std
value: 32.7492
- type: nauc_recall_at_100_diff1
value: 31.468
- type: nauc_recall_at_1000_max
value: 65.0876
- type: nauc_recall_at_1000_std
value: 53.171
- type: nauc_recall_at_1000_diff1
value: 29.1675
- type: nauc_precision_at_1_max
value: 31.467299999999998
- type: nauc_precision_at_1_std
value: -3.4500999999999995
- type: nauc_precision_at_1_diff1
value: 44.9659
- type: nauc_precision_at_3_max
value: 44.257000000000005
- type: nauc_precision_at_3_std
value: 4.6068999999999996
- type: nauc_precision_at_3_diff1
value: 33.8791
- type: nauc_precision_at_5_max
value: 46.2531
- type: nauc_precision_at_5_std
value: 10.1235
- type: nauc_precision_at_5_diff1
value: 30.849
- type: nauc_precision_at_10_max
value: 43.8988
- type: nauc_precision_at_10_std
value: 15.1328
- type: nauc_precision_at_10_diff1
value: 26.8399
- type: nauc_precision_at_20_max
value: 39.724199999999996
- type: nauc_precision_at_20_std
value: 19.8477
- type: nauc_precision_at_20_diff1
value: 22.093899999999998
- type: nauc_precision_at_100_max
value: 27.1358
- type: nauc_precision_at_100_std
value: 26.2318
- type: nauc_precision_at_100_diff1
value: 9.3587
- type: nauc_precision_at_1000_max
value: 12.7943
- type: nauc_precision_at_1000_std
value: 25.520300000000002
- type: nauc_precision_at_1000_diff1
value: 0.2346
- type: nauc_mrr_at_1_max
value: 31.467299999999998
- type: nauc_mrr_at_1_std
value: -3.4500999999999995
- type: nauc_mrr_at_1_diff1
value: 44.9659
- type: nauc_mrr_at_3_max
value: 37.810700000000004
- type: nauc_mrr_at_3_std
value: -0.5135
- type: nauc_mrr_at_3_diff1
value: 41.5755
- type: nauc_mrr_at_5_max
value: 38.205099999999995
- type: nauc_mrr_at_5_std
value: 0.5058
- type: nauc_mrr_at_5_diff1
value: 41.151900000000005
- type: nauc_mrr_at_10_max
value: 38.3527
- type: nauc_mrr_at_10_std
value: 0.8295
- type: nauc_mrr_at_10_diff1
value: 41.3325
- type: nauc_mrr_at_20_max
value: 38.3898
- type: nauc_mrr_at_20_std
value: 0.9883
- type: nauc_mrr_at_20_diff1
value: 41.3869
- type: nauc_mrr_at_100_max
value: 38.3357
- type: nauc_mrr_at_100_std
value: 1.0263
- type: nauc_mrr_at_100_diff1
value: 41.3542
- type: nauc_mrr_at_1000_max
value: 38.3255
- type: nauc_mrr_at_1000_std
value: 1.0194
- type: nauc_mrr_at_1000_diff1
value: 41.361799999999995
- type: main_score
value: 52.861000000000004
task:
type: Retrieval
- dataset:
config: default
name: MTEB SadeemQuestionRetrieval (default)
revision: 3cb0752b182e5d5d740df547748b06663c8e0bd9
split: test
type: sadeem-ai/sadeem-ar-eval-retrieval-questions
metrics:
- type: ndcg_at_1
value: 28.674
- type: ndcg_at_3
value: 60.604
- type: ndcg_at_5
value: 62.092000000000006
- type: ndcg_at_10
value: 63.154999999999994
- type: ndcg_at_20
value: 63.602000000000004
- type: ndcg_at_100
value: 64.242
- type: ndcg_at_1000
value: 64.50399999999999
- type: map_at_1
value: 28.674
- type: map_at_3
value: 52.21
- type: map_at_5
value: 53.052
- type: map_at_10
value: 53.498000000000005
- type: map_at_20
value: 53.620000000000005
- type: map_at_100
value: 53.715999999999994
- type: map_at_1000
value: 53.726
- type: recall_at_1
value: 28.674
- type: recall_at_3
value: 85.112
- type: recall_at_5
value: 88.655
- type: recall_at_10
value: 91.91
- type: recall_at_20
value: 93.681
- type: recall_at_100
value: 97.032
- type: recall_at_1000
value: 99.09
- type: precision_at_1
value: 28.674
- type: precision_at_3
value: 28.371000000000002
- type: precision_at_5
value: 17.730999999999998
- type: precision_at_10
value: 9.191
- type: precision_at_20
value: 4.684
- type: precision_at_100
value: 0.97
- type: precision_at_1000
value: 0.099
- type: mrr_at_1
value: 25.371
- type: mrr_at_3
value: 50.2314
- type: mrr_at_5
value: 51.0212
- type: mrr_at_10
value: 51.481100000000005
- type: mrr_at_20
value: 51.6128
- type: mrr_at_100
value: 51.7119
- type: mrr_at_1000
value: 51.722100000000005
- type: nauc_ndcg_at_1_max
value: 15.7219
- type: nauc_ndcg_at_1_std
value: 2.2991
- type: nauc_ndcg_at_1_diff1
value: 5.2984
- type: nauc_ndcg_at_3_max
value: 37.9971
- type: nauc_ndcg_at_3_std
value: 11.0045
- type: nauc_ndcg_at_3_diff1
value: -38.8501
- type: nauc_ndcg_at_5_max
value: 35.6057
- type: nauc_ndcg_at_5_std
value: 10.8947
- type: nauc_ndcg_at_5_diff1
value: -33.353500000000004
- type: nauc_ndcg_at_10_max
value: 33.5856
- type: nauc_ndcg_at_10_std
value: 11.5392
- type: nauc_ndcg_at_10_diff1
value: -29.2831
- type: nauc_ndcg_at_20_max
value: 32.4619
- type: nauc_ndcg_at_20_std
value: 11.145900000000001
- type: nauc_ndcg_at_20_diff1
value: -26.8202
- type: nauc_ndcg_at_100_max
value: 30.888199999999998
- type: nauc_ndcg_at_100_std
value: 10.2467
- type: nauc_ndcg_at_100_diff1
value: -23.5621
- type: nauc_ndcg_at_1000_max
value: 30.173699999999997
- type: nauc_ndcg_at_1000_std
value: 9.7867
- type: nauc_ndcg_at_1000_diff1
value: -22.1022
- type: nauc_map_at_1_max
value: 15.7219
- type: nauc_map_at_1_std
value: 2.2991
- type: nauc_map_at_1_diff1
value: 5.2984
- type: nauc_map_at_3_max
value: 30.0249
- type: nauc_map_at_3_std
value: 8.110199999999999
- type: nauc_map_at_3_diff1
value: -22.6437
- type: nauc_map_at_5_max
value: 28.654600000000002
- type: nauc_map_at_5_std
value: 7.9832
- type: nauc_map_at_5_diff1
value: -19.4967
- type: nauc_map_at_10_max
value: 27.825400000000002
- type: nauc_map_at_10_std
value: 8.1387
- type: nauc_map_at_10_diff1
value: -17.8584
- type: nauc_map_at_20_max
value: 27.5484
- type: nauc_map_at_20_std
value: 8.0305
- type: nauc_map_at_20_diff1
value: -17.2515
- type: nauc_map_at_100_max
value: 27.3449
- type: nauc_map_at_100_std
value: 7.919099999999999
- type: nauc_map_at_100_diff1
value: -16.8205
- type: nauc_map_at_1000_max
value: 27.3212
- type: nauc_map_at_1000_std
value: 7.9052999999999995
- type: nauc_map_at_1000_diff1
value: -16.7727
- type: nauc_recall_at_1_max
value: 15.7219
- type: nauc_recall_at_1_std
value: 2.2991
- type: nauc_recall_at_1_diff1
value: 5.2984
- type: nauc_recall_at_3_max
value: 82.6956
- type: nauc_recall_at_3_std
value: 27.086700000000004
- type: nauc_recall_at_3_diff1
value: -129.9841
- type: nauc_recall_at_5_max
value: 83.4035
- type: nauc_recall_at_5_std
value: 30.9258
- type: nauc_recall_at_5_diff1
value: -128.9633
- type: nauc_recall_at_10_max
value: 85.344
- type: nauc_recall_at_10_std
value: 44.183699999999995
- type: nauc_recall_at_10_diff1
value: -132.1167
- type: nauc_recall_at_20_max
value: 84.9071
- type: nauc_recall_at_20_std
value: 48.2337
- type: nauc_recall_at_20_diff1
value: -128.3509
- type: nauc_recall_at_100_max
value: 86.99470000000001
- type: nauc_recall_at_100_std
value: 56.7091
- type: nauc_recall_at_100_diff1
value: -126.3125
- type: nauc_recall_at_1000_max
value: 90.02329999999999
- type: nauc_recall_at_1000_std
value: 79.7439
- type: nauc_recall_at_1000_diff1
value: -106.8251
- type: nauc_precision_at_1_max
value: 15.7219
- type: nauc_precision_at_1_std
value: 2.2991
- type: nauc_precision_at_1_diff1
value: 5.2984
- type: nauc_precision_at_3_max
value: 82.6956
- type: nauc_precision_at_3_std
value: 27.086700000000004
- type: nauc_precision_at_3_diff1
value: -129.9841
- type: nauc_precision_at_5_max
value: 83.4035
- type: nauc_precision_at_5_std
value: 30.9258
- type: nauc_precision_at_5_diff1
value: -128.9633
- type: nauc_precision_at_10_max
value: 85.344
- type: nauc_precision_at_10_std
value: 44.183699999999995
- type: nauc_precision_at_10_diff1
value: -132.1167
- type: nauc_precision_at_20_max
value: 84.9071
- type: nauc_precision_at_20_std
value: 48.2337
- type: nauc_precision_at_20_diff1
value: -128.3509
- type: nauc_precision_at_100_max
value: 86.99470000000001
- type: nauc_precision_at_100_std
value: 56.7091
- type: nauc_precision_at_100_diff1
value: -126.3125
- type: nauc_precision_at_1000_max
value: 90.02329999999999
- type: nauc_precision_at_1000_std
value: 79.7439
- type: nauc_precision_at_1000_diff1
value: -106.8251
- type: nauc_mrr_at_1_max
value: 10.6156
- type: nauc_mrr_at_1_std
value: 1.6084
- type: nauc_mrr_at_1_diff1
value: -23.3205
- type: nauc_mrr_at_3_max
value: 26.458599999999997
- type: nauc_mrr_at_3_std
value: 7.3822
- type: nauc_mrr_at_3_diff1
value: -46.1904
- type: nauc_mrr_at_5_max
value: 25.0415
- type: nauc_mrr_at_5_std
value: 7.0684
- type: nauc_mrr_at_5_diff1
value: -43.5901
- type: nauc_mrr_at_10_max
value: 24.1152
- type: nauc_mrr_at_10_std
value: 7.4148000000000005
- type: nauc_mrr_at_10_diff1
value: -42.271300000000004
- type: nauc_mrr_at_20_max
value: 23.7882
- type: nauc_mrr_at_20_std
value: 7.2852
- type: nauc_mrr_at_20_diff1
value: -41.7586
- type: nauc_mrr_at_100_max
value: 23.552999999999997
- type: nauc_mrr_at_100_std
value: 7.1595
- type: nauc_mrr_at_100_diff1
value: -41.3884
- type: nauc_mrr_at_1000_max
value: 23.5271
- type: nauc_mrr_at_1000_std
value: 7.1449
- type: nauc_mrr_at_1000_diff1
value: -41.3491
- type: main_score
value: 63.154999999999994
task:
type: Retrieval
- dataset:
config: ara-ara
name: MTEB XPQARetrieval (ara-ara)
revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f
split: test
type: jinaai/xpqa
metrics:
- type: ndcg_at_1
value: 27.733
- type: ndcg_at_3
value: 27.223000000000003
- type: ndcg_at_5
value: 29.136
- type: ndcg_at_10
value: 32.574999999999996
- type: ndcg_at_20
value: 35.338
- type: ndcg_at_100
value: 39.267
- type: ndcg_at_1000
value: 43.120999999999995
- type: map_at_1
value: 14.713000000000001
- type: map_at_3
value: 21.929000000000002
- type: map_at_5
value: 25.013999999999996
- type: map_at_10
value: 26.926
- type: map_at_20
value: 27.962999999999997
- type: map_at_100
value: 28.647
- type: map_at_1000
value: 28.805999999999997
- type: recall_at_1
value: 14.713000000000001
- type: recall_at_3
value: 25.416
- type: recall_at_5
value: 32.618
- type: recall_at_10
value: 41.062
- type: recall_at_20
value: 50.06
- type: recall_at_100
value: 68.34
- type: recall_at_1000
value: 95.922
- type: precision_at_1
value: 27.733
- type: precision_at_3
value: 18.444
- type: precision_at_5
value: 14.64
- type: precision_at_10
value: 9.28
- type: precision_at_20
value: 5.573
- type: precision_at_100
value: 1.4529999999999998
- type: precision_at_1000
value: 0.194
- type: mrr_at_1
value: 27.7333
- type: mrr_at_3
value: 31.8444
- type: mrr_at_5
value: 33.1644
- type: mrr_at_10
value: 34.2814
- type: mrr_at_20
value: 34.8056
- type: mrr_at_100
value: 35.241
- type: mrr_at_1000
value: 35.3266
- type: nauc_ndcg_at_1_max
value: 39.342
- type: nauc_ndcg_at_1_std
value: 4.1262
- type: nauc_ndcg_at_1_diff1
value: 26.6799
- type: nauc_ndcg_at_3_max
value: 34.3621
- type: nauc_ndcg_at_3_std
value: -1.2518
- type: nauc_ndcg_at_3_diff1
value: 22.1953
- type: nauc_ndcg_at_5_max
value: 33.364
- type: nauc_ndcg_at_5_std
value: -2.278
- type: nauc_ndcg_at_5_diff1
value: 22.8191
- type: nauc_ndcg_at_10_max
value: 34.7684
- type: nauc_ndcg_at_10_std
value: -0.5644
- type: nauc_ndcg_at_10_diff1
value: 21.7363
- type: nauc_ndcg_at_20_max
value: 35.19
- type: nauc_ndcg_at_20_std
value: -0.5218
- type: nauc_ndcg_at_20_diff1
value: 21.9051
- type: nauc_ndcg_at_100_max
value: 35.0375
- type: nauc_ndcg_at_100_std
value: 0.0914
- type: nauc_ndcg_at_100_diff1
value: 21.0693
- type: nauc_ndcg_at_1000_max
value: 36.2854
- type: nauc_ndcg_at_1000_std
value: 0.9521999999999999
- type: nauc_ndcg_at_1000_diff1
value: 22.028
- type: nauc_map_at_1_max
value: 21.7217
- type: nauc_map_at_1_std
value: -4.8469
- type: nauc_map_at_1_diff1
value: 28.691899999999997
- type: nauc_map_at_3_max
value: 29.933300000000003
- type: nauc_map_at_3_std
value: -4.1226
- type: nauc_map_at_3_diff1
value: 23.2263
- type: nauc_map_at_5_max
value: 32.9353
- type: nauc_map_at_5_std
value: -3.7249
- type: nauc_map_at_5_diff1
value: 23.1571
- type: nauc_map_at_10_max
value: 34.3495
- type: nauc_map_at_10_std
value: -2.3154999999999997
- type: nauc_map_at_10_diff1
value: 22.138099999999998
- type: nauc_map_at_20_max
value: 34.6561
- type: nauc_map_at_20_std
value: -2.1671
- type: nauc_map_at_20_diff1
value: 22.157799999999998
- type: nauc_map_at_100_max
value: 34.7339
- type: nauc_map_at_100_std
value: -1.9598
- type: nauc_map_at_100_diff1
value: 22.0112
- type: nauc_map_at_1000_max
value: 34.7776
- type: nauc_map_at_1000_std
value: -1.9099000000000002
- type: nauc_map_at_1000_diff1
value: 22.0438
- type: nauc_recall_at_1_max
value: 21.7217
- type: nauc_recall_at_1_std
value: -4.8469
- type: nauc_recall_at_1_diff1
value: 28.691899999999997
- type: nauc_recall_at_3_max
value: 25.3674
- type: nauc_recall_at_3_std
value: -5.1852
- type: nauc_recall_at_3_diff1
value: 19.712
- type: nauc_recall_at_5_max
value: 27.4254
- type: nauc_recall_at_5_std
value: -4.8346
- type: nauc_recall_at_5_diff1
value: 19.4039
- type: nauc_recall_at_10_max
value: 30.1107
- type: nauc_recall_at_10_std
value: -1.0588
- type: nauc_recall_at_10_diff1
value: 17.908099999999997
- type: nauc_recall_at_20_max
value: 30.488100000000003
- type: nauc_recall_at_20_std
value: -1.688
- type: nauc_recall_at_20_diff1
value: 18.2025
- type: nauc_recall_at_100_max
value: 26.076300000000003
- type: nauc_recall_at_100_std
value: -0.8529
- type: nauc_recall_at_100_diff1
value: 12.2454
- type: nauc_recall_at_1000_max
value: 47.4613
- type: nauc_recall_at_1000_std
value: 13.147300000000001
- type: nauc_recall_at_1000_diff1
value: 20.1172
- type: nauc_precision_at_1_max
value: 39.342
- type: nauc_precision_at_1_std
value: 4.1262
- type: nauc_precision_at_1_diff1
value: 26.6799
- type: nauc_precision_at_3_max
value: 41.9624
- type: nauc_precision_at_3_std
value: 3.6343
- type: nauc_precision_at_3_diff1
value: 15.692999999999998
- type: nauc_precision_at_5_max
value: 41.917300000000004
- type: nauc_precision_at_5_std
value: 3.3890000000000002
- type: nauc_precision_at_5_diff1
value: 14.7503
- type: nauc_precision_at_10_max
value: 41.050599999999996
- type: nauc_precision_at_10_std
value: 5.8458000000000006
- type: nauc_precision_at_10_diff1
value: 10.5962
- type: nauc_precision_at_20_max
value: 38.458999999999996
- type: nauc_precision_at_20_std
value: 6.390700000000001
- type: nauc_precision_at_20_diff1
value: 9.532
- type: nauc_precision_at_100_max
value: 34.36
- type: nauc_precision_at_100_std
value: 8.7814
- type: nauc_precision_at_100_diff1
value: 6.286899999999999
- type: nauc_precision_at_1000_max
value: 31.495
- type: nauc_precision_at_1000_std
value: 14.5312
- type: nauc_precision_at_1000_diff1
value: 3.4628
- type: nauc_mrr_at_1_max
value: 39.342
- type: nauc_mrr_at_1_std
value: 4.1262
- type: nauc_mrr_at_1_diff1
value: 26.6799
- type: nauc_mrr_at_3_max
value: 37.390699999999995
- type: nauc_mrr_at_3_std
value: 2.9843
- type: nauc_mrr_at_3_diff1
value: 24.9747
- type: nauc_mrr_at_5_max
value: 37.072300000000006
- type: nauc_mrr_at_5_std
value: 2.8253
- type: nauc_mrr_at_5_diff1
value: 24.5929
- type: nauc_mrr_at_10_max
value: 37.0102
- type: nauc_mrr_at_10_std
value: 2.9913
- type: nauc_mrr_at_10_diff1
value: 24.4127
- type: nauc_mrr_at_20_max
value: 37.0964
- type: nauc_mrr_at_20_std
value: 2.9813
- type: nauc_mrr_at_20_diff1
value: 24.5641
- type: nauc_mrr_at_100_max
value: 37.12
- type: nauc_mrr_at_100_std
value: 3.08
- type: nauc_mrr_at_100_diff1
value: 24.418699999999998
- type: nauc_mrr_at_1000_max
value: 37.162800000000004
- type: nauc_mrr_at_1000_std
value: 3.0968
- type: nauc_mrr_at_1000_diff1
value: 24.4571
- type: main_score
value: 32.574999999999996
task:
type: Retrieval
- dataset:
config: eng-ara
name: MTEB XPQARetrieval (eng-ara)
revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f
split: test
type: jinaai/xpqa
metrics:
- type: ndcg_at_1
value: 7.6
- type: ndcg_at_3
value: 7.8149999999999995
- type: ndcg_at_5
value: 8.527999999999999
- type: ndcg_at_10
value: 10.605
- type: ndcg_at_20
value: 12.293999999999999
- type: ndcg_at_100
value: 15.572
- type: ndcg_at_1000
value: 22.939
- type: map_at_1
value: 4.04
- type: map_at_3
value: 6.0089999999999995
- type: map_at_5
value: 6.751
- type: map_at_10
value: 7.681
- type: map_at_20
value: 8.233
- type: map_at_100
value: 8.719000000000001
- type: map_at_1000
value: 8.99
- type: recall_at_1
value: 4.04
- type: recall_at_3
value: 7.611
- type: recall_at_5
value: 10.078
- type: recall_at_10
value: 15.347
- type: recall_at_20
value: 20.762
- type: recall_at_100
value: 35.699999999999996
- type: recall_at_1000
value: 87.062
- type: precision_at_1
value: 7.6
- type: precision_at_3
value: 5.289
- type: precision_at_5
value: 4.293
- type: precision_at_10
value: 3.3070000000000004
- type: precision_at_20
value: 2.253
- type: precision_at_100
value: 0.779
- type: precision_at_1000
value: 0.183
- type: mrr_at_1
value: 7.6
- type: mrr_at_3
value: 9.6222
- type: mrr_at_5
value: 10.3822
- type: mrr_at_10
value: 11.3079
- type: mrr_at_20
value: 11.7451
- type: mrr_at_100
value: 12.122
- type: mrr_at_1000
value: 12.2934
- type: nauc_ndcg_at_1_max
value: 45.4432
- type: nauc_ndcg_at_1_std
value: 23.5908
- type: nauc_ndcg_at_1_diff1
value: 43.7038
- type: nauc_ndcg_at_3_max
value: 42.4559
- type: nauc_ndcg_at_3_std
value: 25.163000000000004
- type: nauc_ndcg_at_3_diff1
value: 31.4215
- type: nauc_ndcg_at_5_max
value: 43.180600000000005
- type: nauc_ndcg_at_5_std
value: 26.034000000000002
- type: nauc_ndcg_at_5_diff1
value: 30.0168
- type: nauc_ndcg_at_10_max
value: 41.4998
- type: nauc_ndcg_at_10_std
value: 25.0978
- type: nauc_ndcg_at_10_diff1
value: 26.240999999999996
- type: nauc_ndcg_at_20_max
value: 40.9816
- type: nauc_ndcg_at_20_std
value: 26.053199999999997
- type: nauc_ndcg_at_20_diff1
value: 25.337300000000003
- type: nauc_ndcg_at_100_max
value: 39.2153
- type: nauc_ndcg_at_100_std
value: 26.125
- type: nauc_ndcg_at_100_diff1
value: 22.9574
- type: nauc_ndcg_at_1000_max
value: 37.9948
- type: nauc_ndcg_at_1000_std
value: 26.271099999999997
- type: nauc_ndcg_at_1000_diff1
value: 23.8089
- type: nauc_map_at_1_max
value: 48.9437
- type: nauc_map_at_1_std
value: 24.049699999999998
- type: nauc_map_at_1_diff1
value: 45.0848
- type: nauc_map_at_3_max
value: 45.3565
- type: nauc_map_at_3_std
value: 25.571700000000003
- type: nauc_map_at_3_diff1
value: 33.683
- type: nauc_map_at_5_max
value: 45.7519
- type: nauc_map_at_5_std
value: 26.3703
- type: nauc_map_at_5_diff1
value: 32.1697
- type: nauc_map_at_10_max
value: 45.144099999999995
- type: nauc_map_at_10_std
value: 26.2498
- type: nauc_map_at_10_diff1
value: 30.268299999999996
- type: nauc_map_at_20_max
value: 44.9955
- type: nauc_map_at_20_std
value: 26.8341
- type: nauc_map_at_20_diff1
value: 29.890099999999997
- type: nauc_map_at_100_max
value: 44.7618
- type: nauc_map_at_100_std
value: 27.098
- type: nauc_map_at_100_diff1
value: 29.236099999999997
- type: nauc_map_at_1000_max
value: 44.5937
- type: nauc_map_at_1000_std
value: 27.0781
- type: nauc_map_at_1000_diff1
value: 29.2869
- type: nauc_recall_at_1_max
value: 48.9437
- type: nauc_recall_at_1_std
value: 24.049699999999998
- type: nauc_recall_at_1_diff1
value: 45.0848
- type: nauc_recall_at_3_max
value: 40.0273
- type: nauc_recall_at_3_std
value: 24.3038
- type: nauc_recall_at_3_diff1
value: 26.6893
- type: nauc_recall_at_5_max
value: 39.3244
- type: nauc_recall_at_5_std
value: 24.627299999999998
- type: nauc_recall_at_5_diff1
value: 23.3583
- type: nauc_recall_at_10_max
value: 35.9404
- type: nauc_recall_at_10_std
value: 21.9355
- type: nauc_recall_at_10_diff1
value: 16.5745
- type: nauc_recall_at_20_max
value: 35.0028
- type: nauc_recall_at_20_std
value: 23.885
- type: nauc_recall_at_20_diff1
value: 15.711
- type: nauc_recall_at_100_max
value: 31.068299999999997
- type: nauc_recall_at_100_std
value: 23.876
- type: nauc_recall_at_100_diff1
value: 10.6191
- type: nauc_recall_at_1000_max
value: 23.5704
- type: nauc_recall_at_1000_std
value: 29.7413
- type: nauc_recall_at_1000_diff1
value: 3.2972
- type: nauc_precision_at_1_max
value: 45.4432
- type: nauc_precision_at_1_std
value: 23.5908
- type: nauc_precision_at_1_diff1
value: 43.7038
- type: nauc_precision_at_3_max
value: 41.0985
- type: nauc_precision_at_3_std
value: 27.383499999999998
- type: nauc_precision_at_3_diff1
value: 25.829200000000004
- type: nauc_precision_at_5_max
value: 40.982800000000005
- type: nauc_precision_at_5_std
value: 28.281299999999998
- type: nauc_precision_at_5_diff1
value: 23.612099999999998
- type: nauc_precision_at_10_max
value: 36.5635
- type: nauc_precision_at_10_std
value: 25.4679
- type: nauc_precision_at_10_diff1
value: 19.7839
- type: nauc_precision_at_20_max
value: 34.287800000000004
- type: nauc_precision_at_20_std
value: 26.191
- type: nauc_precision_at_20_diff1
value: 19.5411
- type: nauc_precision_at_100_max
value: 25.4786
- type: nauc_precision_at_100_std
value: 22.0241
- type: nauc_precision_at_100_diff1
value: 13.2463
- type: nauc_precision_at_1000_max
value: 2.8106
- type: nauc_precision_at_1000_std
value: 8.1145
- type: nauc_precision_at_1000_diff1
value: 4.3396
- type: nauc_mrr_at_1_max
value: 45.4432
- type: nauc_mrr_at_1_std
value: 23.5908
- type: nauc_mrr_at_1_diff1
value: 43.7038
- type: nauc_mrr_at_3_max
value: 41.7023
- type: nauc_mrr_at_3_std
value: 25.020599999999998
- type: nauc_mrr_at_3_diff1
value: 35.5867
- type: nauc_mrr_at_5_max
value: 41.4553
- type: nauc_mrr_at_5_std
value: 25.125799999999998
- type: nauc_mrr_at_5_diff1
value: 34.0259
- type: nauc_mrr_at_10_max
value: 40.1864
- type: nauc_mrr_at_10_std
value: 24.3306
- type: nauc_mrr_at_10_diff1
value: 31.7405
- type: nauc_mrr_at_20_max
value: 40.0656
- type: nauc_mrr_at_20_std
value: 24.5064
- type: nauc_mrr_at_20_diff1
value: 31.211
- type: nauc_mrr_at_100_max
value: 39.7816
- type: nauc_mrr_at_100_std
value: 24.424100000000003
- type: nauc_mrr_at_100_diff1
value: 30.990499999999997
- type: nauc_mrr_at_1000_max
value: 39.8505
- type: nauc_mrr_at_1000_std
value: 24.498
- type: nauc_mrr_at_1000_diff1
value: 31.1008
- type: main_score
value: 10.605
task:
type: Retrieval
- dataset:
config: ara-eng
name: MTEB XPQARetrieval (ara-eng)
revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f
split: test
type: jinaai/xpqa
metrics:
- type: ndcg_at_1
value: 7.412000000000001
- type: ndcg_at_3
value: 8.126
- type: ndcg_at_5
value: 8.851
- type: ndcg_at_10
value: 10.529
- type: ndcg_at_20
value: 11.721
- type: ndcg_at_100
value: 15.07
- type: ndcg_at_1000
value: 21.985
- type: map_at_1
value: 3.7310000000000003
- type: map_at_3
value: 6.299
- type: map_at_5
value: 7.23
- type: map_at_10
value: 8.168000000000001
- type: map_at_20
value: 8.554
- type: map_at_100
value: 9.041
- type: map_at_1000
value: 9.272
- type: recall_at_1
value: 3.7310000000000003
- type: recall_at_3
value: 8.053
- type: recall_at_5
value: 10.451
- type: recall_at_10
value: 14.578
- type: recall_at_20
value: 18.497
- type: recall_at_100
value: 33.825
- type: recall_at_1000
value: 83.596
- type: precision_at_1
value: 7.412000000000001
- type: precision_at_3
value: 5.66
- type: precision_at_5
value: 4.5280000000000005
- type: precision_at_10
value: 3.167
- type: precision_at_20
value: 1.9949999999999999
- type: precision_at_100
value: 0.732
- type: precision_at_1000
value: 0.172
- type: mrr_at_1
value: 7.4124
- type: mrr_at_3
value: 9.7934
- type: mrr_at_5
value: 10.2987
- type: mrr_at_10
value: 10.8004
- type: mrr_at_20
value: 11.136
- type: mrr_at_100
value: 11.544
- type: mrr_at_1000
value: 11.691799999999999
- type: nauc_ndcg_at_1_max
value: 46.218599999999995
- type: nauc_ndcg_at_1_std
value: 20.8013
- type: nauc_ndcg_at_1_diff1
value: 23.6016
- type: nauc_ndcg_at_3_max
value: 40.1564
- type: nauc_ndcg_at_3_std
value: 15.2981
- type: nauc_ndcg_at_3_diff1
value: 18.5145
- type: nauc_ndcg_at_5_max
value: 41.6053
- type: nauc_ndcg_at_5_std
value: 15.324399999999999
- type: nauc_ndcg_at_5_diff1
value: 19.275100000000002
- type: nauc_ndcg_at_10_max
value: 42.1222
- type: nauc_ndcg_at_10_std
value: 16.503999999999998
- type: nauc_ndcg_at_10_diff1
value: 17.5452
- type: nauc_ndcg_at_20_max
value: 41.572900000000004
- type: nauc_ndcg_at_20_std
value: 17.2732
- type: nauc_ndcg_at_20_diff1
value: 16.37
- type: nauc_ndcg_at_100_max
value: 39.5201
- type: nauc_ndcg_at_100_std
value: 17.9526
- type: nauc_ndcg_at_100_diff1
value: 16.7166
- type: nauc_ndcg_at_1000_max
value: 38.1
- type: nauc_ndcg_at_1000_std
value: 16.897100000000002
- type: nauc_ndcg_at_1000_diff1
value: 16.2024
- type: nauc_map_at_1_max
value: 38.4858
- type: nauc_map_at_1_std
value: 11.2597
- type: nauc_map_at_1_diff1
value: 31.0795
- type: nauc_map_at_3_max
value: 39.8605
- type: nauc_map_at_3_std
value: 12.9185
- type: nauc_map_at_3_diff1
value: 22.0796
- type: nauc_map_at_5_max
value: 41.8284
- type: nauc_map_at_5_std
value: 14.2185
- type: nauc_map_at_5_diff1
value: 21.151
- type: nauc_map_at_10_max
value: 42.2605
- type: nauc_map_at_10_std
value: 15.4699
- type: nauc_map_at_10_diff1
value: 19.255300000000002
- type: nauc_map_at_20_max
value: 42.3461
- type: nauc_map_at_20_std
value: 16.022100000000002
- type: nauc_map_at_20_diff1
value: 18.574099999999998
- type: nauc_map_at_100_max
value: 42.051899999999996
- type: nauc_map_at_100_std
value: 16.4578
- type: nauc_map_at_100_diff1
value: 18.48
- type: nauc_map_at_1000_max
value: 41.888
- type: nauc_map_at_1000_std
value: 16.379099999999998
- type: nauc_map_at_1000_diff1
value: 18.398999999999997
- type: nauc_recall_at_1_max
value: 38.4858
- type: nauc_recall_at_1_std
value: 11.2597
- type: nauc_recall_at_1_diff1
value: 31.0795
- type: nauc_recall_at_3_max
value: 35.9672
- type: nauc_recall_at_3_std
value: 10.467600000000001
- type: nauc_recall_at_3_diff1
value: 17.453599999999998
- type: nauc_recall_at_5_max
value: 38.8183
- type: nauc_recall_at_5_std
value: 12.134
- type: nauc_recall_at_5_diff1
value: 17.482300000000002
- type: nauc_recall_at_10_max
value: 40.022200000000005
- type: nauc_recall_at_10_std
value: 14.777299999999999
- type: nauc_recall_at_10_diff1
value: 14.9899
- type: nauc_recall_at_20_max
value: 39.0768
- type: nauc_recall_at_20_std
value: 16.7705
- type: nauc_recall_at_20_diff1
value: 13.480400000000001
- type: nauc_recall_at_100_max
value: 32.8179
- type: nauc_recall_at_100_std
value: 17.1501
- type: nauc_recall_at_100_diff1
value: 15.689800000000002
- type: nauc_recall_at_1000_max
value: 24.5043
- type: nauc_recall_at_1000_std
value: 13.0304
- type: nauc_recall_at_1000_diff1
value: 13.047900000000002
- type: nauc_precision_at_1_max
value: 46.218599999999995
- type: nauc_precision_at_1_std
value: 20.8013
- type: nauc_precision_at_1_diff1
value: 23.6016
- type: nauc_precision_at_3_max
value: 46.4718
- type: nauc_precision_at_3_std
value: 22.9463
- type: nauc_precision_at_3_diff1
value: 13.406
- type: nauc_precision_at_5_max
value: 47.6376
- type: nauc_precision_at_5_std
value: 23.3855
- type: nauc_precision_at_5_diff1
value: 12.513
- type: nauc_precision_at_10_max
value: 45.2089
- type: nauc_precision_at_10_std
value: 23.3857
- type: nauc_precision_at_10_diff1
value: 9.8063
- type: nauc_precision_at_20_max
value: 42.873400000000004
- type: nauc_precision_at_20_std
value: 23.5178
- type: nauc_precision_at_20_diff1
value: 7.0671
- type: nauc_precision_at_100_max
value: 32.905499999999996
- type: nauc_precision_at_100_std
value: 21.5548
- type: nauc_precision_at_100_diff1
value: 7.4644
- type: nauc_precision_at_1000_max
value: 11.9931
- type: nauc_precision_at_1000_std
value: 9.6569
- type: nauc_precision_at_1000_diff1
value: 0.9825999999999999
- type: nauc_mrr_at_1_max
value: 46.218599999999995
- type: nauc_mrr_at_1_std
value: 20.8013
- type: nauc_mrr_at_1_diff1
value: 23.6016
- type: nauc_mrr_at_3_max
value: 41.7149
- type: nauc_mrr_at_3_std
value: 18.9033
- type: nauc_mrr_at_3_diff1
value: 18.7117
- type: nauc_mrr_at_5_max
value: 42.2504
- type: nauc_mrr_at_5_std
value: 18.807399999999998
- type: nauc_mrr_at_5_diff1
value: 18.8701
- type: nauc_mrr_at_10_max
value: 42.418
- type: nauc_mrr_at_10_std
value: 18.7045
- type: nauc_mrr_at_10_diff1
value: 18.9132
- type: nauc_mrr_at_20_max
value: 41.9178
- type: nauc_mrr_at_20_std
value: 18.7363
- type: nauc_mrr_at_20_diff1
value: 18.6007
- type: nauc_mrr_at_100_max
value: 41.5807
- type: nauc_mrr_at_100_std
value: 18.724
- type: nauc_mrr_at_100_diff1
value: 18.6174
- type: nauc_mrr_at_1000_max
value: 41.5871
- type: nauc_mrr_at_1000_std
value: 18.706
- type: nauc_mrr_at_1000_diff1
value: 18.6585
- type: main_score
value: 10.529
task:
type: Retrieval
- dataset:
config: ar-ar
name: MTEB STS17 (ar-ar)
revision: faeb762787bd10488a50c8b5be4a3b82e411949c
split: test
type: mteb/sts17-crosslingual-sts
metrics:
- type: cosine_pearson
value: 82.06597171670848
- type: cosine_spearman
value: 82.7809395809498
- type: euclidean_pearson
value: 79.23996991139896
- type: euclidean_spearman
value: 81.5287595404711
- type: main_score
value: 82.7809395809498
- type: manhattan_pearson
value: 78.95407006608013
- type: manhattan_spearman
value: 81.15109493737467
task:
type: STS
- dataset:
config: ar
name: MTEB STS22.v2 (ar)
revision: d31f33a128469b20e357535c39b82fb3c3f6f2bd
split: test
type: mteb/sts22-crosslingual-sts
metrics:
- type: cosine_pearson
value: 54.912880452465004
- type: cosine_spearman
value: 63.09788380910325
- type: euclidean_pearson
value: 57.92665617677832
- type: euclidean_spearman
value: 62.76032598469037
- type: main_score
value: 63.09788380910325
- type: manhattan_pearson
value: 58.0736648155273
- type: manhattan_spearman
value: 62.94190582776664
task:
type: STS
- dataset:
config: ar
name: MTEB STS22 (ar)
revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3
split: test
type: mteb/sts22-crosslingual-sts
metrics:
- type: cosine_pearson
value: 51.72534929358701
- type: cosine_spearman
value: 59.75149627160101
- type: euclidean_pearson
value: 53.894835373598774
- type: euclidean_spearman
value: 59.44278354697161
- type: main_score
value: 59.75149627160101
- type: manhattan_pearson
value: 54.076675975406985
- type: manhattan_spearman
value: 59.610061143235725
task:
type: STS
widget:
- source_sentence: امرأة تكتب شيئاً
sentences:
- مراهق يتحدث إلى فتاة عبر كاميرا الإنترنت
- امرأة تقطع البصل الأخضر.
- مجموعة من كبار السن يتظاهرون حول طاولة الطعام.
- source_sentence: تتشكل النجوم في مناطق تكوين النجوم، والتي تنشأ نفسها من السحب الجزيئية.
sentences:
- لاعب كرة السلة على وشك تسجيل نقاط لفريقه.
- المقال التالي مأخوذ من نسختي من "أطلس البطريق الجديد للتاريخ الوسطى"
- قد يكون من الممكن أن يوجد نظام شمسي مثل نظامنا خارج المجرة
- source_sentence: >-
تحت السماء الزرقاء مع الغيوم البيضاء، يصل طفل لمس مروحة طائرة واقفة على حقل
من العشب.
sentences:
- امرأة تحمل كأساً
- طفل يحاول لمس مروحة طائرة
- اثنان من عازبين عن الشرب يستعدون للعشاء
- source_sentence: رجل في منتصف العمر يحلق لحيته في غرفة ذات جدران بيضاء والتي لا تبدو كحمام
sentences:
- فتى يخطط اسمه على مكتبه
- رجل ينام
- المرأة وحدها وهي نائمة في غرفة نومها
- source_sentence: الكلب البني مستلقي على جانبه على سجادة بيج، مع جسم أخضر في المقدمة.
sentences:
- شخص طويل القامة
- المرأة تنظر من النافذة.
- لقد مات الكلب
license: apache-2.0
---
# GATE-AraBert-V1
This is **GATE | General Arabic Text Embedding** trained using SentenceTransformers in a **multi-task** setup. The system trains on the **AllNLI** and on the **STS** dataset.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2](https://huggingface.co/Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2)
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 768 tokens
- **Similarity Function:** Cosine Similarity
- **Training Datasets:**
- [all-nli](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic-NLi-Pair-Class)
- [sts](https://huggingface.co/datasets/Omartificial-Intelligence-Space/arabic-stsb)
- **Language:** ar
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("Omartificial-Intelligence-Space/GATE-AraBert-v1")
# Run inference
sentences = [
'الكلب البني مستلقي على جانبه على سجادة بيج، مع جسم أخضر في المقدمة.',
'لقد مات الكلب',
'شخص طويل القامة',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
## Evaluation
### Metrics
#### Semantic Similarity
* Dataset: `sts-dev`
* Evaluated with [EmbeddingSimilarityEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
| Metric | Value |
|:--------------------|:----------|
| pearson_cosine | 0.8391 |
| **spearman_cosine** | **0.841** |
| pearson_manhattan | 0.8277 |
| spearman_manhattan | 0.8361 |
| pearson_euclidean | 0.8274 |
| spearman_euclidean | 0.8358 |
| pearson_dot | 0.8154 |
| spearman_dot | 0.818 |
| pearson_max | 0.8391 |
| spearman_max | 0.841 |
#### Semantic Similarity
* Dataset: `sts-test`
* Evaluated with [EmbeddingSimilarityEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| pearson_cosine | 0.813 |
| **spearman_cosine** | **0.8173** |
| pearson_manhattan | 0.8114 |
| spearman_manhattan | 0.8164 |
| pearson_euclidean | 0.8103 |
| spearman_euclidean | 0.8158 |
| pearson_dot | 0.7908 |
| spearman_dot | 0.7887 |
| pearson_max | 0.813 |
| spearman_max | 0.8173 |
## Acknowledgments
The author would like to thank Prince Sultan University for their invaluable support in this project. Their contributions and resources have been instrumental in the development and fine-tuning of these models.
```markdown
## Citation
If you use the GATE, please cite it as follows:
@misc{nacar2025GATE,
title={GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Hybrid Loss Training},
author={Omer Nacar, Anis Koubaa, Serry Taiseer Sibaee and Lahouari Ghouti},
year={2025},
note={Submitted to COLING 2025},
url={https://huggingface.co/Omartificial-Intelligence-Space/GATE-AraBert-v1},
}