Hugging Face's logo

tomaarsen
/
reranker-msmarco-v1.1-ModernBERT-base-bce

Text Classification
sentence-transformers
Safetensors
English
modernbert
cross-encoder
Generated from Trainer
dataset_size:82326
loss:BinaryCrossEntropyLoss
Model card Files Community
tomaarsen HF staff commited on
Commit
ff5b274
·
verified ·
1 Parent(s): e616b2b

Upload correct generated model card

Browse files
Files changed (1) hide show
  1. README.md +404 -136
README.md CHANGED
@@ -1,137 +1,405 @@
1
- ---
2
- tags:
3
- - sentence-transformers
4
- - cross-encoder
5
- - text-classification
6
- pipeline_tag: text-classification
7
- library_name: sentence-transformers
8
- ---
9
-
10
- # CrossEncoder
11
-
12
- This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model trained using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
13
-
14
- ## Model Details
15
-
16
- ### Model Description
17
- - **Model Type:** Cross Encoder
18
- <!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
19
- - **Maximum Sequence Length:** 8192 tokens
20
- - **Number of Output Labels:** 1 label
21
- <!-- - **Training Dataset:** Unknown -->
22
- <!-- - **Language:** Unknown -->
23
- <!-- - **License:** Unknown -->
24
-
25
- ### Model Sources
26
-
27
- - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
28
- - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html)
29
- - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
30
- - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder)
31
-
32
- ## Usage
33
-
34
- ### Direct Usage (Sentence Transformers)
35
-
36
- First install the Sentence Transformers library:
37
-
38
- ```bash
39
- pip install -U sentence-transformers
40
- ```
41
-
42
- Then you can load this model and run inference.
43
- ```python
44
- from sentence_transformers import CrossEncoder
45
-
46
- # Download from the 🤗 Hub
47
- model = CrossEncoder("tomaarsen/reranker-msmarco-v1.1-ModernBERT-base-bce")
48
- # Get scores for pairs...
49
- pairs = [
50
- ['How many calories in an egg', 'There are on average between 55 and 80 calories in an egg depending on its size.'],
51
- ['How many calories in an egg', 'Egg whites are very low in calories, have no fat, no cholesterol, and are loaded with protein.'],
52
- ['How many calories in an egg', 'Most of the calories in an egg come from the yellow yolk in the center.'],
53
- ]
54
- scores = model.predict(pairs)
55
- print(scores.shape)
56
- # [3]
57
-
58
- # ... or rank different texts based on similarity to a single text
59
- ranks = model.rank(
60
- 'How many calories in an egg',
61
- [
62
- 'There are on average between 55 and 80 calories in an egg depending on its size.',
63
- 'Egg whites are very low in calories, have no fat, no cholesterol, and are loaded with protein.',
64
- 'Most of the calories in an egg come from the yellow yolk in the center.',
65
- ]
66
- )
67
- # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
68
- ```
69
-
70
- <!--
71
- ### Direct Usage (Transformers)
72
-
73
- <details><summary>Click to see the direct usage in Transformers</summary>
74
-
75
- </details>
76
- -->
77
-
78
- <!--
79
- ### Downstream Usage (Sentence Transformers)
80
-
81
- You can finetune this model on your own dataset.
82
-
83
- <details><summary>Click to expand</summary>
84
-
85
- </details>
86
- -->
87
-
88
- <!--
89
- ### Out-of-Scope Use
90
-
91
- *List how the model may foreseeably be misused and address what users ought not to do with the model.*
92
- -->
93
-
94
- <!--
95
- ## Bias, Risks and Limitations
96
-
97
- *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
98
- -->
99
-
100
- <!--
101
- ### Recommendations
102
-
103
- *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
104
- -->
105
-
106
- ## Training Details
107
-
108
- ### Framework Versions
109
- - Python: 3.11.6
110
- - Sentence Transformers: 3.5.0.dev0
111
- - Transformers: 4.48.3
112
- - PyTorch: 2.5.0+cu121
113
- - Accelerate: 1.3.0
114
- - Datasets: 2.20.0
115
- - Tokenizers: 0.21.0
116
-
117
- ## Citation
118
-
119
- ### BibTeX
120
-
121
- <!--
122
- ## Glossary
123
-
124
- *Clearly define terms in order to be accessible across audiences.*
125
- -->
126
-
127
- <!--
128
- ## Model Card Authors
129
-
130
- *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
131
- -->
132
-
133
- <!--
134
- ## Model Card Contact
135
-
136
- *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
  -->
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - sentence-transformers
6
+ - cross-encoder
7
+ - text-classification
8
+ - generated_from_trainer
9
+ - dataset_size:82326
10
+ - loss:BinaryCrossEntropyLoss
11
+ base_model: answerdotai/ModernBERT-base
12
+ datasets:
13
+ - microsoft/ms_marco
14
+ pipeline_tag: text-classification
15
+ library_name: sentence-transformers
16
+ metrics:
17
+ - map
18
+ - mrr@10
19
+ - ndcg@10
20
+ model-index:
21
+ - name: CrossEncoder based on answerdotai/ModernBERT-base
22
+ results: []
23
+ ---
24
+
25
+ # CrossEncoder based on answerdotai/ModernBERT-base
26
+
27
+ This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) dataset using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
28
+
29
+ ## Model Details
30
+
31
+ ### Model Description
32
+ - **Model Type:** Cross Encoder
33
+ - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) <!-- at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8 -->
34
+ - **Maximum Sequence Length:** 8192 tokens
35
+ - **Number of Output Labels:** 1 label
36
+ - **Training Dataset:**
37
+ - [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco)
38
+ - **Language:** en
39
+ <!-- - **License:** Unknown -->
40
+
41
+ ### Model Sources
42
+
43
+ - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
44
+ - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html)
45
+ - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
46
+ - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder)
47
+
48
+ ## Usage
49
+
50
+ ### Direct Usage (Sentence Transformers)
51
+
52
+ First install the Sentence Transformers library:
53
+
54
+ ```bash
55
+ pip install -U sentence-transformers
56
+ ```
57
+
58
+ Then you can load this model and run inference.
59
+ ```python
60
+ from sentence_transformers import CrossEncoder
61
+
62
+ # Download from the 🤗 Hub
63
+ model = CrossEncoder("sentence_transformers_model_id")
64
+ # Get scores for pairs...
65
+ pairs = [
66
+ ['what popular gemstone sounds like peridotite', 'Peridot (pronounced pair-a-doe) is the gem variety of olivine. Olivine, which is actually not an official mineral, is composed of two minerals: fayalite and forsterite. Fayalite is the iron rich member with a pure formula of Fe2SiO4. Forsterite is the magnesium rich member with a pure formula of Mg2SiO4.'],
67
+ ['how much does it cost to keep an inmate in jail', 'The cost of each prison varies, depending on the types of inmates who are housed there. For example, it costs $99.12 a day to house an inmate at a reception center, because the inmates residing there are being evaluated and tested medically, psychologically, academically, vocationally, etc. In contrast, a typical adult male facility costs just $44.96 per day to house an inmate (excluding private prisons). When you average all types of state prison facilities together like those listed in the chart below, the daily cost to house an inmate is $55.09.'],
68
+ ['what process is found in both cellular respiration and photosynthesis', 'Cellular respiration takes place in the same way in both plants and animals. Living cells obtain the products of photosynthesis (sugar molecules) and undergo cellular respiration to produce ATP molecules. Some cells respire aerobically, using oxygen, while others undergo anaerobic respiration, without using oxygen. ☞ Cellular respiration takes place in the cytoplasm and mitochondria of the cell. ☛ Photosynthesis uses water, sunlight, and CO 2 from the atmosphere to create glucose molecules, and releases oxygen as a by-product.'],
69
+ ["how to keep someone at arm's length", "keep at arm's length from someone or something. Fig. to retain a degree of physical or social remoteness from someone or something. I try to keep at arm's length from Larry, since our disagreement. I keep Tom at arm's length because we don't get along. 1 keep someone or something around. 2 keep someone or something at arm's length. 3 keep someone or something at bay. 4 keep someone or something hanging. 5 keep someone or something off. 6 keep someone or something on track. 7 keep someone or something under control"],
70
+ ['which is a characteristic of something in the domain bacteria', 'Archaea were only shown to be a separate domain—through analysis of their RNA—in 1977. Many archaea thrive under the extreme conditions of hot sulfur pools or in minerals and rock deep inside the Earth. '],
71
+ ]
72
+ scores = model.predict(pairs)
73
+ print(scores.shape)
74
+ # [5]
75
+
76
+ # ... or rank different texts based on similarity to a single text
77
+ ranks = model.rank(
78
+ 'what popular gemstone sounds like peridotite',
79
+ [
80
+ 'Peridot (pronounced pair-a-doe) is the gem variety of olivine. Olivine, which is actually not an official mineral, is composed of two minerals: fayalite and forsterite. Fayalite is the iron rich member with a pure formula of Fe2SiO4. Forsterite is the magnesium rich member with a pure formula of Mg2SiO4.',
81
+ 'The cost of each prison varies, depending on the types of inmates who are housed there. For example, it costs $99.12 a day to house an inmate at a reception center, because the inmates residing there are being evaluated and tested medically, psychologically, academically, vocationally, etc. In contrast, a typical adult male facility costs just $44.96 per day to house an inmate (excluding private prisons). When you average all types of state prison facilities together like those listed in the chart below, the daily cost to house an inmate is $55.09.',
82
+ 'Cellular respiration takes place in the same way in both plants and animals. Living cells obtain the products of photosynthesis (sugar molecules) and undergo cellular respiration to produce ATP molecules. Some cells respire aerobically, using oxygen, while others undergo anaerobic respiration, without using oxygen. ☞ Cellular respiration takes place in the cytoplasm and mitochondria of the cell. ☛ Photosynthesis uses water, sunlight, and CO 2 from the atmosphere to create glucose molecules, and releases oxygen as a by-product.',
83
+ "keep at arm's length from someone or something. Fig. to retain a degree of physical or social remoteness from someone or something. I try to keep at arm's length from Larry, since our disagreement. I keep Tom at arm's length because we don't get along. 1 keep someone or something around. 2 keep someone or something at arm's length. 3 keep someone or something at bay. 4 keep someone or something hanging. 5 keep someone or something off. 6 keep someone or something on track. 7 keep someone or something under control",
84
+ 'Archaea were only shown to be a separate domain—through analysis of their RNA—in 1977. Many archaea thrive under the extreme conditions of hot sulfur pools or in minerals and rock deep inside the Earth. ',
85
+ ]
86
+ )
87
+ # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
88
+ ```
89
+
90
+ <!--
91
+ ### Direct Usage (Transformers)
92
+
93
+ <details><summary>Click to see the direct usage in Transformers</summary>
94
+
95
+ </details>
96
+ -->
97
+
98
+ <!--
99
+ ### Downstream Usage (Sentence Transformers)
100
+
101
+ You can finetune this model on your own dataset.
102
+
103
+ <details><summary>Click to expand</summary>
104
+
105
+ </details>
106
+ -->
107
+
108
+ <!--
109
+ ### Out-of-Scope Use
110
+
111
+ *List how the model may foreseeably be misused and address what users ought not to do with the model.*
112
+ -->
113
+
114
+ ## Evaluation
115
+
116
+ ### Metrics
117
+
118
+ #### Cross Encoder Reranking
119
+
120
+ * Datasets: `NanoMSMARCO`, `NanoNFCorpus` and `NanoNQ`
121
+ * Evaluated with [<code>CERerankingEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CERerankingEvaluator)
122
+
123
+ | Metric | NanoMSMARCO | NanoNFCorpus | NanoNQ |
124
+ |:------------|:---------------------|:---------------------|:---------------------|
125
+ | map | 0.4939 (+0.0044) | 0.3115 (+0.0411) | 0.5612 (+0.1405) |
126
+ | mrr@10 | 0.4840 (+0.0065) | 0.4840 (-0.0158) | 0.5607 (+0.1340) |
127
+ | **ndcg@10** | **0.5549 (+0.0145)** | **0.3503 (+0.0252)** | **0.6091 (+0.1085)** |
128
+
129
+ #### Cross Encoder Nano BEIR
130
+
131
+ * Dataset: `NanoBEIR_mean`
132
+ * Evaluated with [<code>CENanoBEIREvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CENanoBEIREvaluator)
133
+
134
+ | Metric | Value |
135
+ |:------------|:---------------------|
136
+ | map | 0.4555 (+0.0620) |
137
+ | mrr@10 | 0.5096 (+0.0416) |
138
+ | **ndcg@10** | **0.5048 (+0.0494)** |
139
+
140
+ <!--
141
+ ## Bias, Risks and Limitations
142
+
143
+ *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
144
+ -->
145
+
146
+ <!--
147
+ ### Recommendations
148
+
149
+ *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
150
+ -->
151
+
152
+ ## Training Details
153
+
154
+ ### Training Dataset
155
+
156
+ #### ms_marco
157
+
158
+ * Dataset: [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) at [a47ee7a](https://huggingface.co/datasets/microsoft/ms_marco/tree/a47ee7aae8d7d466ba15f9f0bfac3b3681087b3a)
159
+ * Size: 82,326 training samples
160
+ * Columns: <code>query</code>, <code>passage</code>, and <code>label</code>
161
+ * Approximate statistics based on the first 1000 samples:
162
+ | | query | passage | label |
163
+ |:--------|:-----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------|
164
+ | type | string | string | int |
165
+ | details | <ul><li>min: 11 characters</li><li>mean: 34.37 characters</li><li>max: 99 characters</li></ul> | <ul><li>min: 61 characters</li><li>mean: 415.74 characters</li><li>max: 949 characters</li></ul> | <ul><li>0: ~87.50%</li><li>1: ~12.50%</li></ul> |
166
+ * Samples:
167
+ | query | passage | label |
168
+ |:-------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------|
169
+ | <code>honor meaning</code> | <code>Honor is also employed to signify integrity in a judge, courage in a soldier, and chastity in a woman. To deprive a woman of her honor is, in some cases, punished as a public wrong, and by an action for the recovery of damages done to the relative rights of a husband or a father. Vide Criminal conversation. 2.</code> | <code>0</code> |
170
+ | <code>what happens if xmas day falls on weekend</code> | <code>1 If the holiday falls on a Saturday or Sunday and that day would not otherwise be a working day for the employee, the holiday is transferred to the following Monday or Tuesday so that the employee still gets a paid day off if the employee would usually work on these days.</code> | <code>0</code> |
171
+ | <code>what is the longest recorded flight time of a chicken</code> | <code>According to several chicken trivia sites like Feathersite, Poultryhelp and also including the Guinness World Book of World records, the longest verified flight of a domestic … chicken is 13 seconds and 301.5 feet distance.</code> | <code>0</code> |
172
+ * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss)
173
+
174
+ ### Evaluation Dataset
175
+
176
+ #### ms_marco
177
+
178
+ * Dataset: [ms_marco](https://huggingface.co/datasets/microsoft/ms_marco) at [a47ee7a](https://huggingface.co/datasets/microsoft/ms_marco/tree/a47ee7aae8d7d466ba15f9f0bfac3b3681087b3a)
179
+ * Size: 82,326 evaluation samples
180
+ * Columns: <code>query</code>, <code>passage</code>, and <code>label</code>
181
+ * Approximate statistics based on the first 1000 samples:
182
+ | | query | passage | label |
183
+ |:--------|:-----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------|
184
+ | type | string | string | int |
185
+ | details | <ul><li>min: 11 characters</li><li>mean: 34.76 characters</li><li>max: 99 characters</li></ul> | <ul><li>min: 92 characters</li><li>mean: 420.95 characters</li><li>max: 980 characters</li></ul> | <ul><li>0: ~85.50%</li><li>1: ~14.50%</li></ul> |
186
+ * Samples:
187
+ | query | passage | label |
188
+ |:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------|
189
+ | <code>what popular gemstone sounds like peridotite</code> | <code>Peridot (pronounced pair-a-doe) is the gem variety of olivine. Olivine, which is actually not an official mineral, is composed of two minerals: fayalite and forsterite. Fayalite is the iron rich member with a pure formula of Fe2SiO4. Forsterite is the magnesium rich member with a pure formula of Mg2SiO4.</code> | <code>0</code> |
190
+ | <code>how much does it cost to keep an inmate in jail</code> | <code>The cost of each prison varies, depending on the types of inmates who are housed there. For example, it costs $99.12 a day to house an inmate at a reception center, because the inmates residing there are being evaluated and tested medically, psychologically, academically, vocationally, etc. In contrast, a typical adult male facility costs just $44.96 per day to house an inmate (excluding private prisons). When you average all types of state prison facilities together like those listed in the chart below, the daily cost to house an inmate is $55.09.</code> | <code>0</code> |
191
+ | <code>what process is found in both cellular respiration and photosynthesis</code> | <code>Cellular respiration takes place in the same way in both plants and animals. Living cells obtain the products of photosynthesis (sugar molecules) and undergo cellular respiration to produce ATP molecules. Some cells respire aerobically, using oxygen, while others undergo anaerobic respiration, without using oxygen. ☞ Cellular respiration takes place in the cytoplasm and mitochondria of the cell. ☛ Photosynthesis uses water, sunlight, and CO 2 from the atmosphere to create glucose molecules, and releases oxygen as a by-product.</code> | <code>0</code> |
192
+ * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss)
193
+
194
+ ### Training Hyperparameters
195
+ #### Non-Default Hyperparameters
196
+
197
+ - `eval_strategy`: steps
198
+ - `per_device_train_batch_size`: 128
199
+ - `per_device_eval_batch_size`: 128
200
+ - `learning_rate`: 2e-05
201
+ - `num_train_epochs`: 1
202
+ - `warmup_ratio`: 0.1
203
+ - `seed`: 12
204
+ - `bf16`: True
205
+ - `load_best_model_at_end`: True
206
+
207
+ #### All Hyperparameters
208
+ <details><summary>Click to expand</summary>
209
+
210
+ - `overwrite_output_dir`: False
211
+ - `do_predict`: False
212
+ - `eval_strategy`: steps
213
+ - `prediction_loss_only`: True
214
+ - `per_device_train_batch_size`: 128
215
+ - `per_device_eval_batch_size`: 128
216
+ - `per_gpu_train_batch_size`: None
217
+ - `per_gpu_eval_batch_size`: None
218
+ - `gradient_accumulation_steps`: 1
219
+ - `eval_accumulation_steps`: None
220
+ - `torch_empty_cache_steps`: None
221
+ - `learning_rate`: 2e-05
222
+ - `weight_decay`: 0.0
223
+ - `adam_beta1`: 0.9
224
+ - `adam_beta2`: 0.999
225
+ - `adam_epsilon`: 1e-08
226
+ - `max_grad_norm`: 1.0
227
+ - `num_train_epochs`: 1
228
+ - `max_steps`: -1
229
+ - `lr_scheduler_type`: linear
230
+ - `lr_scheduler_kwargs`: {}
231
+ - `warmup_ratio`: 0.1
232
+ - `warmup_steps`: 0
233
+ - `log_level`: passive
234
+ - `log_level_replica`: warning
235
+ - `log_on_each_node`: True
236
+ - `logging_nan_inf_filter`: True
237
+ - `save_safetensors`: True
238
+ - `save_on_each_node`: False
239
+ - `save_only_model`: False
240
+ - `restore_callback_states_from_checkpoint`: False
241
+ - `no_cuda`: False
242
+ - `use_cpu`: False
243
+ - `use_mps_device`: False
244
+ - `seed`: 12
245
+ - `data_seed`: None
246
+ - `jit_mode_eval`: False
247
+ - `use_ipex`: False
248
+ - `bf16`: True
249
+ - `fp16`: False
250
+ - `fp16_opt_level`: O1
251
+ - `half_precision_backend`: auto
252
+ - `bf16_full_eval`: False
253
+ - `fp16_full_eval`: False
254
+ - `tf32`: None
255
+ - `local_rank`: 0
256
+ - `ddp_backend`: None
257
+ - `tpu_num_cores`: None
258
+ - `tpu_metrics_debug`: False
259
+ - `debug`: []
260
+ - `dataloader_drop_last`: False
261
+ - `dataloader_num_workers`: 0
262
+ - `dataloader_prefetch_factor`: None
263
+ - `past_index`: -1
264
+ - `disable_tqdm`: False
265
+ - `remove_unused_columns`: True
266
+ - `label_names`: None
267
+ - `load_best_model_at_end`: True
268
+ - `ignore_data_skip`: False
269
+ - `fsdp`: []
270
+ - `fsdp_min_num_params`: 0
271
+ - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
272
+ - `fsdp_transformer_layer_cls_to_wrap`: None
273
+ - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
274
+ - `deepspeed`: None
275
+ - `label_smoothing_factor`: 0.0
276
+ - `optim`: adamw_torch
277
+ - `optim_args`: None
278
+ - `adafactor`: False
279
+ - `group_by_length`: False
280
+ - `length_column_name`: length
281
+ - `ddp_find_unused_parameters`: None
282
+ - `ddp_bucket_cap_mb`: None
283
+ - `ddp_broadcast_buffers`: False
284
+ - `dataloader_pin_memory`: True
285
+ - `dataloader_persistent_workers`: False
286
+ - `skip_memory_metrics`: True
287
+ - `use_legacy_prediction_loop`: False
288
+ - `push_to_hub`: False
289
+ - `resume_from_checkpoint`: None
290
+ - `hub_model_id`: None
291
+ - `hub_strategy`: every_save
292
+ - `hub_private_repo`: None
293
+ - `hub_always_push`: False
294
+ - `gradient_checkpointing`: False
295
+ - `gradient_checkpointing_kwargs`: None
296
+ - `include_inputs_for_metrics`: False
297
+ - `include_for_metrics`: []
298
+ - `eval_do_concat_batches`: True
299
+ - `fp16_backend`: auto
300
+ - `push_to_hub_model_id`: None
301
+ - `push_to_hub_organization`: None
302
+ - `mp_parameters`:
303
+ - `auto_find_batch_size`: False
304
+ - `full_determinism`: False
305
+ - `torchdynamo`: None
306
+ - `ray_scope`: last
307
+ - `ddp_timeout`: 1800
308
+ - `torch_compile`: False
309
+ - `torch_compile_backend`: None
310
+ - `torch_compile_mode`: None
311
+ - `dispatch_batches`: None
312
+ - `split_batches`: None
313
+ - `include_tokens_per_second`: False
314
+ - `include_num_input_tokens_seen`: False
315
+ - `neftune_noise_alpha`: None
316
+ - `optim_target_modules`: None
317
+ - `batch_eval_metrics`: False
318
+ - `eval_on_start`: False
319
+ - `use_liger_kernel`: False
320
+ - `eval_use_gather_object`: False
321
+ - `average_tokens_across_devices`: False
322
+ - `prompts`: None
323
+ - `batch_sampler`: batch_sampler
324
+ - `multi_dataset_batch_sampler`: proportional
325
+
326
+ </details>
327
+
328
+ ### Training Logs
329
+ | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_ndcg@10 | NanoNFCorpus_ndcg@10 | NanoNQ_ndcg@10 | NanoBEIR_mean_ndcg@10 |
330
+ |:----------:|:--------:|:-------------:|:---------------:|:--------------------:|:--------------------:|:--------------------:|:---------------------:|
331
+ | -1 | -1 | - | - | 0.0381 (-0.5023) | 0.3036 (-0.0214) | 0.0388 (-0.4619) | 0.1268 (-0.3285) |
332
+ | 0.0002 | 1 | 0.8716 | - | - | - | - | - |
333
+ | 0.0384 | 200 | 0.4492 | - | - | - | - | - |
334
+ | 0.0768 | 400 | 0.382 | - | - | - | - | - |
335
+ | 0.1153 | 600 | 0.3863 | - | - | - | - | - |
336
+ | 0.1537 | 800 | 0.3776 | - | - | - | - | - |
337
+ | 0.1921 | 1000 | 0.364 | 0.3614 | 0.4437 (-0.0967) | 0.2752 (-0.0498) | 0.5125 (+0.0118) | 0.4105 (-0.0449) |
338
+ | 0.2305 | 1200 | 0.3529 | - | - | - | - | - |
339
+ | 0.2690 | 1400 | 0.3491 | - | - | - | - | - |
340
+ | 0.3074 | 1600 | 0.346 | - | - | - | - | - |
341
+ | 0.3458 | 1800 | 0.3411 | - | - | - | - | - |
342
+ | 0.3842 | 2000 | 0.3456 | 0.3465 | 0.5523 (+0.0119) | 0.3359 (+0.0109) | 0.5412 (+0.0406) | 0.4765 (+0.0211) |
343
+ | 0.4227 | 2200 | 0.3476 | - | - | - | - | - |
344
+ | 0.4611 | 2400 | 0.3391 | - | - | - | - | - |
345
+ | 0.4995 | 2600 | 0.3392 | - | - | - | - | - |
346
+ | 0.5379 | 2800 | 0.3355 | - | - | - | - | - |
347
+ | 0.5764 | 3000 | 0.349 | 0.3417 | 0.5289 (-0.0115) | 0.3129 (-0.0122) | 0.5651 (+0.0644) | 0.4690 (+0.0136) |
348
+ | 0.6148 | 3200 | 0.3342 | - | - | - | - | - |
349
+ | 0.6532 | 3400 | 0.3352 | - | - | - | - | - |
350
+ | 0.6916 | 3600 | 0.341 | - | - | - | - | - |
351
+ | 0.7301 | 3800 | 0.3383 | - | - | - | - | - |
352
+ | **0.7685** | **4000** | **0.3353** | **0.3364** | **0.5549 (+0.0145)** | **0.3503 (+0.0252)** | **0.6091 (+0.1085)** | **0.5048 (+0.0494)** |
353
+ | 0.8069 | 4200 | 0.3339 | - | - | - | - | - |
354
+ | 0.8453 | 4400 | 0.3275 | - | - | - | - | - |
355
+ | 0.8838 | 4600 | 0.3283 | - | - | - | - | - |
356
+ | 0.9222 | 4800 | 0.3293 | - | - | - | - | - |
357
+ | 0.9606 | 5000 | 0.3368 | 0.3351 | 0.5314 (-0.0090) | 0.3221 (-0.0030) | 0.5952 (+0.0946) | 0.4829 (+0.0275) |
358
+ | 0.9990 | 5200 | 0.3298 | - | - | - | - | - |
359
+ | -1 | -1 | - | - | 0.5549 (+0.0145) | 0.3503 (+0.0252) | 0.6091 (+0.1085) | 0.5048 (+0.0494) |
360
+
361
+ * The bold row denotes the saved checkpoint.
362
+
363
+ ### Framework Versions
364
+ - Python: 3.11.10
365
+ - Sentence Transformers: 3.5.0.dev0
366
+ - Transformers: 4.49.0.dev0
367
+ - PyTorch: 2.6.0.dev20241112+cu121
368
+ - Accelerate: 1.2.0
369
+ - Datasets: 3.2.0
370
+ - Tokenizers: 0.21.0
371
+
372
+ ## Citation
373
+
374
+ ### BibTeX
375
+
376
+ #### Sentence Transformers
377
+ ```bibtex
378
+ @inproceedings{reimers-2019-sentence-bert,
379
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
380
+ author = "Reimers, Nils and Gurevych, Iryna",
381
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
382
+ month = "11",
383
+ year = "2019",
384
+ publisher = "Association for Computational Linguistics",
385
+ url = "https://arxiv.org/abs/1908.10084",
386
+ }
387
+ ```
388
+
389
+ <!--
390
+ ## Glossary
391
+
392
+ *Clearly define terms in order to be accessible across audiences.*
393
+ -->
394
+
395
+ <!--
396
+ ## Model Card Authors
397
+
398
+ *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
399
+ -->
400
+
401
+ <!--
402
+ ## Model Card Contact
403
+
404
+ *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
405
  -->