File size: 41,866 Bytes
9e5e405 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 |
---
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:2859594
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
base_model: Qwen/Qwen2.5-0.5B-Instruct
widget:
- source_sentence: How old is Garry Marshall?
sentences:
- 'Garry Marshall
On the morning of July 19, 2016, Marshall died at a hospital in Burbank, California
at the age of 81 due to complications of pneumonia after suffering a stroke.[20][21]'
- 'Gregg Marshall
Michael Gregg Marshall (born February 27, 1963) is an American college basketball
coach who currently leads the Shockers team at Wichita State University. Marshall
has coached his teams to appearances in the NCAA Men''s Division I Basketball
Tournament in twelve of his eighteen years as a head coach. He is the most successful
head coach in Wichita State University history (261 wins), and is also the most
successful head coach in Winthrop University history (194 wins).'
- 'Guillotine
For a period of time after its invention, the guillotine was called a louisette.
However, it was later named after Guillotin who had proposed that a less painful
method of execution should be found in place of the breaking wheel, though he
opposed the death penalty and bemoaned the association of the guillotine with
his name.'
- source_sentence: Are there cherry trees in Cherry Springs State Park?
sentences:
- 'Cherry Springs State Park
Awards and press recognition have come to Cherry Springs and its staff. Thom Bemus,
who initiated and coordinates the Stars-n-Parks program, was named DCNR''s 2002Volunteer
of the Year.[66] In 2007the park''s Dark Sky Programming and staff received the
Environmental Education Excellence in Programming award from the Pennsylvania
Recreation and Parks Society.[67] Operations manager Chip Harrison and his wife
Maxine, who directs the Dark Sky Fund, received a 2008award from the Pennsylvania
Outdoor Lighting Council for "steadfast adherence and active promotion of the
principles of responsible outdoor lighting at Cherry Springs State Park".[68]
The DCNR has named Cherry Springs one of "25 Must-See Pennsylvania State Parks",
specifically for having the "darkest night skies on the east coast".[69] Cherry
Springs State Park was featured in the national press in 2003when USA Today named
it one of "10Great Places to get some stars in your eyes",[70] in 2006when National
Geographic Adventure featured it in "Pennsylvania: The Wild, Wild East",[71] and
in The New York Times in 2007.[53] All these were before it was named an International
Dark Sky Park by the International Dark-Sky Association in 2008.[38]'
- 'Cantonese
Although Cantonese shares a lot of vocabulary with Mandarin, the two varieties
are mutually unintelligible because of differences in pronunciation, grammar and
lexicon. Sentence structure, in particular the placement of verbs, sometimes differs
between the two varieties. A notable difference between Cantonese and Mandarin
is how the spoken word is written; both can be recorded verbatim, but very few
Cantonese speakers are knowledgeable in the full Cantonese written vocabulary,
so a non-verbatim formalized written form is adopted, which is more akin to the
Mandarin written form.[4][5] This results in the situation in which a Cantonese
and a Mandarin text may look similar but are pronounced differently.'
- 'Cherry Springs State Park
Cherry Springs State Park is an 82-acre (33ha)[a] Pennsylvania state park in Potter
County, Pennsylvania, United States. The park was created from land within the
Susquehannock State Forest, and is on Pennsylvania Route 44 in West Branch Township.
Cherry Springs, named for a large stand of Black Cherry trees in the park, is
atop the dissected Allegheny Plateau at an elevation of 2,300 feet (701m). It
is popular with astronomers and stargazers for having "some of the darkest night
skies on the east coast" of the United States, and was chosen by the Pennsylvania
Department of Conservation and Natural Resources (DCNR) and its Bureau of Parks
as one of "25 Must-See Pennsylvania State Parks".[4]'
- source_sentence: How many regions are in Belgium?
sentences:
- 'Pine City, Minnesota
Pine City is a city in Pine County, Minnesota, in East Central Minnesota. Pine
City is the county seat of, and the largest city in, Pine County.[7] A portion
of the city is located on the Mille Lacs Indian Reservation. Founded as a railway
town, it quickly became a logging community and the surrounding lakes made it
a resort town. Today, it is an arts town and commuter town to jobs in the Minneapolis–Saint
Paul metropolitan area.[8] It is also a green city.[9] The population was 3,127
at the 2010 census.'
- 'Provinces of Belgium
The country of Belgium is divided into three regions. Two of these regions, the
Flemish Region or Flanders, and Walloon Region, or Wallonia, are each subdivided
into five provinces. The third region, the Brussels-Capital Region, is not divided
into provinces, as it was originally only a small part of a province itself.'
- 'United Belgian States
The United Belgian States was a confederal republic of eight provinces which had
their own governments, were sovereign and independent, and were governed directly
by the Sovereign Congress (; ), the confederal government. The Sovereign Congress
was seated in Brussels and consisted of representatives of each of the eight provinces.
The provinces of the republic were divided into 11 smaller separate territories,
each with their own regional identities:In 1789, a church-inspired popular revolt
broke out in reaction to the emperor''s centralizing and anticlerical policies.
Two factions appeared: the "Statists" who opposed the reforms, and the "Vonckists"
named for Jan Frans Vonck who initially supported the reforms but then joined
the opposition, due to the clumsy way in which the reforms were carried out.'
- source_sentence: Are there black holes near the galactic nucleus?
sentences:
- 'Supermassive black hole
In September 2014, data from different X-ray telescopes has shown that the extremely
small, dense, ultracompact dwarf galaxy M60-UCD1 hosts a 20 million solar mass
black hole at its center, accounting for more than 10% of the total mass of the
galaxy. The discovery is quite surprising, since the black hole is five times
more massive than the Milky Way''s black hole despite the galaxy being less than
five-thousandths the mass of the Milky Way.'
- 'Aquarela do Brasil
"Aquarela do Brasil" (Portuguese:[akwaˈɾɛlɐ du bɾaˈziw], Watercolor of Brazil),
written by Ary Barroso in 1939 and known in the English-speaking world simply
as "Brazil", is one of the most famous Brazilian songs.'
- 'Supermassive black hole
The difficulty in forming a supermassive black hole resides in the need for enough
matter to be in a small enough volume. This matter needs to have very little angular
momentum in order for this to happen. Normally, the process of accretion involves
transporting a large initial endowment of angular momentum outwards, and this
appears to be the limiting factor in black hole growth. This is a major component
of the theory of accretion disks. Gas accretion is the most efficient and also
the most conspicuous way in which black holes grow. The majority of the mass growth
of supermassive black holes is thought to occur through episodes of rapid gas
accretion, which are observable as active galactic nuclei or quasars. Observations
reveal that quasars were much more frequent when the Universe was younger, indicating
that supermassive black holes formed and grew early. A major constraining factor
for theories of supermassive black hole formation is the observation of distant
luminous quasars, which indicate that supermassive black holes of billions of
solar masses had already formed when the Universe was less than one billion years
old. This suggests that supermassive black holes arose very early in the Universe,
inside the first massive galaxies.'
- source_sentence: When did the July Monarchy end?
sentences:
- 'July Monarchy
Despite the return of the House of Bourbon to power, France was much changed from
the era of the ancien régime. The egalitarianism and liberalism of the revolutionaries
remained an important force and the autocracy and hierarchy of the earlier era
could not be fully restored. Economic changes, which had been underway long before
the revolution, had progressed further during the years of turmoil and were firmly
entrenched by 1815. These changes had seen power shift from the noble landowners
to the urban merchants. The administrative reforms of Napoleon, such as the Napoleonic
Code and efficient bureaucracy, also remained in place. These changes produced
a unified central government that was fiscally sound and had much control over
all areas of French life, a sharp difference from the complicated mix of feudal
and absolutist traditions and institutions of pre-Revolutionary Bourbons.'
- 'Wachovia
Wachovia Corporation began on June 16, 1879 in Winston-Salem, North Carolina as
the Wachovia National Bank. The bank was co-founded by James Alexander Gray and
William Lemly.[9] In 1911, the bank merged with Wachovia Loan and Trust Company,
"the largest trust company between Baltimore and New Orleans",[10] which had been
founded on June 15, 1893. Wachovia grew to become one of the largest banks in
the Southeast partly on the strength of its accounts from the R.J. Reynolds Tobacco
Company, which was also headquartered in Winston-Salem.[11] On December 12, 1986,
Wachovia purchased First Atlanta. Founded as Atlanta National Bank on September
14, 1865, and later renamed to First National Bank of Atlanta, this institution
was the oldest national bank in Atlanta. This purchase made Wachovia one of the
few companies with dual headquarters: one in Winston-Salem and one in Atlanta.
In 1991, Wachovia entered the South Carolina market by acquiring South Carolina
National Corporation,[12] founded as the Bank of Charleston in 1834. In 1998,
Wachovia acquired two Virginia-based banks, Jefferson National Bank and Central
Fidelity Bank. In 1997, Wachovia acquired both 1st United Bancorp and American
Bankshares Inc, giving its first entry into Florida. In 2000, Wachovia made its
final purchase, which was Republic Security Bank.'
- 'July Monarchy
The July Monarchy (French: Monarchie de Juillet) was a liberal constitutional
monarchy in France under Louis Philippe I, starting with the July Revolution of
1830 and ending with the Revolution of 1848. It marks the end of the Bourbon Restoration
(1814–1830). It began with the overthrow of the conservative government of Charles
X, the last king of the House of Bourbon.'
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- pearson_cosine
- spearman_cosine
model-index:
- name: SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct
results:
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: sts dev 896
type: sts-dev-896
metrics:
- type: pearson_cosine
value: 0.45729692013517886
name: Pearson Cosine
- type: spearman_cosine
value: 0.49645340246652353
name: Spearman Cosine
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: sts dev 768
type: sts-dev-768
metrics:
- type: pearson_cosine
value: 0.4455125981991164
name: Pearson Cosine
- type: spearman_cosine
value: 0.4896539219726307
name: Spearman Cosine
---
# SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct). It maps sentences & paragraphs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) <!-- at revision 7ae557604adf67be50417f59c2c2f167def9a775 -->
- **Maximum Sequence Length:** 1024 tokens
- **Output Dimensionality:** 896 dimensions
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: Qwen2Model
(1): Pooling({'word_embedding_dimension': 896, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("AlexWortega/qwen1k")
# Run inference
sentences = [
'When did the July Monarchy end?',
'July Monarchy\nThe July Monarchy (French: Monarchie de Juillet) was a liberal constitutional monarchy in France under Louis Philippe I, starting with the July Revolution of 1830 and ending with the Revolution of 1848. It marks the end of the Bourbon Restoration (1814–1830). It began with the overthrow of the conservative government of Charles X, the last king of the House of Bourbon.',
'July Monarchy\nDespite the return of the House of Bourbon to power, France was much changed from the era of the ancien régime. The egalitarianism and liberalism of the revolutionaries remained an important force and the autocracy and hierarchy of the earlier era could not be fully restored. Economic changes, which had been underway long before the revolution, had progressed further during the years of turmoil and were firmly entrenched by 1815. These changes had seen power shift from the noble landowners to the urban merchants. The administrative reforms of Napoleon, such as the Napoleonic Code and efficient bureaucracy, also remained in place. These changes produced a unified central government that was fiscally sound and had much control over all areas of French life, a sharp difference from the complicated mix of feudal and absolutist traditions and institutions of pre-Revolutionary Bourbons.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
## Evaluation
### Metrics
#### Semantic Similarity
* Datasets: `sts-dev-896` and `sts-dev-768`
* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
| Metric | sts-dev-896 | sts-dev-768 |
|:--------------------|:------------|:------------|
| pearson_cosine | 0.4573 | 0.4455 |
| **spearman_cosine** | **0.4965** | **0.4897** |
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Dataset
#### Unnamed Dataset
* Size: 2,859,594 training samples
* Columns: <code>query</code>, <code>response</code>, and <code>negative</code>
* Approximate statistics based on the first 1000 samples:
| | query | response | negative |
|:--------|:---------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string | string |
| details | <ul><li>min: 4 tokens</li><li>mean: 8.76 tokens</li><li>max: 26 tokens</li></ul> | <ul><li>min: 23 tokens</li><li>mean: 141.88 tokens</li><li>max: 532 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 134.02 tokens</li><li>max: 472 tokens</li></ul> |
* Samples:
| query | response | negative |
|:--------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>Was there a year 0?</code> | <code>Year zero<br>Year zero does not exist in the anno Domini system usually used to number years in the Gregorian calendar and in its predecessor, the Julian calendar. In this system, the year 1 BC is followed by AD 1. However, there is a year zero in astronomical year numbering (where it coincides with the Julian year 1 BC) and in ISO 8601:2004 (where it coincides with the Gregorian year 1 BC) as well as in all Buddhist and Hindu calendars.</code> | <code>504<br>Year 504 (DIV) was a leap year starting on Thursday (link will display the full calendar) of the Julian calendar. At the time, it was known as the Year of the Consulship of Nicomachus without colleague (or, less frequently, year 1257 "Ab urbe condita"). The denomination 504 for this year has been used since the early medieval period, when the Anno Domini calendar era became the prevalent method in Europe for naming years.</code> |
| <code>When is the dialectical method used?</code> | <code>Dialectic<br>Dialectic or dialectics (Greek: διαλεκτική, dialektikḗ; related to dialogue), also known as the dialectical method, is at base a discourse between two or more people holding different points of view about a subject but wishing to establish the truth through reasoned arguments. Dialectic resembles debate, but the concept excludes subjective elements such as emotional appeal and the modern pejorative sense of rhetoric.[1][2] Dialectic may be contrasted with the didactic method, wherein one side of the conversation teaches the other. Dialectic is alternatively known as minor logic, as opposed to major logic or critique.</code> | <code>Derek Bentley case<br>Another factor in the posthumous defence was that a "confession" recorded by Bentley, which was claimed by the prosecution to be a "verbatim record of dictated monologue", was shown by forensic linguistics methods to have been largely edited by policemen. Linguist Malcolm Coulthard showed that certain patterns, such as the frequency of the word "then" and the grammatical use of "then" after the grammatical subject ("I then" rather than "then I"), were not consistent with Bentley's use of language (his idiolect), as evidenced in court testimony. These patterns fit better the recorded testimony of the policemen involved. This is one of the earliest uses of forensic linguistics on record.</code> |
| <code>What do Grasshoppers eat?</code> | <code>Grasshopper<br>Grasshoppers are plant-eaters, with a few species at times becoming serious pests of cereals, vegetables and pasture, especially when they swarm in their millions as locusts and destroy crops over wide areas. They protect themselves from predators by camouflage; when detected, many species attempt to startle the predator with a brilliantly-coloured wing-flash while jumping and (if adult) launching themselves into the air, usually flying for only a short distance. Other species such as the rainbow grasshopper have warning coloration which deters predators. Grasshoppers are affected by parasites and various diseases, and many predatory creatures feed on both nymphs and adults. The eggs are the subject of attack by parasitoids and predators.</code> | <code>Groundhog<br>Very often the dens of groundhogs provide homes for other animals including skunks, red foxes, and cottontail rabbits. The fox and skunk feed upon field mice, grasshoppers, beetles and other creatures that destroy farm crops. In aiding these animals, the groundhog indirectly helps the farmer. In addition to providing homes for itself and other animals, the groundhog aids in soil improvement by bringing subsoil to the surface. The groundhog is also a valuable game animal and is considered a difficult sport when hunted in a fair manner. In some parts of Appalachia, they are eaten.</code> |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
```json
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
896,
768
],
"matryoshka_weights": [
1,
1
],
"n_dims_per_step": -1
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 12
- `per_device_eval_batch_size`: 12
- `gradient_accumulation_steps`: 4
- `num_train_epochs`: 1
- `warmup_ratio`: 0.3
- `bf16`: True
- `batch_sampler`: no_duplicates
#### All Hyperparameters
<details><summary>Click to expand</summary>
- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 12
- `per_device_eval_batch_size`: 12
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 4
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 5e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 1
- `max_steps`: -1
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.3
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`:
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional
</details>
### Training Logs
| Epoch | Step | Training Loss | sts-dev-896_spearman_cosine | sts-dev-768_spearman_cosine |
|:------:|:----:|:-------------:|:---------------------------:|:---------------------------:|
| 0.0002 | 10 | 4.4351 | - | - |
| 0.0003 | 20 | 4.6508 | - | - |
| 0.0005 | 30 | 4.7455 | - | - |
| 0.0007 | 40 | 4.5427 | - | - |
| 0.0008 | 50 | 4.3982 | - | - |
| 0.0010 | 60 | 4.3755 | - | - |
| 0.0012 | 70 | 4.4105 | - | - |
| 0.0013 | 80 | 5.2227 | - | - |
| 0.0015 | 90 | 5.8062 | - | - |
| 0.0017 | 100 | 5.7645 | - | - |
| 0.0018 | 110 | 5.9261 | - | - |
| 0.0020 | 120 | 5.8301 | - | - |
| 0.0022 | 130 | 5.7602 | - | - |
| 0.0023 | 140 | 5.9392 | - | - |
| 0.0025 | 150 | 5.7523 | - | - |
| 0.0027 | 160 | 5.8585 | - | - |
| 0.0029 | 170 | 5.7916 | - | - |
| 0.0030 | 180 | 5.8157 | - | - |
| 0.0032 | 190 | 5.7102 | - | - |
| 0.0034 | 200 | 5.5844 | - | - |
| 0.0035 | 210 | 5.5463 | - | - |
| 0.0037 | 220 | 5.5823 | - | - |
| 0.0039 | 230 | 5.5514 | - | - |
| 0.0040 | 240 | 5.5646 | - | - |
| 0.0042 | 250 | 5.5783 | - | - |
| 0.0044 | 260 | 5.5344 | - | - |
| 0.0045 | 270 | 5.523 | - | - |
| 0.0047 | 280 | 5.4969 | - | - |
| 0.0049 | 290 | 5.5407 | - | - |
| 0.0050 | 300 | 5.6171 | - | - |
| 0.0052 | 310 | 5.5581 | - | - |
| 0.0054 | 320 | 5.8903 | - | - |
| 0.0055 | 330 | 5.8675 | - | - |
| 0.0057 | 340 | 5.745 | - | - |
| 0.0059 | 350 | 5.6041 | - | - |
| 0.0060 | 360 | 5.5476 | - | - |
| 0.0062 | 370 | 5.3964 | - | - |
| 0.0064 | 380 | 5.3564 | - | - |
| 0.0065 | 390 | 5.3054 | - | - |
| 0.0067 | 400 | 5.2779 | - | - |
| 0.0069 | 410 | 5.206 | - | - |
| 0.0070 | 420 | 5.2168 | - | - |
| 0.0072 | 430 | 5.1645 | - | - |
| 0.0074 | 440 | 5.1797 | - | - |
| 0.0076 | 450 | 5.2526 | - | - |
| 0.0077 | 460 | 5.1768 | - | - |
| 0.0079 | 470 | 5.3519 | - | - |
| 0.0081 | 480 | 5.2982 | - | - |
| 0.0082 | 490 | 5.3229 | - | - |
| 0.0084 | 500 | 5.3758 | - | - |
| 0.0086 | 510 | 5.2478 | - | - |
| 0.0087 | 520 | 5.1799 | - | - |
| 0.0089 | 530 | 5.1088 | - | - |
| 0.0091 | 540 | 4.977 | - | - |
| 0.0092 | 550 | 4.9108 | - | - |
| 0.0094 | 560 | 4.811 | - | - |
| 0.0096 | 570 | 4.7203 | - | - |
| 0.0097 | 580 | 4.6499 | - | - |
| 0.0099 | 590 | 4.4548 | - | - |
| 0.0101 | 600 | 4.2891 | - | - |
| 0.0102 | 610 | 4.1881 | - | - |
| 0.0104 | 620 | 4.6 | - | - |
| 0.0106 | 630 | 4.5365 | - | - |
| 0.0107 | 640 | 4.3086 | - | - |
| 0.0109 | 650 | 4.0452 | - | - |
| 0.0111 | 660 | 3.9041 | - | - |
| 0.0112 | 670 | 4.3938 | - | - |
| 0.0114 | 680 | 4.3198 | - | - |
| 0.0116 | 690 | 4.1294 | - | - |
| 0.0117 | 700 | 4.077 | - | - |
| 0.0119 | 710 | 3.9174 | - | - |
| 0.0121 | 720 | 4.1629 | - | - |
| 0.0123 | 730 | 3.9611 | - | - |
| 0.0124 | 740 | 3.7768 | - | - |
| 0.0126 | 750 | 3.5842 | - | - |
| 0.0128 | 760 | 3.1196 | - | - |
| 0.0129 | 770 | 3.6288 | - | - |
| 0.0131 | 780 | 3.273 | - | - |
| 0.0133 | 790 | 2.7889 | - | - |
| 0.0134 | 800 | 2.5096 | - | - |
| 0.0136 | 810 | 1.8878 | - | - |
| 0.0138 | 820 | 2.3423 | - | - |
| 0.0139 | 830 | 1.7687 | - | - |
| 0.0141 | 840 | 2.0781 | - | - |
| 0.0143 | 850 | 2.4598 | - | - |
| 0.0144 | 860 | 1.7667 | - | - |
| 0.0146 | 870 | 2.6247 | - | - |
| 0.0148 | 880 | 1.916 | - | - |
| 0.0149 | 890 | 2.0817 | - | - |
| 0.0151 | 900 | 2.3679 | - | - |
| 0.0153 | 910 | 1.418 | - | - |
| 0.0154 | 920 | 2.7353 | - | - |
| 0.0156 | 930 | 1.992 | - | - |
| 0.0158 | 940 | 1.4564 | - | - |
| 0.0159 | 950 | 1.4154 | - | - |
| 0.0161 | 960 | 0.9499 | - | - |
| 0.0163 | 970 | 1.6304 | - | - |
| 0.0164 | 980 | 0.9264 | - | - |
| 0.0166 | 990 | 1.3278 | - | - |
| 0.0168 | 1000 | 1.686 | 0.4965 | 0.4897 |
### Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.3.0
- Transformers: 4.46.2
- PyTorch: 2.1.0+cu118
- Accelerate: 1.1.1
- Datasets: 3.1.0
- Tokenizers: 0.20.3
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--> |