MugheesAwan11/bge-base-citi-dataset-detailed-9k-1_5k-e1

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("MugheesAwan11/bge-base-citi-dataset-detailed-9k-1_5k-e1")
# Run inference
sentences = [
    ' Drive, Attn: Arbitration Opt Out, San Antonio, TX 78245. Your rejection notice must be mailed within 45 days of account opening. Your rejection notice must state that you reject the arbitration provision and include your name, address, account number and personal signature. No one else may sign the rejection notice. Your rejection notice will not apply to the arbitration provision governing any other account that you have or had with us. Rejection of this arbitration provision wont affect your other rights or responsibilities under this Agreement, including use of the account. 68 Appendix 1: Fee Schedule The following Checkbook Order Fee, Safe Deposit Fee, Fee Chart, and Wire Transfer Fee Chart are known as the Fee Schedule. Unless otherwise stated, all fees described in the Fee Schedule are charged to the account associated with the product or service. Checkbook Orders. Fees will be charged for standard and Non-Standard checkbook orders. Non-Standard Checkbook Orders include non-standard design, non-standard lettering',
    'What is the process for rejecting the arbitration provision and what information should be included in the rejection notice?',
    'What is the Annual Percentage Yield for the Citigold Private Client Pendant Exclusive 24K Gold Rabbit on the Moon or IL in the states of NY, CT, MD, VA, DC, CA, NV, NJ and select markets in FL?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.524
cosine_accuracy@3	0.718
cosine_accuracy@5	0.7827
cosine_accuracy@10	0.848
cosine_precision@1	0.524
cosine_precision@3	0.2393
cosine_precision@5	0.1565
cosine_precision@10	0.0848
cosine_recall@1	0.524
cosine_recall@3	0.718
cosine_recall@5	0.7827
cosine_recall@10	0.848
cosine_ndcg@10	0.6849
cosine_ndcg@100	0.7108
cosine_mrr@10	0.6327
cosine_map@100	0.6384

Training Details

Training Dataset

Unnamed Dataset

• Size: 9,000 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 152 tokens mean: 206.96 tokens max: 299 tokens	min: 9 tokens mean: 52.08 tokens max: 281 tokens

• Samples:

  positive	anchor
  adverse effect on the value of any Index Linked Product. 15 Citi Investment Strategies Citi Flexible Allocation 6 Excess Return Index Index General Conditions Section D: Definitions 16 Citi Investment Strategies Citi Flexible Allocation 6 Excess Return Index Index General Conditions Definitions References to the "applicable Index Methodology" are references to the Index Methodology relating to the relevant Index which together with these Index General Conditions completes the Index Conditions for such Index. References to a "Section" shall be references to a section of these Index General Conditions. References to a "Part" shall be references to a part of the applicable Index Methodology. "Adjustment Event" shall, in respect of a Constituent, have the meaning given to it in the Constituent Schedule applicable to such Constituent. "Affected Constituent" shall have the meaning given to it in Section B . "Affiliate" shall mean, in respect of a person, any entity controlled by such person, any entity which controls	What is the meaning of "applicable Index Methodology" in the Index General Conditions? What does "Section" refer to in the Index General Conditions? How is "Part" defined in the applicable Index Methodology? What is the definition of "Adjustment Event" in relation to a Constituent? How is an "Affected Constituent" defined in Section B? What is the definition of "Affiliate" in relation to a person?
  that the Depositary andor the Custodian may in the future identify from the balance of Shares on deposit in the DR program as belonging to the holders of DRs in the DR Balance on the basis of a full or partial reconciliation of the Share-to-DR imbalance created by the Automatic Conversions and Forced Conversions. The is no guarantee that any such reconciliation will be successful or that any such Shares will be available any time in the near or distant future, and as a result there is no indication that the DRs credited to the DR balance have, or will in the future have, any value. The creation of the DR Balance and any credit of DRs in the DR balance to a Beneficial Owner is purely an accommodation to the Beneficial Owner and does not represent any undertaking of any value or service. Neither the Depositary nor the Custodian undertake in any way to take any action on behalf of the holders of DRs credited to the DR balance to retrieve any Shares from third parties	What is the likelihood of the Depositary and/or the Custodian successfully reconciling the Share-to-DR imbalance in the DR program and identifying Shares belonging to DR holders in the DR Balance? Is there any guarantee of the availability or future value of these Shares? Are the DRs credited to the DR balance of any value? Does the creation of the DR Balance and credit of DRs to Beneficial Owners represent any commitment of value or service? Do the Depositary and the Custodian have any responsibility to retrieve Shares from third parties on behalf of DR holders credited to the DR balance?
  of ways to save money while shopping online. Thats why a browser extension like Citi Shop can be a great addition to your online shopping experience. Lets look at how the Citi Shop extension works. Contact helpdeskciti.com What is the Citi Shop Browser Extension? Citi Shop is a free desktop browser extension you can download through the Chrome, Edge or Safari app stores. Once installed, enroll your eligible Citi credit card and let the Citi Shop program automatically search for available offers at more than 5,000 online merchants across the internet. How to Install the Citi Shop Browser Extension First, download the Citi Shop browser extension from the Chrome, Edge or Safari app store for your desktop browser. Once downloaded, you will be required to enroll your eligible Citi credit card. Contact helpdeskciti.com How to Use the Citi Shop Browser Extension Simply shop at your favorite online merchants. The Citi Shop program automatically searches behind the scenes to find money-saving offers percent	What is the Citi Shop Browser Extension and how does it work? How can I install the Citi Shop Browser Extension for my desktop browser? How do I use the Citi Shop Browser Extension to save money while shopping online? Who can I contact for help with the Citi Shop Browser Extension?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 2
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_map@100
0.0355	10	0.7377	-
0.0709	20	0.5614	-
0.1064	30	0.4571	-
0.1418	40	0.2944	-
0.1773	50	0.2584	-
0.2128	60	0.1855	-
0.2482	70	0.1699	-
0.2837	80	0.2212	-
0.3191	90	0.1827	-
0.3546	100	0.1801	-
0.3901	110	0.1836	-
0.4255	120	0.1112	-
0.4610	130	0.1638	-
0.4965	140	0.1355	-
0.5319	150	0.0873	-
0.5674	160	0.1852	-
0.6028	170	0.1424	-
0.6383	180	0.1467	-
0.6738	190	0.1879	-
0.7092	200	0.1382	-
0.7447	210	0.1358	-
0.7801	220	0.0906	-
0.8156	230	0.1173	-
0.8511	240	0.1196	-
0.8865	250	0.1251	-
0.9220	260	0.0922	-
0.9574	270	0.1344	-
0.9929	280	0.0751	-
1.0	282	-	0.6395
1.0284	290	0.166	-
1.0638	300	0.0842	-
1.0993	310	0.098	-
1.1348	320	0.0674	-
1.1702	330	0.071	-
1.2057	340	0.0527	-
1.2411	350	0.0401	-
1.2766	360	0.0575	-
1.3121	370	0.0418	-
1.3475	380	0.054	-
1.3830	390	0.0495	-
1.4184	400	0.0355	-
1.4539	410	0.0449	-
1.4894	420	0.0509	-
1.5248	430	0.0196	-
1.5603	440	0.0634	-
1.5957	450	0.0522	-
1.6312	460	0.0477	-
1.6667	470	0.0583	-
1.7021	480	0.0584	-
1.7376	490	0.0553	-
1.7730	500	0.0358	-
1.8085	510	0.0253	-
1.8440	520	0.0541	-
1.8794	530	0.0488	-
1.9149	540	0.0528	-
1.9504	550	0.0474	-
1.9858	560	0.0311	-
2.0	564	-	0.6384

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.1.2+cu121
• Accelerate: 0.32.1
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}