RishuD7/exigent-bge-base-financial-matryoshka

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("RishuD7/exigent-bge-base-financial-matryoshka")
# Run inference
sentences = [
    '\nThe management of each individual entity within a suppliers organization is responsible for\nimplementing the VAT Supplier Code of Conduct in their respective area of responsibility. They are\nobliged to take all appropriate action and provide the required structures and resources to ensure\nthat all employees in the entity are familiar with the VAT Supplier Code of Conduct and that its\nprinciples are fully implemented.\n.\n\nAll VAT suppliers are encouraged to direct any questions they might have with regard to the\ncontents, interpretation or implementation of the VAT Supplier Code of Conduct to the VAT Strategic\nProcurement function.\n.\n\n.\n \n.\n\nDocument created Release\nName Index Date\n.\n\n.\n \n.\n\nFile name\n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\nPMS Document BPO1FO30EA MEY A 18.11.2014\n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n  \n.\n\nWAT Strategic Procurement BP01FO30E\n.\n\nVakuumventile AG Supplier Code of Conduct Page 3 of 3\n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\nWe, the undersigned, hereby confirm and declare in the name and on behalf of our company that\n.\n\n1. we have received the VAT Supplier Code of Condex;\n.\n\n2. by signing this declaration, we accept and commit to complying with all rules and requirements as\nlaid out in the VAT Supplier Code of Conduct;\n.\n\n3. we accept that this declaration shall be exclusively governed by the material laws of Switzerland,\nexcluding the UN Law of Sales (CISG).\n.\n\nPlacelDate —-Singagore. / tone 2077\nCompany Kien Ann Engineering Pe ad\nStreet 3c 500 kovo Cirle\n.\n\nPost codelcity Singapore 627035\n.\n\nName of authorized signatory Jameson Low\n.\n\nL. Ze\nSignature << : Ly eA\n* fh\n20,\nXn _A\nCETES\n.\n\n1. Please sign one (1) original c Of this document.\n2. Please note that only duly authorized personnel of your company may sign this document.\n3. Please send the duly signed original copy by conventional mail to:\nVAT VAKUUMVENTILE AG, SEELISTRASSE 1, STRATEGISCHER EINKAUF, CH-9469 HAAG\n.\n\n.\n \n.\n\n.\n   \n.\n\n.\n \n.\n\nDocument created Release\n.\n\n.\n \n.\n\nFile name\nName Index Date\n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\n.\n \n.\n\nPMS Document BPO1FO30EA MEY A 18.11.2014\n\n \n\n \n\nAll business is conducted in compliance with governing national and international laws and\nregulations. As a matter of principle, we honor agreements and obligations we have entered into\nvoluntarily. All suppliers are obliged to carefully study the rules and regulations pertinent to their\narea of responsibility and ensure full compliance. In case of doubt or queries, they are obliged to\nseek additional information and guidance from the appropriate channels or persons in charge. VAT\nhas a zero tolerance policy with regard to violations of its Supplier Code of Conduct. Violations may\nlead to appropriate action being taken against the supplier.\n.\n\n2. Fair competition\n',
    'Governing Law',
    'Absolute Maximum Amount of Liability',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0073
cosine_accuracy@3	0.0218
cosine_accuracy@5	0.0303
cosine_accuracy@10	0.0617
cosine_precision@1	0.0073
cosine_precision@3	0.0073
cosine_precision@5	0.0061
cosine_precision@10	0.0062
cosine_recall@1	0.0073
cosine_recall@3	0.0218
cosine_recall@5	0.0303
cosine_recall@10	0.0617
cosine_ndcg@10	0.0289
cosine_mrr@10	0.0192
cosine_map@100	0.0297

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0061
cosine_accuracy@3	0.0218
cosine_accuracy@5	0.0315
cosine_accuracy@10	0.0617
cosine_precision@1	0.0061
cosine_precision@3	0.0073
cosine_precision@5	0.0063
cosine_precision@10	0.0062
cosine_recall@1	0.0061
cosine_recall@3	0.0218
cosine_recall@5	0.0315
cosine_recall@10	0.0617
cosine_ndcg@10	0.0283
cosine_mrr@10	0.0184
cosine_map@100	0.0293

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0073
cosine_accuracy@3	0.0182
cosine_accuracy@5	0.0278
cosine_accuracy@10	0.0581
cosine_precision@1	0.0073
cosine_precision@3	0.0061
cosine_precision@5	0.0056
cosine_precision@10	0.0058
cosine_recall@1	0.0073
cosine_recall@3	0.0182
cosine_recall@5	0.0278
cosine_recall@10	0.0581
cosine_ndcg@10	0.0268
cosine_mrr@10	0.0176
cosine_map@100	0.0294

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0036
cosine_accuracy@3	0.0169
cosine_accuracy@5	0.0278
cosine_accuracy@10	0.063
cosine_precision@1	0.0036
cosine_precision@3	0.0056
cosine_precision@5	0.0056
cosine_precision@10	0.0063
cosine_recall@1	0.0036
cosine_recall@3	0.0169
cosine_recall@5	0.0278
cosine_recall@10	0.063
cosine_ndcg@10	0.0271
cosine_mrr@10	0.0165
cosine_map@100	0.0285

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0048
cosine_accuracy@3	0.0254
cosine_accuracy@5	0.0375
cosine_accuracy@10	0.0605
cosine_precision@1	0.0048
cosine_precision@3	0.0085
cosine_precision@5	0.0075
cosine_precision@10	0.0061
cosine_recall@1	0.0048
cosine_recall@3	0.0254
cosine_recall@5	0.0375
cosine_recall@10	0.0605
cosine_ndcg@10	0.0285
cosine_mrr@10	0.0188
cosine_map@100	0.0302

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,305 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 123 tokens mean: 353.07 tokens max: 512 tokens	min: 3 tokens mean: 5.37 tokens max: 8 tokens

• Samples:

  positive	anchor
  In no event shall CBRE, Client, or their respective affiliates incur liability under this agreement or otherwise relating to the Services beyond the insurance proceeds available with respect to the particular matter under the Insurance Policies required to be carried by CBRE AND Client under Article 6 above including, if applicable, proceeds of self-insurance. Each party shall and shall cause its affiliates to look solely to such insurance proceeds (and any such proceeds paid through self-insurance) to satisfy its claims against the released parties and agrees that it shall have no right of recovery beyond such proceeds; provided, however, that if insurance proceeds under such policies are not paid because a party has failed to maintain such policies, comply with policy requirements or, in the case of self-insurance, unreasonably denied a claim, such party shall be liable for the amounts that otherwise would have been payable under such policies had such party maintained such policies, complied with the policy requirement or not unreasonably denied such claim, as the case may be.	Absolute Maximum Amount of Liability
  4. Rent. 4.01 From and after the Commencement Date, Tenant shall pay Landlord, without any setoff or deduction, unless expressly set forth in this Lease, all Base Rent and Additional Rent due for the Term (collectively referred to as "Rent"). "Additional Rent" means all sums (exclusive of Base Rent) that Tenant is required to pay Landlord under this Lease. Tenant shall pay and be liable for all rental, sales and use taxes (but excluding income taxes), if any, imposed upon or measured by Rent. Base Rent and recurring monthly charges of Additional Rent shall be due and payable in advance on the first day of each calendar month without notice or demand, provided that the installment of Base Rent attributable to the first (1st) full calendar month of the Term following the Abatement Period shall be due concurrently with the execution of this Lease by Tenant. All other items of Rent shall be due and payable on or before thirty (30) days after billing by Landlord. Rent shall be made payable to the entity, and sent to the address, that Landlord designates and shall be made by good and sufficient check or by other means acceptable to Landlord. Landlord may return to Tenant, at any time within fifteen (15) days after receiving same, any payment of Rent (a) made following any Default (irrespective of whether Landlord has commenced the exercise of any remedy), or (b) that is less than the amount due. Each such returned payment (whether made by returning Tenant's actual check, or by issuing a refund in the event Tenant's check was deposited) shall be conclusively presumed not to have been received or approved by Landlord. If Tenant does not pay any Rent when due hereunder, Tenant shall pay Landlord an administration fee in the amount of five percent (5%) of the past due amount. In addition, past due Rent shall accrue interest at a rate equal to the lesser of (i) twelve percent (12%) per annum or (ii) the maximum legal rate, and Tenant shall pay Landlord a fee for any checks returned by Tenant's bank for any reason. Notwithstanding the foregoing, no such late charge or of interest shall be imposed with respect to the first (1st) late payment in any calendar year, but not with respect to more than three (3) such late payments during the initial Term of this Lease.	Late Payment Charges
  Term This Agreement shall come into force and shall last unlimited from such date. Either Party may however terminate this Agreement at any time by giving upon thirty (30) days' written notice to the other Party. The Receiving Party's obligations contained in this Agreement to keep confidential and restrict use of the Disclosing Party's Confidential Information shall sur- vive for a period of five (5) years from the date of its termination for any reason whatsoever. lX. Contractual penalty For the purposes of this Non-Disclosure Agreement, " Confidential Information" includes all technical and/or commercial and/or financial information in the field designated in section 1., which a contracting Party (hereinafter referred to as the "EQ€i1gPedy") makes, or has made, accessible to the other contracting Party (hereinafter referred to as the ".&eiyi!g Partv") in oral, written, tangible or other form (e.9. disk, data carrier) directly or indirectly, in- cluding but not limited to, drawings, models, components, and other material. Confidential In- formation is to be identified as such. Orally communicated or visually, information having been designated as confidential at the time of disclosure will be confirmed as such in writing by the Disclosing Party within 30 (thirty) days from such disclosure being understood thatlhe ./A information will be considered Confidential Information during that period of 30 (thirty) days. /L t'-4 PF 0233 (September 2016) page 1 of 5 ä =. PFEIFFER F . F . VACUUM	Termination for Convenience

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
1.5385	10	7.981	-	-	-	-	-
3.0769	20	0.9258	-	-	-	-	-
4.6154	30	0.1708	0.0285	0.0294	0.0293	0.0302	0.0297

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.1.2+cu121
• Accelerate: 0.32.1
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}