cngcv/bge-base-financial-matryoshka

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cngcv/bge-base-financial-matryoshka")
# Run inference
sentences = [
    'After text generation, the process involves providing test data to NT Q, which then undergoes article correction, including dealing with fragmented articles and errors.',
    'What is the process for providing test data to NT Q after text generation?',
    'What is the significance of the dates in the text?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7755
cosine_accuracy@3	0.8776
cosine_accuracy@5	0.9592
cosine_accuracy@10	0.9796
cosine_precision@1	0.7755
cosine_precision@3	0.2925
cosine_precision@5	0.1918
cosine_precision@10	0.098
cosine_recall@1	0.7755
cosine_recall@3	0.8776
cosine_recall@5	0.9592
cosine_recall@10	0.9796
cosine_ndcg@10	0.8776
cosine_mrr@10	0.8448
cosine_map@100	0.8464

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7959
cosine_accuracy@3	0.898
cosine_accuracy@5	0.9592
cosine_accuracy@10	0.9796
cosine_precision@1	0.7959
cosine_precision@3	0.2993
cosine_precision@5	0.1918
cosine_precision@10	0.098
cosine_recall@1	0.7959
cosine_recall@3	0.898
cosine_recall@5	0.9592
cosine_recall@10	0.9796
cosine_ndcg@10	0.8846
cosine_mrr@10	0.8539
cosine_map@100	0.8551

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6939
cosine_accuracy@3	0.9184
cosine_accuracy@5	0.9592
cosine_accuracy@10	0.9592
cosine_precision@1	0.6939
cosine_precision@3	0.3061
cosine_precision@5	0.1918
cosine_precision@10	0.0959
cosine_recall@1	0.6939
cosine_recall@3	0.9184
cosine_recall@5	0.9592
cosine_recall@10	0.9592
cosine_ndcg@10	0.8397
cosine_mrr@10	0.7993
cosine_map@100	0.8017

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6939
cosine_accuracy@3	0.9184
cosine_accuracy@5	0.9184
cosine_accuracy@10	0.9184
cosine_precision@1	0.6939
cosine_precision@3	0.3061
cosine_precision@5	0.1837
cosine_precision@10	0.0918
cosine_recall@1	0.6939
cosine_recall@3	0.9184
cosine_recall@5	0.9184
cosine_recall@10	0.9184
cosine_ndcg@10	0.8168
cosine_mrr@10	0.7823
cosine_map@100	0.7866

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5918
cosine_accuracy@3	0.7959
cosine_accuracy@5	0.8163
cosine_accuracy@10	0.9184
cosine_precision@1	0.5918
cosine_precision@3	0.2653
cosine_precision@5	0.1633
cosine_precision@10	0.0918
cosine_recall@1	0.5918
cosine_recall@3	0.7959
cosine_recall@5	0.8163
cosine_recall@10	0.9184
cosine_ndcg@10	0.7471
cosine_mrr@10	0.6929
cosine_map@100	0.6978

Training Details

Training Dataset

Unnamed Dataset

• Size: 196 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 15 tokens mean: 46.58 tokens max: 118 tokens	min: 10 tokens mean: 17.25 tokens max: 43 tokens

• Samples:

  positive	anchor
  The document lists several tasks with their statuses, such as "Done", "In progress", and "To be done". These statuses indicate the current progress of each task within the project. For example, "Set up environment" and "Set up development environment" are marked as "Done", suggesting these tasks have been completed, while "Build translation data set" is marked as "In progress", indicating it is currently being worked on.	What is the status of the project tasks mentioned in the document?
  The 'Web Application Construction' task is mentioned to be completed by NT Q, with a duration from July 17, 2023, to July 28, 2023, and is marked as 'Done' with a completion of 10 tasks.	What is the scope of the 'Web Application Construction' task?
  "RE F" could potentially stand for "Reference File" or "Record File," indicating that this text might be part of a larger dataset or document used for reference or record-keeping purposes.	What is the significance of the "RE F" at the beginning of the text?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
1.0	1	0.6908	0.7097	0.8111	0.6240	0.8011
2.0	2	0.7292	0.7692	0.8177	0.6634	0.8162
3.0	3	0.7555	0.8014	0.8541	0.6992	0.8451
4.0	4	0.7866	0.8017	0.8551	0.6978	0.8464

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.42.3
• PyTorch: 2.1.2
• Accelerate: 0.32.1
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}