Hugging Face's logo

adriansanz
/
ST-tramits-SQV-004-5ep

Sentence Similarity
sentence-transformers
Safetensors
xlm-roberta
feature-extraction
Generated from Trainer
dataset_size:5175
loss:MatryoshkaLoss
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-m3
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 1024 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • json

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("adriansanz/sqv-v4")
# Run inference
sentences = [
    "La persona consumidora presenti la reclamació davant de l'entitat acreditada en un termini superior a un any des de la data en què va presentar la reclamació a l'empresa.",
    "Quin és el paper de l'entitat acreditada en la tramitació d'una reclamació?",
    "Quin és el resultat de la modificació substancial de la llicència d'obres en relació a les autoritzacions administratives?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.0574
cosine_accuracy@3 0.1791
cosine_accuracy@5 0.2539
cosine_accuracy@10 0.4243
cosine_precision@1 0.0574
cosine_precision@3 0.0597
cosine_precision@5 0.0508
cosine_precision@10 0.0424
cosine_recall@1 0.0574
cosine_recall@3 0.1791
cosine_recall@5 0.2539
cosine_recall@10 0.4243
cosine_ndcg@10 0.2113
cosine_mrr@10 0.1472
cosine_map@100 0.1688

Information Retrieval

Metric Value
cosine_accuracy@1 0.0591
cosine_accuracy@3 0.167
cosine_accuracy@5 0.2417
cosine_accuracy@10 0.4174
cosine_precision@1 0.0591
cosine_precision@3 0.0557
cosine_precision@5 0.0483
cosine_precision@10 0.0417
cosine_recall@1 0.0591
cosine_recall@3 0.167
cosine_recall@5 0.2417
cosine_recall@10 0.4174
cosine_ndcg@10 0.2074
cosine_mrr@10 0.1442
cosine_map@100 0.1663

Information Retrieval

Metric Value
cosine_accuracy@1 0.0643
cosine_accuracy@3 0.1617
cosine_accuracy@5 0.2417
cosine_accuracy@10 0.4052
cosine_precision@1 0.0643
cosine_precision@3 0.0539
cosine_precision@5 0.0483
cosine_precision@10 0.0405
cosine_recall@1 0.0643
cosine_recall@3 0.1617
cosine_recall@5 0.2417
cosine_recall@10 0.4052
cosine_ndcg@10 0.2063
cosine_mrr@10 0.1464
cosine_map@100 0.17

Information Retrieval

Metric Value
cosine_accuracy@1 0.0557
cosine_accuracy@3 0.1565
cosine_accuracy@5 0.2278
cosine_accuracy@10 0.3913
cosine_precision@1 0.0557
cosine_precision@3 0.0522
cosine_precision@5 0.0456
cosine_precision@10 0.0391
cosine_recall@1 0.0557
cosine_recall@3 0.1565
cosine_recall@5 0.2278
cosine_recall@10 0.3913
cosine_ndcg@10 0.1965
cosine_mrr@10 0.1377
cosine_map@100 0.1621

Information Retrieval

Metric Value
cosine_accuracy@1 0.0678
cosine_accuracy@3 0.1704
cosine_accuracy@5 0.2574
cosine_accuracy@10 0.4174
cosine_precision@1 0.0678
cosine_precision@3 0.0568
cosine_precision@5 0.0515
cosine_precision@10 0.0417
cosine_recall@1 0.0678
cosine_recall@3 0.1704
cosine_recall@5 0.2574
cosine_recall@10 0.4174
cosine_ndcg@10 0.2142
cosine_mrr@10 0.1528
cosine_map@100 0.1754

Information Retrieval

Metric Value
cosine_accuracy@1 0.0522
cosine_accuracy@3 0.1461
cosine_accuracy@5 0.233
cosine_accuracy@10 0.4035
cosine_precision@1 0.0522
cosine_precision@3 0.0487
cosine_precision@5 0.0466
cosine_precision@10 0.0403
cosine_recall@1 0.0522
cosine_recall@3 0.1461
cosine_recall@5 0.233
cosine_recall@10 0.4035
cosine_ndcg@10 0.1961
cosine_mrr@10 0.1339
cosine_map@100 0.1564

Training Details

Training Dataset

json

  • Dataset: json
  • Size: 5,175 training samples
  • Columns: positive and anchor
  • Approximate statistics based on the first 1000 samples:
    positive anchor
    type string string
    details
    • min: 5 tokens
    • mean: 43.23 tokens
    • max: 117 tokens
    • min: 10 tokens
    • mean: 20.25 tokens
    • max: 46 tokens
  • Samples:
    positive anchor
    Aquest tràmit us permet consultar informació de les anotacions d'entrada i sortida que hi consten al registre de l'Ajuntament de Sant Quirze del Vallès. Quin és el format de les dades de sortida del tràmit?
    Tràmit a través del qual la persona interessada posa en coneixement de l’Ajuntament la voluntat de: ... Renunciar a una llicència prèviament atorgada. Quin és el resultat de la renúncia a una llicència urbanística prèviament atorgada?
    D’acord amb el plànol d'ubicació de parades: Mercat de diumenges a Les Fonts Quin és el plànol d'ubicació de parades del mercat de diumenges a Les Fonts?
  • Loss: MatryoshkaLoss with these parameters:
    {
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        1024,
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 5
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.2
  • bf16: True
  • tf32: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 5
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.2
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: True
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_1024_cosine_map@100 dim_128_cosine_map@100 dim_256_cosine_map@100 dim_512_cosine_map@100 dim_64_cosine_map@100 dim_768_cosine_map@100
0.4938 10 3.936 - - - - - -
0.9877 20 2.7857 0.1550 0.1522 0.1557 0.1507 0.1344 0.1503
1.4815 30 1.4901 - - - - - -
1.9753 40 1.3464 0.1580 0.1654 0.1695 0.1580 0.1510 0.1624
2.4691 50 0.7755 - - - - - -
2.9630 60 0.8553 0.1608 0.1705 0.1647 0.1661 0.1564 0.1689
3.4568 70 0.5817 - - - - - -
3.9506 80 0.6587 - - - - - -
4.0 81 - 0.1672 0.1657 0.1620 0.1689 0.1556 0.1669
4.4444 90 0.4847 - - - - - -
4.9383 100 0.6024 0.1688 0.1754 0.1621 0.17 0.1564 0.1663
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.1.1
  • Transformers: 4.44.2
  • PyTorch: 2.4.1+cu121
  • Accelerate: 0.35.0.dev0
  • Datasets: 3.0.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss 

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
4
Safetensors
Model size
568M params
Tensor type
F32
·
Inference Examples
Sentence Similarity
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for adriansanz/ST-tramits-SQV-004-5ep

Base model

BAAI/bge-m3
Finetuned
(185)
this model

Evaluation results

View on Papers With Code