dinho1597/phi-2-telecom-ft

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5 on the telecom-qa-multiple_choice dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • telecom-qa-multiple_choice

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What is one way to enhance the robustness of terahertz in real-time communication?',
    'Enhancing beam tracking, resource allocation, and user association can improve the robustness of terahertz in real-time communication.',
    'Handover refers to the process of changing the serving cell of a UE in RRC_CONNECTED.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: telecom-ir-eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9482
cosine_accuracy@3	0.991
cosine_accuracy@5	0.9924
cosine_accuracy@10	0.9952
cosine_precision@1	0.9482
cosine_recall@1	0.9482
cosine_ndcg@10	0.9753
cosine_mrr@10	0.9686
cosine_map@100	0.9688

Training Details

Training Dataset

telecom-qa-multiple_choice

• Dataset: telecom-qa-multiple_choice at 73aebbb
• Size: 6,552 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 4 tokens mean: 18.54 tokens max: 48 tokens	min: 10 tokens mean: 29.33 tokens max: 100 tokens

• Samples:

  anchor	positive
  What are the two mechanisms in a CDMA (Code Division Multiple Access) system that reduce the chances of significant interference?	In a CDMA system, power control and soft handoff mechanisms are used to reduce the chances of significant interference. Power control ensures that there is no significant intra-cell interference, and soft handoff selects the base station with the best reception to control the user's power, reducing the chance of out-of-cell interference.
  What type of traffic is LPWAN (Low-Power Wide Area Network) suitable for?	LPWANs are suitable for sporadic and intermittent transmissions of very small packets.
  What is the definition of an Authenticator in IEEE Std 802.11-2020?	An Authenticator is an entity at one end of a point-to-point LAN segment that facilitates authentication of the entity attached to the other end of that link.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

telecom-qa-multiple_choice

• Dataset: telecom-qa-multiple_choice at 73aebbb
• Size: 6,552 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 4 tokens mean: 18.83 tokens max: 45 tokens	min: 10 tokens mean: 28.7 tokens max: 99 tokens

• Samples:

  anchor	positive
  How is the continuous-time impulse response of the end-to-end system computed?	The continuous-time impulse response of the end-to-end system is computed as the convolution of the impulse responses of the RIS paths and the propagation channels.
  What is lattice staggering in a multicarrier scheme?	Lattice staggering is a methodology that generates inherent orthogonality between the points in the lattice for the real domain by using different prototype filters for the real and imaginary parts of the scheme.
  What are the benefits of using Mobile Edge Computing (MEC) for computation-intensive applications?	MEC enables applications to be split into small tasks with some of the tasks performed at the local or regional clouds, reducing delay and backhaul usage.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• weight_decay: 0.01
• num_train_epochs: 5
• lr_scheduler_type: cosine_with_restarts
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	telecom-ir-eval_cosine_ndcg@10
0.3659	15	0.5651	0.1124	0.9593
0.7317	30	0.144	0.0674	0.9713
1.0976	45	0.0783	0.0605	0.9730
1.4634	60	0.0633	0.0607	0.9753
1.8293	75	0.0469	0.0570	0.9749
2.1951	90	0.0297	0.0568	0.9755
2.5610	105	0.0287	0.0583	0.9749
2.9268	120	0.0266	0.0594	0.9747
3.2927	135	0.0196	0.0604	0.9738
3.6585	150	0.0191	0.0609	0.9742
4.0244	165	0.0167	0.0608	0.9749
4.3902	180	0.0165	0.0611	0.9746
4.7561	195	0.016	0.0611	0.9746
5.0	205	-	-	0.9753

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}