metadata

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:56041
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: BAAI/bge-base-en-v1.5
widget:
  - source_sentence: >-
      What is the significance of the tables 6.1.6.2.5-1 and 6.1.6.2.6-1 in the
      context of the Namf_Communication Service API?
    sentences:
      - >-
        The 'notifId' attribute in the PolicyDataSubscription type serves as a
        Notification Correlation ID assigned by the NF service consumer. It is
        included when the 'ConditionalSubscriptionwithPartialNotification' or
        the 'ConditionalSubscriptionWithExcludeNotification' feature is
        supported. This ID is used to correlate notifications with the specific
        subscription request, ensuring that the NF service consumer can track
        and manage notifications effectively.
      - >-
        The 'sessRuleReports' attribute in the 'ErrorReport' type is
        specifically used to report failures related to session rules, whereas
        the 'ruleReports' attribute reports failures related to PCC rules.
        'sessRuleReports' contains an array of 'SessionRuleReport' objects,
        which provide details about the session rule failures. Like
        'ruleReports', it is optional and can have one or more entries
        (cardinality 1..N).
      - >-
        Tables 6.1.6.2.5-1 and 6.1.6.2.6-1 are significant in the
        Namf_Communication Service API as they provide the definitions for the
        types 'AssignEbiData' and 'AssignedEbiData', respectively. These tables
        outline the structure, attributes, and possibly the constraints or rules
        associated with these data types, which are essential for understanding
        and implementing the API's functionality related to EBI assignment and
        management.
  - source_sentence: >-
      What document defines the basic principles for online charging, and where
      is this information referenced?
    sentences:
      - >-
        The UDM (Unified Data Management) returns the Ranging and Sidelink
        Positioning Subscription Data for the UE (User Equipment) identified by
        the supi (Subscription Permanent Identifier). This data is retrieved
        using the GET method, which supports the URI query parameters outlined
        in table 6.1.3.37.3.1-1.
      - >-
        The Nsmf_PDUSession_SMContextStatusNotify service operation is used by
        the SMF (Session Management Function) to notify its consumers about the
        status of an SM (Session Management) context related to a PDU (Packet
        Data Unit) Session. In the context of I-SMF (Intermediate SMF) context
        transfer, this service operation is used to indicate the transfer of the
        SM context to a new I-SMF or SMF set. It also allows the SMF to update
        the SMF-derived CN (Core Network) assisted RAN (Radio Access Network)
        parameters tuning in the AMF (Access and Mobility Management Function).
        Additionally, it can report DDN (Downlink Data Notification) failures
        and provide target DNAI (Data Network Access Identifier) information for
        the current or next PDU session.
      - >-
        The basic principles for online charging are defined in TS 32.240 [1].
        This information is referenced in section 5.2.1 of the document, which
        is part of the '5.2 Online charging scenario' chapter.
  - source_sentence: >-
      What are the possible values for the 'ReportingLevel' enumeration, and
      what do they indicate?
    sentences:
      - >-
        If protected User Plane (UP) messages reach the SN before the SN has
        received the SN Counter value in the SN Reconfiguration Complete
        message, the SN chooses the first unused KSN key of the UE to establish
        the security association. This ensures that communication can proceed
        securely even if the SN Counter value has not yet been received. Once
        the SN Counter value is received, the SN verifies it to ensure there is
        no KSN mismatch.
      - >-
        The 'ReportingLevel' enumeration has three possible values:
        'SER_ID_LEVEL', 'RAT_GR_LEVEL', and 'SPON_CON_LEVEL'. 'SER_ID_LEVEL'
        indicates that usage should be reported at the service ID and rating
        group combination level. 'RAT_GR_LEVEL' indicates that usage should be
        reported at the rating group level. 'SPON_CON_LEVEL' indicates that
        usage should be reported at the sponsor identity and rating group
        combination level. These levels help in categorizing and reporting usage
        data based on different granularities.
      - >-
        Structured data types in the Nudr_GroupIDmap Service API are more
        complex than simple data types. While simple data types represent single
        values like integers or strings, structured data types are composed of
        multiple simple data types or other structured data types, forming a
        more complex data structure. For example, a structured data type might
        represent a user profile containing fields for name, age, and address,
        each of which could be a simple data type. This allows for the
        representation of more intricate and hierarchical data within the API.
  - source_sentence: >-
      What is the purpose of the Intermediate Spending Limit Report Request
      procedure described in the document?
    sentences:
      - >-
        The Resource URI variables defined in table 6.1.3.8.2-1 for the
        'sm-data' resource serve to dynamically construct the URI based on
        specific parameters. These variables include {apiRoot}, <apiVersion>,
        and {supi}. The {apiRoot} variable specifies the base URL of the API,
        <apiVersion> indicates the version of the API to be used, and {supi}
        represents the Subscription Permanent Identifier, which is used to
        uniquely identify the subscriber. These variables ensure that the URI is
        correctly formatted and points to the appropriate resource for the given
        subscriber and API version.
      - >-
        The purpose of the Intermediate Spending Limit Report Request procedure
        is to allow the PCF (Policy Control Function) to request the status of
        additional policy counters available at the CHF (Charging Function) or
        to remove the request for the status of policy counters. The PCF can
        modify the list of subscribed policy counters based on its policy
        decisions, and the CHF responds by providing the policy counter status,
        optionally including pending statuses and their activation times, for
        the requested policy counters.
      - >-
        When ABC online charging is employed, the TDF uses Debit / Reserve Units
        Request[Initial], update, or termination to convey charging information
        related to the detected application traffic. The OCS responds with Debit
        / Reserve Units Response, which includes quotas for rating groups or
        instructions on handling the application traffic (e.g., terminate,
        continue, reroute). The TDF must request a quota before service
        delivery. If only certain quotas are authorized by the OCS (e.g., due to
        insufficient credit), the rating groups without authorized quotas are
        handled according to the received Result Code value. The quota
        supervision mechanism is further described in TS 32.299 [50].
  - source_sentence: >-
      What types of data structures are supported by the GET request body on the
      resource described in table 5.2.11.3.4-2, and how do they influence the
      request?
    sentences:
      - >-
        In Direct Communication mode, the NF Service consumer can subscribe to
        status change notifications of NF instances from the NRF. If the NF
        Service consumer is notified by the NRF or detects by itself (e.g.,
        through a lack of response to a request) that the NF producer instance
        is no longer available, it selects another available NF producer
        instance within the same NF Set. In Indirect Communication mode, the SCP
        or NF Service consumer may also subscribe to status change notifications
        from the NRF and select another NF producer instance within the same NF
        Set if the original instance serving the UE becomes unavailable. The
        specific implementation details of how the SCP detects the
        unavailability of an NF producer instance are left to the
        implementation.
      - >-
        The data structures supported by the GET request body on the resource
        are detailed in table 5.2.11.3.4-2. These structures define the format
        and content of the data that can be sent in the request body. They might
        include fields such as 'filterCriteria', 'sortOrder', or 'pagination',
        which influence how the server processes the request and returns the
        appropriate data.
      - >-
        The specific triggers on the Ro interface that can lead to the
        termination of the IMS service include: 1) Reception of an unsuccessful
        Operation Result different from DIAMETER_CREDIT_CONTROL_NOT_APPLICABLE
        in the Debit/Reserve Units Response message. 2) Reception of an
        unsuccessful Result Code different from
        DIAMETER_CREDIT_CONTROL_NOT_APPLICABLE within the multiple units
        operation in the Debit/Reserve Units Response message when only one
        instance of the multiple units operation field is used. 3) Execution of
        the termination action procedure as defined in TS 32.299 when only one
        instance of the Multiple Unit Operation field is used. 4) Execution of
        the failure handling procedures when the Failure Action is set to
        'Terminate' or 'Retry & Terminate'. 5) Reception in the IMS-GWF of an
        Abort-Session-Request message from OCS.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: BGE_base_3gpp-qa-v2_Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.8347103013864849
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9628129405256866
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9806391748898128
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9927196159954319
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8347103013864849
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.32093764684189546
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1961278349779626
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09927196159954321
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8347103013864849
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9628129405256866
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9806391748898128
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9927196159954319
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9235193716202091
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9002603606826465
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9006611894428589
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.8341214467978801
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9630270694669973
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.980835459752681
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9925947074463339
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8341214467978801
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.32100902315566576
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19616709195053625
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09925947074463341
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8341214467978801
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9630270694669973
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.980835459752681
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9925947074463339
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9232781516394674
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8999735171216805
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9003855301087177
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.8326047001302618
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9624382148783927
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9801930729287486
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9922913581128102
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8326047001302618
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3208127382927975
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19603861458574973
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09922913581128105
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8326047001302618
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9624382148783927
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9801930729287486
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9922913581128102
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9223721780180253
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.898869719250338
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.8993021227310489
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.8294462982459271
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9610642208383148
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9796399064970289
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.991720347602648
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8294462982459271
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3203547402794382
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19592798129940583
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09917203476026483
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8294462982459271
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9610642208383148
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9796399064970289
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.991720347602648
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9204835891487085
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8965493659262566
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.897020544909686
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.8210595813779198
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9574775610713585
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9771595795935119
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9906497028960939
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8210595813779198
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3191591870237861
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19543191591870243
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09906497028960942
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8210595813779198
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9574775610713585
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9771595795935119
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9906497028960939
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9158816707476002
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8908051588080549
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.8913320555914594
            name: Cosine Map@100

BGE_base_3gpp-qa-v2_Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: BAAI/bge-base-en-v1.5
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- json
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("iris49/3gpp-embedding-model-v0")
# Run inference
sentences = [
    'What types of data structures are supported by the GET request body on the resource described in table 5.2.11.3.4-2, and how do they influence the request?',
    "The data structures supported by the GET request body on the resource are detailed in table 5.2.11.3.4-2. These structures define the format and content of the data that can be sent in the request body. They might include fields such as 'filterCriteria', 'sortOrder', or 'pagination', which influence how the server processes the request and returns the appropriate data.",
    "The specific triggers on the Ro interface that can lead to the termination of the IMS service include: 1) Reception of an unsuccessful Operation Result different from DIAMETER_CREDIT_CONTROL_NOT_APPLICABLE in the Debit/Reserve Units Response message. 2) Reception of an unsuccessful Result Code different from DIAMETER_CREDIT_CONTROL_NOT_APPLICABLE within the multiple units operation in the Debit/Reserve Units Response message when only one instance of the multiple units operation field is used. 3) Execution of the termination action procedure as defined in TS 32.299 when only one instance of the Multiple Unit Operation field is used. 4) Execution of the failure handling procedures when the Failure Action is set to 'Terminate' or 'Retry & Terminate'. 5) Reception in the IMS-GWF of an Abort-Session-Request message from OCS.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.8347	0.8341	0.8326	0.8294	0.8211
cosine_accuracy@3	0.9628	0.963	0.9624	0.9611	0.9575
cosine_accuracy@5	0.9806	0.9808	0.9802	0.9796	0.9772
cosine_accuracy@10	0.9927	0.9926	0.9923	0.9917	0.9906
cosine_precision@1	0.8347	0.8341	0.8326	0.8294	0.8211
cosine_precision@3	0.3209	0.321	0.3208	0.3204	0.3192
cosine_precision@5	0.1961	0.1962	0.196	0.1959	0.1954
cosine_precision@10	0.0993	0.0993	0.0992	0.0992	0.0991
cosine_recall@1	0.8347	0.8341	0.8326	0.8294	0.8211
cosine_recall@3	0.9628	0.963	0.9624	0.9611	0.9575
cosine_recall@5	0.9806	0.9808	0.9802	0.9796	0.9772
cosine_recall@10	0.9927	0.9926	0.9923	0.9917	0.9906
cosine_ndcg@10	0.9235	0.9233	0.9224	0.9205	0.9159
cosine_mrr@10	0.9003	0.9	0.8989	0.8965	0.8908
cosine_map@100	0.9007	0.9004	0.8993	0.897	0.8913

Training Details

Training Dataset

json

Dataset: json
Size: 56,041 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 15 tokens
mean: 30.56 tokens
max: 66 tokens

min: 42 tokens
mean: 109.65 tokens
max: 298 tokens

	anchor	positive
type	string	string
details	min: 15 tokens mean: 30.56 tokens max: 66 tokens	min: 42 tokens mean: 109.65 tokens max: 298 tokens

Samples:

anchor	positive
`What does the 'dataStatProps' attribute represent in the 'AnalyticsMetadataInfo' type, and what is its data type?`	`The 'dataStatProps' attribute in the 'AnalyticsMetadataInfo' type represents a list of dataset statistical properties of the data used to generate the analytics. It is defined as an optional attribute with a data type of 'array(DatasetStatisticalProperty)' and a cardinality of 1..N, meaning it can contain one or more elements.`
`Why is it important to have standardized methods for resource management in the Nudm_SubscriberDataManagement Service API?`	`Standardized methods for resource management in the Nudm_SubscriberDataManagement Service API are important because they ensure uniformity, predictability, and compatibility across different implementations and systems. This standardization facilitates seamless integration, reduces errors, and enhances the efficiency of managing subscriber data, which is critical for maintaining reliable communication services.`
`What is the purpose of the Nsmf_PDUSession_SMContextStatusNotify service operation in the context of I-SMF context transfer?`	The Nsmf_PDUSession_SMContextStatusNotify service operation is used by the SMF (Session Management Function) to notify its consumers about the status of an SM (Session Management) context related to a PDU (Packet Data Unit) Session. In the context of I-SMF (Intermediate SMF) context transfer, this service operation is used to indicate the transfer of the SM context to a new I-SMF or SMF set. It also allows the SMF to update the SMF-derived CN (Core Network) assisted RAN (Radio Access Network) parameters tuning in the AMF (Access and Mobility Management Function). Additionally, it can report DDN (Downlink Data Notification) failures and provide target DNAI (Data Network Access Identifier) information for the current or next PDU session.

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 32
per_device_eval_batch_size: 16
gradient_accumulation_steps: 16
learning_rate: 2e-05
num_train_epochs: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
fp16: True
load_best_model_at_end: True
optim: adamw_torch_fused
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 16
eval_accumulation_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.0913	10	1.4273	-	-	-	-	-
0.1826	20	0.5399	-	-	-	-	-
0.2740	30	0.1252	-	-	-	-	-
0.3653	40	0.0625	-	-	-	-	-
0.4566	50	0.0507	-	-	-	-	-
0.5479	60	0.0366	-	-	-	-	-
0.6393	70	0.029	-	-	-	-	-
0.7306	80	0.0239	-	-	-	-	-
0.8219	90	0.0252	-	-	-	-	-
0.9132	100	0.0237	-	-	-	-	-
0.9954	109	-	0.9199	0.9195	0.9180	0.9150	0.9081
1.0046	110	0.026	-	-	-	-	-
1.0959	120	0.017	-	-	-	-	-
1.1872	130	0.02	-	-	-	-	-
1.2785	140	0.0125	-	-	-	-	-
1.3699	150	0.0134	-	-	-	-	-
1.4612	160	0.0128	-	-	-	-	-
1.5525	170	0.0123	-	-	-	-	-
1.6438	180	0.0097	-	-	-	-	-
1.7352	190	0.0101	-	-	-	-	-
1.8265	200	0.0124	-	-	-	-	-
1.9178	210	0.0116	-	-	-	-	-
2.0	219	-	0.9220	0.9216	0.9206	0.9184	0.9130
2.0091	220	0.012	-	-	-	-	-
2.1005	230	0.0111	-	-	-	-	-
2.1918	240	0.0101	-	-	-	-	-
2.2831	250	0.0101	-	-	-	-	-
2.3744	260	0.009	-	-	-	-	-
2.4658	270	0.0103	-	-	-	-	-
2.5571	280	0.009	-	-	-	-	-
2.6484	290	0.0083	-	-	-	-	-
2.7397	300	0.0076	-	-	-	-	-
2.8311	310	0.0093	-	-	-	-	-
2.9224	320	0.0104	-	-	-	-	-
2.9954	328	-	0.9234	0.9230	0.9221	0.9201	0.9156
3.0137	330	0.0104	-	-	-	-	-
3.1050	340	0.0089	-	-	-	-	-
3.1963	350	0.0084	-	-	-	-	-
3.2877	360	0.0082	-	-	-	-	-
3.3790	370	0.0089	-	-	-	-	-
3.4703	380	0.0083	-	-	-	-	-
3.5616	390	0.0061	-	-	-	-	-
3.6530	400	0.0065	-	-	-	-	-
3.7443	410	0.0063	-	-	-	-	-
3.8356	420	0.0084	-	-	-	-	-
3.9269	430	0.0083	-	-	-	-	-
3.9817	436	-	0.9235	0.9233	0.9224	0.9205	0.9159

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.11.11
Sentence Transformers: 3.3.1
Transformers: 4.41.2
PyTorch: 2.1.2+cu121
Accelerate: 1.2.1
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}