SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: BAAI/bge-base-en-v1.5
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
1	'Reasoning:\nThe answer effectively addresses the question and offers practical advice supported by the document. It describes using principles like remembering that everyone gets embarrassed, attention shifting, self-help techniques such as muscle relaxation, and maintaining calm to mitigate the impact of a humiliating experience. These strategies are grounded in the document, which mentions using methods like attention shifting and relaxation techniques to cope with and move past the memory of a humiliating experience. Additionally, the answer is relevant, concise, and stays focused on the specific question without unnecessary digressions.\n\nFinal Evaluation:' "Reasoning:\nThe provided answer gives practical advice that is relevant to the question of surviving freshman year in high school. It discusses both social strategies, such as being outgoing and friendly, and academic strategies like having a solid study plan. This advice aligns with several points highlighted in the document, such as being friendly, making new friends, being organized, and taking good notes.\n\nHowever, the answer does not mention a few important aspects covered in the document, such as attending freshman orientation, understanding the school's layout, and participating in extracurricular activities. While the response is concise and mostly relevant, it could benefit from incorporating more of these specific recommendations from the document for a more comprehensive answer.\n\nFinal Evaluation: \nEvaluation:" 'Reasoning:\nThe answer correctly identifies scenarios where a cover letter is deemed necessary based on the provided document. It mentions specific instances such as when the job ad requires one, when requested by the employer or recruiter, when applying directly to someone known, when referred for the role, when familiar with the job and role, and when uncertain about available jobs in the target company. These points are well-supported by the document and effectively address the question, remaining relevant and concise.\n\nFinal Evaluation:'
0	"Reasoning:\nThe provided answer accurately describes the involvement of the ORGANIZATION_2 team in the farewell process when an employee's tenure concludes. The explanation includes:\n\n1. Context Grounding: The answer aligns with the documents, identifying the specific roles of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera.\n2. Relevance: It addresses the specific question about the extent of ORGANIZATION_2's participation in the farewell process.\n3. Conciseness: The answer is relatively concise and to the point, explaining the roles without excessive elaboration.\n4. Specificity: The answer correctly identifies the specific responsibilities mentioned, such as handling paperwork and aiding in tough conversations.\n5. Correctness: The answer summarizes the document efficiently without introducing inaccuracies.\n\nThe answer could, however, be slightly more concise and more directly focus on the varying extent of involvement based on the specific situation, but overall it is well done.\n\nFinal annotation: \nEvaluation:" 'Reasoning:\nThe answer provided aligns well with the information in the document. The document explains that the purpose of the pyramid is to "collect, analyze, and classify data and events in a meaningful way," and it clearly outlines the steps for understanding collected data, detected activities, and identifying malicious activities needing further action. The response effectively captures these points, offering a specific and concise explanation.\n\nEvaluation:' "Reasoning:\nThe given document provides specific instructions for what to do if the sensor fails to start and no error message is returned. It mentions that the likely cause is the absence of a network adapter, which prevents the sensor from generating a PylumID. The steps to resolve this issue are clearly outlined: adding or enabling another network adapter with a physical address, even if it is disconnected, and then restarting the sensor.\n\nThe provided answer, however, incorrectly states that the information does not cover the query and directs the user to other sources, which is factually incorrect given the document's detailed solution. This shows a lack of context grounding, relevance, and conciseness. Therefore, the answer is inaccurate.\n\nEvaluation:"

Label

Examples

'Reasoning:\nThe answer effectively addresses the question and offers practical advice supported by the document. It describes using principles like remembering that everyone gets embarrassed, attention shifting, self-help techniques such as muscle relaxation, and maintaining calm to mitigate the impact of a humiliating experience. These strategies are grounded in the document, which mentions using methods like attention shifting and relaxation techniques to cope with and move past the memory of a humiliating experience. Additionally, the answer is relevant, concise, and stays focused on the specific question without unnecessary digressions.\n\nFinal Evaluation:'
"Reasoning:\nThe provided answer gives practical advice that is relevant to the question of surviving freshman year in high school. It discusses both social strategies, such as being outgoing and friendly, and academic strategies like having a solid study plan. This advice aligns with several points highlighted in the document, such as being friendly, making new friends, being organized, and taking good notes.\n\nHowever, the answer does not mention a few important aspects covered in the document, such as attending freshman orientation, understanding the school's layout, and participating in extracurricular activities. While the response is concise and mostly relevant, it could benefit from incorporating more of these specific recommendations from the document for a more comprehensive answer.\n\nFinal Evaluation: \nEvaluation:"
'Reasoning:\nThe answer correctly identifies scenarios where a cover letter is deemed necessary based on the provided document. It mentions specific instances such as when the job ad requires one, when requested by the employer or recruiter, when applying directly to someone known, when referred for the role, when familiar with the job and role, and when uncertain about available jobs in the target company. These points are well-supported by the document and effectively address the question, remaining relevant and concise.\n\nFinal Evaluation:'

"Reasoning:\nThe provided answer accurately describes the involvement of the ORGANIZATION_2 team in the farewell process when an employee's tenure concludes. The explanation includes:\n\n1. Context Grounding: The answer aligns with the documents, identifying the specific roles of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera.\n2. Relevance: It addresses the specific question about the extent of ORGANIZATION_2's participation in the farewell process.\n3. Conciseness: The answer is relatively concise and to the point, explaining the roles without excessive elaboration.\n4. Specificity: The answer correctly identifies the specific responsibilities mentioned, such as handling paperwork and aiding in tough conversations.\n5. Correctness: The answer summarizes the document efficiently without introducing inaccuracies.\n\nThe answer could, however, be slightly more concise and more directly focus on the varying extent of involvement based on the specific situation, but overall it is well done.\n\nFinal annotation: \nEvaluation:"
'Reasoning:\nThe answer provided aligns well with the information in the document. The document explains that the purpose of the pyramid is to "collect, analyze, and classify data and events in a meaningful way," and it clearly outlines the steps for understanding collected data, detected activities, and identifying malicious activities needing further action. The response effectively captures these points, offering a specific and concise explanation.\n\nEvaluation:'
"Reasoning:\nThe given document provides specific instructions for what to do if the sensor fails to start and no error message is returned. It mentions that the likely cause is the absence of a network adapter, which prevents the sensor from generating a PylumID. The steps to resolve this issue are clearly outlined: adding or enabling another network adapter with a physical address, even if it is disconnected, and then restarting the sensor.\n\nThe provided answer, however, incorrectly states that the information does not cover the query and directs the user to other sources, which is factually incorrect given the document's detailed solution. This shows a lack of context grounding, relevance, and conciseness. Therefore, the answer is inaccurate.\n\nEvaluation:"

Evaluation

Metrics

Label	Accuracy
all	0.6667

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_wix_qa_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_fi")
# Run inference
preds = model("Reasoning:
The answer provided is detailed and outlines the steps to block off time slots in the Wix Booking Calendar. However, the question specifically asks about removing the time from showing on the booking button, not about blocking off time slots. The instructions given do not address the question directly. The document also does not mention any method for removing the time from the booking button, so the answer lacks context grounding and relevance to both the question and the document.

Evaluation:")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	33	91.9544	289

Label	Training Sample Count
0	335
1	345

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (2, 2)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0006	1	0.2534	-
0.0294	50	0.2576	-
0.0588	100	0.256	-
0.0882	150	0.2569	-
0.1176	200	0.2488	-
0.1471	250	0.2107	-
0.1765	300	0.1824	-
0.2059	350	0.1642	-
0.2353	400	0.1422	-
0.2647	450	0.1194	-
0.2941	500	0.073	-
0.3235	550	0.0671	-
0.3529	600	0.0236	-
0.3824	650	0.0198	-
0.4118	700	0.0199	-
0.4412	750	0.0094	-
0.4706	800	0.0101	-
0.5	850	0.0118	-
0.5294	900	0.0055	-
0.5588	950	0.0091	-
0.5882	1000	0.0109	-
0.6176	1050	0.0063	-
0.6471	1100	0.0016	-
0.6765	1150	0.0026	-
0.7059	1200	0.0051	-
0.7353	1250	0.0057	-
0.7647	1300	0.0018	-
0.7941	1350	0.0014	-
0.8235	1400	0.0021	-
0.8529	1450	0.0013	-
0.8824	1500	0.0012	-
0.9118	1550	0.001	-
0.9412	1600	0.0011	-
0.9706	1650	0.001	-
1.0	1700	0.0011	-
1.0294	1750	0.0011	-
1.0588	1800	0.0011	-
1.0882	1850	0.001	-
1.1176	1900	0.001	-
1.1471	1950	0.001	-
1.1765	2000	0.001	-
1.2059	2050	0.001	-
1.2353	2100	0.001	-
1.2647	2150	0.001	-
1.2941	2200	0.001	-
1.3235	2250	0.001	-
1.3529	2300	0.0009	-
1.3824	2350	0.0009	-
1.4118	2400	0.0009	-
1.4412	2450	0.0017	-
1.4706	2500	0.0009	-
1.5	2550	0.0009	-
1.5294	2600	0.0009	-
1.5588	2650	0.0009	-
1.5882	2700	0.0009	-
1.6176	2750	0.0009	-
1.6471	2800	0.0009	-
1.6765	2850	0.0009	-
1.7059	2900	0.0008	-
1.7353	2950	0.0008	-
1.7647	3000	0.0008	-
1.7941	3050	0.0008	-
1.8235	3100	0.0008	-
1.8529	3150	0.0008	-
1.8824	3200	0.0009	-
1.9118	3250	0.0008	-
1.9412	3300	0.0008	-
1.9706	3350	0.0009	-
2.0	3400	0.0008	-

Framework Versions

Python: 3.10.14
SetFit: 1.1.0
Sentence Transformers: 3.1.1
Transformers: 4.44.0
PyTorch: 2.4.0+cu121
Datasets: 3.0.0
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

Netta1994
/

setfit_baai_wix_qa_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_fi