metadata

library_name: setfit
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
base_model: sentence-transformers/paraphrase-mpnet-base-v2
metrics:
  - accuracy
widget:
  - text: >-
      I recently ordered the Bella Silver Pendant, but I haven't received any
      update about the shipment. Can you provide me with the current status of
      my order?
  - text: >-
      What is the metal purity of the Eternal Swirl Rose Gold Hoop Earring, and
      does it come with a certificate of authenticity?
  - text: >-
      Can you suggest some minimalist necklaces from your 'Best Sellers -
      Minimalist' range?
  - text: >-
      I recently ordered the Pearly Round Earring but haven't received any
      shipping updates. Can you please provide me with the tracking information?
  - text: what are the colors available in air jordan 4
pipeline_tag: text-classification
inference: true
model-index:
  - name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.8762886597938144
            name: Accuracy

SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 6 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
product policy	'Are there any exceptions to the return policy for items that were purchased with a special offer promotion?' 'What is your policy on returning sneakers with added paint or dye?' 'Do you offer exchanges for items that were purchased with a special event celebration?'
order tracking	"I recently placed an order for the Regalia Gold Ring but I haven't received any confirmation or tracking details. Could you please update me on the status of my order?" 'What is the process for rerouting a shipment to a different address?' "I recently ordered a Three Crystal Proposal Ring but haven't received any shipping updates yet. Could you please provide me with the current status of my order?"
complaints	"I recently bought the Golden Love Affair Pendant, but it seems to have tarnished very quickly. I'm not satisfied with the quality. What can you do about this?" "I recently purchased the Three Crystal Proposal Ring, but I'm disappointed to find that one of the crystals is loose. Can you assist me with this issue?" 'I received my Kali- Handcrafted Earring today, but I found that one earring is slightly different from the other in design. Can you help me with this issue?'
product faq	'What is the material used for making the All the Stars Pendant Set, and does it come with matching earrings?' 'What is the Bold and Beautiful Link Ring made of, and could you provide information on sizing and care instructions?' 'What is the material used for making the Sheer Heart Ring, and is it available in different sizes?'
product discoveribility	"I'm interested in necklaces that have an adjustable length. What options do you have?" 'Do you have any charm bracelets available at your store?' 'Could you suggest some pendants that would go well with traditional attire?'
product discoverability	'Types of bakery boxes available' 'adidas sneakers under 25k' 'show me 100 cookie boxes under $50'

Evaluation

Metrics

Label	Accuracy
all	0.8763

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("what are the colors available in air jordan 4")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	4	16.2235	36

Label	Training Sample Count
complaints	30
order tracking	30
product discoverability	30
product discoveribility	30
product faq	20
product policy	30

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0007	1	0.1501	-
0.0333	50	0.1076	-
0.0667	100	0.01	-
0.1	150	0.0023	-
0.1333	200	0.0008	-
0.1667	250	0.0007	-
0.2	300	0.0005	-
0.2333	350	0.0005	-
0.2667	400	0.0003	-
0.3	450	0.0005	-
0.3333	500	0.0003	-
0.3667	550	0.0003	-
0.4	600	0.0002	-
0.4333	650	0.0002	-
0.4667	700	0.0003	-
0.5	750	0.0002	-
0.5333	800	0.0002	-
0.5667	850	0.0002	-
0.6	900	0.0002	-
0.6333	950	0.0002	-
0.6667	1000	0.0001	-
0.7	1050	0.0001	-
0.7333	1100	0.0002	-
0.7667	1150	0.0001	-
0.8	1200	0.0001	-
0.8333	1250	0.0001	-
0.8667	1300	0.0002	-
0.9	1350	0.0001	-
0.9333	1400	0.0002	-
0.9667	1450	0.0001	-
1.0	1500	0.0002	-

Framework Versions

Python: 3.9.16
SetFit: 1.0.3
Sentence Transformers: 2.7.0
Transformers: 4.40.2
PyTorch: 2.3.0
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}