DeBERTa-v3-xsmall-Zyda-2-quality

Model Overview

This model is a fine-tuned version of agentlans/deberta-v3-xsmall-zyda-2 designed for text quality assessment. It achieves the following results on the evaluation set:

• Loss: 0.3165
• MSE: 0.3165

Dataset Information

The model was trained on the Text Quality Meta-Analysis Dataset, which is a comprehensive collection of sentences with associated quality metrics derived from multiple sources and methods. This dataset combines text from various sources with quality scores from different models to create a thorough assessment of sentence quality.

In this context, "quality" refers to legible English sentences that are not spam and contain useful information. It does not necessarily indicate grammatical or factual correctness.

Model Description

The model is based on the DeBERTa-v3-xsmall architecture and has been fine-tuned for sequence classification tasks, specifically for assessing the quality of text inputs.

Intended Uses & Limitations

This model is intended for evaluating the quality of text inputs. It can be used for various applications such as content moderation, spam detection, or assessing the credibility of information.

Usage Example

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load model and tokenizer
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model_name = "agentlans/deberta-v3-xsmall-zyda-2-quality"
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=1).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Function to perform inference
def predict_score(text):
    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(device)
    with torch.no_grad():
        logits = model(**inputs).logits
    return logits.item()

# Example usage
input_text = "This product is excellent and works perfectly!"
predicted_score = predict_score(input_text)
print(f"Predicted score: {predicted_score}")

Sample Predictions

Text	Quality Score
Discover the secret to eternal youth with our revolutionary skincare product!	-1.74
Act now! Limited time offer on miracle weight loss pills!	-1.50
Congratulations! You've won a $1,000 gift card! Click here to claim your prize!	-0.86
Get rich quick with our foolproof investment strategy - no experience needed!	-0.77
Your computer is infected! Click here for a free scan and fix your issues now!	-0.29
Unlock the secrets of the universe with our exclusive online astronomy course!	0.14
Earn money from home by participating in online surveys - sign up today!	0.23
The Eiffel Tower can be 15 cm taller during the summer due to thermal expansion.	0.75
Did you know? The average person spends 6 years of their life dreaming.	1.60
Did you know that honey never spoils? Archaeologists have found pots of honey in ancient Egyptian tombs that are over 3,000 years old and still edible.	2.27

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

• Learning rate: 5e-05
• Train batch size: 64
• Eval batch size: 8
• Seed: 42
• Optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
• Learning rate scheduler: Linear
• Number of epochs: 3.0

Training Results

Training Loss	Epoch	Step	Validation Loss	MSE
0.3506	1.0	12649	0.3512	0.3512
0.2800	2.0	25298	0.3187	0.3187
0.2398	3.0	37947	0.3165	0.3165

Framework Versions

• Transformers: 4.46.3
• PyTorch: 2.5.1+cu124
• Datasets: 3.1.0
• Tokenizers: 0.20.3