metadata

license: mit
datasets:
  - CodeHima/TOS_DatasetV3
language:
  - en
metrics:
  - accuracy
  - precision
base_model: FacebookAI/roberta-base
pipeline_tag: text-classification

TOSRoberta-base

Model Overview

Model Name: TOSRoberta-base
Model Type: Sequence Classification
Base Model: RoBERTa-base
Language: English
Task: Classification of unfairness levels in Terms of Service (ToS) documents

Model Card Version: 1.0
Author: CodeHima

Model Description

The TOSRoberta-base model is a fine-tuned version of RoBERTa-base for classifying clauses in Terms of Service (ToS) documents into three categories:

Clearly Fair
Potentially Unfair
Clearly Unfair

This model has been fine-tuned on a custom dataset labeled with the above categories to help identify unfair practices in ToS documents.

Intended Use

Primary Use Case

The primary use case of this model is to classify text from Terms of Service documents into different levels of fairness. It can be particularly useful for legal analysts, researchers, and consumer protection agencies to quickly identify potentially unfair clauses in ToS documents.

Limitations

Dataset Bias: The model has been trained on a specific dataset, which may introduce biases. It may not generalize well to all types of ToS documents.
Context Understanding: The model may struggle with clauses that require deep contextual or legal understanding.

Performance

Training Configuration

Batch Size: 32 (training), 16 (evaluation)
Learning Rate: 1e-5
Epochs: 10
Optimizer: AdamW
Scheduler: Linear with warmup
Training Framework: PyTorch using Hugging Face's transformers library
Mixed Precision Training: Enabled (fp16)
Resource: Trained on a single NVIDIA T4 GPU (15 GB VRAM)

Training Metrics

Epoch	Training Loss	Validation Loss	Accuracy	F1	Precision	Recall
1	0.668100	0.620207	0.740000	0.727	0.728	0.740
2	0.439800	0.463925	0.824762	0.821	0.826	0.825
3	0.373500	0.432604	0.831429	0.832	0.834	0.831
4	0.342800	0.402661	0.854286	0.854	0.853	0.854
5	0.283800	0.434868	0.829524	0.832	0.840	0.830
6	0.218000	0.437268	0.859048	0.859	0.859	0.859
7	0.266800	0.508120	0.820952	0.824	0.834	0.821
8	0.139600	0.486364	0.855238	0.856	0.856	0.855
9	0.085000	0.530111	0.844762	0.846	0.850	0.845
10	0.103600	0.528026	0.842857	0.844	0.847	0.843

Final Validation Accuracy: 85.90%
Final Test Accuracy: 85.65%

Evaluation Metrics

Accuracy: 85.65%
F1 Score: 85.60%
Precision: 85.61%
Recall: 85.65%

Dataset

The model was trained on the CodeHima/TOS_DatasetV3, which includes labeled clauses from ToS documents. The dataset is split into training, validation, and test sets to ensure reliable performance evaluation.

Dataset Labels:

clearly_fair
potentially_unfair
clearly_unfair

How to Use

Here’s how you can use the model with the Hugging Face transformers library:

from transformers import RobertaTokenizer, RobertaForSequenceClassification

# Load the model
model = RobertaForSequenceClassification.from_pretrained('CodeHima/TOSRoberta-base')
tokenizer = RobertaTokenizer.from_pretrained('CodeHima/TOSRoberta-base')

# Predict the unfairness level of a clause
text = "Insert clause text here."
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=128)
outputs = model(**inputs)
predicted_class = outputs.logits.argmax(-1).item()

# Map the predicted class to the corresponding label
label_mapping = {0: 'clearly_fair', 1: 'potentially_unfair', 2: 'clearly_unfair'}
predicted_label = label_mapping[predicted_class]
print(f"Predicted Label: {predicted_label}")

Ethical Considerations

Bias: The model's predictions may reflect biases present in the training data.
Fair Use: Ensure the model is used responsibly, especially in legal contexts where human oversight is critical.

Conclusion

The TOSRoberta-base model is a reliable tool for identifying unfair clauses in Terms of Service documents. While it performs well, it should be used in conjunction with expert analysis, particularly in legally sensitive contexts.

Model Repository: CodeHima/TOSRoberta-base