|
|
--- |
|
|
license: cc-by-sa-4.0 |
|
|
datasets: |
|
|
- SuccubusBot/incoherent-text-dataset |
|
|
language: |
|
|
- en |
|
|
- es |
|
|
- fr |
|
|
- de |
|
|
- zh |
|
|
- ja |
|
|
- ru |
|
|
- ar |
|
|
- hi |
|
|
metrics: |
|
|
- accuracy |
|
|
base_model: |
|
|
- distilbert/distilbert-base-multilingual-cased |
|
|
pipeline_tag: text-classification |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# DistilBERT Incoherence Classifier (Multilingual) |
|
|
|
|
|
This is a fine-tuned DistilBERT-multilingual model for classifying text based on its coherence. It can identify various types of incoherence. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model:** DistilBERT (distilbert-base-multilingual-cased) |
|
|
- **Task:** Text Classification (Coherence Detection) |
|
|
- **Fine-tuning:** The model was fine-tuned using a synthetically generated dataset that features various types of incoherence |
|
|
|
|
|
## Training Metrics |
|
|
|
|
|
| Epoch | Training Loss | Validation Loss | Accuracy | Precision | Recall | F1 | |
|
|
| :---- | :------------ | :------------ | :-------- | :-------- | :-------- | :------- | |
|
|
| 1 | 0.343600 | 0.303963 | 0.880312 | 0.882746 | 0.880312 | 0.879637 | |
|
|
| 2 | 0.245200 | 0.286482 | 0.900850 | 0.901156 | 0.900850 | 0.899612 | |
|
|
| 3 | 0.149700 | 0.313061 | 0.906161 | 0.906049 | 0.906161 | 0.905103 | |
|
|
|
|
|
## Evaluation Metrics |
|
|
|
|
|
The following metrics were measured on the test set: |
|
|
|
|
|
| Metric | Value | |
|
|
| :---------- | :------- | |
|
|
| Loss | 0.316272 | |
|
|
| Accuracy | 0.903329 | |
|
|
| Precision | 0.903704 | |
|
|
| Recall | 0.903329 | |
|
|
| F1-Score | 0.902359 | |
|
|
|
|
|
## Classification Report: |
|
|
|
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
coherent 0.86 0.93 0.90 2051 |
|
|
grammatical_errors 0.88 0.76 0.81 599 |
|
|
random_bytes 1.00 1.00 1.00 599 |
|
|
random_tokens 1.00 1.00 1.00 600 |
|
|
random_words 0.95 0.93 0.94 600 |
|
|
run_on 0.85 0.79 0.82 600 |
|
|
word_soup 0.89 0.83 0.86 599 |
|
|
|
|
|
accuracy 0.90 5648 |
|
|
macro avg 0.92 0.89 0.90 5648 |
|
|
weighted avg 0.90 0.90 0.90 5648 |
|
|
``` |
|
|
|
|
|
## Confusion Matrix |
|
|
|
|
|
 |
|
|
|
|
|
The confusion matrix above shows the performance of the model on each class. |
|
|
|
|
|
## Usage |
|
|
|
|
|
This model can be used for text classification tasks, specifically for detecting and categorizing different types of text incoherence. You can use the `inference_example` function provided in the notebook to test your own text. |
|
|
|
|
|
```py |
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("SuccubusBot/distilbert-multilingual-incoherence-classifier") |
|
|
model = AutoModelForSequenceClassification.from_pretrained("SuccubusBot/distilbert-multilingual-incoherence-classifier") |
|
|
|
|
|
classifier = pipeline("text-classification", model=model, tokenizer=tokenizer) |
|
|
|
|
|
|
|
|
while True: |
|
|
text = input("Enter text (or type 'exit' to quit): ") |
|
|
if text.lower() == "exit": |
|
|
break |
|
|
|
|
|
# Example usage |
|
|
results = classifier(text) |
|
|
|
|
|
# Print the results with confidence scores for all labels |
|
|
for result in results: |
|
|
print(f"Label: {result['label']}, Confidence: {result['score']}") |
|
|
``` |
|
|
|
|
|
## Limitations |
|
|
|
|
|
The model has been trained on a generated dataset, so care must be taken in evaluating it in the real world. More data may need to be collected before evaluating this model in a real-world setting. |
