mihalca/bert_model_ro_fake_news

Model Card for BERT

This model classifies Romanian fake news in ["fake_news", "misinformation", "propaganda", "real_news", "satire"]

Model Details

Model Description

This model is a BERT-based model fine-tuned for the task of detecting fake news in Romanian text. It is designed to classify text into one of several predefined labels indicating the likelihood of the text being fake news.

• Developed by: Bogdan Mihalca
• Model type: BERT
• License: MIT
• Finetuned from model: bert-base-multilingual-cased

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

This model can be directly used to classify Romanian text into five categories: fake news, misinformation, propaganda, real news, and satire. It can be handy for media monitoring platforms, news verifiers, and research on misinformation.

Downstream Use

Potential applications include integration into larger systems for real-time news validation or being part of a research toolkit for studying misinformation.

Out-of-Scope Use

This model is not intended for classifying non-Romanian text, and its accuracy may diminish significantly if used on text outside of the Romanian language. It is also not suitable for detailed sentiment analysis or non-news related content.

Bias, Risks, and Limitations

This model may exhibit bias due to the nature of the training data, which could lead to overfitting on certain types of news or propaganda specific to Romanian contexts. It might not generalize well to new, unseen forms of misinformation or satire.

Recommendations

Users should evaluate the model's performance on their specific use case before deployment, particularly if applied in contexts different from those in which the training data was collected.

How to Get Started with the Model

To get started with this model, load it using HuggingFace's Transformers library and ensure you have the appropriate tokenizer . Example code for loading and using the model is provided in the Python script.

Training Details

Training Data

The model was trained on a combined dataset of Romanian fake news, including data scraped from Veridica.ro and the Fakerom dataset. The dataset includes categories such as fake news, misinformation, propaganda, real news, and satire.

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: The model was trained using fp32 precision.
• Number of epochs: 3
• Batch size: 16
• Warmup steps: 500
• Weight decay: 0.01
• Learning rate scheduler: Warmup with cosine decay.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was tested on a split of the combined dataset, with 20% of the data used for evaluation.

Factors

The evaluation was performed across all five classes of fake news categories, with results disaggregated by each category.

Metrics

The primary evaluation metrics were accuracy, precision, recall, F1 score, and log loss. ROC AUC scores were also computed for each class.

Results

Results

Accuracy: 93.51%
Precision: 93.80%
Recall: 93.51%
F1 Score: 93.56%
Log Loss: 0.225
ROC AUC Per Class:
    Fake News: 98.50%
    Misinformation: 98.89%
    Propaganda: 98.98%
    Real News: 99.55%
    Satire: 99.99%

Summary

The model demonstrates strong performance across all classes, with particularly high ROC AUC scores indicating good separability between classes.