This model classifies Romanian fake news in ["fake_news", "misinformation", "propaganda", "real_news", "satire"]
Model Details
Model Description
This model is a BERT-based model fine-tuned for the task of detecting fake news in Romanian text. It is designed to classify text into one of several predefined labels indicating the likelihood of the text being fake news.
Developed by: Bogdan Mihalca
Model type: BERT
License: MIT
Finetuned from model: bert-base-multilingual-cased
Model Sources [optional]
Repository: [More Information Needed]
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]
Uses
Direct Use
This model can be directly used to classify Romanian text into five categories: fake news, misinformation, propaganda, real news, and satire.
It can be handy for media monitoring platforms, news verifiers, and research on misinformation.
Downstream Use
Potential applications include integration into larger systems for real-time news validation or being part of a research toolkit for studying misinformation.
Out-of-Scope Use
This model is not intended for classifying non-Romanian text, and its accuracy may diminish significantly if used on text outside of the Romanian language.
It is also not suitable for detailed sentiment analysis or non-news related content.
Bias, Risks, and Limitations
This model may exhibit bias due to the nature of the training data, which could lead to overfitting on certain types of news or propaganda specific to Romanian contexts.
It might not generalize well to new, unseen forms of misinformation or satire.
Recommendations
Users should evaluate the model's performance on their specific use case before deployment, particularly if applied in contexts different from those in which the training data was collected.
How to Get Started with the Model
To get started with this model, load it using HuggingFace's Transformers library and ensure you have the appropriate tokenizer
. Example code for loading and using the model is provided in the Python script.
Training Details
Training Data
The model was trained on a combined dataset of Romanian fake news, including data scraped from Veridica.ro and the Fakerom dataset.
The dataset includes categories such as fake news, misinformation, propaganda, real news, and satire.
Training Procedure
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
Training regime: The model was trained using fp32 precision.
Number of epochs: 3
Batch size: 16
Warmup steps: 500
Weight decay: 0.01
Learning rate scheduler: Warmup with cosine decay.
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was tested on a split of the combined dataset, with 20% of the data used for evaluation.
Factors
The evaluation was performed across all five classes of fake news categories, with results disaggregated by each category.
Metrics
The primary evaluation metrics were accuracy, precision, recall, F1 score, and log loss. ROC AUC scores were also computed for each class.
Results
Results
Accuracy: 93.51%
Precision: 93.80%
Recall: 93.51%
F1 Score: 93.56%
Log Loss: 0.225
ROC AUC Per Class:
Fake News: 98.50%
Misinformation: 98.89%
Propaganda: 98.98%
Real News: 99.55%
Satire: 99.99%
Summary
The model demonstrates strong performance across all classes, with particularly high ROC AUC scores indicating good separability between classes.
