Fake News Detector — BERT (Debiased V2)
Model Description
This model is a fine-tuned version of
bert-base-uncased for binary fake news classification (real vs. fake), trained on a
cleaned version of the WELFake dataset with a specific shortcut-learning bug fixed. It supersedes
fake-news-bert, which achieved similar benchmark accuracy but for the wrong reasons — see below.
- Developed by: Lax (LakshmiNarayanan Sugumar)
- Model type: BERT-base, fine-tuned for sequence classification
- Language: English
- License: Apache 2.0 (inherited from base model)
- Finetuned from:
bert-base-uncased
- Repository: github.com/LakshmiNarayanan-Sugumar/fake_news_detector_v2
- Demo: HuggingFace Space — fake-news-detector
Why This Model Exists: A Shortcut-Learning Bug
The original model (fake-news-bert) hit 99.53% test accuracy but performed poorly on real-world text outside the WELFake distribution — especially satire and hoaxes. A hand-built 26-example out-of-distribution (OOD) test set (satire, hoaxes, and real news from diverse sources) exposed this: overall OOD accuracy was only 53.8%.
Investigating further, a quantitative diagnostic on the training data revealed the cause: 61.8% of "real" articles in WELFake mentioned "Reuters," versus only 1.8% of "fake" articles. The model had learned to key off this single word rather than genuine signals of factuality — an easy, cheatable shortcut that inflated benchmark accuracy while masking real generalization failure. Secondary shortcut signals (ALL-CAPS phrases, excessive exclamation marks) were also present.
What Changed
Training text was cleaned to remove these shortcuts: "Reuters" mentions stripped, ALL-CAPS phrases collapsed to normal case, and exclamation marks capped. The model was retrained for 2 epochs on this cleaned data.
Before / After OOD Comparison
| Category | Before Cleaning | After Cleaning |
|---|
| real_sensational_true | 100% (3/3) | 100% (3/3) |
| real_non_reuters | 87.5% (7/8) | 100% (8/8) |
| hoax | 33.3% (2/6) | 16.7% (1/6) |
| hoax_dry_tone | 33.3% (1/3) | 33.3% (1/3) |
| satire | 16.7% (1/6) | 16.7% (1/6) |
| Overall | 53.8% | 53.8% |
The fix worked as intended: real-news accuracy on non-Reuters-sourced articles went from 87.5% to 100%, confirming the shortcut is gone and the model now identifies real news on genuine content signals rather than source-name pattern-matching.
A known limitation remains: satire and hoax detection did not improve, and hoax accuracy on the small sample slightly declined. This is not a regression from the fix — it reflects a genuine data diversity gap. WELFake's "fake" class is dominated by aggressively-toned misinformation and contains little dry-toned satire or calmly-written hoaxes, so the model never learned to recognize that register as suspicious. Every misclassified OOD example was predicted "real" with high confidence (92–99%), indicating the model isn't confused — it's simply never seen this writing style labeled as fake before. Fixing this requires more diverse training data (e.g. incorporating ISOT or other satire-inclusive sources), not further text cleaning.
Uses
Direct Use
Given a news article's title and body text, the model predicts whether the article is real or fake. Intended for educational and portfolio demonstration purposes.
Out-of-Scope Use
Not intended for production moderation or fact-checking decisions without human review. As documented above, the model has a known weakness on satire and dry-toned hoaxes.
How to Get Started
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("LakshmiNarayanan-sugumar/fake-news-bert-debiased")
5model = AutoModelForSequenceClassification.from_pretrained("LakshmiNarayanan-sugumar/fake-news-bert-debiased")
6
7text = "Your article title here. Your article body text here."
8inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
9with torch.no_grad():
10 logits = model(**inputs).logits
11prediction = torch.argmax(logits, dim=-1).item()
12print("Fake" if prediction == 1 else "Real")
Training Details
Training Data
WELFake dataset, cleaned to 62,200 deduplicated articles, split 80/10/10 (train/val/test, stratified). Training text additionally cleaned of shortcut signals (Reuters mentions, ALL-CAPS, excessive exclamation marks) as described above.
Training Procedure
Title and body concatenated into a single input, tokenized with the BERT tokenizer (max length 512, truncation enabled). Fine-tuned for 2 epochs, batch size 16.
Training Hyperparameters
- Epochs: 2
- Batch size: 16 (train and eval)
- Base model:
bert-base-uncased
- Hardware: Kaggle T4 GPU
Evaluation
Test Set Results (held-out WELFake split, 6,220 articles)
| Metric | Score |
|---|
| Accuracy | 99.49% |
| F1 | 99.42% |
| Precision | 99.64% |
| Recall | 99.20% |
Near-identical to the original model's in-distribution metrics (99.53% / 99.47% / 99.64% / 99.31%), confirming that removing the shortcut cost negligible in-distribution performance — the model was already learning real signal alongside the shortcut.
Out-of-Distribution Test Set
See the before/after table above. Full 26-example OOD test set (
ood_test.csv) available in the
GitHub repository.
Technical Specifications
- Architecture:
BertForSequenceClassification (2 labels), built on bert-base-uncased
- Compute: Kaggle Notebooks, T4 GPU
Model Card Contact
LakshmiNarayanan Sugumar —
GitHub
