You can deploy this model and start earning money today!

omkarwaikar/
cad-reviews

Name: cad-reviews
Availability: InStock

ott-deceptive-opinion-spam

1902.10186

FacebookAI/roberta-base

finetune

mit

Views

No views yet

cad-reviews — Deceptive Review Classifier (RoBERTa-base on Ott corpus)

Part of the Content Authenticity Detector project.

Model description

roberta-base fine-tuned on Ott et al.'s Deceptive Opinion Spam Corpus for binary deceptive-review classification.

Dataset: 1,600 hotel reviews — balanced across deceptive/truthful × positive/negative from 20 Chicago hotels.

Label mapping:

0 = truthful
1 = deceptive

Training uses hotel-stratified 5-fold cross-validation (4 hotels held out per fold) to prevent hotel-identity leakage.

Results

5-fold CV (hotel-stratified):

Fold	Accuracy	Macro F1
Fold 1	91.9%	0.919
Fold 2	90.6%	0.906
Fold 3	87.8%	0.877
Fold 4	90.3%	0.903
Fold 5	88.4%	0.884
Mean ± std	89.8% ± 1.7%	0.898 ± 0.017

Baseline (LR + TF-IDF): 5-fold CV acc=87.5%±2.2%, F1=0.875±0.022. RoBERTa: +2.3% accuracy, +2.3% F1.

Model weights are from the best single fold (Fold 1, macro-F1=0.919).

Training

Base: roberta-base
Epochs: 3 (early stopping, patience=2)
Batch size: 16
Max length: 256 tokens (Ott reviews average ~150 words; 256 covers 99%+)
Learning rate: 2e-5
Hardware: NVIDIA RTX 2060 (6GB)
CV strategy: Hotel-stratified 5-fold (4 hotels per fold as held-out set)

Intended use & limitations

Intended: Research and educational demonstrations of NLP-based deceptive review detection.

Limitations:

Trained exclusively on hotel reviews from Chicago, written by Mechanical Turk workers (Ott et al. 2011). Will not generalise to product reviews, restaurant reviews, or other domains.
Small corpus (1,600 examples). Performance estimates have moderate variance across folds.
Deceptive reviews in the wild are stylistically different from deliberately written Turk deception.
Not suitable for production content moderation. Treat all verdicts as signals, not ground truth.

Attention caveat

The accompanying demo uses last-layer attention weights as token highlights. Per Jain & Wallace (2019), attention weights are not faithful explanations — high-attention tokens are not necessarily causal. Highlights are visualisation hints only.

Citation

bibtex

@inproceedings{ott2011finding,
  title={Finding Deceptive Opinion Spam by Any Stretch of the Imagination},
  author={Ott, Myle and Choi, Yejin and Cardie, Claire and Hancock, Jeffrey T.},
  booktitle={ACL 2011},
}

@inproceedings{ott2013negative,
  title={Negative Deceptive Opinion Spam},
  author={Ott, Myle and Cardie, Claire and Hancock, Jeffrey T.},
  booktitle={NAACL-HLT 2013},
}

omkarwaikar/cad-reviews