VeriSci Claim Verifier
NLI-pretrained DeBERTa v3 base sequence classifier for evidence-aware scientific claim verification over SUPPORTS, REFUTES, and NOT_ENOUGH_INFO.
Intended Use
Given a scientific claim and retrieved evidence passage, classify whether the evidence supports, refutes, or does not provide enough information. This model is not a medical device, not a substitute for peer review, and should not be used for clinical, legal, or public-policy decisions without expert review.
Training Data
- Primary verifier data:
allenai/scifact_entailment train split.
- Hard NEI examples:
andreiaalexa/scifact-relevance-pairs title/train not_relevant pairs.
- Validation/test policy: train-overlapping validation claims removed, then remaining validation rows partitioned by normalized claim key. Derived test was not used for training or model selection.
Evaluation
Validation accuracy: 0.8452
Validation macro F1: 0.8269
Derived test accuracy: 0.8876
Derived test macro F1: 0.8503
See evaluation/eval_summary.json and prediction JSONL files for class metrics, confusion matrices, and limitations.
Limitations
- The derived test split is small and originates from SciFact validation, not an official hidden test set.
- The model only judges the provided evidence; retrieval failures can change the end-to-end answer.
- Raw softmax confidence is not a calibrated probability.
- Training data is SciFact-derived and subject to license/domain constraints.
