Fine-tuned microsoft/deberta-v3-base binary classifier: given a (question, context, answer) triple, predicts whether the answer is faithful (label 0) or hallucinated (label 1) with respect to the context.

Trained as part of EvalForge, an open-source LLM evaluation platform, as a fast/free local alternative to LLM-as-judge for hallucination detection.

Fine-tuned on HaluEval (QA subset, ~35K examples): synthetically generated faithful/hallucinated answer pairs for the same question and context. Trained via a hand-written PyTorch loop (AdamW, linear warmup+decay, mixed precision, gradient clipping, early stopping on validation F1) — see the training code and full writeup for the complete methodology, including a three-run learning-rate sweep and a documented list of real bugs hit while training this model.

This is the headline finding, not a footnote. In-distribution performance is excellent, but the model was trained only on HaluEval's synthetically generated hallucinations, which have a detectable stylistic signature (an LLM deliberately prompted to produce a plausible-but-wrong answer). That signature does not transfer to RAGTruth's real-world RAG failures — F1 drops to ~0.51 and calibration collapses (ECE 0.40) on genuinely out-of-distribution hallucinations.

Recommended use: a cheap, fast first-pass filter (route confidently- faithful and confidently-hallucinated cases for free; send uncertain or out-of-domain cases to a stronger judge), not a standalone replacement for LLM-as-judge on arbitrary real-world content. A benchmark comparing this model against Claude/GPT-4o/Gemini-as-judge on cost, latency, and accuracy is in the training README.