Model Name: lettucedect-large-modernbert-en-v1
Organization: KRLabsOrg
Github: https://github.com/KRLabsOrg/LettuceDetect
LettuceDetect is a transformer-based model for hallucination detection on context and answer pairs, designed for Retrieval-Augmented Generation (RAG) applications. This model is built on ModernBERT, which has been specifically chosen and trained becasue of its extended context support (up to 8192 tokens). This long-context capability is critical for tasks where detailed and extensive documents need to be processed to accurately determine if an answer is supported by the provided context.
The model is trained to identify tokens in the answer text that are not supported by the given context. During inference, the model returns token-level predictions which are then aggregated into spans. This allows users to see exactly which parts of the answer are considered hallucinated.
1from lettucedetect.models.inference import HallucinationDetector
2
3# For a transformer-based approach:
4detector = HallucinationDetector(
5 method="transformer", model_path="KRLabsOrg/lettucedect-base-modernbert-en-v1"
6)
7
8contexts = ["France is a country in Europe. The capital of France is Paris. The population of France is 67 million.",]
9question = "What is the capital of France? What is the population of France?"
10answer = "The capital of France is Paris. The population of France is 69 million."
11
12# Get span-level predictions indicating which parts of the answer are considered hallucinated.
13predictions = detector.predict(context=contexts, question=question, answer=answer, output_format="spans")
14print("Predictions:", predictions)
15
16# Predictions: [{'start': 31, 'end': 71, 'confidence': 0.9944414496421814, 'text': ' The population of France is 69 million.'}]
We evaluate our model on the test set of the
RAGTruth dataset. Our large model,
lettucedetect-large-v1, achieves an overall F1 score of 79.22%, outperforming prompt-based methods like GPT-4 (63.4%) and encoder-based models like
Luna (65.4%). It also surpasses fine-tuned LLAMA-2-13B (78.7%) (presented in
RAGTruth) and is competitive with the SOTA fine-tuned LLAMA-3-8B (83.9%) (presented in the
RAG-HAT paper). Overall,
lettucedetect-large-v1 and
lettucedect-base-v1 are very performant models, while being very effective in inference settings.
The results on the example-level can be seen in the table below.
At the span level, our model achieves the best scores across all data types, significantly outperforming previous models. The results can be seen in the table below. Note that here we don't compare to models, like
RAG-HAT, since they have no span-level evaluation presented.
1@misc{Kovacs:2025,
2 title={LettuceDetect: A Hallucination Detection Framework for RAG Applications},
3 author={Ádám Kovács and Gábor Recski},
4 year={2025},
5 eprint={2502.17125},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2502.17125},
9}
