You can deploy this model and start earning money today!

Federico6725/
Llama-HalluES-detector

Name: Llama-HalluES-detector
Availability: InStock

safetensors

modernbert

Views

No views yet

Llama-HalluES-detector

Token-level hallucination detector for Retrieval-Augmented Generation in Spanish. Given a source context and a model response, it flags the tokens of the response that are not supported by the context.

It is the open detector of the Hallu-ES benchmark: a fine-tune of jhu-clsp/mmBERT-base following the LettuceDetect recipe, ported to Spanish.

This model measures faithfulness to the provided source, not world truth. A factually correct statement that the context does not support is counted as a hallucination. Do not use it as a general fact-checker.

Built with Llama

Part of the training corpus consists of outputs generated by Meta Llama 3.1 (Llama-3.1-8B-Instruct), and this model is trained on data that includes those outputs. Use of the Llama materials is governed by the Llama 3.1 Community License Agreement. Per section 1(b) of that agreement, an AI model trained on Llama outputs must carry a name beginning with Llama, which is why this repository is named Llama-HalluES-detector while the benchmark itself keeps the name Hallu-ES.

The training corpus also contains outputs of Qwen2.5-7B-Instruct (Apache 2.0 with additional conditions) and Salamandra-7b-instruct (Apache 2.0).

Usage

The model is a two-class token classifier. Class 1 is the hallucinated class; a response is flagged when any of its tokens passes a probability threshold of 0.5. Context and response are passed as a sentence pair, with truncation applied to the context only — the response must never be truncated, since the labels are token-level.

python

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

MODEL = "Federico6725/Llama-HalluES-detector"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForTokenClassification.from_pretrained(MODEL).eval()

context = ("La Sagrada Familia es una basílica de Barcelona diseñada por "
           "Antoni Gaudí. Las obras comenzaron en 1882.")
response = ("La Sagrada Familia, obra de Gaudí, se empezó a construir en 1882 "
            "y es el monumento más visitado de España.")

enc = tok(context, response, truncation="only_first", max_length=3072,
          return_tensors="pt")
with torch.no_grad():
    probs = torch.softmax(model(**enc).logits, -1)[0, :, 1]

# keep only the tokens belonging to the response (sequence 1)
mask = torch.tensor([sid == 1 for sid in enc.sequence_ids(0)])
resp_probs = probs[mask]

print("hallucinated:", bool((resp_probs >= 0.5).any()))
print("max token probability:", float(resp_probs.max()))

For the full pipeline — rebuilding each response's effective context from the benchmark's source file, including the truncated-context QA condition, and merging contiguous positive tokens into spans — use src/predict.py from the Hallu-ES repository:

bash

python3 src/predict.py input.jsonl output.jsonl
# defaults to this checkpoint; MODEL_DIR overrides it with a local folder

Results

Evaluated on the held-out Hallu-ES test split: 504 responses over 74 sources, none of which appeared in training (the split is at source level, so there is no leakage). The gold standard is the LLM annotator's labels, themselves validated at κ = 0.875 against blind manual annotation.

System	P	R	F1	κ vs LLM annotator	κ vs human
TF-IDF + LogReg baseline	0.801	0.817	0.809	0.615	—
mDeBERTa-v3 (collapsed)	0.498	1.000	(0.665)	—	—
Llama-HalluES-detector	0.857	0.785	0.819	0.655	0.686

Two things belong next to that number:

The margin over the lexical baseline (0.809 → 0.819) is not statistically significant (McNemar over the 114 discordant responses, p = 0.40). The real gain is per task: QA recall rises from 0.439 to 0.805 (p < 0.001).
The κ of 0.686 against human judgement is measured on the stratified sample of 288 manually annotated responses. It retains ~78% of the agreement that the commercial LLM annotator reaches with the same manual annotation (0.875) — at no inference cost.

English and Turkish references (lettucedetect-large 0.792, Turk-LettuceDetect 0.727) are computed on their own corpora and gold standards and are shown for context only; the comparison is informative, not symmetric.

Training


Base model	`jhu-clsp/mmBERT-base` — 307M parameters, 110M non-embedding
Task	Token classification, 2 classes (`0` = supported, `1` = hallucinated)
Training data	9,516 responses / 10,670 valid spans (Hallu-ES `train.jsonl`)
Max length	3,072 tokens; context truncated, response never
Epochs / lr / warmup	3 / 1e-5 / 10%
Effective batch size	16 (8 × 2 gradient accumulation)
Precision	bf16 (weights published here in fp32)
Model selection	Best epoch by F1 on an internal dev set of 400 training responses
Seed	42
Hardware	≈50 min on a Colab A100

No hyperparameter search was performed against the test set. GPU training is not deterministic: re-running the recipe should land within roughly ±0.005 F1 of the reported 0.819.

Requires transformers >= 5.0. This is not a soft recommendation: the config stores per-layer-type RoPE settings under the rope_parameters key, and transformers 4.5x parses it without raising while silently falling back to the default local_rope_theta = 10000 instead of the 160000 mmBERT uses. Since 15 of the 22 layers are sliding-attention, the model will load, run, and return degraded predictions with no warning. The checkpoint was saved with transformers 5.14.1.

Limitations

The F1 gold standard consists of LLM-annotator labels, validated at κ = 0.875 against manual annotation, not direct human gold as in the English baselines.
The manual annotation was produced by the author, who also wrote the annotation criteria and the annotator prompt. It is blind, but not independent of the criteria being tested.
The annotator the model distils is measurably stricter than the manual annotation on soft evaluative additions; disagreements concentrate in data-to-text (κ = 0.722).
Generalisation to generators other than the three 2024 models used to build the corpus (Llama-3.1-8B, Qwen2.5-7B, Salamandra-7b) is not measured.
Domains are limited to encyclopedic text and synthetic business listings. It has not been evaluated on specialised domains (medical, legal) where a hallucination is more costly.
The response-level decision uses a fixed threshold of 0.5 with no human-annotated development set to tune it against.

Intended use and out-of-scope use

Intended for research on faithfulness in Spanish RAG systems, and as a screening signal in RAG pipelines where a flagged response is routed to review.

Out of scope: general fact-checking, any use as sole evidence for an automated decision about a person, and any deployment where a false negative carries serious consequences without a human in the loop.

Citation

bibtex

@mastersthesis{romiti2026halluees,
  author = {Romiti, Federico},
  title  = {Hallu-ES: a span-level hallucination benchmark for
            retrieval-augmented generation in Spanish},
  school = {Universitat Politècnica de València},
  year   = {2026}
}

Benchmark, corpus, splits and evaluation protocol: https://github.com/Federico-rom/hallu-es