ToolACE hallucination DeBERTa v3 small token classifier
This model is a DeBERTa v3 small token classifier trained for span level hallucination detection in tool calling responses.
Labels
normal
contradiction
overgeneration
missing_tool
Question and context tokens are ignored during training. The classifier is trained only on answer tokens.
Training data
The model was trained on the ToolACE RAGTruth style synthetic hallucination dataset.
Dataset repository:
Ali-Bhai/toolace_ragtruth_synthetic
Validation tuned thresholds
{
"contradiction": 0.9,
"overgeneration": 0.0,
"missing_tool": 0.0
}
Final combined test results
| method | precision | recall | f1 |
|---|
| Improved DeBERTa tuned thresholds | 0.976366 | 0.992562 | 0.984397 |
| Template suffix sanity baseline | 1 | 0.96786 | 0.983667 |
| Improved DeBERTa token classifier | 0.96975 | 0.992788 | 0.981134 |
| LettuceDetect transformer | 0.339035 | 0.787189 | 0.473946 |
| LookBackLens style distilgpt2 | 0.346983 | 0.655472 | 0.453761 |
| Unsupported value heuristic | 0.170562 | 0.0187523 | 0.0337895 |
| Empty prediction | 0 | 0 | 0 |
Final per type F1 table
| method | contradiction | missing_tool | overgeneration |
|---|
| Empty prediction | 0 | 0 | 0 |
| Improved DeBERTa token classifier | 0.67071 | 0.983388 | 0.987825 |
| Improved DeBERTa tuned thresholds | 0.700767 | 0.987462 | 0.990821 |
| LettuceDetect transformer | 0.0488308 | 0.287056 | 0.558726 |
| LookBackLens style distilgpt2 | 0.0139008 | 0.414203 | 0.450776 |
| Template suffix sanity baseline | 0 | 1 | 1 |
| Unsupported value heuristic | 0.377397 | 0 | 0.00618375 |
Loading
Install transformers, then load the model.
from transformers import AutoTokenizer, AutoModelForTokenClassification
model_id = "Ali-Bhai/toolace_hallucination_deberta_v3_small"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)
print(model.config.id2label)
Important evaluation note
The final reported method applies validation tuned thresholds to the token classifier outputs.
Validation tuned thresholds:
{
"contradiction": 0.9,
"overgeneration": 0.0,
"missing_tool": 0.0
}
Caveat
This model was trained on synthetic hallucination labels. It should be evaluated carefully before use on naturally occurring production hallucinations.
