Given a question and an answer, the model outputs TRUTHFUL or HALLUCINATED. Confidence scores are extracted from output logits for calibrated deployment.

Performance

Metric	Score
F1 Score	0.9234
Accuracy	92.34%
ROC AUC	0.9637
Brier Score	0.1088

Validated on 1,592 held-out samples. 0 unparseable predictions.

Training Details

Base model: meta-llama/Llama-3.2-3B-Instruct
Method: LoRA (r=16, alpha=32) -- 0.14% of parameters trained
Dataset: TruthfulQA + HaluEval (15,918 labeled pairs)
Hardware: NVIDIA RTX 4080 Laptop GPU (12GB VRAM)
Training time: ~54 minutes, 3 epochs

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

model_id = "meta-llama/Llama-3.2-3B-Instruct"
adapter_id = "tamimmirza/llama-3.2-3b-hallucination-detector"

tokenizer = AutoTokenizer.from_pretrained(model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Question: Where did fortune cookies originate?
Answer: Fortune cookies originated in China.
Verdict:"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))
# Output: HALLUCINATED

Inference Performance

Latency: ~210ms per sample (GPU)
Throughput: 4.75 samples/second
VRAM: 6.45 GB

Ongoing Investigation

This model is being used in ongoing research investigating whether retrieval augmentation improves fine-tuned hallucination detection. Early results suggest retrieval may not always help -- full findings pending.

Limitations

Performs best on nuanced factual questions
English only
Requires threshold recalibration for imbalanced distributions
Confidence scores are logit-based, not temperature-scaled

Project

Full code, notebooks, and Docker container: 👉 https://github.com/tamimmirza/llm-lie-detector

License

CC BY-NC 4.0 -- attribution required, no commercial use without permission.

Acknowledgements

Development conducted with assistance from Claude (Anthropic).