Views
No views yet
unsloth/Qwen3-4B and fine-tuned using LoRA and ORPO on HalluGuard-Preferences-76k, a synthetic preference dataset for hallucination detection derived from FineWeb.HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation
unsloth/Qwen3-4BGROUNDED, HALLUCINATED_INTRINSIC, HALLUCINATED_EXTRINSIC)HalluGuard-Preferences-76k1from transformers import AutoModelForCausalLM, AutoTokenizer
2import json
3
4model_name = "lrsbrgrn/HalluGuard-Qwen3-4B"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModelForCausalLM.from_pretrained(
7 model_name,
8 torch_dtype="auto",
9 device_map="auto"
10)
11
12def create_prompt(document, claim):
13 return json.dumps({
14 "instructions": [
15 "You will be given a document and a claim.",
16 "Decide whether the claim is 'GROUNDED', 'HALLUCINATED_INTRINSIC', or 'HALLUCINATED_EXTRINSIC' based ONLY on the document.",
17 "Definitions:",
18 " - GROUNDED: The claim is fully supported by the document. All relevant parts are directly verifiable from the document.",
19 " - HALLUCINATED_INTRINSIC: The claim contradicts what the document states or clearly implies.",
20 " - HALLUCINATED_EXTRINSIC: The claim includes information that is not stated or implied in the document and cannot be verified using only the document (it requires external knowledge).",
21 "Justification requirements:",
22 " - Your justification MUST be evidence-grounded.",
23 " - Explicitly refer to the relevant parts of the document (by quoting or paraphrasing them).",
24 " - Explain how these parts SUPPORT, CONTRADICT, or FAIL TO SUPPORT the claim.",
25 " - Do NOT use any external knowledge; rely only on the provided document.",
26 "Answer format (VERY IMPORTANT):",
27 " - You MUST respond using EXACTLY the following XML structure:",
28 " <answer>",
29 " <classification>CATEGORY</classification>",
30 " <justification>Your reasoning here</justification>",
31 " </answer>",
32 " - CATEGORY must be ONE of: GROUNDED, HALLUCINATED_INTRINSIC, HALLUCINATED_EXTRINSIC.",
33 " - The <justification> must briefly explain your reasoning and cite evidence from the document.",
34 " - Do NOT add any other text before or after the <answer>...</answer> block.",
35 " - Do NOT add any extra tags or attributes.",
36 ],
37 "document": f"'{document}'",
38 "claim": f"'{claim}'",
39 })
40
41document = "Apple shares hit record highs, briefly valuing the company at $900B, after beating Wall Street forecasts with strong international sales."
42claim = "Apple stock hit record, valuing the company at $900B, after beating Wall Street expectations on international sales."
43
44prompt = create_prompt(document, claim)
45
46messages = [{"role": "user", "content": prompt}]
47text = tokenizer.apply_chat_template(
48 messages,
49 tokenize=False,
50 add_generation_prompt=True,
51 enable_thinking=True
52)
53inputs = tokenizer([text], return_tensors="pt").to(model.device)
54
55generated_ids = model.generate(**inputs, max_new_tokens=32768, temperature=0.6, top_p=0.95, top_k=20)
56output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
57
58# parsing thinking content
59try:
60 # rindex finding 151668 (</think>)
61 index = len(output_ids) - output_ids[::-1].index(151668)
62except ValueError:
63 index = 0
64
65thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
66content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
67
68print("thinking content:", thinking_content)
69print("content:", content)<think>
The claim states that Apple stock hit a record, valuing the company at $900B, after beating Wall Street expectations on international sales. The document says "Apple shares hit record highs, briefly valuing the company at $900B, after beating Wall Street forecasts with strong international sales." All three elements of the claim — record high, $900B valuation, and beating Wall Street on international sales — are directly stated in the document. The claim is therefore grounded.
</think>
<answer>
<classification>GROUNDED</classification>
<justification>The document states that Apple shares hit record highs, briefly valuing the company at $900B, after beating Wall Street forecasts with strong international sales, which directly supports all elements of the claim.</justification>
</answer>sglang>=0.4.6.post1 or vllm>=0.8.5 or to create an OpenAI-compatible API endpoint:python -m sglang.launch_server --model-path lrsbrgrn/HalluGuard-Qwen3-4B --reasoning-parser qwen31vllm serve lrsbrgrn/HalluGuard-Qwen3-4B --enable-reasoning --reasoning-parser deepseek_r1
2
31@inproceedings{bergeron-etal-2026-halluguard,
2 title = "{H}allu{G}uard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation",
3 author = "Bergeron, Loris and
4 Buhnila, Ioana and
5 Francois, Jerome and
6 State, Radu",
7 editor = "Liakata, Maria and
8 Moreira, Viviane P. and
9 Zhang, Jiajun and
10 Jurgens, David",
11 booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
12 month = jul,
13 year = "2026",
14 address = "San Diego, California, United States",
15 publisher = "Association for Computational Linguistics",
16 url = "https://aclanthology.org/2026.findings-acl.835/",
17 pages = "16918--16932",
18 ISBN = "979-8-89176-395-1",
19 abstract = "Large Language Models excel at NLP tasks but remain prone to hallucinations, limiting trust in real-world applications. We present HalluGuard, a 4B-parameter Small Reasoning Model (SRM) designed as a guardrail for Retrieval-Augmented Generation (RAG) pipelines, which classify document-claim pairs as grounded or hallucinated in closed-book, document-grounded settings and produces evidence-grounded justifications. Our approach combines (i) a domain-agnostic synthetic dataset derived from FineWeb and refined through multi-stage curation and data reformation, (ii) synthetic grounded and hallucinated claims, and (iii) preference-based fine-tuning with Odds Ratio Preference Optimization (ORPO) to distill large-model reasoning into a smaller backbone. On the RAGTruth subset of the LLM-AggreFact benchmark, HalluGuard achieves 84.4{\%} balanced accuracy (BAcc), surpassing specialized models, MiniCheck (7B; 84.0{\%}) and Granite Guardian 3.3 (8B; 82.2{\%}) while using roughly half their parameters. Across the benchmark, it reaches 77.1{\%} BAcc, surpassing larger general-purpose LLMs such as GPT-4o (75.9{\%}). HalluGuard and datasets will be released upon acceptance."
20}