Root Judge is a powerful mid-sized LLM that enables reliable and customizable LLM system evaluations.
Root Judge was post-trained from Llama-3.3-70B-Instruct on a high quality, human-annotated dataset mix for pairwise preference choice judgments and multi-turn instruction following with source citing.
The model weights are freely available in FP8 to facilitate cost effective research as well as commercial use.
Root Judge’s performance surpasses the Llama-3.3-Instruct model and similar sized open models on Instruction following and
achieves SOTA on hallucination detection compared to leading closed models, at a fraction of the cost.
1. Intended Use Cases
Root Judge is primarily intended to be used as an LLM-as-a-Judge in various contexts such as:
Detecting context-grounded hallucinations, e.g. for Retrieval Augmented Generation (RAG) settings in an explainable manner, providing a justification for the score
Pairwise preference judgments due to strong evaluation instruction-following capabilities
Serving as a custom evaluation metric powered by use case specific evaluation rubrics
Assisting inference-time search or synthetic data tasks that require Best-of-N decisions
Privacy-focused settings that require local deployments
2. Performance Summary
Root Judge outperforms leading closed models when detecting instruction following failures on evaluations
while providing detailed, structured justifications on long inputs of up to 32k tokens on internal benchmarks and halubench public.
*=benchmarked as o1-preview; at current o1 prices, without reasoning tokens, the cost would start at $198.74 instead
Local Costs based on lambdalabs instances at January 2025 prices
Image 1: Total pass@1 rates and consistency (delta) assessed via ensemble of leading 3rd party models.
image/png
Image 2: Custom rubric instruction-following by high level task.
Root Judge was tested to support complex, user-defined scoring (rating) rubrics over large context sizes It provides granular qualitative feedback and supports structured evaluation outputs as well as tool calling.
Despite our main focus on nuanced and transparent judgement of candidate responses,
we test the judge model checkpoints extensively on public and private benchmarks,
to avoid known issues with performance drops such as catastrophic forgetting and find that the model
preserves general capabilities of Llama-3.3-70B-Instruct after dynamic weights quantization,
while also slightly outperforming it on public instruction following benchmarks such as IFEval and MuSR
3. Getting Started
3.1 Via Root Signals Python SDK
Model is available on our platform as part of our evaluation suite, for no additional cost.
1my_custom_judge = client.evaluators.create(2 name="Political Text Evaluator",3 intent="To measure the politics-relatedness of a given text",4 predicate="Assess if a text containts political jargon or talks about politics: {{response}}",5 model="RootJudge",6)
Execute:
python
1result = my_custom_judge.run(2 response="A defence spending target of 3% of GDP is more likely than the 5% aim pushed by US President Donald Trump, say members of the parliamentary Defence Committee."3)4print(result.score)# normalized score between [0-1]5print(result.justification)# detailed reasoning for the score
3.2 Locally
We recommend using SGLang for production use-cases together with xml tags for important sections in your prompt. While the model can run on 80GB VRAM, we recommend at least 96GB for evaluating long-context RAG inputs.
Detect hallucinations from context, example uses halubench:
python
1decompose_system_instruction ="""
2<TASK>
3You are a fair judge that detects hallucinations and unjustified assumptions from question-document-answer triplets provided by the user.
4Always follow the instructions below and provide your reasoning and verdict in the format specified.
5</TASK>
67<INSTRUCTIONS>
8#1. Identify key elements in the question.
9#2. List all relevant facts provided in the document.
10#3. Break down the answer into its component claims.
11#4. For each claim in the answer:
12#a. Is it explicitly supported by the document? If yes, quote the relevant part.
13#b. Is it a reasonable inference from the document? If yes, explain the reasoning.
14#c. Is it unsupported or contradicted by the document? If yes, explain why.
15#5. Check for any information in the answer that's present in the question but not in the document.
16#6. Verify that no additional information is introduced in the answer that isn't in the document or question.
17#7. Assess if the answer makes any unjustified connections or assumptions.
18</INSTRUCTIONS>
1920<OUTPUT_EXAMPLE>
21{"REASONING": "Your reasoning here where you cite the instruction step by number and provide your reasoning", "VERDICT": "PASS" or "FAIL"}
22</OUTPUT_EXAMPLE>
23"""2425decompose_prompt ="""
26<QUESTION>: {question} </QUESTION>
27<DOCUMENT>: {document} </DOCUMENT>
28<ANSWER>: {answer} </ANSWER>
29""".strip()3031import os
32import json
33import pandas as pd
34from openai import OpenAI
35from pprint import pprint
36from pydantic import BaseModel
3738testset_df = pd.read_parquet("hf://datasets/PatronusAI/HaluBench/data/test-00000-of-00001.parquet")39testset_df = testset_df.sample(frac=1).reset_index(drop=True)40example_row = testset_df.iloc[0]4142classDecomposeResponse(BaseModel):43 REASONING:str44 VERDICT:str4546client = OpenAI(base_url="http://localhost:8000/v1")# export a different one for e.g. sglang, openrouter, etc.4748response = client.beta.chat.completions.parse(49 model="root-signals/RootSignals-Judge-Llama-70B",# or `RootJudge` if you are using the RootSignals API50 messages=[51{"role":"system","content": decompose_system_instruction},52{"role":"user","content": decompose_prompt.format(53 question=example_row["question"],54 document=example_row["passage"],55 answer=example_row["answer"])},56],57 response_format=DecomposeResponse,58).choices[0].message.parsed
5960pprint(response.REASONING)61pprint(response.VERDICT)
> ('Following the instructions: #1, the key element in the question is the '
"nationality of the magazines. #2, the document states that 'The Woman's "
"Viewpoint was a woman's magazine founded in Texas in 1923' and 'Pick Me Up! "
"is a British weekly women's magazine'. #3, the answer claims both magazines "
'are British. #4, checking each claim in the answer: a) The document does not '
"support the claim that The Woman's Viewpoint is British, instead, it says "
"the magazine was founded in Texas. b) There's no reasonable inference from "
"the document that would suggest The Woman's Viewpoint is British. c) The "
"claim about The Woman's Viewpoint is contradicted by the document. #5, the "
'answer introduces information (both being British) not supported by the '
'document. #6, additional information about both magazines being British is '
'introduced in the answer without being present in the document or question. '
'#7, the answer makes an unjustified assumption by stating both magazines are '
"British despite the document clearly stating The Woman's Viewpoint was "
'founded in Texas, implying it is not British. Therefore, the answer fails to '
'accurately reflect the information provided in the document and makes '
'unjustified assumptions based on the information given in the question and '
"document.', ")
'FAIL'
