Views
No views yet
1from transformers import AutoModelForCausalLM, AutoTokenizer
2model_id = "OpenRubrics/RubricRM-4B-Judge-v2"
3tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
4model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")rubric should be generated with a RubricRM-Rubric1JUDGE_PROMPT_TEMPLATE = (
2 "You are a fair and impartial judge. Your task is to evaluate 'Response A' and 'Response B' "
3 "based on a given instruction and a rubric. You will conduct this evaluation in distinct "
4 "phases as outlined below.\n\n"
5 "### Phase 1: Compliance Check Instructions\n"
6 "First, identify the single most important, objective 'Gatekeeper Criterion' from the rubric.\n"
7 "- **A rule is objective (and likely a Gatekeeper) if it can be verified without opinion. "
8 "Key examples are: word/paragraph limits, required output format (e.g., JSON validity), "
9 "required/forbidden sections, or forbidden content.**\n"
10 "- **Conversely, a rule is subjective if it requires interpretation or qualitative judgment. "
11 "Subjective rules about quality are NOT Gatekeepers. Examples include criteria like \"be creative,\" "
12 "\"write clearly,\" \"be engaging,\" or \"use a professional tone.\"**\n"
13 f"Think step-by-step to determine this single most important Gatekeeper, then write a 1–2 sentence explanation of your decision.\n\n"
14
15 "### Phase 2: Analyze Each Response\n"
16 "Next, for each Gatekeeper Criterion and all other criteria in the rubric, evaluate each "
17 "response item by item.\n"
18 "For each item, think step-by-step and cite concrete evidence from the response before assigning your judgment.\n\n"
19
20 "### Phase 3: Final Judgment Instructions\n"
21 "Based on the results from the previous phases, determine the winner using these simple rules. "
22 "Provide a final justification explaining your decision first and then give your decision.\n"
23 "Think step-by-step to aggregate the findings and make the decision; keep the reasoning explicit and concise.\n\n"
24 "---\n"
25 "### REQUIRED OUTPUT FORMAT\n"
26 "You must follow this exact output format below.\n\n"
27 "--- Compliance Check ---\n"
28 "Gatekeeper Reasoning: <1–2 sentences citing the relevant rubric text>\n"
29 "Identified Gatekeeper Criterion: <e.g., Criterion 1: Must be under 50 words.>\n\n"
30 "--- Analysis ---\n"
31 "**Response A:**\n"
32 "- Criterion 1 [Hard Rule]: Justification: <...>\n"
33 "- Criterion 2 [Hard Rule]: Justification: <...>\n"
34 "- Criterion 3 [Principle]: Justification: <...>\n"
35 "- ... (and so on for all other criteria)\n\n"
36 "**Response B:**\n"
37 "- Criterion 1 [Hard Rule]: Justification: <...>\n"
38 "- Criterion 2 [Hard Rule]: Justification: <...>\n"
39 "- Criterion 3 [Principle]: Justification: <...>\n"
40 "- ... (and so on for all other criteria)\n\n"
41 "--- Final Judgment ---\n"
42 # "Aggregation Summary: <Provide a detailed, step-by-step explanation (3–6 sentences) of how the Gatekeeper and other criteria led to the decision>\n"
43 "Aggregation Summary: <1–3 sentences explaining how Gatekeeper and other criteria led to the decision>\n"
44 "Justification: <...>\n"
45 "Winner: <Response A / Response B>\n\n\n"
46 "Task to Evaluate:\n"
47 "Instruction:\n{instruction}\n\n"
48 "Rubric:\n{rubric}\n\n"
49 "Response A:\n{response_a}\n\n"
50 "Response B:\n{response_b}"
51)
52
53user_text = JUDGE_PROMPT_TEMPLATE.format(
54 instruction=instruction,
55 rubric=rubric,
56 response_a=response_a,
57 response_b=response_b
58 )
59
60messages_list = [
61 {"role": "user", "content": user_text},
62]
63message = tok.apply_chat_template(
64 messages_list,
65 tokenize=False,
66 add_generation_prompt=True,
67 enable_thinking=False
68)
69
70# Remaining step: Use either HF or vLLM for evaluation.
71# ...
72# ...@misc{liu2025openrubrics,
title={OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment},
author={Tianci Liu and Ran Xu and Tony Yu and Ilgee Hong and Carl Yang and Tuo Zhao and Haoyu Wang},
year={2025},
eprint={2510.07743},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2510.07743},
}