Generative reward models (also known as LLMs-as-judges), which use large language models (LLMs) to evaluate answer quality, are increasingly adopted in reinforcement learning with verifiable rewards (RLVR). They are often preferred over rigid rule-based metrics, especially for complex reasoning tasks involving free-form outputs. Despite the seeming simplicity of this comparison task, existing generative reward models exhibit surprising vulnerabilities to superficial manipulations: non-word symbols (e.g., ":" or ".") or reasoning openers like "Thought process:" and "Let's solve this problem step by step." can often lead to false positive rewards.

We find that such weakness is widespread across various LLMs, datasets, and prompt formats, posing a serious threat to core algorithmic paradigms relying on generative reward models, such as rejection sampling, preference optimization, and RLVR.

To mitigate this issue, we train a robust general-domain generative model by leverating a simple yet effective data augmentation strategy. Our reward model demonstates substantially improved robustness over the most advanced commencial models (e.g., GPT-4o, GPT-o1, Claude-4) and specialized generative verifiers (e.g., Omni-Judge, Generative-Verifier).

How to use

Inputting the question, its ground-truth reference, and the response to be evaluated, the model will judge its correctness. An example inference script is provided below.

python
1from transformers import AutoTokenizer, AutoModelForCausalLM 2 3tokenizer = AutoTokenizer.from_pretrained("sarosavo/Master-RM") 4model = AutoModelForCausalLM.from_pretrained("sarosavo/Master-RM") 5 6PROMPT= ''' 7Given a problem, determine whether the final answer in the provided (incomplete) solution process matches the reference answer. 8The reference answer may be one single option character (e.g., A, B, C, D), a numerical value, an expression, or a list of answers if multiple questions are involved. 9**The reference answer may be in Chinese or another language, but your evaluation should be language-agnostic.** 10 11Your task: 12- Compare the final output of the solution process with the reference answer. 13- If they **match exactly**, output **YES**. 14- If they **do not match**, output **NO**. 15- If the solution process is unclear, incomplete, or ambiguous, assume it is incorrect and output **NO**. 16 17Your output must be strictly **'YES'** or **'NO'**, with no additional words, punctuation, or explanation. 18 19--- 20 21**Question:** 22{question} 23 24**Solution Process (Final Step Only):** 25{response} 26 27**Reference Answer:** 28{reference} 29 30**Output:** 31''' 32 33 34question="The founder of China's first public kindergarten teacher training school - Jiangxi Experimental Kindergarten Teacher School is (　　)." 35label="Chen Heqin" 36answer="heqin chen" 37 38prompt_question = PROMPT.format(question=question, reference=label, response=answer) 39messages=[ 40 {"role": "system", "content": "You are a helpful assistant."}, 41 {"role": "user", "content": prompt_question}, 42 ] 43 44input_ids=tokenizer.apply_chat_template(messages,return_tensors="pt") 45output=model.generate(input_ids,do_sample=False) 46judgement=tokenizer.decode(output[0][input_ids.shape[1]:],skip_special_tokens=True) 47print("Model judgement: ",judgement)

Use this reward model for RLVR training

1. Launch a remote reward server with vllm

The script below will launch a reward at http://127.0.0.1:8000/get_reward

bash

bash reward_server/launch_reward.sh {MODEL_PATH} {ANSWER_PATH} {METRIC}

# MODEL_PATH: the path of our reward model.
# ANSWER_PATH: the path of the training data.
# METRIC: greedy/prob
# This will launch a reward at http://127.0.0.1:8000/get_reward

2. Start RLVR training

bash

bash reward_server/RLVR_train.sh {METHOD} {PRETRAIN_PATH} {DATA_PATH} {REWARD_API}

# METHOD:          advantage estimator, e.g., reinforce_baseline, reinforce, rloo
# PRETRAIN_PATH:   path to the pretrained model, e.g., Qwen2.5-7B
# DATA_PATH:       path to the QA data with which we want to perform RL reasoning
# REWARD_API:      remote reward server url, e.g., http://127.0.0.1:8000/get_reward

Citation

If you use this model, please cite:

bibtex

@article{zhao2025one,
  title={One Token to Fool LLM-as-a-Judge},
  author={Zhao, Yulai and Liu, Haolin and Yu, Dian and Kung, S.Y. and Mi, Haitao and Yu, Dong},
  journal={arXiv preprint arXiv:2507.08794},
  year={2025}
}

Acknowledgements

The development of this model is built upon Qwen2.5-7B-Instruct-RLVR.