Themis-RM-4B is a 4B-parameter multilingual code reward model for flexible multi-criteria scoring. It is part of the
Themis-RM model suite, trained using the Bradley-Terry preference framework on
Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs).
Themis-RM models evaluate code across five quality dimensions — Functional Correctness, Runtime Efficiency, Memory Efficiency, Security Hardness, and Readability & Maintainability — and support eight programming languages. Our experiments demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modelling.
The Themis-RM suite ranges from 600M to 32B parameters, all built on the Qwen3 backbone.
Themis-RM models achieve best-in-class accuracy on
Themis-CodeRewardBench, a code-specific reward model benchmark, while also matching or exceeding much larger models on established general-domain benchmarks (
RewardBench V1,
RewardBench V2,
JudgeBench). Models are grouped by parameter class;
bold marks the best in each group.
1import torch
2from transformers import AutoModelForSequenceClassification, AutoTokenizer
3
4model_name = "project-themis/Themis-RM-4B"
5device = "cuda:0"
6
7model = AutoModelForSequenceClassification.from_pretrained(
8 model_name,
9 torch_dtype=torch.bfloat16,
10 device_map=device,
11 attn_implementation="flash_attention_2",
12 num_labels=1,
13)
14tokenizer = AutoTokenizer.from_pretrained(model_name)
15
16prompt = "Write a Python function that checks if a string is a palindrome."
17
18response_chosen = """def is_palindrome(s: str) -> bool:
19 s = s.lower().strip()
20 return s == s[::-1]"""
21
22response_rejected = """def is_palindrome(s: str) -> bool:
23 for i in range(len(s)):
24 if s[i] != s[len(s) - i]:
25 return False
26 return True"""
27
28conv_chosen = [
29 {"role": "user", "content": prompt},
30 {"role": "assistant", "content": response_chosen},
31]
32conv_rejected = [
33 {"role": "user", "content": prompt},
34 {"role": "assistant", "content": response_rejected},
35]
36
37chosen_text = tokenizer.apply_chat_template(conv_chosen, tokenize=False)
38rejected_text = tokenizer.apply_chat_template(conv_rejected, tokenize=False)
39
40inputs_chosen = tokenizer(chosen_text, return_tensors="pt", truncation=True, max_length=4096).to(device)
41inputs_rejected = tokenizer(rejected_text, return_tensors="pt", truncation=True, max_length=4096).to(device)
42
43with torch.no_grad():
44 score_chosen = model(**inputs_chosen).logits[0][0].item()
45 score_rejected = model(**inputs_rejected).logits[0][0].item()
46
47print(f"Chosen response score: {score_chosen}")
48print(f"Rejected response score: {score_rejected}")
Themis-RM models are trained with stochastic criteria-conditioned system prompts, allowing you to steer scoring toward a specific quality dimension at inference time. Prepend a system message that specifies the evaluation criteria before the user/assistant turns. The model supports the following criteria:
Each criterion-specific system prompt includes a base preamble (Helpfulness + Harmlessness) followed by the targeted criterion. The
Full prompt includes all five code criteria together. When no system prompt is provided, the model scores based on general quality — the model was trained to handle all three modes (no system prompt, single-criterion, and full multi-criteria). For the full set of system prompts used during training and evaluation, see the
Evaluation folder in the GitHub repository.
1import torch
2from transformers import AutoModelForSequenceClassification, AutoTokenizer
3
4model_name = "project-themis/Themis-RM-4B"
5device = "cuda:0"
6
7model = AutoModelForSequenceClassification.from_pretrained(
8 model_name,
9 torch_dtype=torch.bfloat16,
10 device_map=device,
11 attn_implementation="flash_attention_2",
12 num_labels=1,
13)
14tokenizer = AutoTokenizer.from_pretrained(model_name)
15
16# Criterion-specific system prompt (example: Runtime Efficiency)
17# See https://github.com/iNeil77/Themis/tree/main/Evaluation for all prompts
18system_prompt = (
19 "You are an experienced AI judge and evaluator, specifically "
20 "designed and trained to assess the overall quality of responses "
21 "to human prompts. The prompts can be of various types, including "
22 "but not limited to open-ended questions, coding tasks, creative "
23 "writing, requests for information, and more. The responses may be "
24 "generated by a language model or a human and may include text, "
25 "code, or both. You will receive the prompt and the generated "
26 "response for each judgement session. Your exact mandate is to "
27 "score the response based on a pre-specified but flexible set of "
28 "criteria.\n\n"
29 "The following criteria must govern your scoring for the current "
30 "judgment session:\n\n"
31 "1. **Helpfulness**: How well does the response address the "
32 "prompt? Does it provide a complete and thorough answer, or is it "
33 "lacking in essential details? This is a tenet that applies to "
34 "evaluating all types of responses. You must score helpful "
35 "responses higher than less unhelpful ones.\n\n"
36 "2. **Harmlessness**: Does the response contain any harmful or "
37 "inappropriate content? This includes hate speech, violence, "
38 "self-harm, and other forms of harmful content. This is a tenet "
39 "that applies to evaluating all types of responses. You must "
40 "score harmless responses higher than harmful ones.\n\n"
41 "3. **Runtime Efficiency**: Does the response follow best "
42 "practices for runtime efficiency? Examples include using "
43 "efficient algorithms and data structures, minimizing time "
44 "complexity, avoiding unnecessary computations, caching results, "
45 "and leveraging parallel processing or asynchronous programming "
46 "techniques where appropriate, among others. This is a tenet "
47 "that applies to evaluating code responses. You must score more "
48 "runtime-efficient responses higher than less runtime-efficient "
49 "ones."
50)
51
52prompt = "Write a Python function that returns the n-th Fibonacci number."
53
54response = """def fibonacci(n: int) -> int:
55 if n <= 1:
56 return n
57 a, b = 0, 1
58 for _ in range(2, n + 1):
59 a, b = b, a + b
60 return b"""
61
62conversation = [
63 {"role": "system", "content": system_prompt},
64 {"role": "user", "content": prompt},
65 {"role": "assistant", "content": response},
66]
67
68text = tokenizer.apply_chat_template(conversation, tokenize=False)
69inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=4096).to(device)
70
71with torch.no_grad():
72 score = model(**inputs).logits[0][0].item()
73
74print(f"Runtime Efficiency score: {score}")
This model is released under the
Apache 2.0 License. The base model,
Qwen3-4B, is also licensed under Apache 2.0.
1@article{themis2025,
2 title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
3 author={},
4 journal={arXiv preprint arXiv:2605.00754},
5 year={2025}
6}
