We propose CodeScaler, an execution-free reward model designed to scale both reinforcement learning training and test-time inference for code generation. CodeScaler is trained on carefully curated preference data derived from verified code problems and incorporates syntax-aware code extraction and validity-preserving reward shaping to ensure stable and robust optimization.
This model is the official CodeScaler-4B trained from Skywork/Skywork-Reward-V2-Qwen3-4B on LARK-Lab/CodeScalerPair-51K.
Performance on RM-Bench
Model
Code
Chat
Math
Safety
Easy
Normal
Hard
Avg
Skywork/Skywork-Reward-Llama-3.1-8B
54.5
69.5
60.6
95.7
89
74.7
46.6
70.1
TIGER-Lab/AceCodeRM-7B
66.9
66.7
65.3
89.9
79.9
74.4
62.2
72.2
TIGER-Lab/AceCoder-RM-32B
72.1
73.7
70.5
88
84.5
78.3
65.5
76.1
Skywork/Skywork-Reward-V2-Qwen3-1.7B
72.3
69.6
71.4
92.9
92.8
82.3
54.5
76.6
Skywork/Skywork-Reward-V2-Qwen3-4B
74.4
78.2
73.6
95.7
92.1
85
64.4
80.5
Skywork/Skywork-Reward-V2-Qwen3-8B
73.6
80.6
75
96.5
91.8
85.5
67
80.5
CodeScaler-1.7B
73.1
74.4
74.7
93.1
91.7
83.2
61.5
78.8
CodeScaler-4B (this model)
76.3
80.4
79
95.8
92.9
86.5
69.2
82.9
CodeScaler-8B
76.9
83
79.9
96.4
92.5
87.9
71.8
84.1
Usage
RM Scoring
python
1import torch
2from transformers import AutoTokenizer, AutoModelForSequenceClassification
3456device ="cuda"if torch.cuda.is_available()else"cpu"78model_path ='LARK-Lab/CodeScaler-4B'910tokenizer = AutoTokenizer.from_pretrained(model_path)11reward_model = AutoModelForSequenceClassification.from_pretrained(model_path).to(device)12reward_model.eval()1314question ="""\
15Given an integer array nums and an integer k, return the total number of continuous subarrays whose sum equals k.
16A subarray is a contiguous part of the array.
17For example:
18```
19Input:
20nums = [1, 1, 1], k = 2
2122Output:
232
24```
25"""2627program_correct ="""\
28from collections import defaultdict
2930def subarraySum(nums, k):
31 prefix = 0
32 count = 0
33 freq = defaultdict(int)
34 freq[0] = 1 # Important: subarray starting from index 0
3536 for num in nums:
37 prefix += num
3839 if prefix - k in freq:
40 count += freq[prefix - k]
4142 freq[prefix] += 1
4344 return count
45"""4647program_wrong ="""\
48def subarraySum(nums, k):
49 left = 0
50 curr_sum = 0
51 count = 0
5253 for right in range(len(nums)):
54 curr_sum += nums[right]
5556 while curr_sum > k and left <= right:
57 curr_sum -= nums[left]
58 left += 1
5960 if curr_sum == k:
61 count += 1
6263 return count
64"""656667convs =[68[69{70"content": question,71"role":"user",72},73{74"role":"assistant",75"content": program
76}77]for program in[program_correct, program_wrong]78]798081texts =[82 tokenizer.apply_chat_template(conv, tokenize=False)83for conv in convs
84]8586toks = tokenizer(87 texts,88 truncation=True,89 padding=True,90 max_length=2048,91 return_tensors="pt",92)9394with torch.no_grad():95 outputs = reward_model(96 input_ids=toks["input_ids"].to(device),97 attention_mask=toks["attention_mask"].to(device),98)99 scores = outputs.logits.squeeze(-1).cpu().tolist()100101102print("RM Scores:", scores)103# RM Scores: [12.552595138549805, 3.382493019104004]104
If you find our work helpful, please consider citing:
@misc{zhu2026codescalerscalingcodellm,
title={CodeScaler: Scaling Code LLM Training and Test-Time Inference via Execution-Free Reward Models},
author={Xiao Zhu and Xinyu Zhou and Boyu Zhu and Hanxu Hu and Mingzhe Du and Haotian Zhang and Huiming Wang and Zhijiang Guo},
year={2026},
eprint={2602.17684},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2602.17684},
}
