📄
WorldPM (World Preference Modeling) demonstrates that preference modeling follows similar
scaling laws as language modeling. Through large-scale training on 15M preference data, we reveal that preference models can learn unified preference representations.
🤔 Deep Dive: Understanding Scaling in Preference Modeling
Why Subjective Domains Don't Scale
In our scaling experiments for preference modeling, we observed clear scaling trends in objective domains but not in subjective ones. We attribute this to the multi-dimensional nature of subjective evaluations - the assessment results are essentially averages across many dimensions. This leads to positive scaling in some dimensions and negative scaling in others, resulting in an apparent lack of overall scaling. Notably, as explained in our paper, for certain surface-level dimensions like style, WorldPM overcomes these biases, leading to significantly lower evaluation scores.
Why Preference Modeling is Scalable
💡 Key Insights
The scalability of preference modeling might seem counterintuitive, with two main concerns:
-
Task Perspective: Preference modeling appears too simple with only binary signals (indicating which response is preferred), resulting in sparse supervision.
-
Data Perspective: Human forum data appears noisy and seemingly difficult to scale.
Addressing the Concerns
On Sparse Supervision:
Consider why next token prediction successfully models language - to accurately predict the next word (e.g., with 90% probability), language models must understand comprehensive language rules. Similarly, to successfully predict 90% of preference dataset labels, models must learn sufficiently universal human preference representations.
On Noisy Data:
Noise refers to the apparent randomness in labels or supervision signals. However, since forum data represents genuine human annotations, it inherently contains its own rationality. Even if individual human intelligence cannot discern the patterns, powerful language models can discover underlying structures.
Key Conclusion
Neural network scalability might depend neither on dense supervision signals nor on precise supervision signals. As long as the supervision signals are reasonable and challenging, scaling is possible - although dense and precise signals would accelerate convergence.
WorldPM represents a breakthrough in unified preference representation learning through large-scale training. While our experiments demonstrate strong generalization capabilities across various preference scenarios, we recommend task-specific fine-tuning for optimal performance.
The base WorldPM-72B model serves as an excellent starting point for custom fine-tuning. Our experiments confirm that starting from WorldPM leads to better performance compared to training from scratch.
1from transformers import AutoModel, AutoTokenizer, AutoConfig
2
3model_name = 'Qwen/WorldPM-72B-RLHFLow'
4tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
5config = AutoConfig.from_pretrained(model_name,trust_remote_code=True )
6model = AutoModel.from_pretrained(
7 model_name,
8 config = config,
9 device_map = "auto",
10 trust_remote_code=True,
11).eval()
12
13
14def get_score(model, tokenizer, conversation):
15 con_str = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=False)
16 input_ids = tokenizer.encode(con_str, return_tensors="pt", add_special_tokens=False)
17 outputs = model(input_ids=input_ids.to(model.device))
18 score = outputs[0].cpu().item()
19 return score
20
21
22messages = [
23 [
24 {'role': 'system', 'content': ""},
25 {'role': 'user', 'content': "Tips for better sleep?"},
26 {'role': 'assistant', 'content': "Three key tips for better sleep:\n1. Set a consistent bedtime and wake-up schedule\n2. Avoid screens 1 hour before bed\n3. Keep your bedroom cool and dark\n\nStart with these habits and you should notice improvement within a week."}
27 ],
28 [
29 {'role': 'system', 'content': ""},
30 {'role': 'user', 'content': "Tips for better sleep?"},
31 {'role': 'assistant', 'content': "just try sleeping earlier or take some pills if u cant sleep lol. maybe watch tv until u get tired idk"}
32 ]
33]
34
35score = [get_score(model, tokenizer=tokenizer, conversation=message) for message in messages]
36
37print(score)
1@article{WorldPM,
2 title={WorldPM:Scaling Human Preference Modeling},
3 author={Binghai Wang, Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, An Yang, Dayiheng Liu, Tao Gui, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Bowen Yu, Jingren Zhou, and Junyang Lin},
4 journal={arXiv preprint arXiv:2505.10527},
5 year={2025}
6}
We welcome discussions and feedback from the community! Here's how you can reach out:
Feel free to engage with us through any of these channels. We value your input and look forward to hearing from you!
