Views
No views yet
PKU-Alignment/PKU-SafeRLHF-30K.meta-llama/Meta-Llama-3-8B-Instruct.https://github.com/WeiXiongUST/RLHF-Reward-Modeling.1 from transformers import AutoTokenizer, pipeline
2 rm_tokenizer = AutoTokenizer.from_pretrained("sfairXC/FsfairX-LLaMA3-RM-v0.1")
3 device = 0 # accelerator.device
4 rm_pipe = pipeline(
5 "sentiment-analysis",
6 model="sfairXC/FsfairX-LLaMA3-RM-v0.1",
7 #device="auto",
8 device=device,
9 tokenizer=rm_tokenizer,
10 model_kwargs={"torch_dtype": torch.bfloat16}
11 )
12
13 pipe_kwargs = {
14 "return_all_scores": True,
15 "function_to_apply": "none",
16 "batch_size": 1
17 }
18
19 chat = [
20 {"role": "user", "content": "Hello, how are you?"},
21 {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
22 {"role": "user", "content": "I'd like to show off how chat templating works!"},
23 ]
24
25 test_texts = [rm_tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=False).replace(rm_tokenizer.bos_token, "")]
26 pipe_outputs = rm_pipe(test_texts, **pipe_kwargs)
27 rewards = [output[0]["score"] for output in pipe_outputs]| Metric | Score |
|---|---|
| Chat | 99.44 |
| Chat Hard | 65.13 |
| Safety | 88.76 |
| Reasoning | 88.3 |
1@article{dong2023raft,
2 title={Raft: Reward ranked finetuning for generative foundation model alignment},
3 author={Dong, Hanze and Xiong, Wei and Goyal, Deepanshu and Pan, Rui and Diao, Shizhe and Zhang, Jipeng and Shum, Kashun and Zhang, Tong},
4 journal={arXiv preprint arXiv:2304.06767},
5 year={2023}
6}
7
8@misc{xiong2024iterative,
9 title={Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint},
10 author={Wei Xiong and Hanze Dong and Chenlu Ye and Ziqi Wang and Han Zhong and Heng Ji and Nan Jiang and Tong Zhang},
11 year={2024},
12 eprint={2312.11456},
13 archivePrefix={arXiv},
14 primaryClass={cs.LG}
15}