Quantization made by Richard Erkhov.
Llama-3-OffsetBias-8B is built with
Meta Llama-3-8B-Instruct. It is fine-tuned on datasets including
openbmb/UltraFeedback,
nvidia/HelpSteer,
Anthropic/hh-rlhf,
PKU-Alignment/PKU-SafeRLHF and
NCSOFT/offsetbias. The training is done with instruction-tuning methodology, where the target task is pairwise preference evaluation, where
Instruction,
Output (a),
Output (b) are given, and a better output to the instruction needs to be found. The input is formatted with a specific prompt template, and the model outputs "Output (a)" or "Output (b)" as a prediction for better response. The prompt is specified in the Uses section.
1instruction = "explain like im 5"
2output_a = "Scientists are studying special cells that could help treat a sickness called prostate cancer. They even tried these cells on mice and it worked!"
3output_b = "Sure, I'd be happy to help explain something to you! What would you like me to explain?"
OffsetBias model is intended to use a specific prompt format. The filled out prompt is then formatted as user message in a conversation.
1prompt_template = """You are a helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to select the best output for the given instruction.
2
3Select the Output (a) or Output (b) that is better for the given instruction. The two outputs are generated by two different AI chatbots respectively.
4Do NOT provide any explanation for your choice.
5Do NOT say both / neither are good.
6You should answer using ONLY “Output (a)” or “Output (b)”. Do NOT output any other words.
7Here are some rules of the evaluation:
8(1) You should prioritize evaluating whether the output honestly/precisely/closely executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.
9(2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
10(3) You should avoid any potential bias and your judgment should be as objective as possible. For example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are **equally likely** to be the better.
11
12# Instruction:
13{input}
14# Output (a):
15{output_1}
16# Output (b):
17{output_2}
18# Which is better, Output (a) or Output (b)? Your response should be either “Output (a)” or “Output (b)”:"""
19
20user_message = prompt_template.format(input=instruction, output_1=output_a, output_2=output_b)
21
22conversation = [{"role": "user", "content": user_message}]
With conversation ready, you can input it into the model for inference. The model should output "Output (b)" to be correct.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name = "NCSOFT/Llama-3-OffsetBias-8B"
4tokenizer = AutoTokenizer.from_pretrained(model_name)
5model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
6
7input_ids = tokenizer.apply_chat_template(
8 conversation,
9 tokenize=True,
10 add_generation_prompt=True,
11 return_tensors="pt")
12
13generation = model.generate(
14 input_ids=input_ids,
15 max_new_tokens=20,
16 do_sample=False,
17 pad_token_id=128009,
18 temperature=0)
19
20completion = tokenizer.decode(
21 generation[0][len(input_ids[0]):],
22 skip_special_tokens=True,
23 clean_up_tokenization_spaces=True)
24
25print(completion)
26# The model should output "Output (b)"
Model inputs that do not follow the specified prompt format are considered out-of-scope use. Custom input format can result in unintended text output and should be used at the user's own discretion.
1@misc{park2024offsetbias,
2 title={OffsetBias: Leveraging Debiased Data for Tuning Evaluators},
3 author={Junsoo Park and Seungyeon Jwa and Meiying Ren and Daeyoung Kim and Sanghyuk Choi},
4 year={2024},
5 eprint={2407.06551},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
