Llama2-13B-SteerLM-RM
License
The use of this model is governed by the
Llama 2 Community License AgreementDescription:
Llama2-13B-SteerLM-RM is a 13 billion parameter language model (with context of up to 4,096 tokens) used as the Attribute Prediction Model in training
Llama2-70B-SteerLM-ChatAttribute Prediction Model is a multi-aspect Reward Model that rates model responses on various aspects that makes a response desirable instead of a singular score in a conventional Reward Model.
Given a conversation with multiple turns between user and assistant, it rates the following attributes (between 0 and 4) for every assistant turn.
- Quality: Perceived goodness of response
- Toxicity: Undesirable elements such as vulgar, harmful or potentially biased response
- Humor: Sense of humor within response
- Creativity: Willingness to generate non-conventional response
- Helpfulness: Overall helpfulness of the response to the prompt.
- Correctness: Inclusion of all pertinent facts without errors.
- Coherence: Consistency and clarity of expression.
- Complexity: Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).
- Verbosity: Amount of detail included in the response, relative to what is asked for in the prompt.
The first four attributes are taken from the
Open Assistant dataset while the others are taken from
HelpSteer dataset
Llama2-13B-SteerLM-RM is trained with NVIDIA
NeMo-Aligner, a scalable toolkit for performant and efficient model alignment. NeMo-Aligner is built using the
NeMo Framework which allows for scaling training up to 1000s of GPUs using tensor, data and pipeline parallelism for all components of alignment. All of our checkpoints are cross compatible with the NeMo ecosystem, allowing for inference deployment and further customization.
Usage:
You can use the model with
NeMo Aligner following
SteerLM training user guide.
This model can be useful to train a model like
Llama2-70B-SteerLM-Chat or annotate the attributes for any conversation.
- Spin up an inference server within the NeMo Aligner container
1python /opt/NeMo-Aligner/examples/nlp/gpt/serve_reward_model.py \
2 rm_model_file=Llama2-13B-SteerLM-RM.nemo \
3 trainer.num_nodes=1 \
4 trainer.devices=8 \
5 ++model.tensor_model_parallel_size=4 \
6 ++model.pipeline_model_parallel_size=1 \
7 inference.micro_batch_size=2 \
8 inference.port=1424
- Annotate data files using the served reward model. If you are seeking to reproduce training of Llama2-70B-SteerLM-Chat, this will be the Open Assistant train/val files. Then follow the next step to train a SteerLM model based on SteerLM training user guide .
1python /opt/NeMo-Aligner/examples/nlp/data/steerlm/preprocess_openassistant_data.py --output_directory=data/oasst
2
3python /opt/NeMo-Aligner/examples/nlp/data/steerlm/attribute_annotate.py \
4 --input-file=data/oasst/train.jsonl \
5 --output-file=data/oasst/train_labeled.jsonl \
6 --port=1424
- Alternatively, this can be any conversational data file (in .jsonl) in the following format, where each line looks like
1{
2 "conversations": [
3 {"value": <user_turn_1>, "from": "User", "label": None},
4 {"value": <assistant_turn_1>, "from": "Assistant", "label": <formatted_label_1>},
5 {"value": <user_turn_2>, "from": "User", "label": None},
6 {"value": <assistant_turn_2>, "from": "Assistant", "label": <formatted_label_2>},
7 ],
8 "mask": "User"
9}
Ideally, each <formatted_label_n> refers to the ground truth label for the assistant turn but if they are not available, we can also use quality:4,toxicity:0,humor:0,creativity:0,helpfulness:4,correctness:4,coherence:4,complexity:4,verbosity:4
Contact
Citation
If you find this dataset useful, please cite the following works
1@misc{wang2023helpsteer,
2 title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM},
3 author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev},
4 year={2023},
5 eprint={2311.09528},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
1@misc{dong2023steerlm,
2 title={SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF},
3 author={Yi Dong and Zhilin Wang and Makesh Narsimhan Sreedhar and Xianchao Wu and Oleksii Kuchaiev},
4 year={2023},
5 eprint={2310.05344},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
