You can deploy this model and start earning money today!

xxccho/
helpsteer3-full-rm-llama3.1-8b-epoch3-merged

Name: helpsteer3-full-rm-llama3.1-8b-epoch3-merged
Availability: InStock

safetensors

llama

reward-model

meta-llama/Llama-3.1-8B-Instruct

finetune

apache-2.0

Views

No views yet

helpsteer3-full-rm-llama3.1-8b-epoch3 (merged)

Reward model trained on the full HelpSteer3 preference dataset (all 4 domains: general / code / multilingual / stem; ~38,459 train records), LoRA-merged into the base weights so this repo is a standalone, drop-in model — no PEFT/adapter loading required.

Base model: meta-llama/Llama-3.1-8B-Instruct
Training: LoRA rank 8, alpha 16, target q/k/v/o_proj, merged into base weights
Training data: nvidia/HelpSteer3 (preference subset, all domains)
Epoch: 3, step 5511

Usage

python

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained(
    "xxccho/helpsteer3-full-rm-llama3.1-8b-epoch3-merged", num_labels=1, torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("xxccho/helpsteer3-full-rm-llama3.1-8b-epoch3-merged")

messages = [{"role": "user", "content": "What is 7 x 8?"},
            {"role": "assistant", "content": "7 x 8 = 56."}]
text = tok.apply_chat_template(messages, tokenize=False)
inputs = tok(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    reward = model(**inputs).logits.squeeze(-1).item()

Note on the unmerged adapter-only predecessor

An earlier repo (xxccho/helpsteer3-full-rm-llama3.1-8b-epoch3) published the raw LoRA adapter (base=meta-llama/Llama-3.1-8B-Instruct) instead of merged weights. Loading that repo directly with AutoModelForSequenceClassification.from_pretrained does not correctly restore the trained score head and gives substantially different (degraded) scores. Use this merged repo instead for downstream use, including as the base_model for xxccho/rm-debias-loras-helpsteer3-llama-3.1-8b.