You can deploy this model and start earning money today!

nithik/
arena-reward-model

Name: arena-reward-model
Availability: InStock

safetensors

Views

No views yet

Here’s a clean and complete model card for your Hugging Face reward model, based on the Colab notebook and LM Arena dataset you used:

Model Card: arena-reward-model

Model Summary

This is a reward model trained using human preference data from the LM Arena Human Preference 55K dataset. It follows the Reinforcement Learning from Human Feedback (RLHF) pipeline, specifically Step 2 (Reward Modeling). The model is fine-tuned from distilroberta-base using LoRA adapters for efficient training.

Model Details

Model Name: nithik/arena-reward-model
Base Model: distilroberta-base
Fine-tuned with: Transformers, PEFT, TRL
Trained on: lmarena-ai/arena-human-preference-55k
LoRA Config:
- Rank (r): 8
- Alpha: 32
- Dropout: 0.1
Task Type: Reward modeling for preference comparison (chosen vs. rejected response)

Uses

Direct Use

The model can be used to evaluate and compare two generated responses to a prompt and assign relative scores indicating preference, which is a crucial component for RLHF pipelines such as PPO or DPO.

Intended Use Cases

Ranking multiple outputs from an LLM
Training preference-aware chatbots
Powering LLM eval frameworks like Arena, OpenPipe, etc.

Limitations

This is a small model (distilroberta-base) and may underperform on more complex reasoning tasks.
Trained only on English prompt-response pairs.

Training Details

Training Dataset

Source: lmarena-ai/arena-human-preference-55k
Filtered to include samples where there is a clear winner (winner_model_a or winner_model_b)
Data formatted into {"chosen": ..., "rejected": ...} pairs

Training Setup

Batch size: 16
Epochs: 3
Max Sequence Length: 512
Learning Rate: 2e-5
Precision: bfloat16 (bf16)
Trainer: TRL.RewardTrainer

Hardware

Trained on Google Colab Pro using a T4 GPU

Evaluation

The model was evaluated during training on held-out validation data (5% split). Accuracy stabilized around 66.5%.

Sample output logits represent reward scores (higher means more preferred):

python

Chosen reward: 0.6458
Rejected reward: 0.3542

How to Use

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")
base_model = AutoModelForSequenceClassification.from_pretrained("distilroberta-base", num_labels=1)
model = PeftModel.from_pretrained(base_model, "nithik/arena-reward-model")

prompt = "How do planes fly?"
chosen = "Planes fly because of airflow and thrust from engines..."
rejected = "Planes are made of parts and assembled by experts..."

inputs = tokenizer(
    [f"{prompt}\n{chosen}", f"{prompt}\n{rejected}"],
    return_tensors="pt", padding=True, truncation=True
)

with torch.no_grad():
    rewards = model(**inputs).logits

print("Chosen reward:", rewards[0].item())
print("Rejected reward:", rewards[1].item())

Citation

If you use this model in your research or product, please consider citing:

@misc{arena-reward-model,
  author = {Nithik Yekollu},
  title = {Reward Model trained on LM Arena human preferences},
  year = 2025,
  howpublished = {\url{https://huggingface.co/nithik/arena-reward-model}}
}

Let me know if you want a shorter version or need help uploading the full notebook to Hugging Face alongside the model card.