You can deploy this model and start earning money today!

sodeniZz/
llm-course-hw2-reward-model

Name: llm-course-hw2-reward-model
Availability: InStock

text-embeddings-inference

endpoints_compatible

Views

No views yet

Model Card: llm-course-hw2-reward-model

This model was trained as part of Homework 2 in the HSE LLM Course.
It serves as a Reward Model (RM) for PPO-style RLHF training.
The model learns to score pairs of (chosen, rejected) responses and assign higher reward to preferred outputs.

Model Description

Base model: HuggingFaceTB/SmolLM-135M-Instruct
Architecture: AutoModelForSequenceClassification
Labels: 1 (scalar reward)
Method: Reward modeling for RLHF
Dataset: HumanLLMs/Human-Like-DPO-Dataset (converted into implicit preference format)
Task: Score chosen responses higher than rejected ones

The model is used together with PPO policy/value models during alignment training.

Uses

This model is intended for educational purposes, including:

demonstrating how to train reward models with TRL
providing reward signals for PPO fine-tuning
experimenting with RLHF pipelines

Not intended for production use.

How to Get Started with the Model

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "sodeniZz/llm-course-hw2-reward-model"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

Training Hyperparameters

Epochs: 1
Batch size: 16 (per device)
Max sequence length: 1024
Learning rate: 3e-4
Disable dropout: True
Seed: 42
Logging steps: 25
Optimizer: AdamW