This model was trained to predict human preferences between two chatbot responses, also known as reward modeling or preference modeling. Given two responses (Response A and Response B), the model outputs a score indicating which response is preferred by humans.

Base Model: google/gemma-2b
Adapter Type: LoRA (Low-Rank Adaptation)
Task: Sequence Classification (preference prediction)
Competition: LMSYS Chatbot Arena
Framework: PEFT (Parameter-Efficient Fine-Tuning)

LoRA Configuration

json

{
  "peft_type": "LORA",
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.1,
  "target_modules": ["o_proj", "v_proj", "q_proj", "k_proj"],
  "modules_to_save": ["classifier", "score"],
  "task_type": "SEQ_CLS"
}

Usage

Installation

pip install transformers peft torch

Inference

python

from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    "google/gemma-2b",
    num_labels=1,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "OldKingMeister/gemma-2b-lmsys-arena-final")
tokenizer = AutoTokenizer.from_pretrained("OldKingMeister/gemma-2b-lmsys-arena-final")

# Prepare input - example comparing two responses
text = """Which response is better for the prompt: What is machine learning?

Response A: Machine learning is a subset of AI.

Response B: Machine learning enables systems to learn from experience."""

# Tokenize and predict
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)
    preference_score = outputs.logits.item()

# Score interpretation:
# Positive: Response B is preferred
# Negative: Response A is preferred
print(f"Preference score: {preference_score}")

Training Details

Dataset

Source: LMSYS Chatbot Arena Conversations
Task: Binary preference classification (which response is better)
Format: Conversations with human preference labels

Training Hyperparameters

Parameter	Value
Learning Rate	2e-4
Batch Size	4
Gradient Accumulation Steps	4
Epochs	10
Max Sequence Length	512
LoRA Rank (r)	16
LoRA Alpha	32
LoRA Dropout	0.1

Hardware

GPU: NVIDIA A100 (40GB)
Training Time: ~6 hours
Mixed Precision: fp16

Model Architecture

Base Parameters: 2.5B (frozen)
Trainable Parameters: ~16M (LoRA adapters + classification head)
Total Checkpoint Size: ~44MB

Citation

bibtex

@misc{lmsys-arena-2024,
  title={LMSYS Chatbot Arena Competition},
  howpublished={https://www.kaggle.com/competitions/lmsys-chatbot-arena},
  year={2024}
}

@article{gemma2024,
  title={Gemma: Open Models Based on Gemini Research and Technology},
  author={{Google}},
  year={2024}
}

License

Apache 2.0