🎭 Emotionally Aware: Generates responses with increased emotional expressiveness (+38% emoji usage vs base model)
💬 Conversational Flow: Reduced reliance on template phrases like "As an AI..."
🛡️ Safety Preservation: Maintains base model's harmlessness constraints through reference model regularization
⚡ Efficient Alignment: Achieved with single-epoch training on 10k preference pairs

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("bikmish/llm-course-hw2-dpo")
tokenizer = AutoTokenizer.from_pretrained("bikmish/llm-course-hw2-dpo")

messages = [
    {"role": "user", "content": "Just saw an amazing movie!"},
    {"role": "assistant", "content": "Oh cool! What's it about? 😃"}  # DPO-style response
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))