Direct Preference Optimization
Здесь представлена Direct Preference Optimization (DPO) модель, обученная в рамках курса по LLM от VK.
В качестве референсной модели бралась HuggingFaceTB/SmolLM-135M-Instruct, после чего модель файнтюнилась на датасете HumanLLMs/Human-Like-DPO-Dataset.
Детали
Описание модели
Reinforcement Learning from Human Feedback (RLHF) является достаточно сложной задачей.
Это связано во многом с нестабильностью обучения и особыми требованиями к Reward модели, которую мы хотим использовать.
В статье DPO предлагается простой лосс, сильно упрощающий классическое видение алгоритма.
В этом репозитории представлен пример DPO-модели для генерации ответов, более близких к ответам человека.
Для файнтюна используется HuggingFaceTB/SmolLM-135M-Instruct, прошедшая стадию SFT, и понимающую формат чата пользователя-ассистента.
Ниже показано влияние нашего файнтюна на модель на примере ответа на пользовательский вопрос "What's your morning routine like?"
Пример: ответ модели до файнтюна
What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "Awake Morning Routine" for you:
Morning Routine (10-15 minutes)
- 10:00 AM: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or a standing forward fold.
- 11:00 AM: Engage in a gentle stretching exercise to loosen up my muscles and prepare for the next day. This can include chest opens, shoulder rolls, or neck stretches. Hold each stretch for 20-30 seconds and then relaxation.
- 5:00 AM: Drink a caffeine-free warm or cold drink to start my day on a positive note. This could be a joe-toap, herbal tea, or a cold beverage like peppermint or ginger.
- 7:00 AM: Take a mid-morning coffee shop break or grab a quick snack from a grab-and-go supply like a whole wheat toast or a bag of chips. This will give my body
Пример: ответ модели после файнтюна
The morning routine is a crucial aspect of our daily life that sets the tone for our success and behavior. It involves setting aside time for tasks, managing our energy, and making the most of our time. A consistent and effective morning routine helps to regulate our emotions,motivation, and productivity, which in turn enables us to achieve our goals and success.
Here are some tips on how to establish a good morning routine:
- Start your day with a positive mindset: Believe in yourself and your abilities. Try to focus on your current task and take action, rather than worrying about things that may not get done.
- Plan your day to make the most of your time: Identify your tasks and break them down into smaller ones. Learn to prioritize your tasks and make sure that they are important.
- Use a morning exercise routine: Exercise is essential for a healthy and happy body. Exercise helps to increase energy and speed, and it also improves your mood.
- Eat a nutritious meal: Eat a meal that is rich in nutrients and energy. Avoid consuming large amounts of food, especially when you are first waking up.
- Stay hydrated: Drink plenty of water before starting your day. Drinking enough water helps to reduce fatigue and
Референсы
- Модель: HuggingFaceTB/SmolLM-135M-Instruct
- Датасет: HumanLLMs/Human-Like-DPO-Dataset
- Оригинальная статья: https://arxiv.org/abs/2305.18290
