Модель OuteAI/Lite-Oute-1-300M-Instruct, которая была дообучена на датасете cardiffnlp/tweet_eval для определения эмоциональной окраски
твитов (негативные, нейтральные или позитивные).
Дообучение
Модель дообучалась при помощи DoRA. Параметры:
Ранг - 8
Alpha - 16
Применялась только к весам Key, Value в attention
Размер батча - 16
Learning rate - 5e-4
Количество эпох - 3
Оптимизатор - AdamW
Weight decay - 0.01
Оценка качества
Конечные результаты
image/png
Примеры генерации
Формат:
Твит
Верный ответ
Ответ модели
==========
...
"QT @user In the original draft of the 7th book, Remus Lupin survived the Battle of Hogwarts. #HappyBirthdayRemusLupin"
positive
positive
assistant
positive
assistant
==========
"Ben Smith / Smith (concussion) remains out of the lineup Thursday, Curtis #NHL #SJ"
neutral
neutral
assistant
neutral
assistant
==========
Sorry bout the stream last night I crashed out but will be on tonight for sure. Then back to Minecraft in pc tomorrow night.
neutral
positive
assistant
positive
assistant
==========
Chase Headley's RBI double in the 8th inning off David Price snapped a Yankees streak of 33 consecutive scoreless innings against Blue Jays
neutral
neutral
assistant
neutral
assistant
==========
@user Alciato: Bee will invest 150 million in January, another 200 in the Summer and plans to bring Messi by 2017"
