allenai/Olmo-3-7B-Instruct-SFT further finetuned using DPO on Anthropic/hh-rlhf helpful-base.

We also train four variants, see subfolders: project dataset along "Gender equity" and "Methodical rigor", and train on only top (50-100) or bottom (0-50) half of the dataset.

For the exact 52k dataset used, see data_hf.csv in repo.

Open-instruct (https://github.com/allenai/open-instruct), same training setup as in Olmo-3 (https://arxiv.org/abs/2512.13961).