A LoRA policy from DPO on FinQA, where the preference pairs come from the same verifiable reward as the GRPO arm: sample completions from the base, label each correct/incorrect by whether the final number is right, pair a correct one (chosen) vs an incorrect one (rejected). This is the offline counterpart in finqa-rlvr - same base, same signal as GRPO, only the optimization differs.

result

FinQA validation accuracy (greedy, n=200): 0.530 -> 0.555 (+5 questions). A small positive trend, but within noise at this eval size (per-arm SE ~3.5 pts). The project's honest conclusion is not "DPO beats GRPO" - it is that the method is second-order at this scale; reward design and base headroom dominate. Trained on yavuz-ai/finqa-prefs (1,881 correctness-labelled pairs).

training

DPO (TRL, reference-free with the LoRA adapter), LoRA r=16 / alpha=32, beta 0.1, lr 5e-6, 1 epoch, 1x A100.

use

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen2.5-3b-dpo-finqa")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")