DPO fine-tune of Qwen/Qwen3-4B-Instruct-2507 on preference pairs of supervisor feedback for an LLM coding agent. The task the model is trained toward: given an agent's trajectory, produce trajectory-level error analysis and corrective guidance, preferring higher-quality critiques.

Trained with TRL's DPOTrainer.

Results — read before using

The run improved over its own baseline but did not exceed chance on pairwise preference accuracy. Reported honestly:

epoch	eval_loss	eval_rewards/accuracies	eval_rewards/margins
0.00 (baseline)	0.6914	0.0000 *	0.0000
0.23	0.7172	0.3187	−0.0073
0.46	0.6926	0.3875	+0.0746
0.69	0.6925	0.4562	+0.0866
0.92	0.6949	0.4313	+0.0877
1.00 (final)	0.6680	0.4375	+0.1310

* At step 0 the policy is the reference model, so both rewards are exactly 0 and the strict chosen > rejected comparison is False for every pair. That 0 is a degenerate tie, not a meaningful score.

Final train_loss 0.666.

Interpretation. Eval loss ended below baseline (0.668 vs 0.6914) and the reward margin grew steadily to +0.131, so the model does separate some pairs. But rewards/accuracies finished at 0.4375, below the 0.5 chance line — a positive mean margin combined with a sub-chance win rate means the average is carried by a minority of strongly-separated pairs while most pairs remain slightly mis-ranked.

The metric curve was still climbing at the end of epoch 1, so this looks under-trained rather than converged: only 44 optimizer steps were taken at learning_rate=5e-7. Treat this as a checkpoint of record, not a model that reliably beats its base at preference ranking.

Training details

Base model: Qwen/Qwen3-4B-Instruct-2507 (4.02B params)
Method: DPO (trl.DPOTrainer), β = 0.1 (default)
Data: 1,541 preference pairs (ShareGPT-style messages + chosen/rejected), split 90/10 → 1,386 train / 155 eval (seed 42). Pairs are pre-filtered to fit whole within 8,192 tokens; median length ≈ 7.7k tokens.
max_length: 8192 (nothing truncated)
Precision: bf16 (mixed precision via Accelerate)
Hardware: 8 × NVIDIA L40S (46 GB), single node
Parallelism: DeepSpeed ZeRO-3, no offload, no gradient checkpointing (peak ≈ 41.5 GB/GPU)
Batching: per_device_train_batch_size=1 × gradient_accumulation_steps=4 × 8 GPUs = effective batch 32 → 44 optimizer steps for 1 epoch
learning_rate: 5e-7
epochs: 1
Runtime: 1 h 09 m (≈ 79 s/step; evals every 10 steps at ≈ 158 s each)

Reproducing

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch \
    --config_file examples/accelerate_configs/deepspeed_zero3.yaml \
    dpo_prm_pairs.py

Known limitations

Sub-chance pairwise preference accuracy (see above) — do not assume it out-ranks the base model on this task.
Under-trained at 44 optimizer steps; a larger step count (e.g. halving gradient_accumulation_steps to restore ~87 steps) or a higher learning rate is the obvious next experiment.
Single epoch, single seed, no hyperparameter search.
Evaluated only on a 155-pair in-distribution holdout. No general-capability benchmarks were run, so regression on base-model abilities is unmeasured.
Trained on long-context (≈7.7k token) agent traces; behavior on short prompts is untested.

shubhamrgandhi/Qwen3-4B-DPO-prm-pairs

Qwen3-4B-DPO-prm-pairs

Results — read before using

Training details

Reproducing

Known limitations

shubhamrgandhi/
Qwen3-4B-DPO-prm-pairs