This adapter applies Direct Preference Optimization (DPO) to improve structured output accuracy (JSON / YAML / XML / TOML / CSV) by aligning the model with human preferences.

Training Configuration

Base model: Qwen/Qwen3-4B-Instruct-2507
SFT adapter: lora_structeval_t_qwen3_4b_v2_stage3/checkpoint-100
Method: DPO with QLoRA (4-bit)
Max sequence length: 2048
DPO beta: 0.1
Learning rate: 1e-07
LoRA: r=8, alpha=16
Batch size: 2 (grad accum: 4, effective BS=8)
Epochs: 1

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "DLNorb/dpo_lora_model_stage3"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

Sources & Terms (IMPORTANT)

Training data:

https://huggingface.co/datasets/u-10bei/dpo-dataset-qwen-cot: MIT License

Compliance: Users must comply with each dataset's license (including copyright notice) and the base model's original terms of use.