Full merged 16-bit weights at the repo root (ready to use with Transformers), and
LoRA adapter checkpoints under adapters/ (saved during training).

Training Objective

This model was optimized using Direct Preference Optimization (DPO) to prefer chosen outputs over rejected ones, focusing on structured response quality.

Training Configuration

Base model: Qwen/Qwen3-4B-Instruct-2507
Method: DPO
Epochs: 1
Learning rate: 1e-06
Beta: 0.05
Max sequence length: 1024

Usage (merged weights)

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gen-oze/dpo-qwen"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = "Your question here"
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))