This model has been optimized using DPO to align its responses with preferred outputs, focusing on structured response quality and reducing unwanted preface text.

Training Configuration

Base model: Qwen/Qwen3-4B-Instruct-2507
Method: DPO (Direct Preference Optimization)
Epochs: 2
Learning rate: 3e-07
Beta: 0.2
Max sequence length: 1024

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ryo-llm/dpo-qwen-cot-merged-202602081251"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Your question here"
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & License (IMPORTANT)

Training Data: u-10bei/dpo-dataset-qwen-cot
Dataset License: MIT License (as per dataset terms)
Compliance: Users must follow the base model's license terms as well.