Base model: Qwen/Qwen3-4B-Instruct-2507
Continued training from (base adapter): tussiiiii/qwen3-4b-structured-output-lora-continued-v5-daichira-ver2
This repo: DPO-continued LoRA adapter weights (NOT a fully merged 16-bit model)

Note: This repository does not contain full merged 16-bit weights. You must load the base model and then apply this LoRA adapter.

Training Objective

This adapter was optimized using DPO to align responses to preferred outputs, focusing on improving reasoning style (Chain-of-Thought preferences) and structured response quality based on the provided preference dataset.

Training Configuration

Base model: Qwen/Qwen3-4B-Instruct-2507
Base adapter (continued from): tussiiiii/qwen3-4b-structured-output-lora-continued-v5-daichira-ver2
Method: DPO (Direct Preference Optimization)
Epochs: 1
Learning rate: 1e-07
Beta: 0.1
Max sequence length: 1024
Adapter type: LoRA (trainable, continued)
Trainer: TRL DPOTrainer (Unsloth patched)

Usage

Since this is a LoRA adapter, load the base model and then attach the adapter via PEFT.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen3-4B-Instruct-2507"
adapter_id    = "your_id/your-repo-name"  # <-- set to this repo (e.g. qwen3-4b-structured-output-lora-continued-v5-daichira-ver2-dpo)

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

# Attach LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)

# Test inference
prompt = "Your question here"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & License (IMPORTANT)

Training Data: [u-10bei/dpo-dataset-qwen-cot]
License: MIT License. (As per dataset terms).
Compliance: Users must follow the original base model's license terms.

tussiiiii/qwen3-4b-structured-output-lora-continued-v5-daichira-ver2-dpo

qwen3-4b-structured-output-lora-continued-v5-daichira-ver2-dpo

What this repository contains

Training Objective

Training Configuration

Usage

Sources & License (IMPORTANT)

tussiiiii/
qwen3-4b-structured-output-lora-continued-v5-daichira-ver2-dpo