This repository contains a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) with the u-10bei/dpo-dataset-qwen-cot preference dataset.

✅ This is NOT a merged full model.
You must load the base model and then apply this LoRA adapter via PEFT.

Training Configuration

Base model: Qwen/Qwen3-4B-Instruct-2507
Method: DPO (Direct Preference Optimization)
Epochs: 1
Learning rate: 1e-05
Beta: 0.1
Max sequence length: 1024

Usage (Transformers + PEFT)

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE_MODEL = "Qwen/Qwen3-4B-Instruct-2507"
ADAPTER_ID = "your_id/your-repo-name"  # <-- this repo

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(base, ADAPTER_ID)

# Inference example
prompt = "Produce a TOML document for an API specification."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Sources & License (IMPORTANT)

Training Data: [u-10bei/dpo-dataset-qwen-cot]
License: MIT License. (As per dataset terms).
Compliance: Users must follow the original base model's license terms.

kisyou07/lora_structeval_t_qwen3_4b_lr2e5_ep2_wd001_dpo1

lora-structured-dpo-qwen3-4b-dpo1

Training Configuration

Usage (Transformers + PEFT)

Sources & License (IMPORTANT)

kisyou07/
lora_structeval_t_qwen3_4b_lr2e5_ep2_wd001_dpo1