Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

qwen3-4b-instruct-dpo-from-sft-4096_u-10bei_9datasets – AI Model by WassyO | AlphaNeural AI

You can deploy this model and start earning money today!

WassyO/
qwen3-4b-instruct-dpo-from-sft-4096_u-10bei_9datasets

like

0

text-generation

u-10bei/dpo-dataset-qwen-cot

Qwen/Qwen3-4B-Instruct-2507

Views

No views yet

qwen3-4b-instruct-dpo-from-sft-4096_u-10bei_9datasets

This repository provides a LoRA adapter trained with Direct Preference Optimization (DPO).

Base model: Qwen/Qwen3-4B-Instruct-2507
Initialized from SFT LoRA: /content/lora_sft_best
Preference dataset: u-10bei/dpo-dataset-qwen-cot

This repo contains LoRA adapter weights only. Load it on top of the base model.

Training Objective

Align responses with preferred outputs using DPO (pairwise preference learning).

Training Configuration

Method: DPO (TRL) + Unsloth (4-bit QLoRA base loading)
Epochs: 1
Learning rate: 5e-07
Beta: 0.1
Max sequence length: 1024
Max prompt length: 512

Usage (Load LoRA)

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_id = "Qwen/Qwen3-4B-Instruct-2507"
lora_id = "WassyO/qwen3-4b-instruct-dpo-from-sft-4096_u-10bei_9datasets"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(base, lora_id)

prompt = "Your question here"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & Terms (IMPORTANT)

Training data: u-10bei/dpo-dataset-qwen-cot
Compliance: Users must follow the dataset terms and the base model license.