You can deploy this model and start earning money today!

Qwen3-4B-StructEval-Iterative-Alignment

Name: dpo-qwen-cot-merged20
Availability: InStock

This model is a highly optimized version of Qwen3-4B-Instruct-2507, specifically developed for the StructEval-T task. It features a sophisticated iterative alignment process to maximize performance in structured data reasoning and Chain-of-Thought (CoT) generation.

This repository contains full-merged 16-bit weights. No adapter loading is required.

Training Pipeline

Unlike standard fine-tuning, this model has undergone a four-stage iterative training process to ensure precise alignment and deep reasoning capabilities:

Stage 1: SFT - Foundation building with structured CoT trajectories.
Stage 2: DPO - First alignment to preference reasoning paths.
Stage 3: SFT - Knowledge reinforcement and format refinement.
Stage 4: DPO - Final preference optimization for high-fidelity structured outputs.

Training Objective

The model is engineered to excel in:

Complex Reasoning: Enhanced Chain-of-Thought processing for structural evaluation.
Structural Integrity: Strict adherence to complex data formats (JSON, Tables, etc.).
Consistency: High-reliability outputs across iterative multi-turn interactions.

Training Configuration (Final Stage)

Method: Iterative DPO (Direct Preference Optimization)
Base model: unsloth/Qwen3-4B-Instruct-2507
Epochs: 1
Learning rate: 3e-06
Beta: 0.05
Max sequence length: 2560
Platform: Trained with Unsloth

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "moushi21/dpo-qwen-cot-merged20"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

Sources & Terms (IMPORTANT)

Training data:

u-10bei/structured_data_with_cot_dataset_512_v2
u-10bei/structured_data_with_cot_dataset_512_v4
u-10bei/structured_data_with_cot_dataset_512_v5
u-10bei/structured_data_with_cot_dataset_512
u-10bei/structured_data_with_cot_dataset_v2
u-10bei/structured_data_with_cot_dataset
u-10bei/dpo-dataset-qwen-cot

Dataset License: MIT License. This dataset is used and distributed under the terms of the MIT License. Compliance: Users must comply with the MIT license (including copyright notice) and the base model's original terms of use.