Note: Training was conducted with a 4-bit loading setup (QLoRA-style). Some training artifacts may reference an Unsloth 4-bit base path, but the canonical base model ID for this run is shown above.

This repository contains LoRA adapter weights only. The base model must be loaded separately.

Training Objective

This adapter is trained via supervised fine-tuning (SFT) to improve structured output accuracy (JSON / YAML / XML / TOML / CSV).

CoT masking: enabled (SFT_MASK_COT=1)
Output learn mode: after_marker
Output markers:
- Output:, OUTPUT:, Final:, Answer:, Result:, Response:

Training Configuration

Key settings (from meta.json):

Base model: Qwen/Qwen3-4B-Instruct-2507
Dataset: u-10bei/structured_data_with_cot_dataset_512_v4
Max sequence length: 512
Epochs: 2
Learning rate: 2e-5
LoRA: r=64, alpha=128, dropout=0
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Train batch size (per device): 2
Eval batch size (per device): 2
Gradient accumulation: 8
Eval steps: 50
Logging steps: 10
Save steps: 100
Warmup ratio: 0.05
Weight decay: 0.01
Validation ratio: 0.05
Seed: 3407
Precision: bf16 (USE_BF16=true)
Hardware (logged): NVIDIA A100-SXM4-80GB

Evaluation (logged)

Eval loss: 0.7212356
Eval runtime: 17.3598s
Eval samples/sec: 13.191
Eval steps/sec: 6.625

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "your_id/qwen3-4b-sft-v0-cotv4-lr2e5-ep2"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.bfloat16,  # matches USE_BF16=true
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

Sources & Terms (IMPORTANT)

Training data: u-10bei/structured_data_with_cot_dataset_512_v4 Dataset License/Terms: Please refer to the dataset card on Hugging Face and comply with its terms. Base Model Terms: You must comply with the base model’s original license/terms of use.

Repository License: MIT

sabia0080/qwen3-4b-sft-v0-cotv4-lr2e5-ep2

qwen3-4b-sft-v0-cotv4-lr2e5-ep2

Training Objective

Training Configuration

Evaluation (logged)

Usage

Sources & Terms (IMPORTANT)

sabia0080/
qwen3-4b-sft-v0-cotv4-lr2e5-ep2