You can deploy this model and start earning money today!

qwen3-4b-sft-v13-mixed-rlvmr

Name: qwen3-4b-sft-v13-mixed-rlvmr
Availability: InStock

This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using LoRA + Unsloth.

This repository contains LoRA adapter weights only. The base model must be loaded separately.

Training Objective

This adapter is trained to improve multi-turn agent task performance on ALFWorld (household tasks) and DBBench (database operations).

All training data is tagged with RLVMR meta-reasoning tags (planning, explore, reflection, monitor) to enable subsequent reinforcement learning with verifiable meta-reasoning rewards.

Loss is applied to all assistant turns in the multi-turn trajectory, enabling the model to learn environment observation, action selection, tool use, and recovery from errors.

Data Composition

Source	Samples	Description
DBBench v1-v4 (RLVMR tagged)	~3,060	Multi-turn SQL agent trajectories
DBBench v4 (upsampled)	~1,000	Additional v4 samples
LogicCat (RLVMR tagged)	~4,030	Single-turn MySQL CoT (MIT license)
ALFWorld v5 (RLVMR tagged)	~2,500	Household task trajectories
ALFWorld v4 (RLVMR tagged)	~2,500	Household task trajectories
ALFWorld v3 (RLVMR tagged)	~1,000	Household task trajectories
Total	~14,090	DBBench ~57% / ALFWorld ~43%

Training Configuration

Base model: Qwen/Qwen3-4B-Instruct-2507
Method: LoRA (full precision base, bf16)
Max sequence length: 2048
Epochs: 5
Learning rate: 8e-05
LoRA: r=64, alpha=128, targets=all layers
Effective batch size: 32 (bs=2 x grad_accum=16)
Weight decay: 0.01
Warmup ratio: 0.05
Scheduler: cosine

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "Kaito-F/qwen3-4b-sft-v13-mixed-rlvmr"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

Sources & Terms (IMPORTANT)

Training data:

Kaito-F/alfworld_trajectory_v5_rlvmr_tagged_v2
Kaito-F/alfworld_trajectory_v4_rlvmr_tagged_v2
Kaito-F/alfworld_trajectory_v3_rlvmr_tagged_v2
Kaito-F/dbbench_v4_rlvmr_tagged
Kaito-F/dbbench_v3_rlvmr_tagged
Kaito-F/dbbench_v2_rlvmr_tagged
Kaito-F/dbbench_v1_rlvmr_tagged
Kaito-F/logiccat_dbbench_rlvmr_tagged

This repository does NOT redistribute the dataset. Users must comply with the dataset license and base model terms.