Note:
DPO training is performed on top of a merged SFT model.
Details of the SFT training and data are documented in the model card of
kuririrn/qwen3-4b-agent-trajectory_alf_admissible-lora-constraint_gen-dist_allign.
Datasets
SFT starting point
The starting point for this DPO adapter is an SFT-trained adapter:
Please refer to its model card for detailed information about the
supervised fine-tuning data and configuration.
DPO dataset
kuririrn/alfworld_roundwise_dpo_v2
This dataset provides round-wise DPO samples with fields:
"messages": system + multi-turn history (trajectory)
"chosen": more desirable next action
"rejected": less desirable next action
"reason": categorical label describing why "rejected" is undesirable
(for example, "loop", "task_succeeded_negative", "implausible_location").
The DPO loss is computed only on the next turn (chosen vs rejected)
conditioned on the same messages history.
Prompting and Output Format
The model is used as an environment-interacting agent. A typical prompt
is constructed as a chat-style conversation:
system: explains the agent role and task
user: provides the latest environment observation and instructions
assistant: responds with a single next action in a canonical format
Actions are emitted as a single line:
Use the "ACTION:" tag (uppercase).
Follow with a structured action string compatible with the environment.
Examples:
ACTION: go to countertop 1
ACTION: take apple 1 from countertop 1
ACTION: open drawer 2
ACTION: task succeeded
Internally, the DPO data normalizes historical tags such as "Act:" and
"Think:" to "ACTION:" and "THOUGHT:" to align with the evaluation
environment and reduce errors caused by format mismatch.
Usage
Below is a minimal example for loading the adapter on top of the base model
using transformers and peft (shown as plain text for safety in this README):
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "your_username/your_dpo_repo"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
base,
torch_dtype=torch.float16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)
# Now you can generate actions conditioned on the environment history.
