This model was fine-tuned using DPO to improve multi-turn agent task performance by learning preferences from the u-10bei/sft_alfworld_trajectory_dataset_v2,u-10bei/sft_alfworld_trajectory_dataset_v4 dataset. The DPO training process aims to increase the likelihood of generating 'chosen' responses and decrease the likelihood of 'rejected' responses for given prompts.

Training Configuration (DPO)

Base SFT Model: rokugatsu/LLM2025_Advanced_5_v4
DPO Dataset: u-10bei/sft_alfworld_trajectory_dataset_v2,u-10bei/sft_alfworld_trajectory_dataset_v4
DPO Method: Direct Preference Optimization (DPO)
Max sequence length: 2048
Epochs: 0.125
Learning rate: 2e-06
Beta parameter (DPO loss): 0.1

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

model_id = "rokugatsu/LLM2025_Advanced_DPO_5_v5"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16, # Use bfloat16 if your GPU supports it
    device_map="auto",
)
# The model is already merged, so no need for PeftModel.from_pretrained(model, adapter)

# Example for inference (assuming you have a chat_template)
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)

outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & Terms (IMPORTANT)

Training data: u-10bei/sft_alfworld_trajectory_dataset_v2,u-10bei/sft_alfworld_trajectory_dataset_v4

Dataset License: MIT License. This dataset is used and distributed under the terms of the MIT License. Compliance: Users must comply with the MIT license (including copyright notice) and the base model's original terms of use.