SFT adapter trained on a 50/50 mix of Goodbuy (sim) and Bestbuy (real) tasks.
Part of the Halluminate blog post on sim-to-real transfer for web agents.
Tasks:
goodbuy-bestbuy-mirrored-tasks.
Training logs:
W&B report.
1from peft import PeftModel
2from transformers import Qwen2_5_VLForConditionalGeneration
3
4base = Qwen2_5_VLForConditionalGeneration.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
5model = PeftModel.from_pretrained(base, "victoria-kp/qwen25vl-7b-simtoreal-sft-mixed")
QLoRA (4-bit NF4 base, LoRA rank 64, alpha 128, attention + MLP layers) on
success-only Claude Fable teacher trajectories from the mixed 128+128 training
split. One example per agent step: observation in (screenshot, task
instruction, BID element list, history), teacher reasoning and JSON action
out. Learning rate 5e-5 cosine with 5% warmup, effective batch size 2, max
sequence length 4096. This is the validation-loss-optimal checkpoint
(step 1500, epoch 0.81).
