subfolder	data split	reward model	final reward
`scalar-firsterror-20260817`	first_error (2,662)	Bradley-Terry scalar head	+4.608
`discrete-firsterror-20260817`	first_error (2,662)	judge-token margin	−2.273
`scalar-preerror-20260818`	pre_error (2,669)	Bradley-Terry scalar head	+4.982
`discrete-preerror-20260818`	pre_error (2,669)	judge-token margin	−5.053

Reward numbers are not comparable across rows. The two reward models emit different scales, and the two splits are different states. Compare policies by downstream task performance, not by these values.

first_error — the screen immediately after the first erroneous action; the policy must recover.
pre_error — the screen before the erroneous action; the policy must avoid the mistake.

python

import torch
from transformers import AutoModelForImageTextToText
from peft import PeftModel

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-2B-Instruct",
    revision="89644892e4d85e24eaac8bacfd4f463576704203",
    dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(
    base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818")

Qwen3-VL is a vision-language model, so AutoModelForCausalLM / AutoPeftModelForCausalLM do not work — use AutoModelForImageTextToText. After loading, assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392.

All four were trained from Gyubeum/AndroidFlux_RL_Train_Test @ 4545695c26951a4884e8f18e6468d69cbdf9d174 for 1 epoch with identical hyperparameters — only the reward model and the data split differ. See each subfolder's README.md for full details.

Intended comparison (run elsewhere, on held-out data): base Qwen/Qwen3-VL-2B-Instruct vs. scalar-GRPO vs. discrete-GRPO.