from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "zhiyuanhucs/qwen3.5-9b-bc-sft-delta-force",
    revision="step-800",                    # change to any branch above
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("zhiyuanhucs/qwen3.5-9b-bc-sft-delta-force", revision="step-800", trust_remote_code=True)

IMPORTANT: Disable thinking at inference

Qwen3.5 chat_template.jinja prepends <think>\n to every assistant turn. Training target is <think>\n\n</think>\n\n<|action_start|>...<|action_end|> (empty thinking shell). At inference you must use the non-thinking prefix so the model directly continues the action body:

python

# Using ms-swift (Megatron-SWIFT) inference engine
engine.template._response_prefix = None
engine.template.enable_thinking = False     # makes response_prefix = '<think>\n\n</think>\n\n'

Or in raw transformers:

python

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)

Without this, the 9B model's strong reasoning prior will keep thinking inside <think>...</think> and likely exhaust max_new_tokens before emitting <|action_start|>.

Output format

Every assistant reply is a 200-ms action wrapped in custom tokens:

<|action_start|>X Y Z ; k1 ; k2 ; k3 ; k4 ; k5 ; k6<|action_end|>

X Y Z = cumulative mouse delta + wheel, quantized at 5-pixel step
6 key groups separated by ; (one per 33-ms sub-frame)
letter keys upper-case (W/A/D/Shift/Ctrl/LB/RB/MB/Q/F/...)

4 new special tokens added to tokenizer (vocab size 248320): <|action_start|>, <|action_end|>, <|thought_start|>, <|thought_end|>

Training recipe

Backend: Megatron-SWIFT (mcore-bridge 1.2.1, megatron-core 0.16.1)
Hardware: 4× H200 141GB
Parallelism: TP=4, PP=1, DP=1, Sequence Parallel=true
Batch: micro=1, global=8 (grad_acc=8)
Max len: 32768 (20 images × 1024 vision tokens + text)
LR: 5e-6 cosine → 5e-7, 5% warmup, 3 epochs
Weight decay: 0.01
Precision: bf16
Recompute: full / uniform / n=1
MTP: num_layers=1, scale=0.1 (Multi-Token Prediction, DeepSeek-V3 style)
Optimizer: AdamW (β1=0.9, β2=0.95)
Total steps: 1275

Total training time: 8 h 18 min on 4× H200.

Evaluation

Format-correctness on 50 val samples, inference with --no_thinking (see scripts/42_check_format.py in the training repo):

Metric	step-800 (best)	step-1200	step-1275
contains `<\|action_start\|>`	100%	100%	100%
contains `<\|action_end\|>`	100%	100%	100%
well-formed pair	100%	100%	100%
parses `X Y Z` as 3 ints	100%	100%	100%
uses `;` separator	100%	100%	100%
exact_match (body)	16%	14%	14%
avg edit distance	5.9	5.9	5.9
key-set IoU (micro)	67%	69%	69%
xyz-exact	20%	20%	20%
xyz-direction-correct	34%	34%	34%

9B vs 4B comparison (both best-ckpt, 50 val samples)

	4B step-1200	9B step-800
format	100%	100%
exact_match	14%	16%
avg edit distance	6.2	5.9

Observation

Format learning is saturated by step-200 on this 9B model — 100% well-formed actions from very early, never regresses.
Content plateaus around step-200-800: extending training from 800 → 1275 gives no meaningful improvement (LR has decayed to 5e-7).
Mouse delta regression is mode-collapsed to common values (0, ±10, ±50). This is a CE-loss limitation for continuous targets and affects 4B and 9B equally — scaling parameters from 4B → 9B does not help here.
Keystrokes are learned well (67-70% set IoU). High-frequency keys (W/Shift/D/A/LB/RB/Q/F) are matched; rare keys (Ctrl/F/Tab) are sometimes missed.

To break through the 16% exact-match ceiling, the recommended next step is RL / rejection sampling using edit-distance + keystroke-F1 as reward, not more SFT steps.

Training code

Full training pipeline (dataset preprocessing, training scripts, checkpoint-format verification, scaling to 30B/70B) is at: https://github.com/game-agent-open/training