
Optimizer step	130
In-training validation success rate	70.3%
Backbone	`Qwen/Qwen2.5-1.5B-Instruct` (1.54B params, bf16)
Environment	ALFWorld `alfworld/AlfredTWEnv`
Hardware	2x NVIDIA A100 80GB

Standalone evaluation at this step

128 ALFWorld valid_seen tasks, 3 seeds, temperature 0.4, max 50 env steps:

	value
success rate	58.33% ± 1.61
base `Qwen2.5-1.5B-Instruct`	2.86% ± 0.37
mean episode length	29.9 (base: 49.1)
format rate	96.5% (base: 98.1%)

Training curve

In-training validation success rate (val/success_rate), measured every 5 optimizer steps on a 32-task validation batch (shown every 10 steps below). This is a smaller and noisier probe than the 128-task standalone evaluation.

optimizer step	success rate (%)
0	6.2
10	7.8
20	13.3
30	14.8
40	16.4
50	34.4
60	35.2
70	36.7
80	43.0
90	50.0
100	59.4
110	64.8
120	63.3
130	70.3
140	68.0
150	64.8

Run history

This run (ppo_qwen2.5_1.5b_v3) is the third PPO attempt. An earlier attempt collapsed — its step-130 checkpoint scored 9.90% with a 1.3% action format rate, i.e. the policy stopped emitting parseable actions. The run published here was restarted from checkpoints twice (at step 30 and step 80) for operational reasons, not because of divergence; the validation curve is continuous across those boundaries.

Hyperparameters

group	parameter	value
RL	`adv_estimator`	gae
RL	`actor.use_kl_loss` / `kl_loss_coef`	True / 0.01
RL	`kl_loss_type`	low_var_kl
RL	invalid action penalty	True, coef 0.1
Optim	actor learning rate	1e-6
Optim	critic learning rate	1e-5
Optim	`ppo_mini_batch_size`	256
Data	`train_batch_size`	128
Data	group size (`env.rollout.n`)	1
Data	episodes per step	128 x 1 = 128
Data	`max_prompt_length` / `max_response_length`	2048 / 512
Rollout	engine / TP / `gpu_memory_utilization`	vLLM / 2 / 0.6
Rollout	`val_kwargs.temperature`	0.4
Train	`save_freq` / `test_freq`	10 / 5

Files

path	contents
`model.safetensors`, `config.json`, tokenizer files	bf16 weights, merged from the FSDP shards with `scripts/model_merger.py`. `lm_head` is omitted because `tie_word_embeddings=true` (same layout as the base Qwen release).
`training_state/actor/*.pt`	FSDP-sharded actor weights, Adam optimizer state, RNG/scheduler state

training_state/ lets you resume RL training from this exact optimizer step. The shards are written for world_size=2; resuming on a different number of GPUs requires resharding.

Other checkpoints from this run

step 10 · step 20 · step 30 · step 60 · step 70 · step 80 · step 130 · step 140 · step 150

Not every step was retained: the run used trainer.max_actor_ckpt_to_keep, so some intermediate checkpoints were pruned during training.

Caveats

Do not use is_correct / pass@1 from verl logs; they are hardcoded to 1.0. Use val/success_rate.
Evaluated on ALFWorld valid_seen only; valid_unseen was not run.
A StraTA reproduction is published alongside these models (chanyoungkim/strata-qwen2.5-1.5b-alfworld-step*). It uses a different agent scaffold (140 eval tasks, temperature 0.7, 7168-token prompts, strategy-conditioned prompting) and is therefore not directly comparable to the numbers above.

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "chanyoungkim/ppo-qwen2.5-1.5b-alfworld-step130"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(repo)