Reproduction of StraTA (Strategic Trajectory Abstraction, arXiv:2605.06642, code xxyQwQ/StraTA @ 592238b) on ALFWorld with Qwen/Qwen2.5-1.5B-Instruct.

This checkpoint: optimizer step 50 of 150.


Optimizer step	50 / 150
Validation success rate at this step	18.57% (ALFWorld `valid_seen`, 140 tasks)
Base model success rate	0.71%
Backbone	`Qwen/Qwen2.5-1.5B-Instruct` (1.54B params, bf16)
Environment	ALFWorld (AgentGym HTTP interface)
Hardware	2x NVIDIA A100 80GB
Wall-clock (full 150 steps)	~44 hours

Training curve

Validation success rate on the full ALFWorld valid_seen split (140 tasks), measured every 10 optimizer steps during training. Step 0 is the untrained base model.

optimizer step	success rate (%)
0	0.71
10	0.71
20	3.57
30	1.43
40	14.29
50	18.57
60	6.43
70	22.86
80	37.14
90	42.14
100	43.57
110	50.71
120	69.29
130	85.00
140	83.57
150	86.43

Final evaluation (step 150, 2 seeds)

Run with evaluate-strata-alfworld-text-qwen2.5-1.5b.sh: val_only, 140 tasks, temperature 0.7 / top_p 0.8 / top_k 20, max 50 env steps.

subtask	seed 1	seed 2	mean ± std
pick	97.14	97.14	97.14 ± 0.00
clean	92.59	92.59	92.59 ± 0.00
pick2	91.67	87.50	89.58 ± 2.95
heat	81.25	81.25	81.25 ± 0.00
cool	76.00	72.00	74.00 ± 2.83
look	76.92	69.23	73.08 ± 5.44
ALL	87.86	85.71	86.79 ± 1.52

Auxiliary metrics: action format_rate 1.000 (both seeds), mean episode length 13.2–14.2 steps (base model: 49.7).

Hyperparameters

All algorithm hyperparameters follow the paper: 150 steps, batch size 16, 4 strategies per task, 8 rollouts per strategy, oversampling ratio sigma=8, aggregation ratio delta=0.5, length penalty threshold lambda=0.5, self-judgment reward weight kappa=0.1.

group	parameter	value
StraTA	`num_stras_per_run` (strategies per task)	4
StraTA	`num_trajs_per_stra` (rollouts per strategy)	8
StraTA	`num_cands_per_stra` (oversampling ratio, sigma)	8
StraTA	`delta_coef` (aggregation ratio)	0.5
StraTA	`lambda_coef` (length penalty threshold)	0.5
StraTA	`kappa_coef` (self-judgment reward weight)	0.1
StraTA	`use_diverse_stras` / `use_self_judgment`	True / True
StraTA	`use_length_reward` / `use_format_reward`	True / True
StraTA	strategy embedding model	`sentence-transformers/all-MiniLM-L6-v2` (CPU)
RL	`adv_estimator`	grpo
RL	`algorithm.gamma`	0
RL	`kl_ctrl.kl_coef`	0.001
RL	`actor.use_kl_loss`	False
RL	`clip_ratio_high`	0.28
RL	`loss_agg_mode`	seq-mean-token-sum
RL	`entropy_coeff`	0
RL	`stepwise_advantage`	enabled, `per_step`
Optim	learning rate	1e-6
Optim	`ppo_mini_batch_size` (global)	1024
Optim	`use_dynamic_bsz` / `ppo_max_token_len_per_gpu`	True / 32768
Data	`train_batch_size` (tasks per step)	16
Data	episodes per step	16 x 4 x 8 = 512
Data	`max_prompt_length` / `max_response_length`	7168 / 1024
Data	env `num_histories`	2
Data	max env steps per episode	50
Rollout	engine / temperature	vLLM 0.11.0 (async) / 1.0
Eval	`val_kwargs`	n=1, temperature 0.7, top_p 0.8, top_k 20
Train	total steps	150 (`total_training_steps=151`)

Infrastructure deltas

This run reproduces the upstream 7B recipe on 2x A100 80GB. A full diff of the resolved Hydra config against train-strata-alfworld-text-qwen2.5-7b.sh shows 6 differing keys out of 533, all infrastructure -- no algorithm or optimization parameter was changed:

key	upstream 7B	this run	effect on results
`trainer.n_gpus_per_node`	8	2	none -- verl computes `ppo_mini_batch_size *= rollout.n` then `//= world_size`, so the global mini-batch is invariant
`actor.fsdp_config.param_offload`	True	False	none -- CPU offload only
`actor.fsdp_config.optimizer_offload`	True	False	none
`ref.fsdp_config.param_offload`	True	False	none
`actor.checkpoint.save_contents`	`[hf_model]`	`[model,optimizer,extra,hf_model]`	none -- what gets written
`trainer.save_freq`	50	10	none -- checkpoint cadence

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "chanyoungkim/strata-qwen2.5-1.5b-alfworld-step050"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(repo)

The model expects the StraTA ALFWorld prompt format: it first emits a plan inside <strategy>...</strategy>, then acts, emitting one action per turn inside <action>...</action>.

Files

path	contents
`model.safetensors`, `config.json`, tokenizer files	bf16 weights, ready for `from_pretrained`. `lm_head` is omitted because `tie_word_embeddings=true` (same layout as the base Qwen release).
`training_state/model_world_size_2_rank_{0,1}.pt`	FSDP-sharded fp32 weights
`training_state/optim_world_size_2_rank_{0,1}.pt`	Adam optimizer state
`training_state/extra_state_world_size_2_rank_{0,1}.pt`	RNG and scheduler state
`training_state/fsdp_config.json`	FSDP layout descriptor

training_state/ lets you resume RL training from this exact optimizer step. Note the shards are written for world_size=2 (2 GPUs, FSDP). Resuming on a different number of GPUs requires resharding. To resume, place the files back as checkpoints/<exp>/models/global_step_50/actor/ and point verl at that directory.

The published model.safetensors is a bf16 cast of the fp32 master weights; the original fp32 tensors are preserved inside training_state/.

All checkpoints from this run

step 10 · step 20 · step 30 · step 40 · step 50 · step 60 · step 70 · step 80 · step 90 · step 100 · step 110 · step 120 · step 130 · step 140 · step 150

Caveats

Do not use is_correct or pass@1 from the logs. Both are hardcoded to 1.0 in the workflow and are meaningless. The correct metric is val/average/metrics/success_rate.
Evaluation uses the StraTA inference protocol (one strategy, then one trajectory conditioned on it; no diversity selection, no self-judgment). This is what strata_workflow.run() does when rollout_engine.validate is set, and it matches evaluate-strata-*.sh.
Numbers here are not directly comparable to models evaluated under a different agent scaffold (different prompt format, context window, or sampling temperature).
The evaluation set is ALFWorld valid_seen (140 tasks, all of them -- not a subsample). valid_unseen was not evaluated.

Citation

bibtex

@article{xue2026strata,
  title={StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction},
  author={Xue, Xiangyuan and Zhou, Yifan and Wang, Zidong and Tang, Shengji and Torr, Philip and Ouyang, Wanli and Bai, Lei and Yin, Zhenfei},
  journal={arXiv preprint arXiv:2605.06642},
  year={2026}
}