Lego-RL-Qwen3.5-35B-A3B · OpenHands SDK · 200K
Qwen3.5-35B-A3B trained with online RL inside the unmodified OpenHands SDK harness, at 200K context, on 2,699 real repository issues whose own test suites produce the reward.
SWE-bench Verified: 64.0 → 70.4 (+6.4) — no reward model, no reference-patch similarity, no harness rewrite.
This checkpoint is the OpenHands SDK production run of
Lego-RL (
Faithful · Reliable · Observable), the RL component of the
LegoX series. The agent solves a real issue in a real repository inside a fresh sandbox, the task's own verifier suite decides
{0, 1}, and the trajectory the harness actually produced — token ids, masks, log-probs and MoE expert routes captured
inside the serving path — becomes the gradient step.
Why harness-native training
The scaffold is part of the environment, not the policy. The same weights score very differently depending on which harness runs them, so training under a rewritten control flow optimizes for a deployment you never ship:
| Model | OpenHands SDK | Claude Code | OpenCode |
|---|
| Qwen3.5-35B-A3B (starting point) | 64.0 | 62.4 | 57.2 |
| Qwen3.6-35B-A3B (next-gen base) | 67.4 | 63.4 | 60.6 |
| KAT-Coder-V2.5-Dev (post-trained Qwen3.6) | 67.0 | 66.8 | 64.8 |
| Lego-RL-Qwen3.5-35B-A3B | 70.4 | 68.2 | 66.6 |
SWE-bench Verified (%), one shared protocol: temperature 0.7, 200 turns, 200K context.
Each Lego-RL column is a
separate run trained in that harness from the same starting checkpoint, the same 2,699 tasks and the same 3 epochs.
This repository is the OpenHands SDK run (70.4). The Claude Code and OpenCode runs are reported in the paper; their checkpoints are being released into the
Lego-RL collection. Across the three harnesses RL adds
+6.4 / +5.8 / +9.4 — more than the entire Qwen3.5 → Qwen3.6 base-model generation (+3.4 / +1.0 / +3.4).
Quick start
1. Serve with vLLM
1vllm serve Lego-X/qwen3_5_35b_a3b_ohsdk_200k_rl \
2 --served-model-name vllm_model \
3 --tensor-parallel-size 4 \
4 --enable-expert-parallel \
5 --max-model-len 262144 \
6 --gpu-memory-utilization 0.9 \
7 --enable-chunked-prefill --enable-prefix-caching \
8 --dtype bfloat16 \
9 --enable-auto-tool-choice \
10 --tool-call-parser qwen3_coder \
11 --host 0.0.0.0 --port 8000
[!IMPORTANT]
--tool-call-parser qwen3_coder is not optional. The model was rolled out and trained with this parser; serving it behind hermes (or any other parser) silently degrades tool-call formatting — this exact misconfiguration cost us a full cluster-day during development.
2. Drive it with OpenHands SDK
The model is a coding agent policy: it expects a harness that gives it a repository, a shell and file-edit tools, and it expects room to run. Point OpenHands SDK at the vLLM endpoint as an OpenAI-compatible provider:
1from openhands.sdk import LLM, Agent, Conversation # openhands-sdk >= 1.14
2from pydantic import SecretStr
3
4llm = LLM(
5 model="openai/vllm_model",
6 base_url="http://localhost:8000/v1",
7 api_key=SecretStr("dummy"),
8 temperature=0.7,
9 max_input_tokens=200_000,
10 max_output_tokens=32_768,
11)
12agent = Agent(llm=llm)
13conversation = Conversation(agent=agent, workspace="/path/to/repo")
14conversation.send_message("Fix the failing test in tests/test_units.py")
15conversation.run()
Consult your openhands-sdk version's own docs for the exact constructor signature — the training runs used the openhands-sdk runtime image c-oh-sdk-1.33.0, and evaluation used c-oh-sdk-1.14.0.
Run it with 200 turns and 200K context. The RL policy learned to spend turns: mean turns per trajectory grew from 46.6 to 83.1. Capping turns at the usual 50–100 truncates the second half of its trajectories systematically and costs most of the gain.
3. Reproduce the evaluation
1git clone https://github.com/LegoX/Lego-RL.git && cd Lego-RL
2bash scripts/setup_env.sh
3cp scripts/eval/_template.env scripts/eval/configs/my_eval.env # set MODEL_PATH, DATASET_PATH, kubeconfig
4bash scripts/eval/eval.sh scripts/eval/configs/my_eval.env
Sandboxed execution and verifier rewards come from
Harbor; see the
evaluation docs.
Training
| |
|---|
| Starting checkpoint | Qwen/Qwen3.5-35B-A3B (sparse MoE, 256 experts, 8 active) |
| Harness | OpenHands SDK, unmodified — a thin adapter, not a fork |
| Tasks | Lego-X/Lego-RL-2699 — 2,699 real repository issues, converted from GAIR/OpenSWE |
| Reward | each task's own test suite, run in a fresh sandbox: {0, 1}. No reward model, no patch similarity, no LLM judge |
| Algorithm | GSPO (sequence-level surrogate), group-relative advantage over G = 8 rollouts per task |
| Batch | 64 prompts × 8 responses = 512 trials/step; 3 epochs = 126 steps |
| Optimizer | lr 1e-6 constant, KL loss 1e-3 (low-var), clip [3e-4, 4e-4], rollout temperature 1.0 |
| Context | 200K (30K prompt + 170K response), 200 turns |
| Backend | VeOmni FSDP, Ulysses SP = 8, R3 rollout routing replay; fully-async with partial rollout (staleness 1) |
| Hardware | 3 nodes × 8 GPUs (2 training + 1 rollout) |
Three properties made this run trainable rather than merely runnable:
- Faithful capture. An in-process proxy at the serving boundary records token ids, masks, log-probs and expert routes at generation time — nothing is re-tokenized from a stored transcript, which matters because the harness rewrites its own history (compaction, re-serialized tool arguments, injected reminders). Replaying rollout-time MoE routing (R3) lifted rollout↔training correlation from
0.9946 to 0.9993; across all three production runs the median per-step correlation stayed ≥ 0.998.
- Reliable execution. Reward-hacking paths are closed inside the environment, per phase: no network and no fix-bearing git history during the agent phase, both restored for grading; test files are withheld and any edit to them rolled back. Trajectories that died of infrastructure faults (2.4% here) are masked out of the loss rather than scored as failures.
- Difficulty-filtered task pool. 36,884 OpenSWE candidates → 2,699 (12.4% of the executable pool), keeping only tasks solved 1–3 times out of 4 by a reference rollout model. An unfiltered pool never improved at all: 72.7% of its tasks were never solved and 13.4% always were, so under six tasks in a hundred produced any in-group reward variance.
The training set is disjoint from SWE-bench Verified at both the repository and the instance level.
What RL actually changed
420 trajectories from the first step vs. 420 from the last, in the OpenHands SDK run:
| Behavior | Before | After |
|---|
| Re-reads a file after editing it | 73.6% | 98.1% |
| Files inspected before the first edit | 3.45 | 6.92 |
| Runs the test suite on its own initiative | 85.0% | 93.6% |
| Mean turns per trajectory | 46.6 | 83.1 |
| Malformed tool calls | 1.07% | 0.15% |
| pass@8 | 83.2 | 87.9 |
| pass⁸ (all 8 attempts solve it) | 28.3 | 39.4 |
The honest reading: the model became more reliable on tasks it could already sometimes solve, more than it became capable of new ones. pass⁸ moved 11.1 points while pass@8 moved 4.7. Mid-trajectory error recovery barely moved (63.9% → 66.8%) — a terminal binary reward gives no independent signal for recovering from a failed command.
Intended use and limitations
Use it as an agent policy, not as a chat model: it was optimized inside a harness that hands it a repository, a shell, and file-editing tools, and its behavior (long inspect-edit-verify loops, heavy test invocation) only pays off in that setting.
- Harness. Trained in OpenHands SDK. It also runs under Claude Code / OpenCode / any OpenAI- or Anthropic-compatible harness, but the paper trains a separate policy per harness, and each does best in the one it was trained in.
- Budget. 200K context and 200 turns; response lengths roughly doubled during training (43.5K → 90.9K tokens per trajectory). Short budgets truncate it.
- Domain. Python-heavy repository issue-resolution, in the SWE-bench/OpenSWE distribution. No claim outside it.
- Inherited base behavior. Safety, multilingual and general-knowledge behavior come from Qwen3.5-35B-A3B and were not targeted by this RL. The architecture retains the base model's vision tower; training and evaluation were text-only.
- Sandbox it. The policy writes files and executes shell commands on purpose. Run it in a container.
Related work in the LegoX series
Acknowledgement
Built on
verl (trainer + rollout) and
Harbor (sandboxed execution + verifier reward), with
OpenHands as the harness.
Citation
1@misc{du2026legorlharnessnativereinforcementlearning,
2 title={LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
3 author={Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai and Shaowei Wang and Jierun Chen and Chaofan Tao and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong and Xiaohui Li and Haoli Bai},
4 year={2026},
5 eprint={2608.17393},
6 archivePrefix={arXiv},
7 primaryClass={cs.AI},
8 url={https://arxiv.org/abs/2608.17393},
9}
