TB2 evaluation progress

Reproduction contents

• The patched sf-agent source used for evaluation.
• The 89 Terminal-Bench 2.0 task definitions, tests, instructions, and Dockerfiles.
• Evaluation, serving, node-preparation, and result-summary scripts.
• The tmax dependency files and the exact upstream tmax commit.
• The Harbor prebuilt-image compose override.
• A detailed partial-reward and RL-effect analysis in ANALYSIS.md.

Installation

• Linux, Docker, and an eight-GPU node.
• Python 3.12 and uv.
• A GPU-compatible PyTorch/vLLM installation (the original environment used vLLM 0.19.1).
• Qwen3.6-27B and/or Ornith-1.0-35B weights.
• The 89 prebuilt image tarballs, or locally built images from the included task Dockerfiles.

Important configuration distinction

v1: generous-resource configuration

• --n-concurrent 8
• --override-cpus 24
• --override-memory-mb 49152
• --agent-timeout-multiplier 12
• temperature=1.0, k=1

Native-resource configuration

• Task CPU, memory, and timeout from each task.toml
• --n-concurrent 20
• --agent-timeout-multiplier 1
• temperature=1.0, k=1

Current interpretation

• Under v1, Qwen base and RL checkpoints are close (roughly 58–60%), so the measured RL gain is small.
• Under native limits, Qwen RL checkpoints score above the Qwen base, but the absolute baseline is too low. This improvement is configuration-dependent and is not yet sufficient evidence for a card-level RL gain.
• Ornith later checkpoints decline under native limits; the full checkpoint curve is still being evaluated.
• Single-sample (k=1, temperature 1.0) task outcomes are noisy. A final claim should use repeated runs (k>=3, preferably k=5) under one fixed, validated configuration.

Failure-trajectory findings

• Most failed trajectories call mark_task_complete despite verifier failure.
• Qwen failures commonly stop after roughly 25–27 steps, below the 100-step cap.
• Many failures are executable solutions with incorrect output or semantics, rather than infrastructure crashes.
• About one quarter of failed tasks are near misses with at least 70% of verifier subtests passing.
• Across matched Qwen base/step100/step140 tasks, RL produced both stable wins and stable regressions; the net pass@1 gain was small under v1.

Operational notes

• vLLM requires the matching bundled CUDA libraries; the serving environment used unset LD_LIBRARY_PATH.
• Merged checkpoints are loaded as language-model-only Qwen VLM architectures using --hf-overrides and copied base config.json.
• Docker images are loaded from 89 local tarballs and retagged by scripts/node_prep.sh.
• Re-run node_prep.sh before every new Harbor job on a reused node. Previous runs may remove upstream image tags.
• Prune stale Docker networks when a recycled node reports all predefined address pools have been fully subnetted.
• The Harbor prebuilt compose file was locally configured with pull_policy: never to avoid Docker Hub pulls.

Files

• scripts/eval_harbor.sh: generic local/Hugging Face checkpoint evaluator.
• scripts/eval_harbor_qwenbase.sh: original v1 generous-resource Qwen base evaluator.
• scripts/eval_merged.sh: earlier sf-agent merged-checkpoint evaluator.
• scripts/node_prep.sh: local image loading, retagging, network setup.
• scripts/install.sh: installs the pinned tmax and bundled sf-agent source.
• scripts/summarize_results.py: reports accuracy, reward mean, and verifier-subtest partial mean.
• src/sf-agent/: patched sf-agent source and all 89 TB2 task definitions.
• vendor/tmax/: tmax dependency metadata and original evaluation notes.
• patches/: Harbor compose override used for local-only prebuilt images.
• ANALYSIS.md: partial-reward, RL-effect, and trajectory-failure analysis.
• hold_1node.sbatch: four-day, eight-GPU Slurm hold job.
• results/progress.tsv: live snapshot at publication time.