It is part of a Terminal-Bench training loop, but the reported result below is only a four-task raw-command TBLite proxy gate. It is not a full Terminal-Bench score and should not be cited as a full benchmark win.

Method

base model: allenai/tmax-2b
initialization: burtenshaw/tmax2-rawcmd-tmaxsft-lora-lr1e5-r8-s80
method: continued LoRA SFT with TRL SFTTrainer
target format: raw bash command completions
loss: completion-only SFT
learning rate: 2e-6
training steps: 100
seed: 180

The training mixture used 512 replay command rows from successful allenai/tmax-sft traces plus 320 generated non-eval semantic oracle rows. The generated rows covered balanced log rollup, JSON rollup, protein cleaning, and schedule-window task families. Generated commands were executed and deterministically verified before inclusion.

The TBLite proxy evaluation tasks were kept eval-only and were not used as training trajectories or solutions.

Proxy Evaluation

Evaluation used a separate raw-command TBLite proxy over:

log-summary
jsonl-aggregator
protein-sequence
schedule-vacation

Model or adapter	Strict success	Success rate	Valid command rate	Avg turns
`allenai/tmax-2b`	0/4	0.00	0.7083	6.0
`burtenshaw/tmax2-rawcmd-tmaxsft-lora-lr1e5-r8-s80`	1/4	0.25	0.9000	5.0
this adapter	2/4	0.50	0.8889	4.5

This adapter improved strict proxy success over the prior released TMax2 adapter, but it slightly reduced valid-command rate relative to that adapter. The promotion criterion was strict proxy success.

Reproducibility

training job: SLURM 22305377
eval job: SLURM 22305734
run root: /fsx/benjamin_burtenshaw/post-training-agent-experiments/2026-06-24-tmax2-semantic-oracle-sft
eval summary: /fsx/benjamin_burtenshaw/post-training-agent-experiments/2026-06-24-tmax2-semantic-oracle-sft/evals/combined_summary.json
Trackio dashboard: https://huggingface.co/spaces/burtenshaw/terminal-bench-loop-trackio

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "allenai/tmax-2b"
adapter_id = "burtenshaw/tmax2-semantic-oracle-tmaxsft-lora-lr2e6-s100"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Caveats

This is a research adapter for terminal-agent post-training experiments. It was selected by a small proxy gate and should be validated with stronger held-out Terminal-Bench-style tasks before broader use.