A 5.67B-parameter SWE agent model trained from random init in 182.2 B200-hours (8×B200, 22.7 h wall-clock) on multi-turn SWE agent trajectories only — no web-text pretraining stage, no instruction tuning, no RL.

At $5 per B200-hour that is roughly $911 of compute. It resolves 11.0% of SWE-bench Verified (pass@1 over K=10 samples).

This is a research artifact from a compute-scaling study of SWE agents, not a general-purpose assistant. See Limitations.

The ladder

Model	Params	Compute	pass@1
`nanoswe/nanoswe-depth16-260812`	0.54B	2.44 B200-h	0.54%
`nanoswe/nanoswe-depth20-260812`	0.90B	6.32 B200-h	2.19%
`nanoswe/nanoswe-12h-260812`	1.38B	12 B200-h	5.0%
`nanoswe/nanoswe-depth32-260812`	2.82B	56.96 B200-h	8.47%
`nanoswe/nanoswe-192h-260812`	5.67B	182.2 B200-h	11.0%

Quickstart

The architecture (NanoChatForCausalLM) is out-of-tree, so install the vLLM plugin first — after that vllm serve works with no flags and no trust_remote_code:

bash

pip install "nanoswe-vllm @ git+https://github.com/nanosweb/nanoswe-vllm@v0.1.0"
vllm serve nanoswe/nanoswe-192h-260812 --max-model-len 32768

bash

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "nanoswe/nanoswe-192h-260812",
  "messages": [{"role": "user", "content": "List the files in the current directory."}],
  "temperature": 0.7, "max_tokens": 2048}'

In-process:

python

import nanoswe_vllm  # noqa: F401  — registers the architecture
from vllm import LLM, SamplingParams

llm = LLM("nanoswe/nanoswe-192h-260812", max_model_len=32768)
out = llm.chat([{"role": "user", "content": "Fix the failing test in tests/test_utils.py"}],
               SamplingParams(temperature=0.7, max_tokens=2048))
print(out[0].outputs[0].text)

The plugin wheel is also mirrored in this repo (nanoswe_vllm-0.1.0-py3-none-any.whl) if you would rather not install from git.

AutoModelForCausalLM is not supported — there is no transformers-native implementation of this architecture. AutoTokenizer.from_pretrained works normally.

Chat format

Nine special tokens, IDs 32759–32767; the template is in tokenizer_config.json and is applied automatically by /v1/chat/completions:

<|bos|><|user_start|>{system}\n\n{user}<|user_end|><|assistant_start|>{assistant}<|assistant_end|>

A leading system message is merged into the first user turn. Generation stops at <|assistant_end|> (id 32763).

Evaluation


Benchmark	SWE-bench Verified
pass@1	11.0%
Sampling	K=10 decodes/instance, temperature 0.7
Scaffold	mini-swe-agent, step limit 100, 2048 max tokens/turn
Serving	vLLM 0.20.1, bf16

Sphinx instances are graded with the harness's leaked tox.ini hunk stripped from the submission: the image ships that edit uncommitted, git add -A sweeps it into every patch, and the grader reverse-applies it — which silently scored every sphinx submission as unresolved. Seed-to-seed variation across identical recipes is ≈0.5 pp, so differences below ~1 pp against another single-seed run are not resolved.

Training


Parameters	5,667,427,610
Layers × width	42 × 2688 (21 heads, head_dim 128, no GQA)
Context	32,768
Vocab	32,768 (nanoswe RustBPE, web-text-trained)
Compute	182.2 B200-hours on 8×B200 (≈$911 at $5/B200-h)
Steps × batch	11,295 × 2,097,152 tokens ≈ 23.7B tokens
Precision	bf16 weights, fp8 (tensorwise) matmuls
Objective	next-token CE, token-normalized, on assistant spans

Architecture is derived from nanochat and keeps its distinctive pieces: a smear gate (each layer mixes in the previous token's pre-block hidden state), value embeddings on alternating layers, a mid-trunk backout, per-layer residual/x0 lambdas, QK-norm with a 1.2 split scale, parameterless RMSNorm, relu² MLP, logit softcap at 15, untied embeddings, RoPE θ=1e6, and an SSSL window pattern (three short-window layers per full-attention layer).

Data. Three curriculum phases over nanoswe/nanoswe-trajs-260812 with two transition segments: pI (4,406 steps) swe-smith-extra 50.5% / swe-zero 22.1% / swe-smith-extra-short 11.4% / swe-zero-extra 9.2% / swe-hero-extra 6.8%; pII (1,633 steps) swe-smith-extra verified-only 74.8% / swe-zero 25.2%; pIII (3,922 steps) mini-coder-trajs-400k 56.3% / swe-zero 43.7%.

Contamination

The corpus is audited against a strict rule: no training trajectory may come from any of the 12 SWE-bench source repositories, forks included — not merely "no eval instance is copied into training". Provenance is recoverable from every row's instance_id, which encodes the upstream owner__repo, with 0 unparseable rows corpus-wide, so a clean scan is genuinely clean rather than unprovable.

A full scan of all 1,587,217 trajectories (10,548 distinct repos) on 2026-06-21 found 4,516 violating rows (1,370 distinct instances) from 7 of the 12 repos; they were purged, leaving 1,582,701 rows with 0 violations. This model was trained on 2026-06-25/26, i.e. entirely on the cleaned corpus.

Consequently there is no eval-instance overlap either: no row from any SWE-bench source repo survives in training, so no evaluated instance_id can appear.

Limitations

Not a chat model. It was trained exclusively on SWE agent trajectories from random init; on general prompts it produces agent-shaped output (shell commands, patches) or degenerates. It has no safety tuning of any kind.
Needs a matching scaffold. Useful output requires an agent loop resembling the training trajectories — a bash-style tool interface with observations fed back as user turns.
Absolute capability is low. 11.0% pass@1 is far below frontier coding models; the point of the artifact is the compute/accuracy scaling point, not the accuracy.
Serving requires the plugin, which reads vLLM attention-metadata internals and is validated against vLLM 0.20.x.

License

MIT.