nanoswe-depth32-260812

A 2.82B-parameter SWE agent model trained from random init in 56.96 B200-hours (8×B200, 7.0 h wall-clock) on multi-turn SWE agent trajectories only — no web-text pretraining stage, no instruction tuning, no RL.

At $5 per B200-hour that is roughly $285 of compute. It resolves 8.47% of SWE-bench Verified (pass@1 over K=10 samples).

This is a research artifact from a compute-scaling study of SWE agents, not a general-purpose assistant. See Limitations.

The ladder

Model	Params	Compute	pass@1
nanoswe/nanoswe-depth16-260812	0.54B	2.44 B200-h	0.54%
nanoswe/nanoswe-depth20-260812	0.90B	6.32 B200-h	2.19%
nanoswe/nanoswe-12h-260812	1.38B	12 B200-h	5.0%
nanoswe/nanoswe-depth32-260812	2.82B	56.96 B200-h	8.47%
nanoswe/nanoswe-192h-260812	5.67B	182.2 B200-h	11.0%

Quickstart

The architecture (NanoChatForCausalLM) is out-of-tree, so install the vLLM plugin first — after that vllm serve works with no flags and no trust_remote_code:

pip install "nanoswe-vllm @ git+https://github.com/nanosweb/nanoswe-vllm@v0.1.0"
vllm serve nanoswe/nanoswe-depth32-260812 --max-model-len 32768

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "nanoswe/nanoswe-depth32-260812",
  "messages": [{"role": "user", "content": "List the files in the current directory."}],
  "temperature": 0.7, "max_tokens": 2048}'

In-process:

import nanoswe_vllm  # noqa: F401  — registers the architecture
from vllm import LLM, SamplingParams

llm = LLM("nanoswe/nanoswe-depth32-260812", max_model_len=32768)
out = llm.chat([{"role": "user", "content": "Fix the failing test in tests/test_utils.py"}],
               SamplingParams(temperature=0.7, max_tokens=2048))
print(out[0].outputs[0].text)

The plugin wheel is also mirrored in this repo (nanoswe_vllm-0.1.0-py3-none-any.whl) if you would rather not install from git.

AutoModelForCausalLM is not supported — there is no transformers-native implementation of this architecture. AutoTokenizer.from_pretrained works normally.

Chat format

Nine special tokens, IDs 32759–32767; the template is in tokenizer_config.json and is applied automatically by /v1/chat/completions:

<|bos|><|user_start|>{system}\n\n{user}<|user_end|><|assistant_start|>{assistant}<|assistant_end|>

A leading system message is merged into the first user turn. Generation stops at <|assistant_end|> (id 32763).

Evaluation

Benchmark	SWE-bench Verified
pass@1	8.47%
Sampling	K=10 decodes/instance, temperature 0.7
Scaffold	mini-swe-agent, step limit 100, 2048 max tokens/turn
Serving	vLLM 0.20.1, bf16

Sphinx instances are graded with the harness's leaked tox.ini hunk stripped from the submission: the image ships that edit uncommitted, git add -A sweeps it into every patch, and the grader reverse-applies it — which silently scored every sphinx submission as unresolved. Seed-to-seed variation across identical recipes is ≈0.5 pp, so differences below ~1 pp against another single-seed run are not resolved.

Training

Parameters	2,818,575,450
Layers × width	32 × 2048 (16 heads, head_dim 128, no GQA)
Context	32,768
Vocab	32,768 (nanoswe RustBPE, web-text-trained)
Compute	56.96 B200-hours on 8×B200 (≈$285 at $5/B200-h)
Steps × batch	8,532 × 1,572,864 tokens ≈ 13.4B tokens
Precision	bf16 weights, fp8 (tensorwise) matmuls
Objective	next-token CE, token-normalized, on assistant spans

Architecture is derived from nanochat and keeps its distinctive pieces: a smear gate (each layer mixes in the previous token's pre-block hidden state), value embeddings on alternating layers, a mid-trunk backout, per-layer residual/x0 lambdas, QK-norm with a 1.2 split scale, parameterless RMSNorm, relu² MLP, logit softcap at 15, untied embeddings, RoPE θ=1e6, and an SSSL window pattern (three short-window layers per full-attention layer).

Data. The tail recipe over nanoswe/nanoswe-trajs-260812: a single pretraining phase on the broad SWE-smith / swe-zero mixture followed by a short annealed tail on the mini-coder-trajs-400k + swe-zero mixture.

Contamination

The corpus is audited against a strict rule: no training trajectory may come from any of the 12 SWE-bench source repositories, forks included — not merely "no eval instance is copied into training". Provenance is recoverable from every row's instance_id, which encodes the upstream owner__repo, with 0 unparseable rows corpus-wide, so a clean scan is genuinely clean rather than unprovable.

A full scan of all 1,587,217 trajectories (10,548 distinct repos) on 2026-06-21 found 4,516 violating rows (1,370 distinct instances) from 7 of the 12 repos; they were purged, leaving 1,582,701 rows with 0 violations. This model was trained on 2026-07-08/09, i.e. entirely on the cleaned corpus.

Consequently there is no eval-instance overlap either: no row from any SWE-bench source repo survives in training, so no evaluated instance_id can appear.

Limitations

• Not a chat model. It was trained exclusively on SWE agent trajectories from random init; on general prompts it produces agent-shaped output (shell commands, patches) or degenerates. It has no safety tuning of any kind.
• Needs a matching scaffold. Useful output requires an agent loop resembling the training trajectories — a bash-style tool interface with observations fed back as user turns.
• Absolute capability is low. 8.47% pass@1 is far below frontier coding models; the point of the artifact is the compute/accuracy scaling point, not the accuracy.
• Serving requires the plugin, which reads vLLM attention-metadata internals and is validated against vLLM 0.20.x.

License

MIT.