Persistent cloud controller

The AutoCAD Windows EC2 instances are workers, not benchmark controllers. Each worker hosts one interactive AutoCAD desktop and its loopback broker. The model loop, SSM port-forward process, trace writer, evaluator dispatch, and batch state must run on a separate persistent controller.

Running that control plane on a developer machine means sleep, network loss, or reboot can pause every rollout while leaving costly Windows workers alive. For long batches, use a persistent controller in your own environment. The AWS example uses Amazon Linux, encrypted storage, no inbound security-group rules, and Systems Manager as its administration path.

Deployment contract

The controller uses an operator-owned instance role; temporary local AWS credentials are not copied to it.
Model API credentials are delivered with the operator's secret manager and exposed only through a root-owned systemd EnvironmentFile.
autocad-bench-resume.timer scans every benchmark run every two minutes and starts the one-shot resume service only when reconciliation is due. An empty scan is a healthy idle result, not a crash or restart condition.
Each active batch incrementally publishes changed state, compact turns, and immutable frames to its own S3 prefix every three seconds. The five-minute autocad-bench-sync.timer remains a low-priority full-archive fallback.
Every batch participates in a shared, file-lock-backed controller semaphore. The default host-wide ceiling is 65 live rollouts across all batches. It also reserves slots for pre-upgrade wrapper processes until they exit, so an old batch and a new batch cannot accidentally exceed the machine-wide ceiling.
Shared execution runs all sandbox supervisors and async model loops inside one controller Python runtime. Rollouts still have independent Windows VMs, authenticated SSM tunnels, broker sessions, output roots, credentials, and cleanup, but no longer pin two duplicate Python interpreters apiece.
Long rollouts persist each frame immediately and retain only the current screenshot in memory. Exact recovery uses one atomically replaced resume-checkpoint.json; append-only viewer traces contain compact artifact references instead of repeating the entire growing request history. At most eight heavyweight evaluators run concurrently. Snapshot and resume services run with low I/O priority so durability cannot starve live tunnels.
autocad-bench-evaluation-retry.timer retries transient vision-judge failures from persisted DWG and inspection artifacts; it never holds a Windows worker open for a model-only retry.
EC2 termination protection is enabled on the controller.
A resumed sandbox always uses allow_recovery=False and skips the normal exact-task handoff reset. If admission cannot prove that the existing broker is healthy, it fails closed and leaves the worker running.

The checked-in examples are under infra/cloud-controller/: an EC2 trust policy, placeholder controller policy, bootstrap script, and batch systemd unit. Every account, role, bucket, config, and output path must be supplied by the operator.

Exact resume behavior

autocad-bench-resume discovers rollout directories that lack result.json or contain a recoverable infrastructure result, asks the selected infrastructure plugin for their original worker IDs, and reconnects only to available workers. The built-in AWS plugin resolves EC2 instance IDs from batch tags or legacy lifecycle logs. The command accepts a single batch root or the shared runs/benchmarks parent, so newly created runs are covered without editing a systemd unit. The pilot validates the saved task, attempt, provider, model, and broker session identities before issuing input. Model-terminal results also remain eligible while trusted AutoCAD evaluation is incomplete. A retry_pending vision judgment is excluded because it no longer needs a licensed Windows session and is owned by the evaluation retry timer.

The Windows broker retains the authoritative absolute action counter and current screenshot. The rolling exact-resume checkpoint retains the last provider request and OpenAI's compact history, OpenRouter's append-only input prefix and committed output items, Anthropic's byte-bounded/compacted Messages prefix, or Mantle Kimi K2.5's append-only Chat Completions prefix and committed tool call. OpenRouter, Anthropic, and Mantle Chat resume checksum-verify and rehydrate referenced images before restoring the conversation. Resume reconstructs only the interrupted tool result from the broker counter, translates new loop-relative action indices back to absolute broker indices, and appends to the existing trace. It never replays an action blindly. Legacy compact traces without the checkpoint retain the prior last-JSONL-row fallback.

The controller capacity regression is executable without AWS:

python -m autocad_bench.orchestration.controller_stress \
  --workers 65 --actions 30 --frame-bytes 131072 \
  --max-rss-growth-mb 1024

If the old controller had already checkpointed and released AutoCAD, the recovery path retrieves the checkpointed DWG and event logs without sending more model actions, then evaluates that partial or completed artifact. A released session without a checkpoint cannot be resumed and must receive a fresh replacement rollout.

Worker cleanup is deliberately asymmetric:

successful rollout + artifact retrieval + automatic evaluation: request worker termination;
reconnect, retrieval, or evaluation failure: keep the worker unchanged for diagnosis or retry.

Execution and evaluation are independent contracts. A model-complete rollout is never relabeled as a model failure because the vision API timed out. Trusted inspection and deterministic scoring finish first; transient vision failures are persisted as retry_pending and retried by the controller with exponential backoff. Infrastructure errors remain eligible for exact-session resume even if an earlier controller already wrote a terminal result.json.

Commands

Inspect a recovery plan without touching a worker:

uv run autocad-bench-resume \
  --run-root runs/benchmarks/<terra-run> \
  --run-root runs/benchmarks/<luna-run> \
  --dry-run

Execute the same plan under systemd on the controller:

sudo systemctl start autocad-bench-resume.timer
sudo systemctl status autocad-bench-resume.timer
jq . /opt/autocad-bench/runs/cloud-resume-state.json

The dashboard remains a client/viewer. It can run on a Mac without owning benchmark execution; closing the viewer has no effect on rollouts.