docs/cloud-controller.md

# Persistent cloud controller

The AutoCAD Windows EC2 instances are workers, not benchmark controllers. Each worker hosts one interactive AutoCAD desktop and its loopback broker. The model loop, SSM port-forward process, trace writer, evaluator dispatch, and batch state must run on a separate persistent controller.

Running that control plane on a developer machine means sleep, network loss, or
reboot can pause every rollout while leaving costly Windows workers alive. For
long batches, use a persistent controller in your own environment. The AWS
example uses Amazon Linux, encrypted storage, no inbound security-group rules,
and Systems Manager as its administration path.

## Deployment contract

- The controller uses an operator-owned instance role; temporary local AWS
  credentials are not copied to it.
- Model API credentials are delivered with the operator's secret manager and
  exposed only through a root-owned systemd `EnvironmentFile`.
- `autocad-bench-resume.timer` scans every benchmark run every two minutes and
  starts the one-shot resume service only when reconciliation is due. An empty
  scan is a healthy idle result, not a crash or restart condition.
- Each active batch incrementally publishes changed state, compact turns, and
  immutable frames to its own S3 prefix every three seconds. The five-minute
  `autocad-bench-sync.timer` remains a low-priority full-archive fallback.
- Every batch participates in a shared, file-lock-backed controller semaphore.
  The default host-wide ceiling is 65 live rollouts across all batches. It also
  reserves slots for pre-upgrade wrapper processes until they exit, so an old
  batch and a new batch cannot accidentally exceed the machine-wide ceiling.
- Shared execution runs all sandbox supervisors and async model loops inside
  one controller Python runtime. Rollouts still have independent Windows VMs,
  authenticated SSM tunnels, broker sessions, output roots, credentials, and
  cleanup, but no longer pin two duplicate Python interpreters apiece.
- Long rollouts persist each frame immediately and retain only the current
  screenshot in memory. Exact recovery uses one atomically replaced
  `resume-checkpoint.json`; append-only viewer traces contain compact artifact
  references instead of repeating the entire growing request history. At most
  eight heavyweight evaluators run concurrently. Snapshot and resume services
  run with low I/O priority so durability cannot starve live tunnels.
- `autocad-bench-evaluation-retry.timer` retries transient vision-judge failures from persisted DWG and inspection artifacts; it never holds a Windows worker open for a model-only retry.
- EC2 termination protection is enabled on the controller.
- A resumed sandbox always uses `allow_recovery=False` and skips the normal exact-task handoff reset. If admission cannot prove that the existing broker is healthy, it fails closed and leaves the worker running.

The checked-in examples are under `infra/cloud-controller/`: an EC2 trust
policy, placeholder controller policy, bootstrap script, and batch systemd
unit. Every account, role, bucket, config, and output path must be supplied by
the operator.

## Exact resume behavior

`autocad-bench-resume` discovers rollout directories that lack `result.json` or
contain a recoverable infrastructure result, asks the selected infrastructure
plugin for their original worker IDs, and reconnects only to available workers.
The built-in AWS plugin resolves EC2 instance IDs from batch tags or legacy
lifecycle logs. The command accepts a single batch root or the shared
`runs/benchmarks` parent, so newly created runs are covered without editing a
systemd unit. The pilot validates the saved task, attempt, provider, model, and
broker session identities before issuing input.
Model-terminal results also remain eligible while trusted AutoCAD evaluation is
incomplete. A `retry_pending` vision judgment is excluded because it no longer
needs a licensed Windows session and is owned by the evaluation retry timer.

The Windows broker retains the authoritative absolute action counter and current screenshot. The rolling exact-resume checkpoint retains the last provider request and OpenAI's compact history, OpenRouter's append-only input prefix and committed output items, Anthropic's byte-bounded/compacted Messages prefix, or Mantle Kimi K2.5's append-only Chat Completions prefix and committed tool call. OpenRouter, Anthropic, and Mantle Chat resume checksum-verify and rehydrate referenced images before restoring the conversation. Resume reconstructs only the interrupted tool result from the broker counter, translates new loop-relative action indices back to absolute broker indices, and appends to the existing trace. It never replays an action blindly. Legacy compact traces without the checkpoint retain the prior last-JSONL-row fallback.

The controller capacity regression is executable without AWS:

```bash
python -m autocad_bench.orchestration.controller_stress \
  --workers 65 --actions 30 --frame-bytes 131072 \
  --max-rss-growth-mb 1024
```

If the old controller had already checkpointed and released AutoCAD, the recovery path retrieves the checkpointed DWG and event logs without sending more model actions, then evaluates that partial or completed artifact. A released session without a checkpoint cannot be resumed and must receive a fresh replacement rollout.

Worker cleanup is deliberately asymmetric:

- successful rollout + artifact retrieval + automatic evaluation: request worker termination;
- reconnect, retrieval, or evaluation failure: keep the worker unchanged for diagnosis or retry.

Execution and evaluation are independent contracts. A model-complete rollout is
never relabeled as a model failure because the vision API timed out. Trusted
inspection and deterministic scoring finish first; transient vision failures are
persisted as `retry_pending` and retried by the controller with exponential
backoff. Infrastructure errors remain eligible for exact-session resume even if
an earlier controller already wrote a terminal `result.json`.

## Commands

Inspect a recovery plan without touching a worker:

```bash
uv run autocad-bench-resume \
  --run-root runs/benchmarks/<terra-run> \
  --run-root runs/benchmarks/<luna-run> \
  --dry-run
```

Execute the same plan under systemd on the controller:

```bash
sudo systemctl start autocad-bench-resume.timer
sudo systemctl status autocad-bench-resume.timer
jq . /opt/autocad-bench/runs/cloud-resume-state.json
```

The dashboard remains a client/viewer. It can run on a Mac without owning benchmark execution; closing the viewer has no effect on rollouts.