docs/deterministic-sandbox-contract.md

# Deterministic AutoCAD sandbox contract

This document defines the only supported path from an AutoCAD benchmark AMI to
a model-controlled desktop. “Deterministic” means that every accepted VM proves
the same immutable environment and handoff invariants, every ambiguous mutation
is replay-safe, and every wait/retry is bounded. It does not mean Windows, AWS,
or AutoCAD can never fail; it means those failures either occur before model
usage or are retained as explicit rollout failures.

## Release status

- Active source/runtime contract: `windows-autocad-2019-v10`.
- Candidate source/runtime contract: `windows-autocad-2019-v11`.
- Worker images and rollback images are built and owned by the operator.

The harness accepts v10 for historical result reproduction and v11 for new
runs, while requiring each configuration and live image to agree exactly.
Broker v9 images are intentionally rejected before agent handoff.

## State machine

```text
launch
  -> EC2 running + both status checks OK
  -> SSM online
  -> current-boot interactive readiness
  -> authenticated exact-process tunnel (up to 3 fresh tunnel processes)
  -> exact task lease/reset (up to 2 wrapper attempts; broker recovery inside each)
  -> verified sequence-0 screenshot
  -> agent handoff
  -> actions/controller-owned checkpoint save/evidence retrieval
  -> session release
  -> instance termination request
```

Runtime admission deliberately does not run a disposable AutoCAD reset. The
immutable broker/session checks are followed directly by the exact task reset,
which removes the former gap where a smoke passed but the real reset failed.
Release-image validation may still use a disposable smoke as an offline AMI gate.

Before the model child starts, a reset may use the broker's two clean AutoCAD
launch attempts, then one wrapper-level replay of the same idempotent task reset.
A newly launched clone may then be discarded and replaced once. These layers are
bounded and occur before model/API usage. After `sandbox_agent_started`, the
sample is never restarted or silently replaced.

## Required handoff evidence

The model receives the VM only when all of the following are true:

1. EC2 is `running`; instance and system checks are both `ok`.
2. SSM reports the exact instance online.
3. the readiness timestamp and startup ID belong to the current Windows boot;
4. the installed deployment ID, manifest checksum, broker binary checksum,
   broker version, and input-driver version all match;
5. Administrator is logged into the DCV console, the broker is in that same
   nonzero interactive Windows session, and the desktop is exactly 1920x1080;
6. the PyAutoGUI worker has successfully injected its real pointer prewarm on
   `WinSta0\\Default`;
7. authenticated `/health` through the SSM tunnel identifies the exact broker
   PID and Windows session that passed readiness; and
8. the rollout's own replay-safe lease/reset has passed the full AutoCAD handoff
   path: clean process boundary, restored template, rendered ribbon, maximized
   window, AutoCAD foreground ownership, no unknown modal popup, stable initial
   pixels, and a sequence-zero PNG with a valid checksum.

The benchmark wall clock begins only after item 8. After that point, AutoCAD is
not required to stay maximized or foreground during model work. The model does
not save or choose a filename. At checkpoint, the trusted broker focuses the
leased AutoCAD process, cancels transient commands, verifies that ActiveX is
bound to that exact process, and invokes `Save` or `SaveAs` to a
controller-owned attempt path. It retries the COM save five times and accepts
the checkpoint only after a stable, signature-valid DWG exists.

## Bounded recovery and retry matrix

| Boundary | Bound | Safe retry identity / behavior |
|---|---:|---|
| EC2 `RunInstances` | 6 requests | retries quota-release, capacity, throttling, and transient signing failures |
| Other EC2/SSM calls | 3 requests | read calls or idempotent start/stop/terminate calls only |
| AutoCAD reset handoff | 2 complete launches | kill process tree, restore template, retry the entire handoff |
| Current-clone admission | 2 checks | first check plus one diagnostic broker/process-tree recovery |
| Fresh clone admission | 2 clones by default, max 3 | failed clone is terminated before a replacement is launched; no model call occurred |
| SSM port-forward startup | 3 fresh processes | each process must make authenticated `/health` pass within 45 seconds |
| SSM port-forward during rollout | 8 restarts | one stable local port; watchdog kills the complete process group and reconnects |
| Exact task reset | 2 wrapper attempts | same lease and idempotent reset identity; each broker reset retains its own two clean AutoCAD handoffs |
| Broker HTTP operation | 8 transport attempts | one operation deadline, fresh HTTP connection, unchanged idempotency identity |
| Direct-provider pre-action response | 10 consecutive failures | OpenAI and Anthropic retries are safe because no desktop action has occurred; any valid provider response resets the transport-error streak, while malformed/missing tool output has its own consecutive limit and gets a larger output cap |
| Evidence retrieval | 300 seconds total | checkpoint, DWG, events, frame events, and status each have a sub-cap |
| Release | 60 seconds | idempotent release; EC2 termination still follows if it times out |
| Batch straggler cleanup | 1 reconciliation pass | one tag-scoped `TerminateInstances`, with bounded transient AWS retries and no polling |

No loop polls forever. No retry expands the number of evaluated model samples.

## Exactly-once mutation rules

- Lease identity is `(task_id, attempt_id)`. A lost lease response is replayed
  from the existing active lease; a different identity conflicts.
- Reset uses stable `request_id=reset-{session_id}` and returns the cached initial
  screenshot on replay. Reusing another reset key conflicts.
- Every action uses a stable SHA-256 request ID derived from attempt ID, model
  tool-use ID, and batch offset, plus the next expected action index. An identical
  retry returns the cached action result without input or logging side effects.
- Checkpoint and release are state-idempotent. Artifact, event, frame-event, and
  status retrievals are read-only.
- A broker HTTP error is never retried as if it were a transport failure.

This removes the unsafe choice between duplicating an input and abandoning an
input whose response was lost.

## Action outcome rules

The complete action batch is validated before its first input. Coordinates must
fit the live display, typed text must be printable ASCII, `done` must be last,
and the non-empty batch must fit the remaining 5,000-action rollout budget. There
is no separate per-tool-call action-count cap.
Local validation errors are returned to the model without touching the desktop.

After input is injected, whole-desktop animation is diagnostic rather than
fatal. If pixels do not settle within the post-action window, the broker caches
and returns the freshest screenshot with
`post_action_visual_stable=false`. This is essential: throwing after input commit
would create an ambiguous mutation. Initial handoff stability remains a hard
admission requirement.

An input receipt proves only that input reached the interactive Windows desktop.
It never proves that AutoCAD semantically accepted the command. The model must
inspect the resulting screenshot, restore focus when necessary, and recover from
the visible state.

## Failure classification

- `admission_failed`: no model request was made; clone may receive its one fresh
  replacement.
- `transport_exhausted`: supervised tunnel and bounded broker retries could not
  restore communication; preserve partial trace/evidence and fail the sample.
- `model_contract_error`: provider repeatedly omitted or malformed the required
  action call; preserve its billable usage and fail the sample.
- `action_rejected`: the full batch was rejected before injection and reported
  back to the model, or an unexpected Windows-side rejection failed closed.
- `artifact_unavailable`: final DWG save/checkpoint evidence was absent; the task
  fails even if screenshots looked correct.
- `cleanup_error`: bounded evidence/release/instance cleanup failed; retain it in
  result/lifecycle state and run tag-scoped fleet reconciliation.

## Observed scale failures and their v10 controls

| Observed failure | v10 control |
|---|---|
| SSM tunnel died and all later broker calls failed | watchdog restart on stable port, process-group kill, per-outage restart allowance, fresh HTTP connections, and up to 48 deadline-bounded transport attempts with the same idempotency key |
| reset smoke returned HTTP 500 with orphan broker/Python/AutoCAD processes | scheduled-task stop/start gates, bounded tree kill, verified empty runtime/attempt directories, one recovery only |
| post-action animation caused `desktop_unstable` after input | return and cache freshest frame with explicit unstable receipt |
| lost action response risked duplicate keyboard/mouse input | stable request ID, expected index, cached exact replay |
| lost lease/reset response left an unusable active desktop | replay-safe lease identity and cached reset response |
| non-ASCII typing failed after earlier batch items ran | whole-batch printable-ASCII validation before injection plus broker defense-in-depth |
| OpenAI/Anthropic response omitted/truncated `computer_action` | ten consecutive safe pre-action retries, usage aggregation, larger output allowance, explicit format repair; a valid tool response resets the streak |
| transient AWS signature failure aborted preflight/control plane | bounded retry of transient signed AWS calls |
| terminated vCPU had not returned to quota | bounded `RunInstances` retry for `VcpuLimitExceeded` |
| one bad cold clone required a manual retry pass | one automatic terminate-and-replace admission attempt before model handoff |
| finalization hung for many minutes | 300-second total evidence budget and 60-second release bound |
| cleanup API error leaked tagged instances | idempotent terminate retries plus one batch tag reconciliation pass |

## Release gate

Source tests are necessary but not sufficient. A v10 AMI becomes benchmarkable
only after the Windows gates in
[`incremental-ami-v10-runbook.md`](./incremental-ami-v10-runbook.md) pass on an
untouched clone. Until then, v10 is a source candidate and v9 remains only a
rollback image, not a compatible runtime for the hardened harness.