Deterministic AutoCAD sandbox contract

This document defines the only supported path from an AutoCAD benchmark AMI to a model-controlled desktop. “Deterministic” means that every accepted VM proves the same immutable environment and handoff invariants, every ambiguous mutation is replay-safe, and every wait/retry is bounded. It does not mean Windows, AWS, or AutoCAD can never fail; it means those failures either occur before model usage or are retained as explicit rollout failures.

Release status

Active source/runtime contract: windows-autocad-2019-v10.
Candidate source/runtime contract: windows-autocad-2019-v11.
Worker images and rollback images are built and owned by the operator.

The harness accepts v10 for historical result reproduction and v11 for new runs, while requiring each configuration and live image to agree exactly. Broker v9 images are intentionally rejected before agent handoff.

State machine

launch
  -> EC2 running + both status checks OK
  -> SSM online
  -> current-boot interactive readiness
  -> authenticated exact-process tunnel (up to 3 fresh tunnel processes)
  -> exact task lease/reset (up to 2 wrapper attempts; broker recovery inside each)
  -> verified sequence-0 screenshot
  -> agent handoff
  -> actions/controller-owned checkpoint save/evidence retrieval
  -> session release
  -> instance termination request

Runtime admission deliberately does not run a disposable AutoCAD reset. The immutable broker/session checks are followed directly by the exact task reset, which removes the former gap where a smoke passed but the real reset failed. Release-image validation may still use a disposable smoke as an offline AMI gate.

Before the model child starts, a reset may use the broker's two clean AutoCAD launch attempts, then one wrapper-level replay of the same idempotent task reset. A newly launched clone may then be discarded and replaced once. These layers are bounded and occur before model/API usage. After sandbox_agent_started, the sample is never restarted or silently replaced.

Required handoff evidence

The model receives the VM only when all of the following are true:

EC2 is running; instance and system checks are both ok.
SSM reports the exact instance online.
the readiness timestamp and startup ID belong to the current Windows boot;
the installed deployment ID, manifest checksum, broker binary checksum, broker version, and input-driver version all match;
Administrator is logged into the DCV console, the broker is in that same nonzero interactive Windows session, and the desktop is exactly 1920x1080;
the PyAutoGUI worker has successfully injected its real pointer prewarm on WinSta0\\Default;
authenticated /health through the SSM tunnel identifies the exact broker PID and Windows session that passed readiness; and
the rollout's own replay-safe lease/reset has passed the full AutoCAD handoff path: clean process boundary, restored template, rendered ribbon, maximized window, AutoCAD foreground ownership, no unknown modal popup, stable initial pixels, and a sequence-zero PNG with a valid checksum.

The benchmark wall clock begins only after item 8. After that point, AutoCAD is not required to stay maximized or foreground during model work. The model does not save or choose a filename. At checkpoint, the trusted broker focuses the leased AutoCAD process, cancels transient commands, verifies that ActiveX is bound to that exact process, and invokes Save or SaveAs to a controller-owned attempt path. It retries the COM save five times and accepts the checkpoint only after a stable, signature-valid DWG exists.

Bounded recovery and retry matrix

Boundary	Bound	Safe retry identity / behavior
EC2 `RunInstances`	6 requests	retries quota-release, capacity, throttling, and transient signing failures
Other EC2/SSM calls	3 requests	read calls or idempotent start/stop/terminate calls only
AutoCAD reset handoff	2 complete launches	kill process tree, restore template, retry the entire handoff
Current-clone admission	2 checks	first check plus one diagnostic broker/process-tree recovery
Fresh clone admission	2 clones by default, max 3	failed clone is terminated before a replacement is launched; no model call occurred
SSM port-forward startup	3 fresh processes	each process must make authenticated `/health` pass within 45 seconds
SSM port-forward during rollout	8 restarts	one stable local port; watchdog kills the complete process group and reconnects
Exact task reset	2 wrapper attempts	same lease and idempotent reset identity; each broker reset retains its own two clean AutoCAD handoffs
Broker HTTP operation	8 transport attempts	one operation deadline, fresh HTTP connection, unchanged idempotency identity
Direct-provider pre-action response	10 consecutive failures	OpenAI and Anthropic retries are safe because no desktop action has occurred; any valid provider response resets the transport-error streak, while malformed/missing tool output has its own consecutive limit and gets a larger output cap
Evidence retrieval	300 seconds total	checkpoint, DWG, events, frame events, and status each have a sub-cap
Release	60 seconds	idempotent release; EC2 termination still follows if it times out
Batch straggler cleanup	1 reconciliation pass	one tag-scoped `TerminateInstances`, with bounded transient AWS retries and no polling

No loop polls forever. No retry expands the number of evaluated model samples.

Exactly-once mutation rules

Lease identity is (task_id, attempt_id). A lost lease response is replayed from the existing active lease; a different identity conflicts.
Reset uses stable request_id=reset-{session_id} and returns the cached initial screenshot on replay. Reusing another reset key conflicts.
Every action uses a stable SHA-256 request ID derived from attempt ID, model tool-use ID, and batch offset, plus the next expected action index. An identical retry returns the cached action result without input or logging side effects.
Checkpoint and release are state-idempotent. Artifact, event, frame-event, and status retrievals are read-only.
A broker HTTP error is never retried as if it were a transport failure.

This removes the unsafe choice between duplicating an input and abandoning an input whose response was lost.

Action outcome rules

The complete action batch is validated before its first input. Coordinates must fit the live display, typed text must be printable ASCII, done must be last, and the non-empty batch must fit the remaining 5,000-action rollout budget. There is no separate per-tool-call action-count cap. Local validation errors are returned to the model without touching the desktop.

After input is injected, whole-desktop animation is diagnostic rather than fatal. If pixels do not settle within the post-action window, the broker caches and returns the freshest screenshot with post_action_visual_stable=false. This is essential: throwing after input commit would create an ambiguous mutation. Initial handoff stability remains a hard admission requirement.

An input receipt proves only that input reached the interactive Windows desktop. It never proves that AutoCAD semantically accepted the command. The model must inspect the resulting screenshot, restore focus when necessary, and recover from the visible state.

Failure classification

admission_failed: no model request was made; clone may receive its one fresh replacement.
transport_exhausted: supervised tunnel and bounded broker retries could not restore communication; preserve partial trace/evidence and fail the sample.
model_contract_error: provider repeatedly omitted or malformed the required action call; preserve its billable usage and fail the sample.
action_rejected: the full batch was rejected before injection and reported back to the model, or an unexpected Windows-side rejection failed closed.
artifact_unavailable: final DWG save/checkpoint evidence was absent; the task fails even if screenshots looked correct.
cleanup_error: bounded evidence/release/instance cleanup failed; retain it in result/lifecycle state and run tag-scoped fleet reconciliation.

Observed scale failures and their v10 controls

Observed failure	v10 control
SSM tunnel died and all later broker calls failed	watchdog restart on stable port, process-group kill, per-outage restart allowance, fresh HTTP connections, and up to 48 deadline-bounded transport attempts with the same idempotency key
reset smoke returned HTTP 500 with orphan broker/Python/AutoCAD processes	scheduled-task stop/start gates, bounded tree kill, verified empty runtime/attempt directories, one recovery only
post-action animation caused `desktop_unstable` after input	return and cache freshest frame with explicit unstable receipt
lost action response risked duplicate keyboard/mouse input	stable request ID, expected index, cached exact replay
lost lease/reset response left an unusable active desktop	replay-safe lease identity and cached reset response
non-ASCII typing failed after earlier batch items ran	whole-batch printable-ASCII validation before injection plus broker defense-in-depth
OpenAI/Anthropic response omitted/truncated `computer_action`	ten consecutive safe pre-action retries, usage aggregation, larger output allowance, explicit format repair; a valid tool response resets the streak
transient AWS signature failure aborted preflight/control plane	bounded retry of transient signed AWS calls
terminated vCPU had not returned to quota	bounded `RunInstances` retry for `VcpuLimitExceeded`
one bad cold clone required a manual retry pass	one automatic terminate-and-replace admission attempt before model handoff
finalization hung for many minutes	300-second total evidence budget and 60-second release bound
cleanup API error leaked tagged instances	idempotent terminate retries plus one batch tag reconciliation pass

Release gate

Source tests are necessary but not sufficient. A v10 AMI becomes benchmarkable only after the Windows gates in incremental-ami-v10-runbook.md pass on an untouched clone. Until then, v10 is a source candidate and v9 remains only a rollback image, not a compatible runtime for the hardened harness.