# AutoCAD Benchmark Contract v0.6

Status: active 2D + 3D corpus contract; v9 remains historical

This contract freezes the assumptions used by the first local benchmark implementation. Infrastructure-specific values may be revised before the first live AutoCAD rollout, but every result must record the contract version it used.

## Software versions

| Component | Pinned value |
|---|---|
| Contract | `autocad-bench-contract-v0.6` |
| Python | `3.12.12` |
| Verifiers | `0.2.0` |
| AutoCAD | AutoCAD 2019 `R23.0.46.0.0` |
| Output DWG format | AutoCAD 2018 |
| Windows | Windows Server 2022 Datacenter, build `20348` |
| Worker image | Operator-owned; must pass the broker and evaluator admission contracts |
| Evaluator source contract | `autocad-2019-r23.0.46-v4` (Model + all-layout inspection; 3D render pinned to `Layout1`) |
| Broker | v10 for current runs; v9 for historical result reproduction |
| Initial pilot model | OpenAI API `gpt-5.6-sol`, reasoning effort `xhigh` |

Python dependencies are locked in `uv.lock`. The Windows and AutoCAD identifiers above were read from the running AMI and are part of the evaluator/cache version boundary.

## Task contract

- Input: one task-specific problem statement and one full-resolution dimensioned reference PNG.
- Workspace: one clean, pre-named `attempt.dwg` open in AutoCAD.
- 2D target: all required geometry, dimensions, and annotations shown by the reference.
- 3D target: two coordinated deliverables in one DWG. Model Space must contain one true dimensionally accurate AutoCAD part or assembly inferred from the reference's complementary views; disconnected 2D projections are not a substitute. `Layout1` must contain the complete paper-space presentation and reproduce every isometric, orthographic, section, and detail view actually shown, together on the shown sheet and in the shown arrangement, with its dimensions, callouts, notes, center/hidden lines, and hatching. The required presentation must not be moved to another layout or DWG. Tasks do not assume a fixed front/top/side set.
- Output: one final readable DWG.
- Units: millimetres unless an in-AutoCAD gold audit marks a task otherwise.
- Manifest model wall-time budget: 900 seconds for 2D tasks and 1,800 seconds for 3D tasks, starting only after the verified initial handoff; EC2 boot and AutoCAD reset do not consume it.
- Effective action budget: 5,000 actions for every task. Historical manifests and configurations may retain larger declared values for reproducibility, but the harness clamps every new or resumed rollout to the repository-wide ceiling. Every sub-action counts, regardless of how actions are grouped into model tool calls.
- Primary reward: final-DWG similarity only.
- Intermediate states, action efficiency, wall time, and token usage do not affect correctness reward.
- Keyboard focus follows the live Windows UI. The model must click its intended target and recover when injected input has no visible effect. It receives a fresh frame after each complete action batch, not between sub-actions.
- Controller acknowledgements and focus telemetry are diagnostic delivery evidence only; they never assert semantic success.

## Model-visible capabilities

Allowed actions:

- `click`
- `double_click`
- `drag`
- `move`
- `scroll`
- `key`
- `type`
- `wait`
- `done`

The AutoCAD in-application command bar and normal AutoCAD keyboard shortcuts are allowed.

The model emits exactly one `computer_action` tool call per turn. Its `actions` array contains any non-empty ordered action batch that fits the remaining 5,000-action rollout budget; there is no separate per-call item cap. The harness validates the complete array before executing its first item, then executes the items immediately and sequentially. The model receives one aggregate outcome and the final post-batch screenshot only after all items finish. It never receives intermediate screenshots or results, so any action that depends on an intermediate focus, dialog, menu, render, or command state must be sent in a one-action batch. `done` may appear only as the final batch item.

Prohibited model capabilities:

- MCP
- Command Prompt, PowerShell, or another OS shell
- Direct filesystem APIs or tools
- AutoCAD COM/.NET APIs
- AutoLISP
- Script or macro execution
- Clipboard-based script injection
- Process/window inspection APIs
- Network access from the Windows desktop
- Access to gold DWGs or scorer metadata

## Desktop contract

- Resolution: 1920x1080.
- Display scale: 100%.
- Locale: English.
- Workspace: fixed AutoCAD workspace, currently the 2D Drafting & Annotation layout. AutoCAD's 3D commands remain available through the in-application command line for 3D tasks.
- Window state: the AutoCAD top-level window must be maximized and cover the complete Windows work area for the initial agent screenshot. The taskbar may remain visible in the full 1920x1080 desktop capture. AutoCAD Clean Screen mode is not required; the ribbon and command bar remain available. After handoff, actions and screenshots follow the live desktop and may show Save As/File Explorer dialogs.
- Theme, ribbon, command bar, units, template, object snaps, grid, and dynamic input: fixed by the versioned Windows image.
- Third-party plugins: disabled.
- Startup/recovery/update dialogs: disabled in the image where possible.
- Gold corpus: absent from the desktop.
- Outbound network: blocked.
- Initial concurrency: one rollout and one desktop.

## Artifact and verification contract

- Only the final `attempt.dwg` is scored.
- The evaluator may use trusted AutoCAD automation outside the model session.
- Candidate and gold files are not compared byte-for-byte.
- For 2D tasks, the active judge uses the reference, trusted gold/candidate renders, and deterministic evidence for geometry, dimension, annotation, and presentation fidelity.
- For 3D tasks, the active judge requires both true model-space 3D geometry and the complete presentation on `Layout1`. The trusted evaluator records 3D entity counts, XYZ model extents, every paper layout and floating viewport, and renders only `Layout1` when it exists, even when it is empty. A submission containing only a solid, only projected/copy-drawn views, or a presentation placed on another layout is incomplete. The vector-semantic comparison remains diagnostic rather than authoritative for 3D geometry.
- Missing, unreadable, empty, underlay-based, OLE-based, raster-based, or external-reference-based submissions receive zero.
- Every run must retain Kuro-compatible `events.json` and screenshot-aligned `frame_events.json` input logs. They are audit/training artifacts and remain excluded from the correctness reward.
- `events.json` is NDJSON using the Kuro Windows event names and fields. Broker-generated input carries `sourcePid=-1`, `actionIndex`, and `provenance="broker_action"`.
- `frame_events.json` is NDJSON with one row per broker-captured action frame: `{frame, video_t_ms, events}`. Each embedded event also carries `video_t_ms` and `t_in_frame_ms`. For multi-action tool calls, intermediate rows are retained for audit but are not returned to the model.
- Both logs and the final DWG must be retrieved before the session is released.

## Dataset contract

- Corpus size: 50 tasks with contiguous IDs from `task-001` to `task-050`:
  21 2D and 29 3D.
- Development split: 7 tasks: 3 2D and 4 3D.
- Evaluation split: 43 tasks: 18 2D and 25 3D.
- Gold paths remain scorer-private.
- Public task data contains an opaque task ID, task-specific prompt, reference image, drawing type, level, units, and budgets only.

## Built-in AWS reference architecture

- Validation instance type: `g4dn.xlarge` with NVIDIA Tesla T4.
- Interactive display: Amazon DCV console, pinned to 1920x1080.
- Private control transport: loopback HTTP reached through AWS Systems Manager port forwarding.
- Network ingress: none; the benchmark security group has no inbound rules.
- Trusted evaluator: interactive full AutoCAD scheduled task. Core Console is not used because it crashed or hung on this image.
- Gold render path: `PublishToWeb PNG.pc3`, `monochrome.ctb`, fixed nearest 1600x1200 raster media, extents/fit/centered, lineweights disabled.

Every AMI, VPC resource, IAM role, region, credential source, and storage
bucket is supplied by the operator.

## Pending benchmark confirmation

Before a live rollout, record:

- Whether all gold drawings use millimetres.
- Complete the remaining manual unit and AutoCAD-open audit for every pending gold drawing, including the new 3D corpus, before treating its scores as release-grade.