AutoCAD Benchmark Contract v0.6

Status: active 2D + 3D corpus contract; v9 remains historical

This contract freezes the assumptions used by the first local benchmark implementation. Infrastructure-specific values may be revised before the first live AutoCAD rollout, but every result must record the contract version it used.

Software versions

Component	Pinned value
Contract	`autocad-bench-contract-v0.6`
Python	`3.12.12`
Verifiers	`0.2.0`
AutoCAD	AutoCAD 2019 `R23.0.46.0.0`
Output DWG format	AutoCAD 2018
Windows	Windows Server 2022 Datacenter, build `20348`
Worker image	Operator-owned; must pass the broker and evaluator admission contracts
Evaluator source contract	`autocad-2019-r23.0.46-v4` (Model + all-layout inspection; 3D render pinned to `Layout1`)
Broker	v10 for current runs; v9 for historical result reproduction
Initial pilot model	OpenAI API `gpt-5.6-sol`, reasoning effort `xhigh`

Python dependencies are locked in uv.lock. The Windows and AutoCAD identifiers above were read from the running AMI and are part of the evaluator/cache version boundary.

Task contract

Input: one task-specific problem statement and one full-resolution dimensioned reference PNG.
Workspace: one clean, pre-named attempt.dwg open in AutoCAD.
2D target: all required geometry, dimensions, and annotations shown by the reference.
3D target: two coordinated deliverables in one DWG. Model Space must contain one true dimensionally accurate AutoCAD part or assembly inferred from the reference's complementary views; disconnected 2D projections are not a substitute. Layout1 must contain the complete paper-space presentation and reproduce every isometric, orthographic, section, and detail view actually shown, together on the shown sheet and in the shown arrangement, with its dimensions, callouts, notes, center/hidden lines, and hatching. The required presentation must not be moved to another layout or DWG. Tasks do not assume a fixed front/top/side set.
Output: one final readable DWG.
Units: millimetres unless an in-AutoCAD gold audit marks a task otherwise.
Manifest model wall-time budget: 900 seconds for 2D tasks and 1,800 seconds for 3D tasks, starting only after the verified initial handoff; EC2 boot and AutoCAD reset do not consume it.
Effective action budget: 5,000 actions for every task. Historical manifests and configurations may retain larger declared values for reproducibility, but the harness clamps every new or resumed rollout to the repository-wide ceiling. Every sub-action counts, regardless of how actions are grouped into model tool calls.
Primary reward: final-DWG similarity only.
Intermediate states, action efficiency, wall time, and token usage do not affect correctness reward.
Keyboard focus follows the live Windows UI. The model must click its intended target and recover when injected input has no visible effect. It receives a fresh frame after each complete action batch, not between sub-actions.
Controller acknowledgements and focus telemetry are diagnostic delivery evidence only; they never assert semantic success.

Model-visible capabilities

Allowed actions:

click
double_click
drag
move
scroll
key
type
wait
done

The AutoCAD in-application command bar and normal AutoCAD keyboard shortcuts are allowed.

The model emits exactly one computer_action tool call per turn. Its actions array contains any non-empty ordered action batch that fits the remaining 5,000-action rollout budget; there is no separate per-call item cap. The harness validates the complete array before executing its first item, then executes the items immediately and sequentially. The model receives one aggregate outcome and the final post-batch screenshot only after all items finish. It never receives intermediate screenshots or results, so any action that depends on an intermediate focus, dialog, menu, render, or command state must be sent in a one-action batch. done may appear only as the final batch item.

Prohibited model capabilities:

MCP
Command Prompt, PowerShell, or another OS shell
Direct filesystem APIs or tools
AutoCAD COM/.NET APIs
AutoLISP
Script or macro execution
Clipboard-based script injection
Process/window inspection APIs
Network access from the Windows desktop
Access to gold DWGs or scorer metadata

Desktop contract

Resolution: 1920x1080.
Display scale: 100%.
Locale: English.
Workspace: fixed AutoCAD workspace, currently the 2D Drafting & Annotation layout. AutoCAD's 3D commands remain available through the in-application command line for 3D tasks.
Window state: the AutoCAD top-level window must be maximized and cover the complete Windows work area for the initial agent screenshot. The taskbar may remain visible in the full 1920x1080 desktop capture. AutoCAD Clean Screen mode is not required; the ribbon and command bar remain available. After handoff, actions and screenshots follow the live desktop and may show Save As/File Explorer dialogs.
Theme, ribbon, command bar, units, template, object snaps, grid, and dynamic input: fixed by the versioned Windows image.
Third-party plugins: disabled.
Startup/recovery/update dialogs: disabled in the image where possible.
Gold corpus: absent from the desktop.
Outbound network: blocked.
Initial concurrency: one rollout and one desktop.

Artifact and verification contract

Only the final attempt.dwg is scored.
The evaluator may use trusted AutoCAD automation outside the model session.
Candidate and gold files are not compared byte-for-byte.
For 2D tasks, the active judge uses the reference, trusted gold/candidate renders, and deterministic evidence for geometry, dimension, annotation, and presentation fidelity.
For 3D tasks, the active judge requires both true model-space 3D geometry and the complete presentation on Layout1. The trusted evaluator records 3D entity counts, XYZ model extents, every paper layout and floating viewport, and renders only Layout1 when it exists, even when it is empty. A submission containing only a solid, only projected/copy-drawn views, or a presentation placed on another layout is incomplete. The vector-semantic comparison remains diagnostic rather than authoritative for 3D geometry.
Missing, unreadable, empty, underlay-based, OLE-based, raster-based, or external-reference-based submissions receive zero.
Every run must retain Kuro-compatible events.json and screenshot-aligned frame_events.json input logs. They are audit/training artifacts and remain excluded from the correctness reward.
events.json is NDJSON using the Kuro Windows event names and fields. Broker-generated input carries sourcePid=-1, actionIndex, and provenance="broker_action".
frame_events.json is NDJSON with one row per broker-captured action frame: {frame, video_t_ms, events}. Each embedded event also carries video_t_ms and t_in_frame_ms. For multi-action tool calls, intermediate rows are retained for audit but are not returned to the model.
Both logs and the final DWG must be retrieved before the session is released.

Dataset contract

Corpus size: 50 tasks with contiguous IDs from task-001 to task-050: 21 2D and 29 3D.
Development split: 7 tasks: 3 2D and 4 3D.
Evaluation split: 43 tasks: 18 2D and 25 3D.
Gold paths remain scorer-private.
Public task data contains an opaque task ID, task-specific prompt, reference image, drawing type, level, units, and budgets only.

Built-in AWS reference architecture

Validation instance type: g4dn.xlarge with NVIDIA Tesla T4.
Interactive display: Amazon DCV console, pinned to 1920x1080.
Private control transport: loopback HTTP reached through AWS Systems Manager port forwarding.
Network ingress: none; the benchmark security group has no inbound rules.
Trusted evaluator: interactive full AutoCAD scheduled task. Core Console is not used because it crashed or hung on this image.
Gold render path: PublishToWeb PNG.pc3, monochrome.ctb, fixed nearest 1600x1200 raster media, extents/fit/centered, lineweights disabled.

Every AMI, VPC resource, IAM role, region, credential source, and storage bucket is supplied by the operator.

Pending benchmark confirmation

Before a live rollout, record:

Whether all gold drawings use millimetres.
Complete the remaining manual unit and AutoCAD-open audit for every pending gold drawing, including the new 3D corpus, before treating its scores as release-grade.