docs/parallel-rollouts.md

Parallel rollout operations

Contract

autocad-bench-batch runs one task/model assignment in one fresh AutoCAD VM. An assignment never shares a Windows desktop, broker session, SSM tunnel, trace directory, or DWG artifact with another assignment. The same model may be assigned to multiple distinct tasks in one benchmark.

The batch has two modes:

• Plan mode is the default. It performs read-only AWS, quota, credential, and model checks and prints JSON. It does not call RunInstances.
• Execute mode requires --execute, a new --output-dir, and a completely passing preflight. Any preflight issue prevents the first launch.

The runner admits the configured concurrent wave against the quota class for the target instance family before launching: Standard On-Demand for A/C/D/H/I/M/R/T/Z instances or G/VT On-Demand for G/VT instances. It then starts at most max_concurrency sandbox workers, queuing any remaining assignments. In shared mode those workers are isolated threads and async model loops in one Python runtime rather than one supervisor plus one pilot interpreter per assignment. Each sandbox still owns its instance lifecycle and requests termination in finally. Instances also receive BatchId and RolloutId tags. At batch exit, a reconciler finds any matching non-terminated instances and issues one TerminateInstances request without polling.

Every config labels the broker version baked into its operator-owned image. Preflight requires that label to equal a supported harness version before making an AWS call.

Provider paths

Provider	Adapter	Authentication	Images and actions
OpenAI	Responses API	OPENAI_API_KEY	Reference PNG plus current frame every turn; strict forced computer_action function
OpenRouter	Responses API	OPENROUTER_API_KEY	Append-only stateless Responses history with sticky routing; Anthropic routes add automatic caching and the structured-output beta header
Anthropic	Official Messages SDK	ANTHROPIC_API_KEY	Byte-bounded Messages history; strict forced computer_action tool; stable-prefix and automatic caching; deterministic client compaction for Fable, official server compaction for other Claude models
Fireworks	OpenAI-compatible Chat Completions	FIREWORKS_API_KEY	Kimi K2.6 Fast router; strict forced tools; bounded thinking; automatic prefix caching with stable rollout affinity; deterministic compaction
AWS Mantle	OpenAI-compatible Chat Completions for Kimi K2.5; Responses for models that advertise it	AWS profile, SigV4	moonshotai.kimi-k2.5 uses strict client-side tools, multimodal Chat messages, append-only prefix reuse, and deterministic compaction
Bedrock runtime	Converse	AWS profile	Bedrock image blocks and forced normalized tool

Secrets remain in the parent environment. They are never placed in CLI arguments, batch state, run state, model-input traces, or logs by the harness. The temporary broker token is injected only into its rollout child environment by the sandbox wrapper. When an Anthropic/OpenRouter/Fireworks/AWS model uses an OpenAI vision judge, the wrapper retains OPENAI_API_KEY for the post-rollout stage and removes it from the model child environment.

Every provider receives the same task prompt, fixed reference, current 1920x1080 desktop, normalized action-batch schema, and 5,000-action rollout budget. Direct OpenAI and Bedrock reconstruct a compact controller history. Direct Anthropic normally preserves exact validated user/tool-result and assistant/tool-use messages until Anthropic's official compact-2026-01-12 context edit runs at the documented 150,000-token default threshold. Claude Fable 5 is deliberately isolated from that server sampling path: after 90,000 effective input tokens or 24 completed turns, whichever comes first, the client drops only complete tool-use/tool-result pairs and replaces the old raw turns with a deterministic checkpoint retaining the 16 most recent controller outcomes. It then reattaches the full task, fixed reference, and current desktop. Fable requests omit both the compaction beta header and context_management; no additional model call is used to summarize history. Other Anthropic models retain official server compaction, including non-null compaction-block replay and failed-compaction no-op handling. The immutable system and task/reference prefixes have explicit five-minute cache breakpoints, while top-level automatic caching advances over each newly appended turn.

OpenRouter's Responses endpoint is stateless, so its adapter also sends an append-only history: the immutable first user message, the selected provider function_call, its matching controller function_call_output, and the next user desktop message. A rollout-stable session_id and prompt_cache_key keep provider routing sticky. Claude routes additionally use top-level automatic five-minute caching; other routes reuse the same exact-prefix layout with whatever implicit caching their selected provider supports. OpenRouter Responses does not expose per-block explicit cache breakpoints. Exact resume checkpoints retain this history with artifact references and rehydrate the reference and historical screenshots from their verified local files before continuing.

Kimi K2.5 does not support Mantle Responses, so its dedicated mantle-kimi-k2.5-chat profile uses signed /v1/chat/completions requests. The first user message contains the task, fixed reference, and live desktop; later turns append the validated assistant tool call, controller tool result, and newest desktop without rewriting the reusable prefix. Mantle has no separate documented cache-control request field for this model. Cache reuse is therefore implicit and is audited through usage.prompt_tokens_details.cached_tokens. The harness deterministically cuts a fresh task/reference/current-desktop checkpoint at 24 retained messages or before the complete serialized request exceeds 3 MiB, whichever comes first, retaining a bounded summary of recent controller outcomes. A Mantle HTTP 400 body-length rejection forces the same compaction and retries the unconsumed model turn. Exact resume checkpoints checksum-verify and rehydrate all referenced images before continuing.

Fireworks uses the same append-only Chat Completions state machine through the pinned fireworks-kimi-k2p6-fast-chat profile. The exact model ID is accounts/fireworks/routers/kimi-k2p6-fast; Fast is selected by that router ID, so the request deliberately omits the mutually exclusive Priority service_tier. Kimi gets a 1,024-token thinking budget inside a 2,048-token output ceiling, preserving the returned reasoning_content with its assistant tool call on the next turn. Automatic caching is strengthened with stable x-session-affinity, x-multi-turn-session-id, and prompt_cache_key values unique to the rollout. The adapter records cached-token, Fireworks performance, and rate-limit telemetry, retries Fast checkpoint-swap HTTP 425 responses, and compacts at 24 messages or before the serialized request exceeds 8 MiB so the fixed reference and current desktop remain below Fireworks' 10 MB base64-image limit.

Prompt caching reduces repeated processing and cost but does not reduce HTTP request bytes. Anthropic requests therefore have a 24 MiB client-side safety limit below the provider's 32 MiB hard limit. Fable forces its deterministic client checkpoint before pruning if that guard is reached; other Anthropic models remove historical live screenshots while preserving immutable PNG artifacts, tool calls, tool results, and the newest desktop. An HTTP 413 triggers one immediate retry only after a byte-reducing rewrite; the harness never resends an identical oversized payload. Provider HTTP 5xx failures retry the same unconsumed model turn persistently with exponential backoff capped at 60 seconds, even when a provider advisory header says not to retry. HTTP 408/409 and connection/read failures retain the global ten-consecutive-failure budget. Client or server compaction subsequently bounds text/tool history. Exact checkpoints store the resulting model-visible state, Fable checkpoint counters and outcomes, and checksum-verified artifact references so controller interruption does not require replaying an action.

OpenRouter remains append-only: prior screenshots stay in its logical Responses request and are prefix-cached when the selected provider supports it; the new tool result, budget, and current screenshot form the uncached tail.

OpenAI, OpenRouter, direct Anthropic, Fireworks, and Mantle Kimi K2.5 use an action-specific strict schema that requires every field applicable to the selected action and rejects unknown fields. Other Mantle Responses models and Bedrock retain controller-side validation until their deployed tool surfaces are verified to accept the strict flag. Each tool call may contain any non-empty action batch that fits the remaining rollout budget, with no separate per-call item cap. Actions execute sequentially with only the final post-batch frame returned to the model. Invalid batches are rejected before execution and returned to the same model as recoverable observations. The default limit is 10 consecutive invalid turns; any completely valid batch resets the streak, while the run state preserves the lifetime invalid count separately.

Every model-provider adapter retries HTTP 429 or provider-equivalent throttling responses indefinitely with bounded exponential backoff while honoring a longer retry-after response header. Provider HTTP 5xx responses also retry indefinitely with exponential backoff capped at 60 seconds because retrying an unconsumed model turn cannot duplicate a desktop action. HTTP 408/409 and connection/read failures permit ten consecutive failed requests; any actual non-error provider response resets that transport/API-error streak. Missing or malformed computer_action responses independently permit ten consecutive model responses, and a valid action response resets that streak. Rate-limit, transient-request, and malformed-response counters do not consume one another. Direct SDK retry layers are disabled where supported so hidden provider retries cannot violate this shared accounting contract.

Admission sequence

1. Parse the JSON config with unknown fields rejected.
2. Require unique rollout IDs and unique task/provider/model assignments.
3. Require exactly expected_rollouts enabled entries unless the operator explicitly passes --allow-partial. Configs with task_suite must also match that ordered suite exactly.
4. Verify all model-provider credentials and the configured vision-judge credential exist.
5. Load and checksum every required gold-cache entry for the pinned evaluator version; require schema-v3 or schema-v4 model/layout metadata for 3D tasks. Schema v4 is required when isolated physical 3D extents are part of deterministic scoring.
6. Verify AMI, subnets, security group, instance type, and Session Manager plugin.
7. Count pending/running vCPUs in the target quota class and admit min(enabled_rollouts, max_concurrency) instances against Standard quota L-1216C47A or G/VT quota L-DB2E81BA.
8. Retrieve direct-provider model IDs when keys are available.
9. Retrieve the live Mantle /v1/models and Bedrock capability catalogs for AWS entries; require exact IDs and advertised image input.
10. Only after all checks pass, create the output root and start sandbox workers.

AWS can still reject an individual launch because regional capacity and quota consumption can change after preflight. Launch itself receives bounded quota-release/capacity/signing retries. A clone must then pass immutable current-boot identity, authenticated tunnel health, and the exact assignment's lease/reset before its child starts. Tunnel startup tries at most three fresh SSM processes; the idempotent exact reset is replayed once, with the broker retaining two clean AutoCAD handoff attempts per reset call. If a new clone exhausts identity or exact-reset admission, it is terminated and replaced once by default before the first model request. This does not change the frozen sample count because no sample started. After sandbox_agent_started, a failed rollout is recorded exactly once and is never silently replaced; other assignments continue.

Output layout

batch-output/
  batch-state.json
  trace-publisher.json
  logs/
    openai-5-6-sol.log
  rollouts/
    openai-5-6-sol/
      run-state.json
      resume-checkpoint.json
      reference.png
      frames/
      model_turns.jsonl
      model_turn_summaries.jsonl
      input_receipts.jsonl
      events.json
      frame_events.json
      attempt.dwg
      result.json
      evaluation/
        status.json
        metadata.json
        render.png
        score.json
        diff.png
        vision-judge/
          score.json
          request.json
          responses.json

batch-state.json is atomically rewritten as rollout states change and includes every task assignment, concurrency, and an optional pinned pricing contract. The Vue dashboard's benchmark registry aggregates the batch immediately, including queued tasks before their rollout directories exist. A launched sandbox with no pilot trace yet is shown as starting, not as handed off. Per-task traces become available from the same benchmark detail view as soon as each pilot creates them.

Each sandbox begins evaluation immediately after its pilot exits and before instance cleanup. It invokes the trusted evaluator on that same licensed Windows host, retrieves metadata and the candidate render through request-scoped presigned S3 URLs, compares against the pinned local gold cache, and runs the configured vision judge. evaluation/status.json is atomically updated throughout. Invalid DWGs produce deterministic invalid/zero results without requiring a render; evaluator transport, cache, or judge failures mark evaluation failed and make an otherwise successful sandbox exit nonzero. Independent rollout slots do not wait for the rest of the batch.

Token cost is calculated per model response. Cached input is charged separately from uncached input, reasoning output is reported as a diagnostic subset of output, and long-context multipliers are applied only to responses whose input crosses the configured threshold.

Commands

Read-only plan:

uv run autocad-bench-batch \
  --config configs/my-benchmark.json \
  --skip-direct-model-check

Live catalog check:

uv run autocad-bench-models --region us-east-1

Launch only after every configured model ID and provider credential has been verified:

uv run autocad-bench-batch \
  --config configs/my-benchmark.json \
  --output-dir runs/benchmarks/my-run \
  --execute

--allow-partial is an explicit override for a deliberately smaller experiment. It is not used to hide unavailable requested models.