docs/parallel-rollouts.md

# Parallel rollout operations

## Contract

`autocad-bench-batch` runs one task/model assignment in one fresh AutoCAD VM. An assignment never shares a Windows desktop, broker session, SSM tunnel, trace directory, or DWG artifact with another assignment. The same model may be assigned to multiple distinct tasks in one benchmark.

The batch has two modes:

- Plan mode is the default. It performs read-only AWS, quota, credential, and model checks and prints JSON. It does not call `RunInstances`.
- Execute mode requires `--execute`, a new `--output-dir`, and a completely passing preflight. Any preflight issue prevents the first launch.

The runner admits the configured concurrent wave against the quota class for the target instance family before launching: Standard On-Demand for A/C/D/H/I/M/R/T/Z instances or G/VT On-Demand for G/VT instances. It then starts at most `max_concurrency` sandbox workers, queuing any remaining assignments. In shared mode those workers are isolated threads and async model loops in one Python runtime rather than one supervisor plus one pilot interpreter per assignment. Each sandbox still owns its instance lifecycle and requests termination in `finally`. Instances also receive `BatchId` and `RolloutId` tags. At batch exit, a reconciler finds any matching non-terminated instances and issues one `TerminateInstances` request without polling.

Every config labels the broker version baked into its operator-owned image.
Preflight requires that label to equal a supported harness version before
making an AWS call.

## Provider paths

| Provider | Adapter | Authentication | Images and actions |
|---|---|---|---|
| OpenAI | Responses API | `OPENAI_API_KEY` | Reference PNG plus current frame every turn; strict forced `computer_action` function |
| OpenRouter | Responses API | `OPENROUTER_API_KEY` | Append-only stateless Responses history with sticky routing; Anthropic routes add automatic caching and the structured-output beta header |
| Anthropic | Official Messages SDK | `ANTHROPIC_API_KEY` | Byte-bounded Messages history; strict forced `computer_action` tool; stable-prefix and automatic caching; deterministic client compaction for Fable, official server compaction for other Claude models |
| Fireworks | OpenAI-compatible Chat Completions | `FIREWORKS_API_KEY` | Kimi K2.6 Fast router; strict forced tools; bounded thinking; automatic prefix caching with stable rollout affinity; deterministic compaction |
| AWS Mantle | OpenAI-compatible Chat Completions for Kimi K2.5; Responses for models that advertise it | AWS profile, SigV4 | `moonshotai.kimi-k2.5` uses strict client-side tools, multimodal Chat messages, append-only prefix reuse, and deterministic compaction |
| Bedrock runtime | Converse | AWS profile | Bedrock image blocks and forced normalized tool |

Secrets remain in the parent environment. They are never placed in CLI arguments, batch state, run state, model-input traces, or logs by the harness. The temporary broker token is injected only into its rollout child environment by the sandbox wrapper. When an Anthropic/OpenRouter/Fireworks/AWS model uses an OpenAI vision judge, the wrapper retains `OPENAI_API_KEY` for the post-rollout stage and removes it from the model child environment.

Every provider receives the same task prompt, fixed reference, current 1920x1080 desktop, normalized action-batch schema, and 5,000-action rollout budget. Direct OpenAI and Bedrock reconstruct a compact controller history. Direct Anthropic normally preserves exact validated user/tool-result and assistant/tool-use messages until Anthropic's official `compact-2026-01-12` context edit runs at the documented 150,000-token default threshold. Claude Fable 5 is deliberately isolated from that server sampling path: after 90,000 effective input tokens or 24 completed turns, whichever comes first, the client drops only complete tool-use/tool-result pairs and replaces the old raw turns with a deterministic checkpoint retaining the 16 most recent controller outcomes. It then reattaches the full task, fixed reference, and current desktop. Fable requests omit both the compaction beta header and `context_management`; no additional model call is used to summarize history. Other Anthropic models retain official server compaction, including non-null compaction-block replay and failed-compaction no-op handling. The immutable system and task/reference prefixes have explicit five-minute cache breakpoints, while top-level automatic caching advances over each newly appended turn.

OpenRouter's Responses endpoint is stateless, so its adapter also sends an append-only history: the immutable first user message, the selected provider `function_call`, its matching controller `function_call_output`, and the next user desktop message. A rollout-stable `session_id` and `prompt_cache_key` keep provider routing sticky. Claude routes additionally use top-level automatic five-minute caching; other routes reuse the same exact-prefix layout with whatever implicit caching their selected provider supports. OpenRouter Responses does not expose per-block explicit cache breakpoints. Exact resume checkpoints retain this history with artifact references and rehydrate the reference and historical screenshots from their verified local files before continuing.

Kimi K2.5 does not support Mantle Responses, so its dedicated `mantle-kimi-k2.5-chat` profile uses signed `/v1/chat/completions` requests. The first user message contains the task, fixed reference, and live desktop; later turns append the validated assistant tool call, controller tool result, and newest desktop without rewriting the reusable prefix. Mantle has no separate documented cache-control request field for this model. Cache reuse is therefore implicit and is audited through `usage.prompt_tokens_details.cached_tokens`. The harness deterministically cuts a fresh task/reference/current-desktop checkpoint at 24 retained messages or before the complete serialized request exceeds 3 MiB, whichever comes first, retaining a bounded summary of recent controller outcomes. A Mantle HTTP 400 body-length rejection forces the same compaction and retries the unconsumed model turn. Exact resume checkpoints checksum-verify and rehydrate all referenced images before continuing.

Fireworks uses the same append-only Chat Completions state machine through the pinned `fireworks-kimi-k2p6-fast-chat` profile. The exact model ID is `accounts/fireworks/routers/kimi-k2p6-fast`; Fast is selected by that router ID, so the request deliberately omits the mutually exclusive Priority `service_tier`. Kimi gets a 1,024-token thinking budget inside a 2,048-token output ceiling, preserving the returned `reasoning_content` with its assistant tool call on the next turn. Automatic caching is strengthened with stable `x-session-affinity`, `x-multi-turn-session-id`, and `prompt_cache_key` values unique to the rollout. The adapter records cached-token, Fireworks performance, and rate-limit telemetry, retries Fast checkpoint-swap HTTP 425 responses, and compacts at 24 messages or before the serialized request exceeds 8 MiB so the fixed reference and current desktop remain below Fireworks' 10 MB base64-image limit.

Prompt caching reduces repeated processing and cost but does not reduce HTTP request bytes. Anthropic requests therefore have a 24 MiB client-side safety limit below the provider's 32 MiB hard limit. Fable forces its deterministic client checkpoint before pruning if that guard is reached; other Anthropic models remove historical live screenshots while preserving immutable PNG artifacts, tool calls, tool results, and the newest desktop. An HTTP 413 triggers one immediate retry only after a byte-reducing rewrite; the harness never resends an identical oversized payload. Provider HTTP 5xx failures retry the same unconsumed model turn persistently with exponential backoff capped at 60 seconds, even when a provider advisory header says not to retry. HTTP 408/409 and connection/read failures retain the global ten-consecutive-failure budget. Client or server compaction subsequently bounds text/tool history. Exact checkpoints store the resulting model-visible state, Fable checkpoint counters and outcomes, and checksum-verified artifact references so controller interruption does not require replaying an action.

OpenRouter remains append-only: prior screenshots stay in its logical Responses request and are prefix-cached when the selected provider supports it; the new tool result, budget, and current screenshot form the uncached tail.

OpenAI, OpenRouter, direct Anthropic, Fireworks, and Mantle Kimi K2.5 use an action-specific strict schema that requires every field applicable to the selected action and rejects unknown fields. Other Mantle Responses models and Bedrock retain controller-side validation until their deployed tool surfaces are verified to accept the strict flag. Each tool call may contain any non-empty action batch that fits the remaining rollout budget, with no separate per-call item cap. Actions execute sequentially with only the final post-batch frame returned to the model. Invalid batches are rejected before execution and returned to the same model as recoverable observations. The default limit is 10 consecutive invalid turns; any completely valid batch resets the streak, while the run state preserves the lifetime invalid count separately.

Every model-provider adapter retries HTTP 429 or provider-equivalent throttling responses indefinitely with bounded exponential backoff while honoring a longer `retry-after` response header. Provider HTTP 5xx responses also retry indefinitely with exponential backoff capped at 60 seconds because retrying an unconsumed model turn cannot duplicate a desktop action. HTTP 408/409 and connection/read failures permit ten consecutive failed requests; any actual non-error provider response resets that transport/API-error streak. Missing or malformed `computer_action` responses independently permit ten consecutive model responses, and a valid action response resets that streak. Rate-limit, transient-request, and malformed-response counters do not consume one another. Direct SDK retry layers are disabled where supported so hidden provider retries cannot violate this shared accounting contract.

## Admission sequence

1. Parse the JSON config with unknown fields rejected.
2. Require unique rollout IDs and unique task/provider/model assignments.
3. Require exactly `expected_rollouts` enabled entries unless the operator explicitly passes `--allow-partial`.
   Configs with `task_suite` must also match that ordered suite exactly.
4. Verify all model-provider credentials and the configured vision-judge credential exist.
5. Load and checksum every required gold-cache entry for the pinned evaluator version; require schema-v3 or schema-v4 model/layout metadata for 3D tasks. Schema v4 is required when isolated physical 3D extents are part of deterministic scoring.
6. Verify AMI, subnets, security group, instance type, and Session Manager plugin.
7. Count pending/running vCPUs in the target quota class and admit `min(enabled_rollouts, max_concurrency)` instances against Standard quota `L-1216C47A` or G/VT quota `L-DB2E81BA`.
8. Retrieve direct-provider model IDs when keys are available.
9. Retrieve the live Mantle `/v1/models` and Bedrock capability catalogs for AWS entries; require exact IDs and advertised image input.
10. Only after all checks pass, create the output root and start sandbox workers.

AWS can still reject an individual launch because regional capacity and quota consumption can change after preflight. Launch itself receives bounded quota-release/capacity/signing retries. A clone must then pass immutable current-boot identity, authenticated tunnel health, and the exact assignment's lease/reset before its child starts. Tunnel startup tries at most three fresh SSM processes; the idempotent exact reset is replayed once, with the broker retaining two clean AutoCAD handoff attempts per reset call. If a new clone exhausts identity or exact-reset admission, it is terminated and replaced once by default before the first model request. This does not change the frozen sample count because no sample started. After `sandbox_agent_started`, a failed rollout is recorded exactly once and is never silently replaced; other assignments continue.

## Output layout

```text
batch-output/
  batch-state.json
  trace-publisher.json
  logs/
    openai-5-6-sol.log
  rollouts/
    openai-5-6-sol/
      run-state.json
      resume-checkpoint.json
      reference.png
      frames/
      model_turns.jsonl
      model_turn_summaries.jsonl
      input_receipts.jsonl
      events.json
      frame_events.json
      attempt.dwg
      result.json
      evaluation/
        status.json
        metadata.json
        render.png
        score.json
        diff.png
        vision-judge/
          score.json
          request.json
          responses.json
```

`batch-state.json` is atomically rewritten as rollout states change and includes every task assignment, concurrency, and an optional pinned pricing contract. The Vue dashboard's benchmark registry aggregates the batch immediately, including queued tasks before their rollout directories exist. A launched sandbox with no pilot trace yet is shown as `starting`, not as handed off. Per-task traces become available from the same benchmark detail view as soon as each pilot creates them.

Each sandbox begins evaluation immediately after its pilot exits and before instance cleanup. It invokes the trusted evaluator on that same licensed Windows host, retrieves metadata and the candidate render through request-scoped presigned S3 URLs, compares against the pinned local gold cache, and runs the configured vision judge. `evaluation/status.json` is atomically updated throughout. Invalid DWGs produce deterministic invalid/zero results without requiring a render; evaluator transport, cache, or judge failures mark evaluation failed and make an otherwise successful sandbox exit nonzero. Independent rollout slots do not wait for the rest of the batch.

Token cost is calculated per model response. Cached input is charged separately from uncached input, reasoning output is reported as a diagnostic subset of output, and long-context multipliers are applied only to responses whose input crosses the configured threshold.

## Commands

Read-only plan:

```bash
uv run autocad-bench-batch \
  --config configs/my-benchmark.json \
  --skip-direct-model-check
```

Live catalog check:

```bash
uv run autocad-bench-models --region us-east-1
```

Launch only after every configured model ID and provider credential has been
verified:

```bash
uv run autocad-bench-batch \
  --config configs/my-benchmark.json \
  --output-dir runs/benchmarks/my-run \
  --execute
```

`--allow-partial` is an explicit override for a deliberately smaller experiment. It is not used to hide unavailable requested models.