File size: 5,692 Bytes
0be8f22 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | # Infrastructure plugins
The batch controller does not launch workers directly. It resolves the
`infrastructure.backend` in a batch configuration and delegates admission,
launch-command construction, shared-process execution, trace publishing, and
final cleanup to that backend.
The built-in `aws` plugin preserves the existing EC2 + SSM behavior. External
packages can provide workers on another cloud, an on-premises Windows pool, or
a local virtualization system without changing the benchmark harness or
evaluator.
AWS dependencies are optional in packaged installations:
```bash
pip install "autocad-bench[aws]"
```
The base controller, harness, plugin API, and resume planner import without
`boto3` or `botocore`; a third-party backend supplies only its own SDK.
## Configuration
`backend` defaults to `aws` for compatibility, but every AWS resource is
operator-supplied. There are no repository-owned resource defaults:
```json
{
"infrastructure": {
"backend": "aws",
"broker_version": "windows-autocad-2019-v10",
"image_id": "ami-...",
"subnet_ids": ["subnet-..."],
"security_group_id": "sg-...",
"instance_profile_name": "AutoCADBenchWorkerProfile",
"instance_type": "g4dn.xlarge",
"aws_region": "us-east-1"
},
"evaluation": {
"bucket": "my-autocad-bench-bucket"
}
}
```
`aws_profile` is optional. When omitted, boto3 and the AWS CLI use their
standard credential chain. Automatic AWS evaluation is enabled by default and
requires an explicit bucket in the operator's account. Set
`"evaluation": {"enabled": false}` only when deliberately running without the
trusted automatic evaluator.
Third-party backends put backend-owned values in `settings`:
```json
{
"infrastructure": {
"backend": "my-windows-pool",
"broker_version": "windows-autocad-2019-v10",
"settings": {
"pool": "autocad-2019",
"controller_url_env": "WINDOWS_POOL_URL"
}
}
}
```
Do not put credentials in `settings`; reference environment variables or the
backend's native credential provider.
## Plugin contract
Implement `autocad_bench.infrastructure.InfrastructurePlugin`. The controller
calls its methods in this order:
1. `validate_spec` checks configuration without changing external state.
2. `preflight` performs read-only capacity and worker-image checks.
3. `build_rollout_command` wraps the infrastructure-neutral harness command in
one command that owns the worker lifecycle.
4. `run_shared` runs that command when the controller is configured for
in-process execution. Subprocess mode executes the returned command
directly.
5. `create_trace_publisher` optionally mirrors live artifacts when trace
streaming is configured.
6. `reap_batch` performs idempotent best-effort cleanup after all rollouts.
The complete protocol and request models live in
`src/autocad_bench/infrastructure/`. A backend must:
- isolate simultaneous rollouts from each other;
- make the configured desktop broker reachable by the child harness;
- execute the exact `RolloutLaunchRequest.child_command`;
- preserve the requested output directory and process exit code;
- never expose evaluator inputs or credentials to the model-visible desktop;
- make cleanup idempotent, including after a partially failed launch.
The plugin receives evaluation arguments separately because the lifecycle
wrapper, not the model harness, owns trusted DWG retrieval and evaluation.
Trace publishing is enabled with `AUTOCAD_BENCH_TRACE_DESTINATION` and an
optional `AUTOCAD_BENCH_TRACE_PREFIX`. The AWS plugin also accepts the legacy
`AUTOCAD_BENCH_TRACE_S3_BUCKET` and `AUTOCAD_BENCH_TRACE_S3_PREFIX` names.
## Registration
Publish the implementation as a Python entry point whose name matches
`InfrastructurePlugin.name`:
```toml
[project.entry-points."autocad_bench.infrastructure"]
my-windows-pool = "my_autocad_backend:WindowsPoolPlugin"
```
After installing that package in the same environment as AutoCAD Bench, use
`"backend": "my-windows-pool"` in the batch config. Entry points are loaded
only when selected. Applications embedding the controller can instead use
`register_infrastructure_plugin()` for an in-process registration.
The batch state and preflight report record `infrastructure_backend`, making
mixed-backend result directories auditable.
## Interrupted-run recovery
Recovery is an optional second protocol. Backends that support
`autocad-bench-resume` implement
`autocad_bench.infrastructure.InfrastructureRecoveryPlugin` in addition to the
standard lifecycle contract:
1. `validate_recovery_spec` validates reconnect-only settings.
2. `create_recovery_session` creates the recovery control-plane client.
3. `list_active_workers` maps rollout IDs to backend worker IDs for a batch.
4. `worker_id_from_log` supports legacy runs that predate durable batch worker
indexing.
5. `worker_status` determines whether a discovered worker can be reconnected.
6. `build_recovery_command` wraps the infrastructure-neutral harness resume
command.
7. `terminate_worker` idempotently releases a completed or exhausted worker.
New batch states persist the selected infrastructure spec. Recovery plans copy
that spec into every entry, use the backend-neutral `worker_id`, and can contain
entries from multiple backends. AWS plans retain `instance_id`,
`instance_state`, and `instance_termination_requested` as compatibility aliases
for existing controller tooling.
A backend that intentionally does not support interrupted-session recovery can
implement only `InfrastructurePlugin`. Normal batch cleanup still works, while
`autocad-bench-resume` fails explicitly instead of applying another backend's
recovery semantics.
|