docs/generation-providers.md

# Generation provider runtime

## Status

This document describes the provider-neutral runtime layer used by the
generation foundation. It has two concrete optional adapters: the audited WAN
2.2 worker and FLUX.2 Klein worker. See
[generation-wan.md](generation-wan.md) and
[generation-flux.md](generation-flux.md) for their strict contracts. Other
models remain unregistered and unconfigured.

## Architecture

```
trusted server configuration
        |
GenerationProviderAdapter
        |
RemoteWorkerClient
        |
configured remote worker
        |
GenerationOutputIngestor
        |
CleanupService -> CanonicalAssetService -> workspace-owned media asset
```

`GenerationService` remains the only application-facing orchestration layer.
It retains authorization, idempotency, request/job ownership, and state
transitions. REST, MCP, SDK, n8n, and browser clients cannot provide a worker
URL, bearer token, worker job ID, output URL, or filesystem path.

## Provider contract

Concrete adapters subclass `GenerationProviderAdapter` and implement the
following asynchronous operations:

- `info()` — non-secret worker and model discovery metadata.
- `health()` — liveness only.
- `ready()` — inference readiness and loaded model IDs.
- `validate_request()` — strict typed provider input validation.
- `submit()` — worker submission. An adapter may use an idempotency protocol
  only when its audited worker actually supports one; otherwise it must make
  lost-response ambiguity fail safely without blind resubmission.
- `get_job()` — provider-job polling/reconciliation.
- `cancel()` — cancellation result (`requested`, `cancelled`, `unsupported`,
  or `failed`).
- `retrieve_output()` and `stream_output()` — a validated output descriptor
  and scoped byte stream.
- `normalize_error()` and `close()` — safe error normalization and client
  lifecycle cleanup.

An adapter must not expose arbitrary provider payloads. Its capabilities and
typed request schema define the complete public contract. It must use the
existing `GenerationService`, `GenerationRepository`, canonical-asset service,
and generation state machine; it must not write generation tables or files
directly.

## Worker HTTP client

`RemoteWorkerClient` accepts a base URL and optional bearer token only from
trusted server configuration used by a future adapter. It provides fixed
worker endpoints:

- `GET /health`
- `GET /ready`
- `GET /v1/info`
- `POST /v1/generate`
- `GET /v1/jobs/{worker_job_id}`
- `POST /v1/jobs/{worker_job_id}/cancel`
- a worker-relative output download path supplied by a validated output
  descriptor

The client disables redirects and proxy environment settings, applies connect,
request, and read timeouts, closes owned HTTP connections during application
shutdown, and does not log request or response bodies. Bearer tokens and
`Authorization` headers are never included in exceptions, persistence, audit
records, or API responses. Metadata is bounded to JSON-safe values and removes
credential-like fields, bearer strings, and HTTP(S) URLs so signed download or
upload links cannot enter job records.

The worker URL is validated as an absolute HTTP(S) URL. Public workers require
HTTPS; literal non-public IP addresses are rejected, while loopback HTTP is
allowed only for an operator-configured local development worker. Clients never
control the hostname. Output fetches accept only a strict relative path under
that configured origin: absolute URLs, redirects, query strings, fragments,
backslashes, and traversal segments are rejected.

DNS ownership remains an operator responsibility: production worker hostnames
must be controlled by the deployment and must not resolve to untrusted internal
services. A future deployment should also enforce its egress allowlist at the
network layer.

## Worker metadata and model discovery

`WorkerInfo` represents worker identity plus a list of discovered models.
`WorkerModelInfo` records a model ID, display name, and media types. Legacy
single-model `/v1/info` responses (`id`, `name`, and `type`/`media_types`) are
normalized into a one-item discovery list; multi-model workers can return a
`models` list. A compact single-model worker can instead return a safe map of
named underlying variants; it remains one discovered top-level model and its
adapter verifies the map where required.

Health is not model availability. A model is advertised only when all of the
following are true:

1. a server-owned `GenerationModelRegistration` exists;
2. its provider adapter is configured and available;
3. `/health` reports `healthy`;
4. `/v1/info` discovers the exact model ID and output modality;
5. `/ready` reports `ready`, `model_loaded: true`, and the exact model ID.

The `GenerationModelRegistry` stores provider ID, model capability,
non-secret configuration reference, safe metadata, and derived availability.
Every registration starts unavailable. Optional WAN and FLUX registrations
remain unavailable until their configured worker passes health, readiness, and
exact-identity checks.

Public provider discovery follows the same rule: a configured adapter is not
reported as available until at least one of its registered models has passed
those checks. This prevents a URL/token configuration from being mistaken for
a ready, authorized model.

## Health and readiness states

Worker liveness is normalized to `healthy`, `starting`, `unavailable`,
`unhealthy`, or `unknown`. Readiness is normalized to `ready`, `starting`,
`unavailable`, or `unknown`. A liveness response alone can never enable a
model. A failed runtime refresh marks that provider's registered models
unavailable instead of retaining stale availability.

## Error model and retries

Remote failures are converted to safe categories:

- `invalid_request`
- `authentication_error`
- `authorization_error`
- `worker_unavailable`
- `worker_not_ready`
- `timeout`
- `rate_limited`
- `provider_error`
- `inference_error`
- `output_error`
- `cancellation_error`
- `unknown_error`

The dedicated `GenerationRetryPolicy` is transport-level only; it does not
create jobs or attempts and therefore cannot conflict with the durable job
state machine. It allows bounded exponential-backoff retries for connection
failures, timeouts, `429`, `502`, `503`, and `504`, when the operation is
idempotent. Submissions carry MediaRouter's idempotency key before the client
will retry them. It does not retry validation failures, `400`, `401`, `403`,
invalid worker-job `404`s, generic `500`s, or programming/unknown exceptions.

The generic settings are optional and do not enable a worker:

```dotenv
AI_WORKER_CONNECT_TIMEOUT_SECONDS=10
AI_WORKER_REQUEST_TIMEOUT_SECONDS=60
AI_WORKER_READ_TIMEOUT_SECONDS=300
AI_WORKER_MAX_RETRIES=3
AI_WORKER_RETRY_BACKOFF_SECONDS=0.5
```

No provider URL, token, model, or credential is configured by these settings.

## Provider-job ownership and cancellation

When the trusted generation dispatcher receives a worker job ID, it must call
`GenerationService.bind_provider_job`. The repository checks the current
workspace and provider and the database enforces a unique `(provider,
external_job_id)` binding. That prevents polling, output ingestion, or
cancellation for one workspace from being attached to another workspace's job.
PostgreSQL startup verifies the required uniqueness index, so deployments that
have not applied `0004_generation_provider_runtime_postgres.sql` fail before
they can process generation work.

Cancellation first resolves the tenant-owned job. Queued/retrying work is
cancelled locally. For an active bound worker job, `GenerationService` calls
the adapter. A worker result of:

- `cancelled` transitions the MediaRouter job to `cancelled`;
- `requested` transitions it to `cancel_requested` only;
- `unsupported` returns a capability error without changing the active job;
- `failed` returns a safe cancellation error without claiming success.

A successful empty `204` cancellation response means only `requested`, not
`cancelled`. This prevents a worker that is still using GPU time from being
reported as stopped.

## Output handling

Workers return a `WorkerOutput` descriptor with modality, MIME type, opaque
provider output ID, worker-relative download path, optional checksum/size, and
safe metadata. Filesystem paths and absolute output URLs are rejected.

`GenerationService.ingest_completed_provider_output` is an internal dispatcher
hook. It verifies the tenant-owned job, its bound provider, its exact worker
job ID, and a completed worker status. `GenerationOutputIngestor` then:

1. streams bytes into a controlled temporary request workspace;
2. enforces the configured maximum output size and non-empty output;
3. verifies the optional SHA-256 and byte count;
4. publishes an exclusively created, service-generated filename;
5. registers the file with `CanonicalAssetService` for the same workspace;
6. atomically attaches the canonical asset ID and a safe metadata subset to
   the generation job through `GenerationRepository`.

The output filename never uses a provider/client name. A retry finds and
verifies an existing canonical output instead of overwriting it. Supported
runtime output MIME types are safe raster-image and video types listed in
`GenerationOutputIngestor`; a future provider must extend that reviewed mapping
before advertising a new type.

## Implementing a future provider

1. Add a strictly typed adapter under `app/generation/providers/`; do not add
   provider logic to REST, MCP, SDK, n8n, or frontend code.
2. Add only server-side configuration for its trusted worker endpoint and
   `SecretStr` token. Do not make them client-selectable.
3. Register provider capabilities and a `GenerationModelRegistration` only
   after its worker API, request schema, cancellation, output streaming, and
   reconciliation behavior have been tested.
4. Build `RemoteWorkerClient` from the generic timeout/retry settings and the
   provider's backend-only configuration.
5. On dispatch, transition through the existing job state machine, start the
   existing attempt record, submit with the request idempotency key, then bind
   the returned worker job ID through `GenerationService.bind_provider_job`.
6. During reconciliation, use `get_job`; for a completed result call
   `GenerationService.ingest_completed_provider_output`. Never persist a raw
   worker URL, output path, bearer token, or arbitrary provider response.
7. Add mocked protocol, retry, workspace-isolation, cancellation, output, and
   zero-configuration startup tests before enabling the provider.

WAN and FLUX are registered through the provider-neutral contract but remain
unavailable until their respective trusted configuration and runtime
verification succeed.