docs/generation-providers.md

Generation provider runtime

Status

This document describes the provider-neutral runtime layer used by the generation foundation. It has two concrete optional adapters: the audited WAN 2.2 worker and FLUX.2 Klein worker. See generation-wan.md and generation-flux.md for their strict contracts. Other models remain unregistered and unconfigured.

Architecture

trusted server configuration
        |
GenerationProviderAdapter
        |
RemoteWorkerClient
        |
configured remote worker
        |
GenerationOutputIngestor
        |
CleanupService -> CanonicalAssetService -> workspace-owned media asset

GenerationService remains the only application-facing orchestration layer. It retains authorization, idempotency, request/job ownership, and state transitions. REST, MCP, SDK, n8n, and browser clients cannot provide a worker URL, bearer token, worker job ID, output URL, or filesystem path.

Provider contract

Concrete adapters subclass GenerationProviderAdapter and implement the following asynchronous operations:

• info() — non-secret worker and model discovery metadata.
• health() — liveness only.
• ready() — inference readiness and loaded model IDs.
• validate_request() — strict typed provider input validation.
• submit() — worker submission. An adapter may use an idempotency protocol only when its audited worker actually supports one; otherwise it must make lost-response ambiguity fail safely without blind resubmission.
• get_job() — provider-job polling/reconciliation.
• cancel() — cancellation result (requested, cancelled, unsupported, or failed).
• retrieve_output() and stream_output() — a validated output descriptor and scoped byte stream.
• normalize_error() and close() — safe error normalization and client lifecycle cleanup.

An adapter must not expose arbitrary provider payloads. Its capabilities and typed request schema define the complete public contract. It must use the existing GenerationService, GenerationRepository, canonical-asset service, and generation state machine; it must not write generation tables or files directly.

Worker HTTP client

RemoteWorkerClient accepts a base URL and optional bearer token only from trusted server configuration used by a future adapter. It provides fixed worker endpoints:

• GET /health
• GET /ready
• GET /v1/info
• POST /v1/generate
• GET /v1/jobs/{worker_job_id}
• POST /v1/jobs/{worker_job_id}/cancel
• a worker-relative output download path supplied by a validated output descriptor

The client disables redirects and proxy environment settings, applies connect, request, and read timeouts, closes owned HTTP connections during application shutdown, and does not log request or response bodies. Bearer tokens and Authorization headers are never included in exceptions, persistence, audit records, or API responses. Metadata is bounded to JSON-safe values and removes credential-like fields, bearer strings, and HTTP(S) URLs so signed download or upload links cannot enter job records.

The worker URL is validated as an absolute HTTP(S) URL. Public workers require HTTPS; literal non-public IP addresses are rejected, while loopback HTTP is allowed only for an operator-configured local development worker. Clients never control the hostname. Output fetches accept only a strict relative path under that configured origin: absolute URLs, redirects, query strings, fragments, backslashes, and traversal segments are rejected.

DNS ownership remains an operator responsibility: production worker hostnames must be controlled by the deployment and must not resolve to untrusted internal services. A future deployment should also enforce its egress allowlist at the network layer.

Worker metadata and model discovery

WorkerInfo represents worker identity plus a list of discovered models. WorkerModelInfo records a model ID, display name, and media types. Legacy single-model /v1/info responses (id, name, and type/media_types) are normalized into a one-item discovery list; multi-model workers can return a models list. A compact single-model worker can instead return a safe map of named underlying variants; it remains one discovered top-level model and its adapter verifies the map where required.

Health is not model availability. A model is advertised only when all of the following are true:

1. a server-owned GenerationModelRegistration exists;
2. its provider adapter is configured and available;
3. /health reports healthy;
4. /v1/info discovers the exact model ID and output modality;
5. /ready reports ready, model_loaded: true, and the exact model ID.

The GenerationModelRegistry stores provider ID, model capability, non-secret configuration reference, safe metadata, and derived availability. Every registration starts unavailable. Optional WAN and FLUX registrations remain unavailable until their configured worker passes health, readiness, and exact-identity checks.

Public provider discovery follows the same rule: a configured adapter is not reported as available until at least one of its registered models has passed those checks. This prevents a URL/token configuration from being mistaken for a ready, authorized model.

Health and readiness states

Worker liveness is normalized to healthy, starting, unavailable, unhealthy, or unknown. Readiness is normalized to ready, starting, unavailable, or unknown. A liveness response alone can never enable a model. A failed runtime refresh marks that provider's registered models unavailable instead of retaining stale availability.

Error model and retries

Remote failures are converted to safe categories:

• invalid_request
• authentication_error
• authorization_error
• worker_unavailable
• worker_not_ready
• timeout
• rate_limited
• provider_error
• inference_error
• output_error
• cancellation_error
• unknown_error

The dedicated GenerationRetryPolicy is transport-level only; it does not create jobs or attempts and therefore cannot conflict with the durable job state machine. It allows bounded exponential-backoff retries for connection failures, timeouts, 429, 502, 503, and 504, when the operation is idempotent. Submissions carry MediaRouter's idempotency key before the client will retry them. It does not retry validation failures, 400, 401, 403, invalid worker-job 404s, generic 500s, or programming/unknown exceptions.

The generic settings are optional and do not enable a worker:

AI_WORKER_CONNECT_TIMEOUT_SECONDS=10
AI_WORKER_REQUEST_TIMEOUT_SECONDS=60
AI_WORKER_READ_TIMEOUT_SECONDS=300
AI_WORKER_MAX_RETRIES=3
AI_WORKER_RETRY_BACKOFF_SECONDS=0.5

No provider URL, token, model, or credential is configured by these settings.

Provider-job ownership and cancellation

When the trusted generation dispatcher receives a worker job ID, it must call GenerationService.bind_provider_job. The repository checks the current workspace and provider and the database enforces a unique (provider, external_job_id) binding. That prevents polling, output ingestion, or cancellation for one workspace from being attached to another workspace's job. PostgreSQL startup verifies the required uniqueness index, so deployments that have not applied 0004_generation_provider_runtime_postgres.sql fail before they can process generation work.

Cancellation first resolves the tenant-owned job. Queued/retrying work is cancelled locally. For an active bound worker job, GenerationService calls the adapter. A worker result of:

• cancelled transitions the MediaRouter job to cancelled;
• requested transitions it to cancel_requested only;
• unsupported returns a capability error without changing the active job;
• failed returns a safe cancellation error without claiming success.

A successful empty 204 cancellation response means only requested, not cancelled. This prevents a worker that is still using GPU time from being reported as stopped.

Output handling

Workers return a WorkerOutput descriptor with modality, MIME type, opaque provider output ID, worker-relative download path, optional checksum/size, and safe metadata. Filesystem paths and absolute output URLs are rejected.

GenerationService.ingest_completed_provider_output is an internal dispatcher hook. It verifies the tenant-owned job, its bound provider, its exact worker job ID, and a completed worker status. GenerationOutputIngestor then:

1. streams bytes into a controlled temporary request workspace;
2. enforces the configured maximum output size and non-empty output;
3. verifies the optional SHA-256 and byte count;
4. publishes an exclusively created, service-generated filename;
5. registers the file with CanonicalAssetService for the same workspace;
6. atomically attaches the canonical asset ID and a safe metadata subset to the generation job through GenerationRepository.

The output filename never uses a provider/client name. A retry finds and verifies an existing canonical output instead of overwriting it. Supported runtime output MIME types are safe raster-image and video types listed in GenerationOutputIngestor; a future provider must extend that reviewed mapping before advertising a new type.

Implementing a future provider

1. Add a strictly typed adapter under app/generation/providers/; do not add provider logic to REST, MCP, SDK, n8n, or frontend code.
2. Add only server-side configuration for its trusted worker endpoint and SecretStr token. Do not make them client-selectable.
3. Register provider capabilities and a GenerationModelRegistration only after its worker API, request schema, cancellation, output streaming, and reconciliation behavior have been tested.
4. Build RemoteWorkerClient from the generic timeout/retry settings and the provider's backend-only configuration.
5. On dispatch, transition through the existing job state machine, start the existing attempt record, submit with the request idempotency key, then bind the returned worker job ID through GenerationService.bind_provider_job.
6. During reconciliation, use get_job; for a completed result call GenerationService.ingest_completed_provider_output. Never persist a raw worker URL, output path, bearer token, or arbitrary provider response.
7. Add mocked protocol, retry, workspace-isolation, cancellation, output, and zero-configuration startup tests before enabling the provider.

WAN and FLUX are registered through the provider-neutral contract but remain unavailable until their respective trusted configuration and runtime verification succeed.