Spaces:
Running
Running
File size: 7,158 Bytes
1b2323a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 | # AI Studio
AI Studio is MediaRouter's authenticated, provider-neutral AI workspace. The
frontend routes are `/ai` and `/projects/{projectId}/ai`; both call only the
MediaRouter BFF/API. Provider URLs and credentials never enter browser state.
## Implemented capability boundary
`GET /v1/ai/capabilities` is authoritative. It derives ready tools and models
from the existing generation provider/model registries. A tool is visible only
when the frontend feature flag, user permission, API contract, provider, and
model are all available.
This phase exposes only operations backed by real adapters:
- `generate_image`: FLUX text-to-image and optional image transformation.
- `generate_video`: WAN image-to-video.
Voice, music, transform, transcription, OCR, multimodal understanding, and
structured text creation are not advertised because no corresponding
production adapter exists. There is no mock provider or placeholder output.
Each advertised tool includes its category, input/output media types,
permission, project/asset support, available models, limits, and provider
display metadata. The frontend registry adds presentation metadata only; it
cannot make a backend operation available.
## API
- `GET /v1/ai/capabilities`
- `GET /v1/ai/jobs?offset=0&limit=25`
- `POST /v1/ai/jobs` with `Idempotency-Key`
- `GET /v1/ai/jobs/{generation_id}`
- `POST /v1/ai/jobs/{generation_id}/cancel`
Submission is a discriminated Pydantic union. Image and video parameters are
closed schemas with bounded prompts, dimensions, duration, steps, guidance,
seed, project ID, and source asset IDs. Unknown fields are rejected.
The provider-neutral request is translated by `AiStudioService` into the
existing typed generation request. The backend selects only a ready model of
the requested modality. The browser cannot supply credentials, worker URLs, or
unregistered provider payloads.
## Generation and job lifecycle
AI Studio reuses `generation_requests`, `generation_jobs`, and the existing
dispatcher:
`queued -> processing -> completed | failed | cancelling -> cancelled`
`retrying` is reported only when the durable generation state machine schedules
a safe retry. Progress remains `null` because the current adapters do not
provide trustworthy normalized progress. Completion is reported only when
output validation and canonical asset registration have succeeded.
The `product_surface = ai_studio` marker isolates AI history from legacy
`/v1/generation` requests without creating a second job system. Requests retain
one logical job through retries. Workspace-scoped idempotency rejects reuse of a
key with a different operation, parameters, asset, project, or product surface.
Cancellation is truthful: queued work can become cancelled immediately;
running work remains cancelling until the provider confirms cancellation.
## Projects and assets
Project context is optional globally and authoritative on the project route.
The server verifies project membership and active status. Every source asset is
resolved through `CanonicalAssetService`; it must belong to the authenticated
workspace and, when project context is present, to that exact project.
Outputs are downloaded and validated by the existing generation ingestor,
registered as canonical `media_assets`, and associated with the selected
project. Source assets are never overwritten. The frontend stores asset IDs,
not binaries, worker paths, credentials, or durable signed URLs. Completed
outputs can be opened in the asset library and their project can be opened in
Content Studio.
## Authorization, limits, and audit
The API uses existing authentication, tenant sessions, security policy, and
rate limiter. Scopes are:
- `ai:read`
- `ai:generate`
- `ai:transform`
- `ai:analyze`
- `ai:create`
Only `ai:read` and `ai:generate` currently authorize an advertised operation.
The remaining scopes reserve explicit boundaries for future real adapters.
Rate limiting uses the existing job classification and workspace/user/API-key
identity; no second limiter exists.
Audit events are bounded and omit prompts and provider payloads:
- `ai.generation_requested`
- `ai.generation_completed`
- `ai.generation_failed`
- `ai.generation_cancelled`
Structured logs contain generation/job IDs, operation/model, provider-neutral
status/error category, and timing where available. Secrets, prompts, signed
URLs, and full provider responses are excluded.
Usage and cost fields are present as `null`. They must remain null until a
provider supplies authoritative usage or approved pricing metadata.
## Frontend ownership and integration
TanStack Query owns capabilities, jobs, history, and output refresh.
Zustand owns only selected category/tool/project and temporary prompt/source
input. Active jobs poll by state; completed output invalidates project asset
queries. AI jobs are merged into `WorkspaceActivityDrawer`.
Commands and global search entries are derived from discovered, available
capabilities. Deep links use `/ai?tool={operation}`. Desktop uses tool
navigation plus workspace/history; smaller layouts collapse to normal
responsive grid/stack behavior. Cards and controls are keyboard accessible,
focus visible, semantically labelled, and do not rely on color alone.
## Provider abstraction and failure behavior
AI Studio delegates validation, submission, polling, cancellation, error
normalization, output ingestion, and retry decisions to the existing generation
adapters. Provider unavailable, timeout, rate limit, invalid request, rejection,
network error, partial/invalid output, and cancellation retain truthful typed
failure states. Provider response objects are not exposed.
## MCP, SDK, and n8n
MCP registers `ai.capabilities`, `ai.generate`, `ai.list_jobs`, `ai.get_job`,
and `ai.cancel_job` as thin authorized calls over `AiStudioService`.
The TypeScript and Python SDKs expose capabilities, models, generate, list,
get-generation, and cancel-generation through `client.ai`.
The n8n package is intentionally unchanged. Its current generic operation node
does not provide the capability-driven AI UX and explicit idempotency contract
needed for safe generation. Adding a superficial static AI operation would
violate the no-fake/no-duplicate architecture requirement.
## Migration order
PostgreSQL migrations remain explicit and are never applied on production
startup. Apply:
1. security migrations `0001` through `0004`
2. project migrations `0001_projects_foundation.sql`
3. `0002_project_resources.sql`
4. `0003_editor_persistence_rendering.sql`
5. `0004_ai_studio.sql`
6. social migrations in their documented order
`0004_ai_studio.sql` is additive. It adds project context and product-surface
classification to generation requests, ownership triggers, and query indexes.
SQLite metadata creation remains limited to local/test configuration.
## Deferred runtime certification
Provider credential tests, Docker/Linux runtime, Supabase PostgreSQL/RLS,
FFmpeg, and Python 3.10 production-image certification are intentionally
deferred to the final cross-product production certification phase. Production
settings must not be weakened for Termux.
|