Buckets:
| # Inference Engine — Planning | |
| ## Goal | |
| A resident generation server that runs as a normal AI Toolkit job (like a | |
| captioner job: `job: extension`, `process[0].type: InferenceEngine`), stays up | |
| until stopped from the UI, and serves generation requests for every arch the | |
| toolkit knows (image, video, audio). Each request names its model config; the | |
| engine loads/reloads only the components that differ from what is resident, | |
| drops what the request does not need, and streams progress + per-step latents | |
| back over the request's connection, ending with the finished file. | |
| This is Phase 3 of `toolkit/models/v2/PLANNING.md` ("live server with model | |
| hot-swap"); everything below builds on the unified v2 load API | |
| (`ModelClass.load(...)`, `aitk_post_load`, `component_load_kwargs`) that is | |
| already in place for every non-legacy arch. | |
| ## Survey (2026-09-10) | |
| What exists and is reused as-is: | |
| - **Job plumbing**: `jobs/ExtensionJob.py` -> `get_all_extensions_process_dict` | |
| -> our process class. The UI launches jobs via `ui/cron/actions/startJob.ts` | |
| (`python run.py <config>` with `AITK_JOB_ID`, `CUDA_VISIBLE_DEVICES`, | |
| detached, stdout to the job log). Stop = `Job.stop` flag in sqlite, which | |
| `BaseCaptioner` polls in a daemon thread and turns into a SIGINT. The DB | |
| status/step/info writers (`_update_status`, `_update_key`, retry-on-lock, | |
| stop watcher) live in `extensions_built_in/captioner/BaseCaptioner.py`. | |
| - **Generation path**: `BaseModel.generate_images([GenerateImageConfig])` does | |
| prompt encoding, control images, LoRA/network handling, then | |
| `generate_single_image` (per arch), then `gen_config.save_image_atomic` | |
| (png/jpg/webp image, animated webp video, wav/mp3 audio; `toolkit/audio/ | |
| make_video.py` has the mp4 writer). `testing/test_model_loading.py` proves | |
| this flow for 34 archs and holds realistic per-arch sample settings. | |
| - **Per-step hook point**: every arch samples through a diffusers-style | |
| scheduler object owned by the generation pipeline (`pipeline.scheduler.step` | |
| — diffusers pipelines, the vendored wan22/ltx/minimax/ace pipelines, the | |
| legacy monolith). Wrapping that one method yields (step, timestep, latents) | |
| for all of them without touching per-arch code. | |
| - **Component loading**: `OstrisModelMixin.load` / `load_model` is the single | |
| funnel every v2 component goes through (DiTs, TEs, VAEs, vision encoders, | |
| connectors, vocoders). Legacy monolith archs adopt their pipeline | |
| components into v2 wrappers at one post-load funnel (`adopt_component`). | |
| - **UI front**: `ui/cron/fileServer.ts` owns the public port and already | |
| spawns Next.js on an ephemeral loopback port and proxies to it (incl. | |
| websocket upgrades). Next.js route handlers stream `Response` bodies (the | |
| `/api/monitor` SSE route is the pattern). Bearer auth is a middleware on | |
| `/api/*`. `/api/files/*` serves job output folders. The job detail page | |
| already has a per-job `plugin.html` iframe tab. | |
| - **Latent previews**: ComfyUI's `comfy/latent_formats.py` (read-only | |
| reference) has per-family linear `latent_rgb_factors` tables for SD1/SDXL/ | |
| SD3/Flux/Flux2/Wan21/Wan22/LTXV/LTXAV/MiniMaxH3/Cosmos/HunyuanVideo. | |
| - **HTTP server libs**: `fastapi` 0.140 and `uvicorn` 0.51 are importable in | |
| this env but only as transitive deps of gradio; they are NOT pinned in | |
| `requirements.txt`. | |
| ## Decisions (proposed — confirm before build) | |
| 1. **Transport = one chunked HTTP response per generation request** (not | |
| WebSocket). A `POST /generate` stays open for the whole generation and | |
| streams binary frames; the Next.js proxy route passes the body through as | |
| a `ReadableStream`, so it works with or without the fileServer front, in | |
| docker, on Windows/macOS, and from `curl`. WebSocket would require the | |
| fileServer to be mandatory (Next route handlers cannot upgrade). Cancel is | |
| a separate `POST /cancel/{request_id}` (also fired when the client aborts | |
| the stream). | |
| 2. **Frame format**: length-prefixed binary frames | |
| `u32 header_len | json header | u64 payload_len | payload`. Header | |
| `{type, request_id, ...}`; types: `start` (resolved config, output kind, | |
| latent layout + rgb factors), `status` (load/quantize/encode stages, | |
| from `print_and_status_update`), `progress` (step/total/eta), `latent` | |
| (raw fp16 tensor bytes + shape), `result` (file path/url + media meta), | |
| `error`, `end`. Text frames have empty payload. | |
| 3. **Latents stream raw, projection happens client-side.** The `start` frame | |
| carries `latent_layout` (`BCHW` / `BCFHW` / `audio`), the arch's | |
| `latent_rgb_factors` (+bias) and any scaling, so the browser projects | |
| latents to a canvas with a channels x 3 matmul per pixel and needs no | |
| per-arch code. Server options `stream.latents = "raw" | "none"`, | |
| `stream.every_n_steps`, and for video `stream.max_frames` / | |
| `stream.downsample` to cap frame size (a 30-step video run at full latent | |
| res is hundreds of MB; images are ~0.5 MB/step). Archs without known | |
| factors still stream latents; the client falls back to a per-channel | |
| grayscale/PCA preview. A server-side `preview="jpeg"` mode (full or | |
| TAE decode) is a later add, not v1. | |
| 4. **Endpoint discovery = ephemeral loopback port + a file in the job | |
| folder.** Engine binds `127.0.0.1:0`, writes | |
| `<training_folder>/<job_name>/engine.json` (`host, port, token, pid, | |
| started_at`) and logs it. No fixed port, nothing network-exposed, no | |
| collisions, works identically in docker (same container as the UI) and | |
| on Windows (uvicorn has no named-pipe support, so a loopback TCP port is | |
| the portable stand-in for "no port"). No DB schema change (a new Job | |
| column would crash the jobs list for anyone who has not run `update_db`). | |
| The UI already resolves `<training_folder>/<job.name>` for `plugin.html`. | |
| Requests carry the per-launch `token` (proxy adds it) so other local | |
| processes cannot drive the engine. | |
| 5. **One engine process per GPU set, one generation at a time, FIFO queue** | |
| (`GET /queue`, `POST /cancel`). Multiple engine jobs on different GPUs are | |
| independent; the UI picks by job. | |
| 6. **Hot-swap via a component pool hooked into the mixin funnel** (no | |
| rewrites of the 35 holder `load_model`s). See "Component pool". | |
| 7. **LoRA in v1 = part of the transformer's pool identity** (`lora_path`, | |
| `assistant_lora_path`, network multiplier apply through the existing | |
| holder paths; a different LoRA means the transformer entry reloads; TEs/ | |
| VAEs stay shared). Hot-swappable unmerged LoRA on a resident transformer | |
| is v2. | |
| 8. **Server = FastAPI + uvicorn**, run on a thread inside the job process | |
| (generation stays on the main thread). Requires adding `fastapi` and | |
| `uvicorn` to `requirements.txt` (Jaret's call; fallback is a stdlib | |
| `ThreadingHTTPServer` with chunked responses — strictly worse, but zero | |
| deps). | |
| 9. **Model registry served by the engine.** `GET /models` returns every | |
| registered arch with capabilities + defaults (modality, control-image | |
| requirement, size divisibility, native res, default steps/CFG/frames/fps, | |
| latent layout, rgb factors). The table lives in python | |
| (`toolkit/models/registry.py`, seeded from `MODEL_TESTS` in | |
| `testing/test_model_loading.py` + per-holder class attrs) so the UI reads | |
| one source of truth at runtime instead of a second copy in `options.tsx`. | |
| ## Architecture | |
| ``` | |
| extensions_built_in/inference_engine/ | |
| __init__.py # InferenceEngineExtension (uid "InferenceEngine") | |
| InferenceEngine.py # process: UI status/stop plumbing, server lifecycle | |
| server.py # FastAPI app: /health /models /generate /cancel /queue /unload | |
| engine.py # Engine: request queue, holder cache, pool orchestration | |
| protocol.py # frame encoder (python) — mirrored by ui/src/utils/engineStream.ts | |
| latent_preview.py # latent layout + rgb factor tables per arch family | |
| toolkit/models/v2/pool.py # ComponentPool (identity, residency, eviction) | |
| toolkit/models/registry.py # arch capabilities/defaults (shared with tests + UI) | |
| toolkit/ui_job_status.py # DB status/stop helpers lifted from BaseCaptioner | |
| # (engine uses it; captioner can adopt later, optional) | |
| testing/test_inference_engine.py | |
| ui/src/app/api/inference/[...path]/route.ts # proxy to the engine (streams) | |
| ui/src/app/generate/page.tsx # Generate page | |
| ui/src/components/generate/* # model picker, prompt form, preview canvas, gallery | |
| ui/src/utils/engineStream.ts # frame parser + latent->canvas projection | |
| ``` | |
| ### Process (`InferenceEngine.py`) | |
| - Config: | |
| ```yaml | |
| job: extension | |
| config: | |
| name: inference_engine | |
| process: | |
| - type: InferenceEngine | |
| sqlite_db_path: ./aitk_db.db | |
| device: cuda | |
| engine: | |
| output_folder: <training_folder>/<name>/outputs # results, served by /api/files | |
| keep_resident_gb: 0 # 0 = keep only what the last request needs | |
| idle_unload_seconds: 0 # 0 = never auto-unload | |
| default_model: {arch: zimage, name_or_path: Tongyi-MAI/Z-Image-Turbo} # optional warm load | |
| ``` | |
| - `run()`: status "running / Starting server" -> start uvicorn on a thread | |
| (port 0) -> write `engine.json` -> optional warm load -> main thread loops | |
| on the request queue, executing generations; `info` reflects | |
| "Idle — zimage (Tongyi-MAI/Z-Image-Turbo)" / "Generating 12/30" / "Loading | |
| qwen_image: Quantizing transformer". Stop flag -> SIGINT (captioner | |
| pattern) -> cancel in-flight request, shut the server, delete | |
| `engine.json`, free the pool, status "stopped". | |
| - Also runnable headless: `python run.py config/inference_engine.yaml` | |
| prints the endpoint; no UI needed (useful for tests). | |
| ### Engine (`engine.py`) | |
| - `GenerateRequest`: `{model: ModelConfig kwargs (arch, name_or_path, | |
| extras_name_or_path, quantize, qtype, quantize_te, qtype_te, low_vram, | |
| layer_offloading..., lora_path, dtype), sample: GenerateImageConfig kwargs | |
| (prompt, negative_prompt, width, height, num_inference_steps, | |
| guidance_scale, seed, num_frames, fps, ctrl_img[_1..3] as server path or | |
| uploaded asset id, output_ext), stream: {latents, every_n_steps, | |
| max_frames}}`. Uploads: `POST /assets` stores control images under the job | |
| folder and returns an id/path. | |
| - Holder handling: `get_model_class(model_config)`, build the holder with | |
| the pool active, `load_model()`, then `pool.release_unused()`. The holder | |
| is kept as "active" and reused verbatim for the next request when its | |
| ModelConfig is identical (fast path: no rebuild). A new ModelConfig | |
| rebuilds the holder; the pool makes the rebuild cost = only the changed | |
| components. | |
| - Sampling: `sd.generate_images([gen])` with the step hook installed; | |
| `add_status_update_hook` feeds `status` frames; the `_after_sample_img` | |
| hook plus the saved file feed `result`. Output goes to | |
| `engine.output_folder` with a request-id filename; the frame carries the | |
| path and the `/api/files` url. Cancel = flag checked in the step hook, | |
| raising a `GenerationCancelled` that `generate_images` cleanup tolerates | |
| (it already restores device/rng state in its epilogue; verify the | |
| exception path leaves the holder usable). | |
| ### Step hook (`BaseModel`) | |
| - `BaseModel.sample_step_hook: Optional[Callable]` and, inside | |
| `generate_images`, a context manager that wraps `pipeline.scheduler.step` | |
| to call `hook(step_index, num_steps, timestep, latents)` with the | |
| scheduler's returned sample. Zero cost when no hook is set. Custom loops | |
| that bypass the scheduler (found by the per-arch streaming test) get an | |
| explicit `self._emit_sample_step(...)` call. Legacy monolith: same wrap on | |
| its pipeline scheduler. | |
| - Latent extraction from the DTO/carrier: video latents `BCFHW`, ltx/minimax | |
| AV extras (audio rows) ride the DTO — the `latent` frame sends the main | |
| tensor; audio extras optional. | |
| ### Component pool (`toolkit/models/v2/pool.py`) | |
| - Identity key = `(component class, resolved source [repo/subfolder or file | |
| path after comfy resolution], dtype, qtype (incl. ARA suffix), offload | |
| fraction, final device, lora identity for transformers)`. Computed inside | |
| `OstrisModelMixin.load` from exactly the args it already receives, so a | |
| holder's `component_load_kwargs(role)` output determines sharing (a TE | |
| loaded quantized convrot8 on cuda:0 is one entry; the same TE bf16 on cpu | |
| is another). | |
| - Wiring: a process-global `ComponentPool.current` context; `load` checks | |
| it first (hit = return resident module, skip source+post_load), registers | |
| on miss. Legacy `adopt_component` funnel registers too (consume side stays | |
| full reload for legacy archs). Tokenizers/processors pooled by repo. | |
| - Residency ops: `mark_used(key)` per request; `release_unused(keep)` = | |
| MemoryManager.detach/free, drop refs, `gc.collect()`, | |
| `torch.cuda.empty_cache()`; `bytes(key)` for accounting; | |
| `keep_resident_gb` LRU keep-warm; eviction before a load when | |
| `torch.cuda.mem_get_info` says the incoming component will not fit. | |
| - Safety: entries must not be mutated by holders in ways that change their | |
| identity. Audit as part of step 3: holders that merge LoRAs into weights, | |
| `.to(device)` moves under low_vram (device is in the key — a moved module | |
| must be moved back or re-keyed on release), `MemoryManager.attach` (offload | |
| is in the key), FakeTextEncoder swaps (`toolkit/unloader.py`), per-arch | |
| patches applied in `load_model` after `load` (e.g. vision-tower drops: | |
| fine, they are class-level config and thus part of the class identity). | |
| - Counters exposed on `/health`: hits/misses/evictions, resident entries with | |
| bytes and last-used — the tests assert on these. | |
| ### Server (`server.py`) | |
| - `GET /health` `{ok, busy, queue, active: {arch, name_or_path, config}, | |
| pool: [...], vram: {used, total}}`; `GET /models`; `POST /generate` | |
| (stream); `POST /cancel/{id}`; `GET /queue`; `POST /unload` | |
| (`{all: true}` or component keys); `POST /assets` (multipart upload); | |
| `GET /outputs` (list results). Token check on everything except `/health`. | |
| ### UI | |
| - Proxy: `ui/src/app/api/inference/[...path]/route.ts` — finds the running | |
| `job_type = "inference"` job (query param `job` to pick one when several | |
| GPUs run engines), reads its `engine.json` (cached a few seconds), | |
| forwards method/headers/body with the engine token, returns | |
| `new Response(upstream.body)` so chunked frames flow through untouched. | |
| `export const dynamic = 'force-dynamic'`, no body size limit issues | |
| (control images go through `/assets`, multipart). | |
| - `/generate` page (sidebar "Generate"): engine strip (pick/start/stop the | |
| engine job on a GPU — creates a job via `/api/jobs` with | |
| `job_type: 'inference'` and the config above, then start + startQueue, | |
| exactly like `CaptionDatasetModal.saveJob`; shows health), model panel | |
| (arch select from `/models`, name_or_path creatable select with the arch's | |
| defaults, quantize/qtype/low_vram/LoRA), prompt panel (prompt/negative, | |
| size with divisibility snapping, steps/CFG/seed/frames/fps, control image | |
| upload), preview panel (canvas fed by the stream, step/eta, cancel), | |
| results gallery (files from the engine output folder via `/api/files`, | |
| click to open; video/audio players). | |
| - Jobs list / action bar: `job_type === 'inference'` rows show "Inference | |
| Engine" with a link to `/generate`; stop only (no edit/restart-from-step | |
| UI). Dashboard active-job widget same treatment as caption. | |
| - `ui/src/utils/engineStream.ts`: async frame reader over `fetch` + | |
| `ReadableStream`, event emitter, `projectLatentToImageData(latent, layout, | |
| rgbFactors)`; video: shows a chosen frame (scrubber) — audio: heatmap. | |
| ### Tests (`testing/test_inference_engine.py`) | |
| Mirrors `test_model_loading.py` conventions (`--arch`, `--all` = subprocess | |
| per arch, `--allow-download`, HF offline default, SKIP classification, | |
| outputs under `testing/.engine_test_outputs/<arch>`), reusing its | |
| `MODEL_TESTS` entries for per-arch model/sample settings: | |
| 1. `--arch X`: start the engine in-process on an ephemeral port, POST | |
| `/generate`, consume the stream, assert `start` -> >=1 `progress` -> | |
| >=1 `latent` (shape/dtype/layout sanity, finite values) -> `result` | |
| (file exists, > 1 KB, right extension for the modality) -> `end`; | |
| record timings + VRAM like the loading test. | |
| 2. `--swap A,B[,C]`: sequential requests; assert pool hits for shared | |
| components (e.g. zimage -> zimage_l2p shares Qwen3 TE + KLVAE; wan21 -> | |
| wan22_5b shares UMT5; qwen_image -> qwen_image_edit shares TE + VAE), | |
| assert VRAM after B does not include A's DiT, assert a second A request | |
| is a holder fast path (no loads). | |
| 3. `--cancel`: cancel mid-run, assert the stream ends with `error/cancelled` | |
| and the next request on the same holder succeeds. | |
| 4. `--all`: (1) for every registry arch, then a summary + `report.md`. | |
| ## Steps | |
| Each step lands runnable + tested before the next. | |
| - [x] **1. Skeleton job + server** (2026-09-10) — `InferenceEngineExtension` | |
| (uid `InferenceEngine`), process with the sqlite status/stop plumbing | |
| lifted into `toolkit/ui_job_status.py`, FastAPI + uvicorn on | |
| `127.0.0.1:0` (`server.serve_in_thread`), `engine.json` in the job | |
| folder, `/health` `/models` `/generate` `/cancel` `/queue` `/unload` | |
| `/assets` `/outputs`, headless yaml. `fastapi`/`uvicorn` pinned in | |
| requirements_base.txt. | |
| - [x] **2. Streaming** — binary frame protocol (`protocol.py`, mirrored by | |
| `ui/src/utils/engineStream.ts`), `BaseModel.sample_step_hook` via a | |
| scheduler.step wrap in `generate_images` (+ `_emit_sample_step` for | |
| hand-rolled loops; ace_step wired), latent layouts BCHW/BCFHW/BLC with | |
| flux-style packed-token unpacking and video frame subsampling, per-arch | |
| preview tables (`latent_preview.py`, numbers from ComfyUI's | |
| latent_formats), cancel (step hook raises; holder verified reusable: | |
| the rerun after a cancel reproduced the same file byte-for-byte). | |
| Verified: zimage — 8 progress + 8 latent frames (1x16x64x64, Flux | |
| preview) + result; `--cancel` PASS. | |
| - [x] **3. Component pool + hot-swap** — `toolkit/models/v2/pool.py`; | |
| `OstrisModelMixin.load_model` consults/registers the active pool by | |
| source key (class, name_or_path, dtype, config, subfolder, comfy flag, | |
| kwargs); `aitk_post_load` is idempotent for an identical policy and | |
| detaches offloading before re-applying a different one. Engine: drop | |
| holder → begin_request → build → release_unused; CUDA OOM during a | |
| build frees untouched entries and retries once. Verified: | |
| zimage→zimage_l2p→zimage PASS (TE shared on the way back; l2p's | |
| transformer/VAE bypass the mixin so they reload — see step 4); | |
| wan21→wan22_5b: UMT5 TE shared (pool hit), wan21 DiT+VAE released. | |
| - [ ] **4. Coverage** — engine test across the registry. PASS so far | |
| (2026-09-10, 5090 with ~16 GB free next to a training job, convrot8): | |
| zimage (image, Flux preview), wan21 + wan22_5b (video BCFHW, animated | |
| webp), ace_step_15 (audio BLC, mp3), sd1 (legacy monolith: the hook | |
| is installed in its own generate_images via toolkit/sample_step_hook), | |
| zimage `--cancel`. Known gaps: holders whose component loads | |
| bypass `load_model` (zimage_l2p's from_pretrained + pixel conversion, | |
| legacy monolith adopt path) do not pool; omnigen2 builds its scheduler | |
| inside get_generation_pipeline (hook still applies — wrapping happens | |
| after); custom loops without scheduler.step need `_emit_sample_step` | |
| as they are found. Full `--all` sweep not run (shared GPU); run it | |
| when a free card is available. | |
| - [x] **5. UI** — `/api/inference/[...path]` streaming proxy | |
| (`ui/src/server/inferenceEngine.ts` resolves the running inference | |
| job's engine.json, checks the pid, injects the token), `/generate` | |
| page (engine start/stop per GPU, model + prompt forms fed by | |
| `/models`, live latent canvas with frame scrubber, results gallery via | |
| `/api/files`), sidebar entry, inference rows in the queue table / | |
| dashboard widget / job page, `startJob.ts` injects | |
| `engine.job_folder`. Type-checked and built in an isolated copy of the | |
| UI (the live `next start` on this box was not rebuilt — run | |
| `npm run build` to pick it up, the worker change needs it too). | |
| Proxy verified end to end against that isolated build with a copy of | |
| the db: status → health → models → streamed generate (first frame | |
| arrives immediately, progress frames live, result at the end) → stop | |
| via the Job.stop flag (engine exits 0, row 'stopped', engine.json | |
| removed, status reports no engine). | |
| - [ ] **6. Hardening** — done: FIFO queue + `/queue`, OOM-retry eviction, | |
| stale engine.json ignored when its pid is dead, `wait=1` JSON mode. | |
| Open: idle unload, `keep_resident_gb` LRU keep-warm, pre-load VRAM-fit | |
| eviction (today untouched entries are only freed after the build or | |
| on OOM, so a tight card pays a reload instead of sharing when the | |
| DiT loads first), Windows relay launch check, docker compose smoke, | |
| pooling for the custom-path holders. | |
| ## Open questions | |
| 1. Chunked HTTP stream vs WebSocket (decision 1). | |
| 2. Raw latents + client-side projection vs server-rendered previews | |
| (decision 3). Both can coexist later; which is v1? | |
| 3. `engine.json` in the job folder vs a new Job DB column (decision 4). | |
| 4. OK to add `fastapi` + `uvicorn` to `requirements.txt`? | |
| 5. LoRA as transformer identity in v1 (decision 7) — acceptable? | |
| 6. Should the per-arch registry (`toolkit/models/registry.py`) also become | |
| the source for `ui/src/app/jobs/new/options.tsx` in the same work, or | |
| stay engine-only for now? | |
| 7. Does the engine also need a non-streamed `POST /generate?wait=1` JSON | |
| mode for external scripts (cheap to add alongside)? | |
Xet Storage Details
- Size:
- 21.9 kB
- Xet hash:
- a31e8954a9241ab25341da2a58ed371b436bda61217814af28521a0bd9ec1408
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.