Deploy GPU detection cluster (Docker Space)

.dockerignore

@@ -0,0 +1,9 @@
+node_modules
+.git
+.idea
+.claude
+dist
+.DS_Store
+.npmrc
+*.md
+!README.md

.gitignore

@@ -0,0 +1,5 @@
+node_modules
+dist
+.DS_Store
+/.idea/
+/.claude/

ARTICLE.md

@@ -0,0 +1,530 @@
+# WebGPU + AI in the Browser: Real-Time Object Detection on Video
+
+*Part 3 of the WebGPU Video Processing series — builds on [Part 2: Hardware-Accelerated Video Processing](https://github.com/apssouza22/webgpu-video-ai/blob/main/ARTICLE.md)*
+
+> **Live demo:** [apssouza22.github.io/webgpu-video-ai](https://apssouza22.github.io/webgpu-video-ai) — open it in Chrome or Edge with a GPU, no install required.
+> **Source code:** [github.com/apssouza22/webgpu-video-ai](https://github.com/apssouza22/webgpu-video-ai)
+
+---
+
+In Part 2 we built a complete encode/decode pipeline: MediaBunny decoded video frames, WebGPU composited layers, and WebCodecs exported a final MP4 — all without a server. That pipeline processed video offline, one frame at a time, as fast as the CPU allowed.
+
+Part 3 adds a harder constraint: **real-time**. We want to run an AI object detection model on every video frame as it plays and draw bounding boxes on the GPU — without ever stalling the video. That means three things must happen concurrently: decoding, inference, and rendering. This article explains the architecture that makes that possible and why each piece of the stack was chosen.
+
+---
+
+## Why AI in the browser?
+
+Running models in the browser used to mean a Python server, a REST API, and a round-trip that made real-time interaction impractical. Three recent additions change that:
+
+- **WebGPU** exposes the GPU's thousands of cores directly to web code, making matrix math fast enough for neural networks.
+- **ONNX Runtime Web** compiles a standard model format to run on that GPU from JavaScript.
+- **Transformers.js** wraps ONNX Runtime with a Hugging Face–compatible API, so loading a model is one `pipeline()` call.
+
+Together they let you download a model once to the browser cache and run inference locally, frame after frame, with no server. You can see this running right now at the [live demo](https://apssouza22.github.io/webgpu-video-ai) — the model downloads on first load, then detects objects in the video entirely on your local GPU.
+
+---
+
+## System architecture overview
+
+Before diving into each component, here is how all the pieces fit together. Three execution contexts run concurrently and communicate through a small set of typed messages and GPU command queues:
+
+```mermaid
+flowchart TD
+    subgraph MT["MAIN THREAD"]
+        CP["CompositionPlayer - RAF loop 60fps"]
+        VP["VideoPlayer"]
+        MB["MediaBunny decoder"]
+        GC["GpuCompositor"]
+        GVR["GpuVideoRenderer - pass 1"]
+        GIR["GpuImageRenderer - pass 2"]
+        GDR["GpuDetectionRenderer - pass 3"]
+        VOD["VideoObjectDetection"]
+        AP["AudioPlayer - Web Audio API"]
+        CV["Canvas output"]
+    end
+
+    subgraph DW["DETECTION WORKER"]
+        DWT["detection.worker.ts"]
+        CIB["createImageBitmap resize"]
+        OC["OffscreenCanvas 576x576"]
+        TJ["Transformers.js pipeline()"]
+        ONNX["ONNX Runtime Web"]
+    end
+
+    subgraph GPU["GPU HARDWARE"]
+        RQ["Render command queue - video and detection WGSL"]
+        CQ["Compute command queue - ONNX inference WGSL"]
+    end
+
+    CP --> VP
+    CP --> AP
+    VP --> MB
+    MB -->|VideoFrame| GC
+    GC --> GVR
+    GC --> GIR
+    GC --> GDR
+    GVR --> CV
+    GIR --> CV
+    GDR --> CV
+    GDR --> RQ
+    VP --> VOD
+    VOD -->|"postMessage(frame, [frame]) zero-copy"| DWT
+    DWT --> CIB --> OC --> TJ --> ONNX --> CQ
+    DWT -->|"detect-result bounding boxes"| VOD
+    VOD -->|detections| GDR
+```
+
+The main thread never blocks on inference. The GPU hardware executes both rendering and inference command buffers, potentially in parallel on separate compute units.
+
+---
+
+## WebGPU: why it matters for AI
+
+Graphics and AI share a fundamental operation: multiply a large matrix by a vector, millions of times per second. A CPU executes these sequentially, one element at a time. A GPU executes them in parallel across thousands of shader invocations that all run simultaneously.
+
+```mermaid
+flowchart LR
+    subgraph CPU["CPU - Sequential  O(N^3)"]
+        c1["row 0"] --> c2["row 1"] --> c3["row 2"] --> c4["row N"]
+    end
+    subgraph GPU["GPU - Parallel  O(N) amortized"]
+        g1["core 0 - row 0"]
+        g2["core 1 - row 1"]
+        g3["core 2 - row 2"]
+        g4["core N - row N"]
+    end
+```
+
+The difference is not incremental. A modern GPU can sustain hundreds of TFLOPS on matrix operations while a CPU manages a few TFLOPS at best. For a neural network, every layer is a matrix multiply. More parallelism means more layers per second, which means more frames per second.
+
+WebGPU exposes this hardware through a low-overhead command model. Instead of issuing GPU calls one by one, you record a `GPUCommandEncoder`, accumulate all operations into it, and submit the whole batch in a single call:
+
+```typescript
+const encoder = this.device.createCommandEncoder();
+const pass = encoder.beginRenderPass({ ... });
+pass.setPipeline(this.pipeline);
+pass.setBindGroup(0, bindGroup);
+pass.draw(6);
+pass.end();
+this.device.queue.submit([encoder.finish()]);
+```
+
+The GPU processes the command buffer asynchronously. The CPU continues without waiting. This is the design that makes it possible to both render video and run AI inference at the same time.
+
+### Comparison with WebGL
+
+WebGL was designed for rendering, not compute. Doing matrix math in WebGL requires abusing fragment shaders — packing data into textures and writing shaders that treat each pixel as a matrix element. It works, but it is awkward, limited in precision, and cannot express modern network architectures cleanly.
+
+WebGPU adds a dedicated **compute shader** stage (`@compute`). ONNX Runtime Web uses compute shaders for every operation in a neural network: matrix multiplication, convolution, attention, normalization. The shader code is written in WGSL, the same language used for rendering, and runs on the same hardware. The AI pipeline and the rendering pipeline live in the same GPU process, sharing the same device and queue.
+
+---
+
+## Transformers.js and ONNX: how they work
+
+### ONNX: a universal model format
+
+ONNX (Open Neural Network Exchange) is a file format that describes a model as a computation graph. Nodes are operations (matrix multiply, ReLU, softmax), and edges are tensors that flow between them.
+
+```mermaid
+flowchart TD
+    INPUT["Input image - 576x576x3"]
+    BACKBONE["Backbone ResNet-D\nConv2D and BatchNorm\nconvolutional feature extraction"]
+    ENCODER["Encoder DETR head\nMulti-head self-attention\nMatMul and Softmax and Add"]
+    DECODER["Decoder\nCross-attention over object queries"]
+    BOX["Box head\nxmin, ymin, xmax, ymax"]
+    CLASS["Class head\n60 object labels"]
+
+    INPUT --> BACKBONE
+    BACKBONE -->|"feature maps"| ENCODER
+    ENCODER -->|"encoded tokens"| DECODER
+    DECODER --> BOX
+    DECODER --> CLASS
+```
+
+Training frameworks — PyTorch, TensorFlow, JAX — can all export to ONNX. ONNX Runtime then runs that graph on any supported backend: CPU, CUDA, CoreML, or **WebGPU**. When loaded in the browser, ONNX Runtime Web reads the graph, allocates GPU buffers for each tensor, and compiles the operations into WGSL shaders at load time — which is why the first inference ("warmup") always takes longer than subsequent ones.
+
+The model used here, `onnx-community/rfdetr_medium-ONNX`, is a real-time detection transformer. It takes a `576×576` image as input and returns bounding boxes with class labels and confidence scores. The ONNX file is hosted on Hugging Face and downloaded into the browser's cache the first time you load the [demo](https://apssouza22.github.io/webgpu-video-ai).
+
+### Transformers.js: pipeline abstraction
+
+Transformers.js wraps ONNX Runtime Web with a high-level API modeled after the Hugging Face `transformers` Python library. In [`detection.worker.ts`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/detection/detection.worker.ts), loading the model is a single call:
+
+```typescript
+const MODEL_ID = 'onnx-community/rfdetr_medium-ONNX';
+
+detector = await pipeline('object-detection', MODEL_ID, {
+  device: 'webgpu',   // compile ONNX graph to WGSL compute shaders
+  dtype: 'fp16',      // faster WebGPU inference (fp32 is more accurate)
+  progress_callback: (progress) => {
+    // streams download progress: file name + percentage
+    post({ type: 'status', message: `Downloading ${progress.file} (${percent}%)…` });
+  },
+});
+```
+
+`device: 'webgpu'` tells Transformers.js to compile the ONNX graph onto the GPU via WebGPU. The `progress_callback` is useful because the model is tens of megabytes: you can display a real download bar rather than a silent spinner — which is exactly what the [demo](https://apssouza22.github.io/webgpu-video-ai) shows in the status area during first load.
+
+After the `await` resolves, `detector` is a callable function. You pass it an image (here an `OffscreenCanvas`) and get back structured results:
+
+```typescript
+const results = await detect(canvas, { threshold: 0.5, percentage: true });
+// [{ label: 'person', score: 0.91, box: { xmin: 0.12, ymin: 0.08, xmax: 0.44, ymax: 0.97 } }, ...]
+```
+
+`percentage: true` returns box coordinates normalized to 0–1, which matches the coordinate system the WebGPU detection shader uses directly — no conversion needed.
+
+---
+
+## The parallel processing problem
+
+Here is the core tension. Video at 30 fps gives you 33 ms per frame. Inference alone exceeds that budget on most hardware:
+
+```mermaid
+gantt
+    title Per-frame time budget at 30fps (33ms window)
+    dateFormat x
+    axisFormat %Lms
+
+    section Main thread
+        Decode frame (MediaBunny)  : 0, 5
+        Cache image overlays       : 5, 6
+        GPU render 3 passes        : 6, 8
+
+    section Worker thread
+        AI inference RF-DETR       : crit, 0, 80
+```
+
+RF-DETR on a mid-range GPU takes 30–80 ms. If inference ran synchronously on the main thread, the video would stutter every single frame.
+
+The key insight is that inference and rendering **do not need to happen in the same frame**. The detections from frame N are still valid for frames N+1 and N+2 — objects do not teleport. You can display slightly stale boxes and refresh them as new inference results arrive asynchronously. That observation unlocks the architecture.
+
+---
+
+## Web Workers: the isolation layer
+
+A Web Worker is a JavaScript execution context that runs on a separate OS thread. It shares no memory with the main thread, has its own event loop, and cannot access the DOM. What it *can* access: WebGPU, `OffscreenCanvas`, `VideoFrame`, and `fetch`.
+
+Isolating the model inside a Worker gives you three critical properties:
+
+**1. The main thread is never blocked.** Model loading, WGSL shader compilation, and inference all happen on the worker thread. The `requestAnimationFrame` loop that drives video playback continues at 60 fps regardless of what the model is doing.
+
+**2. The GPU is shared.** Both the worker's WebGPU inference pipeline and the main thread's rendering pipeline target the same physical GPU. They share no JavaScript objects, but the hardware executes their command buffers concurrently when possible.
+
+**3. `VideoFrame` transfer is zero-copy.** `VideoFrame` is a transferable object. When you pass it in the `postMessage` transfer list, the underlying GPU texture buffer moves to the worker without copying pixel data:
+
+```mermaid
+sequenceDiagram
+    participant MT as Main Thread
+    participant W as Detection Worker
+
+    MT->>MT: decode VideoFrame (owns GPU buffer)
+    MT->>W: postMessage(frame, [frame]) - zero copy, no memcpy
+    Note over MT: frame is neutered, cannot be used
+    Note over W: frame received, owns the GPU buffer
+    W->>W: createImageBitmap resize to 576x576
+    W->>W: ONNX inference on GPU
+    W-->>MT: detect-result with bounding boxes
+    Note over W: worker calls frame.close()
+```
+
+A 1280×720 frame at 4 bytes per pixel is 3.5 MB. Copying that on every frame would add ~100 MB/s of memory bandwidth overhead — the zero-copy transfer makes the overhead negligible:
+
+```typescript
+// ObjectDetector.ts — main thread
+this.worker.postMessage(
+  { type: 'detect', id, threshold, frame },
+  [frame],   // ← transfer list: frame is moved, not copied
+);
+// frame is now null/neutered on the main thread
+```
+
+### The typed message protocol
+
+The worker communicates through a narrow, explicitly typed protocol defined in [`workerMessages.ts`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/detection/workerMessages.ts):
+
+```mermaid
+sequenceDiagram
+    participant MT as Main Thread
+    participant W as Detection Worker
+
+    MT->>W: type init
+    W-->>MT: type status - Downloading model
+    W-->>MT: type status - Compiling RF-DETR shaders
+    W-->>MT: type ready
+
+    MT->>W: type detect, id 42, threshold 0.5, frame VideoFrame
+    Note over W: resize to 576x576, run ONNX graph
+    W-->>MT: type detect-result, id 42, results array
+
+    MT->>W: type detect, id 43, frame VideoFrame
+    W-->>MT: type error, id 43, error message
+```
+
+The numeric `id` field is essential: it matches each response to its originating request. The `ObjectDetector` class on the main thread stores pending promises in a `Map<id, {resolve, reject}>`:
+
+```typescript
+// ObjectDetector.ts
+async detect(frame: VideoFrame, options: { threshold: number }): Promise<DetectionResult[]> {
+  const id = this.nextId++;
+  return new Promise((resolve, reject) => {
+    this.pending.set(id, { resolve, reject });
+    this.worker.postMessage({ type: 'detect', id, threshold: options.threshold, frame }, [frame]);
+  });
+}
+
+// on 'detect-result' message:
+const pending = this.pending.get(message.id);
+pending?.resolve(message.results);
+this.pending.delete(message.id);
+```
+
+This is a clean bridge from a message-passing interface to async/await without a library.
+
+---
+
+## The detection pipeline end to end
+
+### Step 1 — Frame preprocessing inside the worker
+
+RF-DETR expects a `576×576` image. Video frames are typically `1280×720` or larger. The worker resizes the transferred `VideoFrame` using `createImageBitmap` with explicit resize parameters, then draws it onto a persistent `OffscreenCanvas`:
+
+```typescript
+// detection.worker.ts
+const MODEL_INPUT_SIZE = 576;
+
+async function runDetection(frame: VideoFrame, threshold: number) {
+  const { canvas, ctx } = getPreprocessSurface(); // reused 576×576 OffscreenCanvas
+
+  const bitmap = await createImageBitmap(frame, {
+    resizeWidth: MODEL_INPUT_SIZE,
+    resizeHeight: MODEL_INPUT_SIZE,   // GPU-accelerated resize
+  });
+  ctx.drawImage(bitmap, 0, 0);
+  bitmap.close();
+  frame.close(); // release the transferred VideoFrame
+
+  return detect(canvas, { threshold, percentage: true });
+}
+```
+
+`createImageBitmap` with resize parameters is GPU-accelerated in browsers that support it. The `OffscreenCanvas` is created once with `{ willReadFrequently: true }` so the browser optimizes its backing store for the one CPU readback that Transformers.js needs to build the input tensor.
+
+```mermaid
+flowchart TD
+    VF["VideoFrame 1280x720\ntransferred from main thread"]
+    CIB["createImageBitmap\nresizeWidth 576, resizeHeight 576\nGPU-accelerated downscale"]
+    OC["OffscreenCanvas 576x576\nreused each frame, willReadFrequently true"]
+    T1["1. Read pixels from canvas"]
+    T2["2. Build Float32 input tensor"]
+    T3["3. Upload tensor to GPU buffer"]
+    T4["4. Run ONNX graph via WGSL compute shaders"]
+    T5["5. Readback output tensor to CPU"]
+    RES["DetectionResult array\nlabel, score, box coords normalized 0 to 1"]
+
+    VF -->|"frame.close() after bitmap"| CIB
+    CIB --> OC
+    OC --> T1 --> T2 --> T3 --> T4 --> T5 --> RES
+```
+
+### Step 2 — Throttling: one frame in flight
+
+Running inference takes longer than a single video frame. The `VideoObjectDetection` class in [`VideoObjectDetection.ts`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/player/VideoObjectDetection.ts) implements a simple but effective throttle: if a detection is already running, the incoming frame is held as a "pending" replacement. When inference completes, it immediately starts on the pending frame, dropping any intermediate frames:
+
+```typescript
+schedule(videoFrame: VideoFrame): void {
+  if (this.detectionBusy) {
+    this.pendingDetectionFrame?.close();            // discard previous pending
+    this.pendingDetectionFrame = new VideoFrame(videoFrame);
+    return;
+  }
+  void this.runDetection(new VideoFrame(videoFrame));
+}
+```
+
+The state machine looks like this:
+
+```mermaid
+flowchart TD
+    A["schedule videoFrame"] --> B{detectionBusy?}
+    B -- NO --> C["runDetection frame\ndetectionBusy = true"]
+    B -- YES --> D["pendingFrame = frame\nclose previous pending if any"]
+    C --> E["await ONNX inference"]
+    E --> F["store detections\ndetectionBusy = false"]
+    F --> G{pendingFrame?}
+    G -- YES --> H["runDetection pendingFrame\npendingFrame = null"]
+    G -- NO --> I["idle - await next schedule"]
+    H --> E
+    D -.->|waits implicitly| F
+```
+
+At any given moment there is at most one detection in flight and one pending. The detection rate tracks hardware capability automatically — a fast GPU detects every frame, a slow GPU skips frames without any explicit rate-limiting code.
+
+### Step 3 — Version-gating stale results
+
+There is a subtle race: between submitting a detection and receiving the result, the user might seek to a different position in the video. Applying old bounding boxes to new content looks wrong. An incrementing version counter handles this:
+
+```typescript
+private async runDetection(detectionFrame: VideoFrame): Promise<void> {
+  const detectionVersion = ++this.detectionVersion;
+  // ...inference (may take 50ms)...
+  if (detectionVersion !== this.detectionVersion) {
+    return; // a newer detection was started — discard this result
+  }
+  this.detections = toGpuDetections(results);
+}
+```
+
+Every call to `runDetection` increments the counter. On completion, it checks that the counter has not moved. If it has, the stale result is silently dropped.
+
+---
+
+## Rendering detections on the GPU
+
+The bounding boxes returned by the model are JavaScript objects. To draw them without looping over pixels in JavaScript, they are packed into a WebGPU uniform buffer and sent to a WGSL shader that draws the boxes entirely on the GPU.
+
+### The uniform buffer layout
+
+```
+Offset 0  ┌─────────────────────────────────┐
+          │  count    (u32,  4 bytes)        │
+Offset 4  │  lineWidth (f32, 4 bytes)        │
+Offset 8  │  _pad      (vec2f, 8 bytes)      │
+Offset 16 ├─────────────────────────────────┤  ← boxes start
+          │  box[0].bounds (vec4f, 16 bytes) │  xmin, ymin, xmax, ymax
+          │  box[0].color  (vec4f, 16 bytes) │  r, g, b, alpha
+Offset 48 ├─────────────────────────────────┤
+          │  box[1].bounds (16 bytes)        │
+          │  box[1].color  (16 bytes)        │
+          ├─────────────────────────────────┤
+          │  ...                             │
+          │  box[31]                         │
+Offset 1040└─────────────────────────────────┘
+Total: 16 + 32 × 32 = 1,040 bytes (fixed, allocated once at startup)
+```
+
+Fixed at 32 boxes maximum — enough for any real scene, and the constant size means the buffer is allocated once at startup and reused every frame with a single `writeBuffer` call.
+
+### The fragment shader draws borders analytically
+
+Instead of uploading one quad per bounding box (which would require a dynamic vertex buffer), the shader uses a **full-screen triangle pass** and tests each pixel analytically against all boxes in the uniform buffer:
+
+```wgsl
+fn onBorder(uv: vec2f, bounds: vec4f, lineWidth: f32) -> bool {
+  let insideX = step(bounds.x, uv.x) * step(uv.x, bounds.z);
+  let insideY = step(bounds.y, uv.y) * step(uv.y, bounds.w);
+
+  let nearLeft   = abs(uv.x - bounds.x) < lineWidth;
+  let nearRight  = abs(uv.x - bounds.z) < lineWidth;
+  let nearTop    = abs(uv.y - bounds.y) < lineWidth;
+  let nearBottom = abs(uv.y - bounds.w) < lineWidth;
+
+  return (nearLeft  && insideY > 0.5) ||
+         (nearRight && insideY > 0.5) ||
+         (nearTop   && insideX > 0.5) ||
+         (nearBottom && insideX > 0.5);
+}
+
+@fragment
+fn fragmentMain(input: VertexOutput) -> @location(0) vec4f {
+  var color = vec4f(0.0);  // transparent by default
+  for (var i = 0u; i < detection.count; i++) {
+    let box = detection.boxes[i];
+    if (onBorder(input.uv, box.bounds, detection.lineWidth)) {
+      color = vec4f(box.color.rgb, 1.0);
+    }
+  }
+  return color;
+}
+```
+
+Every fragment (pixel) independently tests whether it falls on the border of any detection box. The GPU runs these tests in parallel across all pixels simultaneously — ~921,600 pixels × 5 box tests = ~4.6M comparisons completed in a single render pass in under 1ms. No geometry uploads, no dynamic buffers, no JavaScript loops over pixels.
+
+### The compositing order
+
+The [`GpuCompositor`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/gpu/GpuCompositor.ts) issues three render passes per frame in a fixed order, all targeting the same canvas texture:
+
+```mermaid
+flowchart TD
+    CT["Canvas texture 1280x720"]
+    P1["Pass 1 - GpuVideoRenderer\nloadOp clear\nVideoFrame via importExternalTexture\nalpha blend"]
+    S1["video frame rendered"]
+    P2["Pass 2 - GpuImageRenderer\nloadOp load\nHTMLImageElement to rgba8unorm cached texture\nalpha blend"]
+    S2["video plus image overlays"]
+    P3["Pass 3 - GpuDetectionRenderer\nloadOp load\nuniform buffer with box list\nalpha blend"]
+    OUT["Final composited frame\nvideo, overlays, and detection boxes"]
+
+    CT --> P1 --> S1 --> P2 --> S2 --> P3 --> OUT
+```
+
+Each pass's `loadOp: 'load'` means "start from what the previous pass wrote." WebGPU serializes passes within a queue submission, so each pass sees the accumulated result of all prior passes. The detection layer always renders last, so boxes are always visible on top of any overlay.
+
+---
+
+## The warmup problem
+
+ONNX Runtime Web compiles WGSL shaders for each operation in the model graph on first use. On a complex model like RF-DETR, this takes one to three seconds and produces a visible freeze if it happens during playback. The fix is one inference before the user presses play:
+
+```typescript
+// main.ts
+setStatus('Warming up RF-DETR shaders (first inference)…');
+await player.getVideoPlayer().warmupDetection(0);
+```
+
+This decodes frame 0, runs it through the full detection pipeline, and discards the result. The side effect is that all WGSL shaders are compiled and cached by the WebGPU driver. Every subsequent inference hits already-compiled pipelines and runs at steady-state speed. You can observe the warmup message in the status bar when you first open the [demo](https://apssouza22.github.io/webgpu-video-ai).
+
+---
+
+## Required HTTP headers
+
+Transformers.js uses `SharedArrayBuffer` internally for efficient WASM memory management. `SharedArrayBuffer` is gated behind cross-origin isolation — the page must be served with:
+
+```
+Cross-Origin-Opener-Policy: same-origin
+Cross-Origin-Embedder-Policy: require-corp
+```
+
+Without them, `SharedArrayBuffer` is unavailable and Transformers.js either falls back to a slower code path or fails outright. In Vite, these are set in the dev server configuration:
+
+```typescript
+// vite.config.ts
+server: {
+  headers: {
+    'Cross-Origin-Opener-Policy': 'same-origin',
+    'Cross-Origin-Embedder-Policy': 'require-corp',
+  },
+}
+```
+
+You must also set them in production. A missing COEP header is one of the most common reasons a Transformers.js app works in development but silently fails after deployment. Check them first when debugging.
+
+---
+
+## Key lessons
+
+- **Transfer, do not copy.** `VideoFrame` is a transferable. Pass it with `[frame]` in the `postMessage` transfer list to move the GPU buffer to the worker with zero memory copy. Copying 3.5 MB of pixel data per frame destroys throughput.
+
+- **Warm up before you need it.** ONNX shader compilation happens on first inference. Run one dummy call at startup — the [demo](https://apssouza22.github.io/webgpu-video-ai) does this before revealing the play button — so playback never triggers a compile stall mid-video.
+
+- **One detection in flight.** Queuing detections faster than the model can process them causes memory pressure and bursty latency. A single pending-frame slot gives you the freshest possible result at any hardware speed, without any explicit rate-limiting.
+
+- **Version-gate results.** Any async operation that feeds the render loop needs a staleness check. An incrementing counter is enough. Without it, seeking causes stale boxes to flash on screen.
+
+- **Analyze pixels in the shader, not JavaScript.** Drawing bounding boxes as a full-screen analytical test is faster than uploading per-box geometry because the GPU runs the test across all pixels simultaneously with zero JavaScript overhead per pixel.
+
+- **COEP/COOP are not optional.** Set them in both development and production. Check them first when debugging a Transformers.js deployment.
+
+---
+
+## What this architecture enables
+
+The pipeline described here runs a 60-class object detector on live video at whatever rate the hardware supports — from a few detections per second on integrated graphics to near-frame-rate on a discrete GPU — without ever stalling video playback. You can see the detection FPS counter in the [live demo](https://apssouza22.github.io/webgpu-video-ai) updating in real time as the model runs.
+
+More broadly, the same structure applies to any model you want to run on video: pose estimation, segmentation, depth prediction, face recognition. The worker isolation, the zero-copy transfer, the warmup, and the GPU rendering layer are all model-agnostic. Swap the Transformers.js `pipeline()` call and update the WGSL uniform struct, and the architecture carries the new task without structural changes.
+
+The browser is no longer a thin client that sends video to a server for processing. It is a capable ML runtime with direct access to the GPU, a mature async concurrency model, and zero-copy primitives that make real-time inference practical. The gap between what you can build in the browser and what requires a backend is narrowing fast.
+
+---
+
+*Full source code: [github.com/apssouza22/webgpu-video-ai](https://github.com/apssouza22/webgpu-video-ai)*
+*Live demo: [apssouza22.github.io/webgpu-video-ai](https://apssouza22.github.io/webgpu-video-ai)*

CLAUDE.md

@@ -0,0 +1,46 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+```bash
+npm run dev      # Vite (5180) + detection broker API (8787)
+npm run build    # Type-check (tsc) + Vite bundle → docs/
+npm run preview  # Serve the production build locally
+npm run dev:web  # Vite only
+npm run dev:api  # API broker only (8787)
+```
+
+No test suite exists — tested manually in the browser and via `curl`.
+
+## Architecture
+
+This is a **browser GPU detection cluster**: RF-DETR inference runs in a Web Worker on WebGPU, while a Node broker exposes HTTP so backends can submit detection jobs.
+
+### Data flow
+
+```
+curl / Python / app
+  → Node broker (server/) — task queue, image fetch
+  → SSE → browser host (src/host/detectionNodeHost.ts)
+  → ObjectDetector → detection.worker.ts (Transformers.js, WebGPU)
+  → results POST back to broker → HTTP response
+```
+
+### Key modules
+
+| Path | Role |
+|------|------|
+| `src/host/detectionNodeHost.ts` | Registers host, listens for SSE tasks, runs inference |
+| `src/detection/ObjectDetector.ts` | Main-thread API to the detection worker |
+| `src/detection/detection.worker.ts` | RF-DETR on `OffscreenCanvas`; model: `onnx-community/rfdetr_medium-ONNX` |
+| `server/index.ts` | Express broker: `/v1/detect`, host registration, SSE stream |
+
+### Web Worker and CORS requirements
+
+The detection worker uses Transformers.js with `device: 'webgpu'`. Vite sends `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp` (see `vite.config.ts`) for SharedArrayBuffer / WASM.
+
+### Build output
+
+Production build targets `docs/` with base path `/webgpu-video-ai/` for GitHub Pages. Transformers.js is excluded from `optimizeDeps` because models download at runtime.

Dockerfile

@@ -0,0 +1,24 @@
+# Hugging Face Spaces (Docker SDK) — serves UI + detection broker on port 7860
+FROM node:22-bookworm-slim AS build
+
+WORKDIR /app
+COPY package.json package-lock.json ./
+RUN npm ci
+COPY . .
+RUN npm run build:space
+
+FROM node:22-bookworm-slim
+
+WORKDIR /app
+ENV NODE_ENV=production
+ENV PORT=7860
+ENV SERVE_STATIC=1
+
+COPY package.json package-lock.json ./
+RUN npm ci
+COPY --from=build /app/docs ./docs
+COPY --from=build /app/server ./server
+COPY --from=build /app/shared ./shared
+
+EXPOSE 7860
+CMD ["npm", "start"]

README.md

@@ -1,12 +1,115 @@
 ---
-title: Webgpu Cluster
-emoji: 🏃
-colorFrom: red
-colorTo: pink
+title: GPU Detection Cluster
+emoji: 🎮
+colorFrom: blue
+colorTo: purple
 sdk: docker
+app_port: 7860
 pinned: false
-license: mit
-short_description: 'A demo of how to create a webgpu cluster '
+models:
+  - onnx-community/rfdetr_medium-ONNX
+  - HuggingFaceTB/SmolVLM-500M-Instruct
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# GPU detection cluster
+
+Browser-hosted **RF-DETR** object detection exposed over HTTP. Inference runs on **WebGPU** in a dedicated worker; a **Node broker** queues tasks and lets you call detection from `curl`, Python, or any backend.
+
+```text
+curl / Python  →  Node broker (task queue)  →  SSE  →  browser host (WebGPU)
+```
+
+## Quick start
+
+```bash
+npm install
+npm run dev
+```
+
+1. Open **http://localhost:5180**
+2. Choose a **model to share**, pick a host id (e.g. `my-gpu-node`), and click **Start hosting** (loads the model on WebGPU; keep the tab open)
+3. Open **http://localhost:5180/monitor.html** to see all registered hosts and copy curl examples
+4. From another terminal:
+
+```bash
+curl -X POST 'http://localhost:5180/v1/detect' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "host": "my-gpu-node",
+    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
+    "threshold": 0.5
+  }'
+```
+
+Response:
+
+```json
+{
+  "task_id": "...",
+  "host": "my-gpu-node",
+  "threshold": 0.5,
+  "detections": [
+    { "label": "cat", "score": 0.92, "box": { "xmin": 10, "ymin": 20, "xmax": 200, "ymax": 180 } }
+  ]
+}
+```
+
+`npm run dev` runs **Vite** (port 5180) and the **API** (port 8787). Vite proxies `/v1` and `/api` to the broker.
+
+## Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `POST` | `/v1/detect` | Run object detection (waits for host; body below) |
+| `POST` | `/v1/describe` | Describe an image with SmolVLM (waits for host; body below) |
+| `GET` | `/v1/hosts` | List registered hosts and online status |
+| `GET` | `/v1/models` | Cluster models (id, label, whether the host app implements them) |
+| — | `/monitor.html` | Dashboard — live host list and curl examples |
+| `GET` | `/v1/tasks/:id` | Task status / results |
+| `POST` | `/api/hosts/register` | Called by the browser host |
+| `GET` | `/api/hosts/stream?host_id=` | SSE — browser host receives tasks |
+
+**`POST /v1/detect` body:**
+
+- `host` (required) — id from the browser host page
+- `image_url` or `image_base64` (required) — broker fetches URLs server-side
+- `threshold` (optional, default `0.5`)
+
+## Scripts
+
+- `npm run dev` — Vite + API together
+- `npm run dev:api` — API only (port 8787)
+- `npm run dev:web` — Vite only (proxies API when both run)
+- `npm run build` — typecheck + production bundle to `docs/`
+- `npm run start:api` — run broker without watch
+
+## Cluster models
+
+Available models are defined in `shared/clusterModels.ts`. The host page dropdown and `GET /v1/models` both read from that list. Add a new entry there, then wire loading in `src/host/detectionNodeHost.ts` (`ensureModelLoaded`) and any broker routes.
+
+| Model id | Endpoint | Worker |
+|----------|----------|--------|
+| `rfdetr-medium` | `POST /v1/detect` | `src/detection/detection.worker.ts` |
+| `smolvlm-500m` | `POST /v1/describe` | `src/videodescription/videodescription.worker.ts` |
+
+Object detection uses `onnx-community/rfdetr_medium-ONNX` via `@huggingface/transformers` with `device: 'webgpu'`. Video description uses `HuggingFaceTB/SmolVLM-500M-Instruct` with quantized vision/decoder weights (same setup as the [SmolVLM realtime WebGPU demo](https://huggingface.co/spaces/webml-community/smolvlm-realtime-webgpu)). Models download from Hugging Face on first load. Inference runs in a **Web Worker**; images arrive as base64 from the broker and are converted to `VideoFrame` on the host page.
+
+**`POST /v1/describe` body:**
+
+- `host` (required)
+- `image_url` or `image_base64` (required)
+- `instruction` (optional, default `"What do you see?"`)
+- `max_new_tokens` (optional, default `100`)
+
+## Requirements
+
+- Browser with **WebGPU** (Chrome or Edge desktop recommended)
+- Dev server COOP/COEP headers in `vite.config.ts` (required for Transformers.js)
+
+## Hugging Face Spaces
+
+Deploy as a **Docker Space** (broker + UI on port 7860). Step-by-step instructions: [SPACES.md](./SPACES.md).
+
+## License
+
+MIT

SPACES.md

@@ -0,0 +1,80 @@
+# Deploy to Hugging Face Spaces
+
+This app needs a **Docker Space** (Node broker + static WebGPU UI on one origin with COOP/COEP headers).
+
+## 1. Log in
+
+```bash
+hf auth login
+```
+
+Create a token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) (write access).
+
+## 2. Create the Space
+
+Replace `YOUR_USERNAME` with your Hugging Face username:
+
+```bash
+hf repos create YOUR_USERNAME/gpu-video-cluster \
+  --type space \
+  --space-sdk docker \
+  --exist-ok
+```
+
+Optional: pick hardware (default is free CPU):
+
+```bash
+hf repos create YOUR_USERNAME/gpu-video-cluster \
+  --type space \
+  --space-sdk docker \
+  --flavor cpu-basic \
+  --exist-ok
+```
+
+## 3. Push the repo
+
+From the project root:
+
+```bash
+hf upload . YOUR_USERNAME/gpu-video-cluster \
+  --repo-type space \
+  --exclude ".git/*" \
+  --exclude "node_modules/*" \
+  --exclude ".idea/*" \
+  --exclude ".claude/*" \
+  --commit-message "Deploy GPU detection cluster"
+```
+
+The Space builds from `Dockerfile`, runs `npm run build:space`, and starts `npm start` on port **7860**.
+
+## 4. Use the Space
+
+1. Open `https://YOUR_USERNAME-gpu-video-cluster.hf.space/`
+2. Pick a host id and click **Start hosting** (WebGPU runs in the browser tab)
+3. Open `/monitor.html` for curl examples
+4. Call the API on the same URL, e.g.:
+
+```bash
+curl -X POST 'https://YOUR_USERNAME-gpu-video-cluster.hf.space/v1/detect' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "host": "my-gpu-node",
+    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
+    "threshold": 0.5
+  }'
+```
+
+## Local Docker test
+
+```bash
+docker build -t gpu-video-cluster .
+docker run --rm -p 7860:7860 gpu-video-cluster
+```
+
+Then open http://localhost:7860/
+
+## Notes
+
+- **WebGPU** runs in the visitor’s browser (Chrome/Edge desktop). The Space container only hosts the broker and static files.
+- First model load downloads weights from the Hub into the browser cache.
+- GitHub Pages builds use `npm run build` (base `/webgpu-video-ai/`). Spaces use `npm run build:space` (`vite build --mode space`, base `/`) via the Dockerfile.

index.html

@@ -0,0 +1,128 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>GPU detection cluster host</title>
+    <style>
+      :root {
+        color-scheme: dark;
+        font-family: system-ui, sans-serif;
+      }
+      body {
+        margin: 0;
+        padding: 24px;
+        background: #111;
+        color: #eee;
+      }
+      main {
+        max-width: 720px;
+        margin: 0 auto;
+      }
+      h1 {
+        margin: 0 0 8px;
+        font-size: 1.4rem;
+      }
+      p {
+        color: #bdbdbd;
+        line-height: 1.5;
+      }
+      a {
+        color: #9eb6ff;
+      }
+      .row {
+        display: flex;
+        flex-wrap: wrap;
+        gap: 12px;
+        margin: 16px 0;
+      }
+      input,
+      select,
+      button {
+        font: inherit;
+      }
+      input,
+      select {
+        flex: 1;
+        min-width: 200px;
+        padding: 10px 12px;
+        border: 1px solid #333;
+        border-radius: 8px;
+        background: #0d1117;
+        color: #eee;
+      }
+      select {
+        cursor: pointer;
+      }
+      .field {
+        flex: 1;
+        min-width: 200px;
+      }
+      .field label {
+        display: block;
+        margin-bottom: 6px;
+        font-size: 12px;
+        color: #8b949e;
+        text-transform: uppercase;
+        letter-spacing: 0.04em;
+      }
+      .field select {
+        width: 100%;
+        box-sizing: border-box;
+      }
+      button {
+        padding: 10px 16px;
+        border: 0;
+        border-radius: 8px;
+        background: #4f46e5;
+        color: #fff;
+        cursor: pointer;
+      }
+      button:disabled {
+        opacity: 0.5;
+        cursor: not-allowed;
+      }
+      pre {
+        margin: 0;
+        padding: 16px;
+        overflow-x: auto;
+        border: 1px solid #2b2b2b;
+        border-radius: 12px;
+        background: #0d1117;
+        font: 12px/1.5 ui-monospace, monospace;
+        color: #c9d1d9;
+        white-space: pre-wrap;
+      }
+      #status {
+        margin-top: 16px;
+        font: 13px/1.5 ui-monospace, monospace;
+        color: #7eb8ff;
+        white-space: pre-wrap;
+      }
+    </style>
+  </head>
+  <body>
+    <main>
+      <p style="margin: 0 0 16px; font-size: 14px">
+        <a href="/monitor.html">Cluster monitor</a> — view all connected hosts
+      </p>
+      <h1>GPU detection cluster host</h1>
+      <p>
+        Keep this tab open. Inference runs here on WebGPU; the Node broker forwards jobs from
+        curl, Python, or other apps (<code>/v1/detect</code> or <code>/v1/describe</code>).
+      </p>
+      <div class="row">
+        <input id="host-id" type="text" placeholder="Host id" autocomplete="off" />
+        <div class="field">
+          <label for="model-select">Model to share</label>
+          <select id="model-select" aria-label="Model to share with the cluster"></select>
+        </div>
+        <button id="register-btn" type="button">Start hosting</button>
+      </div>
+      <p>Example once hosting is active:</p>
+      <pre id="curl-example"></pre>
+      <p id="status">Initializing…</p>
+    </main>
+    <script type="module" src="/src/pages/detectionNodeHost.ts"></script>
+  </body>
+</html>

monitor.html

@@ -0,0 +1,244 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>GPU cluster monitor</title>
+    <style>
+      :root {
+        color-scheme: dark;
+        font-family: system-ui, sans-serif;
+      }
+      body {
+        margin: 0;
+        padding: 24px;
+        background: #111;
+        color: #eee;
+      }
+      main {
+        max-width: 960px;
+        margin: 0 auto;
+      }
+      header {
+        display: flex;
+        flex-wrap: wrap;
+        align-items: flex-start;
+        justify-content: space-between;
+        gap: 16px;
+        margin-bottom: 24px;
+      }
+      h1 {
+        margin: 0 0 8px;
+        font-size: 1.4rem;
+      }
+      p {
+        color: #bdbdbd;
+        line-height: 1.5;
+        margin: 0 0 8px;
+      }
+      a {
+        color: #9eb6ff;
+      }
+      .nav {
+        display: flex;
+        gap: 16px;
+        font-size: 14px;
+      }
+      .stats {
+        display: flex;
+        flex-wrap: wrap;
+        gap: 12px;
+        margin-bottom: 24px;
+      }
+      .stat {
+        flex: 1;
+        min-width: 140px;
+        padding: 14px 16px;
+        border: 1px solid #2b2b2b;
+        border-radius: 12px;
+        background: #0d1117;
+      }
+      .stat-label {
+        font-size: 12px;
+        color: #8b949e;
+        text-transform: uppercase;
+        letter-spacing: 0.04em;
+      }
+      .stat-value {
+        margin-top: 6px;
+        font-size: 1.25rem;
+        font-weight: 600;
+      }
+      .stat-value.ok {
+        color: #3fb950;
+      }
+      .stat-value.warn {
+        color: #d29922;
+      }
+      .stat-value.err {
+        color: #f85149;
+      }
+      #hosts {
+        display: flex;
+        flex-direction: column;
+        gap: 16px;
+      }
+      .host-card {
+        border: 1px solid #2b2b2b;
+        border-radius: 12px;
+        background: #0d1117;
+        overflow: hidden;
+      }
+      .host-header {
+        display: flex;
+        flex-wrap: wrap;
+        align-items: center;
+        justify-content: space-between;
+        gap: 12px;
+        padding: 16px;
+        border-bottom: 1px solid #21262d;
+      }
+      .host-id {
+        font: 600 1rem/1.2 ui-monospace, monospace;
+      }
+      .badge {
+        display: inline-block;
+        padding: 4px 10px;
+        border-radius: 999px;
+        font-size: 12px;
+        font-weight: 600;
+      }
+      .badge.online {
+        background: #23863633;
+        color: #3fb950;
+      }
+      .badge.offline {
+        background: #6e768133;
+        color: #8b949e;
+      }
+      .host-meta {
+        padding: 0 16px 16px;
+        font-size: 13px;
+        color: #8b949e;
+      }
+      .host-meta span + span::before {
+        content: ' · ';
+      }
+      .curl-details {
+        border-top: 1px solid #21262d;
+      }
+      .curl-details summary {
+        padding: 12px 16px;
+        font-size: 13px;
+        color: #9eb6ff;
+        cursor: pointer;
+        list-style: none;
+        user-select: none;
+      }
+      .curl-details summary::-webkit-details-marker {
+        display: none;
+      }
+      .curl-details summary::before {
+        content: '▸ ';
+        display: inline-block;
+        transition: transform 0.15s ease;
+      }
+      .curl-details[open] summary::before {
+        transform: rotate(90deg);
+      }
+      .curl-details .summary-hide {
+        display: none;
+      }
+      .curl-details[open] .summary-show {
+        display: none;
+      }
+      .curl-details[open] .summary-hide {
+        display: inline;
+      }
+      .curl-body {
+        border-top: 1px solid #21262d;
+      }
+      pre {
+        margin: 0;
+        padding: 16px;
+        overflow-x: auto;
+        background: #010409;
+        font: 12px/1.5 ui-monospace, monospace;
+        color: #c9d1d9;
+        white-space: pre-wrap;
+      }
+      .empty {
+        padding: 32px;
+        text-align: center;
+        border: 1px dashed #30363d;
+        border-radius: 12px;
+        color: #8b949e;
+      }
+      #refresh-note {
+        margin-top: 16px;
+        font-size: 12px;
+        color: #6e7681;
+      }
+      #error {
+        margin-bottom: 16px;
+        padding: 12px 16px;
+        border-radius: 8px;
+        background: #490202;
+        color: #ffa198;
+        font-size: 14px;
+        display: none;
+      }
+      button.copy {
+        margin: 12px 16px 0;
+        padding: 6px 12px;
+        border: 1px solid #30363d;
+        border-radius: 6px;
+        background: #21262d;
+        color: #c9d1d9;
+        font: inherit;
+        font-size: 12px;
+        cursor: pointer;
+      }
+      button.copy:hover {
+        background: #30363d;
+      }
+    </style>
+  </head>
+  <body>
+    <main>
+      <header>
+        <div>
+          <h1>GPU cluster monitor</h1>
+          <p>
+            Registered browser hosts and live connection status from
+            <code>GET /v1/hosts</code>.
+          </p>
+        </div>
+        <nav class="nav">
+          <a href="/">Register a host</a>
+        </nav>
+      </header>
+
+      <div id="error"></div>
+
+      <section class="stats" aria-label="Cluster summary">
+        <div class="stat">
+          <div class="stat-label">Broker</div>
+          <div id="broker-status" class="stat-value">…</div>
+        </div>
+        <div class="stat">
+          <div class="stat-label">Hosts online</div>
+          <div id="online-count" class="stat-value">—</div>
+        </div>
+        <div class="stat">
+          <div class="stat-label">Hosts registered</div>
+          <div id="total-count" class="stat-value">—</div>
+        </div>
+      </section>
+
+      <div id="hosts"></div>
+      <p id="refresh-note">Refreshing every 3s</p>
+    </main>
+    <script type="module" src="/src/pages/clusterMonitor.ts"></script>
+  </body>
+</html>

package.json

@@ -0,0 +1,31 @@
+{
+  "name": "gpu-video-cluster",
+  "private": true,
+  "version": "1.0.0",
+  "type": "module",
+  "description": "Browser GPU cluster for RF-DETR object detection via HTTP",
+  "scripts": {
+    "dev": "concurrently -k -n web,api -c blue,green \"vite\" \"tsx watch server/index.ts\"",
+    "dev:web": "vite",
+    "dev:api": "tsx watch server/index.ts",
+    "build": "tsc && tsc -p tsconfig.server.json && vite build",
+    "build:space": "tsc && tsc -p tsconfig.server.json && vite build --mode space",
+    "preview": "vite preview",
+    "start": "SERVE_STATIC=1 tsx server/index.ts",
+    "start:api": "tsx server/index.ts"
+  },
+  "dependencies": {
+    "@huggingface/transformers": "^4.2.0",
+    "cors": "^2.8.5",
+    "express": "^5.1.0"
+  },
+  "devDependencies": {
+    "@types/cors": "^2.8.17",
+    "@types/express": "^5.0.3",
+    "@webgpu/types": "^0.1.66",
+    "concurrently": "^9.2.0",
+    "tsx": "^4.20.3",
+    "typescript": "~5.9.3",
+    "vite": "^7.3.2"
+  }
+}

server/imageLoader.ts

@@ -0,0 +1,33 @@
+import type {DetectRequestBody} from './types.js';
+
+export async function resolveImagePayload(
+  body: DetectRequestBody,
+): Promise<{imageBase64: string; mimeType: string}> {
+  if (body.image_base64) {
+    const base64 = body.image_base64.replace(/^data:[^;]+;base64,/, '');
+    return {
+      imageBase64: base64,
+      mimeType: body.image_type ?? 'image/jpeg',
+    };
+  }
+
+  if (body.image_url) {
+    const response = await fetch(body.image_url);
+    if (!response.ok) {
+      throw new Error(`Failed to fetch image_url (${response.status})`);
+    }
+
+    const buffer = Buffer.from(await response.arrayBuffer());
+    const mimeType =
+      response.headers.get('content-type')?.split(';')[0]?.trim() ??
+      body.image_type ??
+      'image/jpeg';
+
+    return {
+      imageBase64: buffer.toString('base64'),
+      mimeType,
+    };
+  }
+
+  throw new Error('Provide image_url or image_base64');
+}

server/index.ts

@@ -0,0 +1,354 @@
+import path from 'node:path';
+import {fileURLToPath} from 'node:url';
+import cors from 'cors';
+import express from 'express';
+import type {Request, Response} from 'express';
+import {
+  CLUSTER_MODELS,
+  DEFAULT_CLUSTER_MODEL_ID,
+  isValidClusterModelId,
+} from '../shared/clusterModels.js';
+import {resolveImagePayload} from './imageLoader.js';
+import {initSse, writeSse, writeSseComment} from './sse.js';
+import {store} from './store.js';
+import type {
+  CompleteTaskBody,
+  DescribeRequestBody,
+  DetectRequestBody,
+  RegisterHostBody,
+} from './types.js';
+
+const PORT = Number(process.env.PORT ?? 8787);
+const TASK_TIMEOUT_MS = Number(process.env.TASK_TIMEOUT_MS ?? 120_000);
+const SERVE_STATIC = process.env.SERVE_STATIC === '1';
+const staticDir = path.resolve(
+  path.dirname(fileURLToPath(import.meta.url)),
+  '../docs',
+);
+
+const app = express();
+
+app.use((_req, res, next) => {
+  res.setHeader('Cross-Origin-Opener-Policy', 'same-origin');
+  res.setHeader('Cross-Origin-Embedder-Policy', 'require-corp');
+  next();
+});
+
+app.use(cors());
+app.use(express.json({limit: '32mb'}));
+
+app.get('/health', (_req, res) => {
+  res.json({status: 'ok'});
+});
+
+app.get('/v1/hosts', (_req, res) => {
+  res.json({hosts: store.listHosts()});
+});
+
+app.get('/v1/models', (_req, res) => {
+  res.json({
+    models: CLUSTER_MODELS.map((model) => ({
+      id: model.id,
+      label: model.label,
+      description: model.description,
+      implemented: model.implemented,
+    })),
+  });
+});
+
+app.post('/api/hosts/register', (req, res) => {
+  const body = req.body as RegisterHostBody;
+  if (!body?.id?.trim()) {
+    res.status(400).json({error: 'id is required'});
+    return;
+  }
+
+  const modelId = body.model?.trim() || DEFAULT_CLUSTER_MODEL_ID;
+  if (!isValidClusterModelId(modelId)) {
+    res.status(400).json({
+      error: `Unknown model '${modelId}'. Supported: ${CLUSTER_MODELS.map((m) => m.id).join(', ')}`,
+    });
+    return;
+  }
+
+  const host = store.registerHost(body.id.trim(), modelId);
+  res.json({
+    id: host.id,
+    model: host.model,
+    online: store.isHostOnline(host.id),
+  });
+});
+
+app.get('/api/hosts/stream', (req, res) => {
+  const hostId = String(req.query.host_id ?? '').trim();
+  if (!hostId) {
+    res.status(400).json({error: 'host_id query parameter is required'});
+    return;
+  }
+
+  if (!store.getHost(hostId)) {
+    res.status(404).json({error: `Host '${hostId}' not found`});
+    return;
+  }
+
+  initSse(res);
+
+  const heartbeat = setInterval(() => {
+    store.heartbeat(hostId);
+    writeSseComment(res, 'ping');
+  }, 15_000);
+
+  store.attachHostStream(hostId, {
+    write: (event, data) => writeSse(res, event, data),
+    close: () => {
+      clearInterval(heartbeat);
+      if (!res.writableEnded) {
+        res.end();
+      }
+    },
+  });
+
+  writeSse(res, 'ready', {host_id: hostId});
+
+  req.on('close', () => {
+    clearInterval(heartbeat);
+    store.detachHostStream(hostId);
+  });
+});
+
+app.post('/api/tasks/:taskId/processing', (req, res) => {
+  const taskId = req.params.taskId;
+  store.markProcessing(taskId);
+  res.json({status: 'ok'});
+});
+
+app.post('/api/tasks/:taskId/complete', (req, res) => {
+  const taskId = req.params.taskId;
+  const body = req.body as CompleteTaskBody;
+
+  try {
+    if (body.status === 'done') {
+      const task = store.completeTask(taskId, {
+        status: 'done',
+        results: body.results ?? [],
+      });
+      res.json({status: 'ok', task_id: task.id});
+      return;
+    }
+
+    const task = store.completeTask(taskId, {
+      status: 'error',
+      error: body.error ?? 'Detection failed',
+    });
+    res.json({status: 'ok', task_id: task.id, error: task.error});
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    res.status(404).json({error: message});
+  }
+});
+
+app.get('/v1/tasks/:taskId', (req, res) => {
+  const task = store.getTask(req.params.taskId);
+  if (!task) {
+    res.status(404).json({error: 'Task not found'});
+    return;
+  }
+
+  res.json({
+    id: task.id,
+    kind: task.kind,
+    host: task.hostId,
+    status: task.status,
+    results: task.results,
+    description: task.description,
+    error: task.error,
+    created_at: task.createdAt,
+    started_at: task.startedAt,
+    completed_at: task.completedAt,
+  });
+});
+
+app.post('/v1/detect', async (req, res) => {
+  await handleDetect(req, res);
+});
+
+app.post('/v1/describe', async (req, res) => {
+  await handleDescribe(req, res);
+});
+
+async function handleDetect(req: Request, res: Response): Promise<void> {
+  const body = req.body as DetectRequestBody;
+
+  if (!body?.host?.trim()) {
+    res.status(400).json({
+      error: {
+        message: "Missing required field 'host'. Open the app in a browser and register a host id.",
+        type: 'invalid_request_error',
+      },
+    });
+    return;
+  }
+
+  const hostId = body.host.trim();
+
+  if (!store.getHost(hostId)) {
+    res.status(404).json({
+      error: {
+        message: `Host '${hostId}' not found. Register it in the browser host page first.`,
+        type: 'invalid_request_error',
+      },
+    });
+    return;
+  }
+
+  if (!store.isHostOnline(hostId)) {
+    res.status(503).json({
+      error: {
+        message: `Host '${hostId}' is offline. Keep the browser host tab open with the model loaded.`,
+        type: 'host_unavailable',
+      },
+    });
+    return;
+  }
+
+  try {
+    const image = await resolveImagePayload(body);
+    const threshold = body.threshold ?? 0.5;
+
+    const task = store.createDetectTask(hostId, {
+      imageBase64: image.imageBase64,
+      mimeType: image.mimeType,
+      threshold,
+    });
+
+    const completed = await store.waitForTask(task.id, TASK_TIMEOUT_MS);
+
+    if (completed.status === 'error') {
+      res.status(500).json({
+        error: {
+          message: completed.error ?? 'Detection failed',
+          type: 'detection_error',
+        },
+        task_id: completed.id,
+      });
+      return;
+    }
+
+    res.json({
+      task_id: completed.id,
+      host: completed.hostId,
+      threshold: completed.threshold,
+      detections: completed.results ?? [],
+    });
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    const status = message.includes('timed out') ? 504 : 400;
+    res.status(status).json({
+      error: {
+        message,
+        type: status === 504 ? 'timeout' : 'invalid_request_error',
+      },
+    });
+  }
+}
+
+async function handleDescribe(req: Request, res: Response): Promise<void> {
+  const body = req.body as DescribeRequestBody;
+
+  if (!body?.host?.trim()) {
+    res.status(400).json({
+      error: {
+        message: "Missing required field 'host'. Open the app in a browser and register a host id.",
+        type: 'invalid_request_error',
+      },
+    });
+    return;
+  }
+
+  const hostId = body.host.trim();
+
+  if (!store.getHost(hostId)) {
+    res.status(404).json({
+      error: {
+        message: `Host '${hostId}' not found. Register it in the browser host page first.`,
+        type: 'invalid_request_error',
+      },
+    });
+    return;
+  }
+
+  if (!store.isHostOnline(hostId)) {
+    res.status(503).json({
+      error: {
+        message: `Host '${hostId}' is offline. Keep the browser host tab open with the model loaded.`,
+        type: 'host_unavailable',
+      },
+    });
+    return;
+  }
+
+  try {
+    const image = await resolveImagePayload(body);
+    const instruction = body.instruction?.trim() || 'What do you see?';
+    const maxNewTokens = body.max_new_tokens ?? 100;
+
+    const task = store.createDescribeTask(hostId, {
+      imageBase64: image.imageBase64,
+      mimeType: image.mimeType,
+      instruction,
+      maxNewTokens,
+    });
+
+    const completed = await store.waitForTask(task.id, TASK_TIMEOUT_MS);
+
+    if (completed.status === 'error') {
+      res.status(500).json({
+        error: {
+          message: completed.error ?? 'Description failed',
+          type: 'description_error',
+        },
+        task_id: completed.id,
+      });
+      return;
+    }
+
+    res.json({
+      task_id: completed.id,
+      host: completed.hostId,
+      instruction: completed.instruction,
+      max_new_tokens: completed.maxNewTokens,
+      description: completed.description ?? '',
+    });
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    const status = message.includes('timed out') ? 504 : 400;
+    res.status(status).json({
+      error: {
+        message,
+        type: status === 504 ? 'timeout' : 'invalid_request_error',
+      },
+    });
+  }
+}
+
+if (SERVE_STATIC) {
+  app.use(express.static(staticDir));
+  app.get('/monitor.html', (_req, res) => {
+    res.sendFile(path.join(staticDir, 'monitor.html'));
+  });
+  app.get('/', (_req, res) => {
+    res.sendFile(path.join(staticDir, 'index.html'));
+  });
+}
+
+app.listen(PORT, '0.0.0.0', () => {
+  console.log(`Detection broker listening on http://0.0.0.0:${PORT}`);
+  if (SERVE_STATIC) {
+    console.log(`  Static UI: / and /monitor.html`);
+  }
+  console.log(`  POST /v1/detect`);
+  console.log(`  POST /v1/describe`);
+  console.log(`  GET  /v1/hosts`);
+  console.log(`  GET  /v1/models`);
+  console.log(`  GET  /api/hosts/stream?host_id=...`);
+});

server/sse.ts

@@ -0,0 +1,17 @@
+import type {Response} from 'express';
+
+export function initSse(res: Response): void {
+  res.setHeader('Content-Type', 'text/event-stream');
+  res.setHeader('Cache-Control', 'no-cache');
+  res.setHeader('Connection', 'keep-alive');
+  res.flushHeaders?.();
+}
+
+export function writeSse(res: Response, event: string, data: unknown): void {
+  res.write(`event: ${event}\n`);
+  res.write(`data: ${JSON.stringify(data)}\n\n`);
+}
+
+export function writeSseComment(res: Response, comment: string): void {
+  res.write(`: ${comment}\n\n`);
+}

server/store.ts

@@ -0,0 +1,248 @@
+import {randomUUID} from 'node:crypto';
+import type {
+  DetectionResultJson,
+  DetectionTask,
+  HostRecord,
+  HostStream,
+} from './types.js';
+
+type TaskWaiter = {
+  resolve: (task: DetectionTask) => void;
+  reject: (error: Error) => void;
+  timeout: ReturnType<typeof setTimeout>;
+};
+
+export class BrokerStore {
+  private readonly hosts = new Map<string, HostRecord>();
+  private readonly tasks = new Map<string, DetectionTask>();
+  private readonly waiters = new Map<string, TaskWaiter>();
+
+  registerHost(id: string, model: string): HostRecord {
+    const existing = this.hosts.get(id);
+    const host: HostRecord = {
+      id,
+      model,
+      registeredAt: existing?.registeredAt ?? Date.now(),
+      lastHeartbeat: Date.now(),
+      stream: existing?.stream,
+    };
+    this.hosts.set(id, host);
+    return host;
+  }
+
+  attachHostStream(id: string, stream: HostStream): void {
+    const host = this.hosts.get(id);
+    if (!host) {
+      throw new Error(`Host '${id}' is not registered`);
+    }
+
+    host.stream?.close();
+    host.stream = stream;
+    host.lastHeartbeat = Date.now();
+    this.flushPendingTasks(id);
+  }
+
+  detachHostStream(id: string): void {
+    const host = this.hosts.get(id);
+    if (host) {
+      host.stream = undefined;
+    }
+  }
+
+  heartbeat(id: string): void {
+    const host = this.hosts.get(id);
+    if (host) {
+      host.lastHeartbeat = Date.now();
+    }
+  }
+
+  listHosts(): Array<{
+    id: string;
+    model: string;
+    online: boolean;
+    registeredAt: number;
+    lastHeartbeat: number;
+  }> {
+    return [...this.hosts.values()].map((host) => ({
+      id: host.id,
+      model: host.model,
+      online: Boolean(host.stream),
+      registeredAt: host.registeredAt,
+      lastHeartbeat: host.lastHeartbeat,
+    }));
+  }
+
+  getHost(id: string): HostRecord | undefined {
+    return this.hosts.get(id);
+  }
+
+  isHostOnline(id: string): boolean {
+    const host = this.hosts.get(id);
+    return Boolean(host?.stream);
+  }
+
+  createDetectTask(
+    hostId: string,
+    payload: {imageBase64: string; mimeType: string; threshold: number},
+  ): DetectionTask {
+    return this.createTask(hostId, {
+      kind: 'detect',
+      imageBase64: payload.imageBase64,
+      mimeType: payload.mimeType,
+      threshold: payload.threshold,
+    });
+  }
+
+  createDescribeTask(
+    hostId: string,
+    payload: {
+      imageBase64: string;
+      mimeType: string;
+      instruction: string;
+      maxNewTokens: number;
+    },
+  ): DetectionTask {
+    return this.createTask(hostId, {
+      kind: 'describe',
+      imageBase64: payload.imageBase64,
+      mimeType: payload.mimeType,
+      instruction: payload.instruction,
+      maxNewTokens: payload.maxNewTokens,
+    });
+  }
+
+  private createTask(
+    hostId: string,
+    payload: {
+      kind: DetectionTask['kind'];
+      imageBase64: string;
+      mimeType: string;
+      threshold?: number;
+      instruction?: string;
+      maxNewTokens?: number;
+    },
+  ): DetectionTask {
+    const host = this.hosts.get(hostId);
+    if (!host) {
+      throw new Error(`Host '${hostId}' not found. Register the browser host first.`);
+    }
+
+    const task: DetectionTask = {
+      id: randomUUID(),
+      hostId,
+      kind: payload.kind,
+      status: 'pending',
+      imageBase64: payload.imageBase64,
+      mimeType: payload.mimeType,
+      threshold: payload.threshold,
+      instruction: payload.instruction,
+      maxNewTokens: payload.maxNewTokens,
+      createdAt: Date.now(),
+    };
+
+    this.tasks.set(task.id, task);
+    this.dispatchTask(task);
+    return task;
+  }
+
+  waitForTask(taskId: string, timeoutMs: number): Promise<DetectionTask> {
+    const task = this.tasks.get(taskId);
+    if (!task) {
+      return Promise.reject(new Error(`Task '${taskId}' not found`));
+    }
+
+    if (task.status === 'done' || task.status === 'error') {
+      return Promise.resolve(task);
+    }
+
+    return new Promise<DetectionTask>((resolve, reject) => {
+      const timeout = setTimeout(() => {
+        this.waiters.delete(taskId);
+        reject(new Error(`Task '${taskId}' timed out after ${timeoutMs}ms`));
+      }, timeoutMs);
+
+      this.waiters.set(taskId, {
+        resolve,
+        reject,
+        timeout,
+      });
+    });
+  }
+
+  markProcessing(taskId: string): void {
+    const task = this.tasks.get(taskId);
+    if (!task) {
+      return;
+    }
+
+    task.status = 'processing';
+    task.startedAt = Date.now();
+  }
+
+  completeTask(
+    taskId: string,
+    outcome:
+      | {status: 'done'; results?: DetectionResultJson[]; description?: string}
+      | {status: 'error'; error: string},
+  ): DetectionTask {
+    const task = this.tasks.get(taskId);
+    if (!task) {
+      throw new Error(`Task '${taskId}' not found`);
+    }
+
+    task.completedAt = Date.now();
+
+    if (outcome.status === 'done') {
+      task.status = 'done';
+      task.results = outcome.results;
+      task.description = outcome.description;
+    } else {
+      task.status = 'error';
+      task.error = outcome.error;
+    }
+
+    const waiter = this.waiters.get(taskId);
+    if (waiter) {
+      clearTimeout(waiter.timeout);
+      this.waiters.delete(taskId);
+      if (task.status === 'error') {
+        waiter.reject(new Error(task.error ?? 'Detection failed'));
+      } else {
+        waiter.resolve(task);
+      }
+    }
+
+    return task;
+  }
+
+  getTask(taskId: string): DetectionTask | undefined {
+    return this.tasks.get(taskId);
+  }
+
+  private dispatchTask(task: DetectionTask): void {
+    const host = this.hosts.get(task.hostId);
+    if (!host?.stream) {
+      return;
+    }
+
+    host.stream.write('task', {
+      id: task.id,
+      kind: task.kind,
+      threshold: task.threshold,
+      instruction: task.instruction,
+      max_new_tokens: task.maxNewTokens,
+      image_base64: task.imageBase64,
+      mime_type: task.mimeType,
+    });
+  }
+
+  private flushPendingTasks(hostId: string): void {
+    for (const task of this.tasks.values()) {
+      if (task.hostId === hostId && task.status === 'pending') {
+        this.dispatchTask(task);
+      }
+    }
+  }
+}
+
+export const store = new BrokerStore();

server/types.ts

@@ -0,0 +1,76 @@
+export interface DetectionBox {
+  xmin: number;
+  ymin: number;
+  xmax: number;
+  ymax: number;
+}
+
+export interface DetectionResultJson {
+  label: string;
+  score: number;
+  box: DetectionBox;
+}
+
+export interface HostRecord {
+  id: string;
+  model: string;
+  registeredAt: number;
+  lastHeartbeat: number;
+  stream?: HostStream;
+}
+
+export interface HostStream {
+  write: (event: string, data: unknown) => void;
+  close: () => void;
+}
+
+export type TaskStatus = 'pending' | 'processing' | 'done' | 'error';
+
+export type ClusterTaskKind = 'detect' | 'describe';
+
+export interface DetectionTask {
+  id: string;
+  hostId: string;
+  kind: ClusterTaskKind;
+  status: TaskStatus;
+  imageBase64: string;
+  mimeType: string;
+  threshold?: number;
+  instruction?: string;
+  maxNewTokens?: number;
+  results?: DetectionResultJson[];
+  description?: string;
+  error?: string;
+  createdAt: number;
+  startedAt?: number;
+  completedAt?: number;
+}
+
+export interface DetectRequestBody {
+  host: string;
+  threshold?: number;
+  image_url?: string;
+  image_base64?: string;
+  image_type?: string;
+}
+
+export interface DescribeRequestBody {
+  host: string;
+  instruction?: string;
+  max_new_tokens?: number;
+  image_url?: string;
+  image_base64?: string;
+  image_type?: string;
+}
+
+export interface RegisterHostBody {
+  id: string;
+  model?: string;
+}
+
+export interface CompleteTaskBody {
+  status: 'done' | 'error';
+  results?: DetectionResultJson[];
+  description?: string;
+  error?: string;
+}

shared/clusterModels.ts

@@ -0,0 +1,38 @@
+/** Models a browser host can expose to the cluster. Add entries as new workers are wired up. */
+export interface ClusterModelDefinition {
+  id: string;
+  label: string;
+  description: string;
+  /** When false, the host UI shows the option but cannot start hosting yet. */
+  implemented: boolean;
+}
+
+export const CLUSTER_MODELS: readonly ClusterModelDefinition[] = [
+  {
+    id: 'rfdetr-medium',
+    label: 'Object detection',
+    description: 'RF-DETR Medium (COCO) on WebGPU',
+    implemented: true,
+  },
+  {
+    id: 'smolvlm-500m',
+    label: 'Video description',
+    description: 'SmolVLM-500M-Instruct vision-language model on WebGPU',
+    implemented: true,
+  },
+] as const;
+
+export const DEFAULT_CLUSTER_MODEL_ID = CLUSTER_MODELS[0].id;
+
+export function getClusterModel(id: string): ClusterModelDefinition | undefined {
+  return CLUSTER_MODELS.find((model) => model.id === id);
+}
+
+export function isValidClusterModelId(id: string): boolean {
+  return CLUSTER_MODELS.some((model) => model.id === id);
+}
+
+export function isImplementedClusterModel(id: string): boolean {
+  const model = getClusterModel(id);
+  return Boolean(model?.implemented);
+}

src/detection/ObjectDetector.ts

@@ -0,0 +1,122 @@
+import type {DetectionResult, WorkerRequest, WorkerResponse} from './workerMessages';
+
+export type {DetectionResult} from './workerMessages';
+
+type PendingRequest = {
+  resolve: (results: DetectionResult[]) => void;
+  reject: (error: Error) => void;
+};
+
+export class ObjectDetector {
+  private readonly worker: Worker;
+  private readonly pending = new Map<number, PendingRequest>();
+  private nextId = 0;
+  private readonly ready: Promise<void>;
+  private readySettled = false;
+  private resolveReady!: () => void;
+  private rejectReady!: (error: Error) => void;
+  private onStatus?: (message: string) => void;
+
+  private constructor(worker: Worker, onStatus?: (message: string) => void) {
+    this.worker = worker;
+    this.onStatus = onStatus;
+
+    this.ready = new Promise<void>((resolve, reject) => {
+      this.resolveReady = resolve;
+      this.rejectReady = reject;
+    });
+
+    this.worker.addEventListener('message', (event: MessageEvent<WorkerResponse>) => {
+      this.handleMessage(event.data);
+    });
+    this.worker.addEventListener('error', (event) => {
+      this.failReady(new Error(event.message || 'Detection worker failed'));
+    });
+
+    this.worker.postMessage({type: 'init'} satisfies WorkerRequest);
+  }
+
+  static async create(onStatus?: (message: string) => void): Promise<ObjectDetector> {
+    const worker = new Worker(new URL('./detection.worker.ts', import.meta.url), {
+      type: 'module',
+    });
+    const detector = new ObjectDetector(worker, onStatus);
+    await detector.ready;
+    return detector;
+  }
+
+  async warmup(frame: VideoFrame, threshold = 0.5): Promise<void> {
+    await this.detect(frame, {threshold});
+  }
+
+  async detect(
+    frame: VideoFrame,
+    options: {threshold: number},
+  ): Promise<DetectionResult[]> {
+    await this.ready;
+
+    const id = this.nextId++;
+
+    return new Promise<DetectionResult[]>((resolve, reject) => {
+      this.pending.set(id, {resolve, reject});
+      this.worker.postMessage(
+        {type: 'detect', id, threshold: options.threshold, frame} satisfies WorkerRequest,
+        [frame],
+      );
+    });
+  }
+
+  dispose(): void {
+    for (const {reject} of this.pending.values()) {
+      reject(new Error('Detection worker terminated'));
+    }
+    this.pending.clear();
+    this.worker.terminate();
+  }
+
+  private handleMessage(message: WorkerResponse): void {
+    if (message.type === 'status') {
+      this.onStatus?.(message.message);
+      return;
+    }
+
+    if (message.type === 'ready') {
+      if (!this.readySettled) {
+        this.readySettled = true;
+        this.resolveReady();
+      }
+      return;
+    }
+
+    if (message.type === 'detect-result') {
+      const pending = this.pending.get(message.id);
+      if (pending) {
+        this.pending.delete(message.id);
+        pending.resolve(message.results);
+      }
+      return;
+    }
+
+    if (message.type === 'error') {
+      const error = new Error(message.error);
+
+      if (message.id !== undefined) {
+        const pending = this.pending.get(message.id);
+        if (pending) {
+          this.pending.delete(message.id);
+          pending.reject(error);
+        }
+        return;
+      }
+
+      this.failReady(error);
+    }
+  }
+
+  private failReady(error: Error): void {
+    if (!this.readySettled) {
+      this.readySettled = true;
+      this.rejectReady(error);
+    }
+  }
+}

src/detection/detection.worker.ts

@@ -0,0 +1,123 @@
+/// <reference lib="webworker" />
+
+import {pipeline, type ObjectDetectionOutput} from '@huggingface/transformers';
+import type {WorkerRequest, WorkerResponse} from './workerMessages';
+
+const MODEL_ID = 'onnx-community/rfdetr_medium-ONNX';
+/** RF-DETR preprocessor input size (see model preprocessor_config.json). */
+const MODEL_INPUT_SIZE = 576;
+
+type DetectFn = (
+  input: OffscreenCanvas,
+  options?: {threshold?: number; percentage?: boolean},
+) => Promise<ObjectDetectionOutput>;
+
+let detector: DetectFn | null = null;
+let loadPromise: Promise<void> | null = null;
+let preprocessCanvas: OffscreenCanvas | null = null;
+let preprocessCtx: OffscreenCanvasRenderingContext2D | null = null;
+
+function getPreprocessSurface(): {
+  canvas: OffscreenCanvas;
+  ctx: OffscreenCanvasRenderingContext2D;
+} {
+  if (!preprocessCanvas || !preprocessCtx) {
+    preprocessCanvas = new OffscreenCanvas(MODEL_INPUT_SIZE, MODEL_INPUT_SIZE);
+    const ctx = preprocessCanvas.getContext('2d', {willReadFrequently: true});
+    if (!ctx) {
+      throw new Error('OffscreenCanvas 2D context not available in worker');
+    }
+    preprocessCtx = ctx;
+  }
+  return {canvas: preprocessCanvas, ctx: preprocessCtx};
+}
+
+function post(message: WorkerResponse): void {
+  self.postMessage(message);
+}
+
+async function loadDetector(): Promise<DetectFn> {
+  if (detector) {
+    return detector;
+  }
+
+  if (!loadPromise) {
+    loadPromise = (async () => {
+      post({type: 'status', message: 'Loading RF-DETR model (WebGPU)…'});
+      detector = (await pipeline('object-detection', MODEL_ID, {
+        device: 'webgpu',
+        dtype: 'fp32',
+        progress_callback: (progress) => {
+          if (progress.status === 'progress' && progress.file) {
+            const percent =
+              progress.total && progress.total > 0
+                ? Math.round((progress.loaded / progress.total) * 100)
+                : null;
+            post({
+              type: 'status',
+              message: percent
+                ? `Downloading ${progress.file} (${percent}%)…`
+                : `Downloading ${progress.file}…`,
+            });
+          }
+        },
+      })) as DetectFn;
+      post({type: 'status', message: 'Compiling RF-DETR shaders…'});
+    })();
+  }
+
+  await loadPromise;
+  return detector!;
+}
+
+async function runDetection(
+  frame: VideoFrame,
+  threshold: number,
+): Promise<ObjectDetectionOutput> {
+  const detect = await loadDetector();
+  const {canvas, ctx} = getPreprocessSurface();
+
+  try {
+    const bitmap = await createImageBitmap(frame, {
+      resizeWidth: MODEL_INPUT_SIZE,
+      resizeHeight: MODEL_INPUT_SIZE,
+    });
+    try {
+      ctx.drawImage(bitmap, 0, 0);
+    } finally {
+      bitmap.close();
+    }
+  } finally {
+    frame.close();
+  }
+
+  return detect(canvas, {threshold, percentage: true});
+}
+
+self.onmessage = async (event: MessageEvent<WorkerRequest>) => {
+  const message = event.data;
+
+  try {
+    if (message.type === 'init') {
+      await loadDetector();
+      post({type: 'ready'});
+      return;
+    }
+
+    if (message.type === 'detect') {
+      const results = await runDetection(message.frame, message.threshold);
+      post({type: 'detect-result', id: message.id, results});
+      return;
+    }
+  } catch (error) {
+    if (message.type === 'detect') {
+      message.frame.close();
+    }
+
+    post({
+      type: 'error',
+      error: error instanceof Error ? error.message : String(error),
+      id: message.type === 'detect' ? message.id : undefined,
+    });
+  }
+};

src/detection/workerMessages.ts

@@ -0,0 +1,13 @@
+import type {ObjectDetectionOutput} from '@huggingface/transformers';
+
+export type DetectionResult = ObjectDetectionOutput[number];
+
+export type WorkerRequest =
+  | {type: 'init'}
+  | {type: 'detect'; id: number; threshold: number; frame: VideoFrame};
+
+export type WorkerResponse =
+  | {type: 'status'; message: string}
+  | {type: 'ready'}
+  | {type: 'detect-result'; id: number; results: DetectionResult[]}
+  | {type: 'error'; error: string; id?: number};

src/pages/clusterMonitor.ts

@@ -0,0 +1,232 @@
+const API_BASE = import.meta.env.VITE_API_BASE ?? '';
+const REFRESH_MS = 3000;
+
+const SAMPLE_IMAGE_URL =
+  'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png';
+
+interface HostInfo {
+  id: string;
+  model: string;
+  online: boolean;
+  registeredAt: number;
+  lastHeartbeat: number;
+}
+
+const brokerStatusEl = document.getElementById('broker-status')!;
+const onlineCountEl = document.getElementById('online-count')!;
+const totalCountEl = document.getElementById('total-count')!;
+const hostsEl = document.getElementById('hosts')!;
+const errorEl = document.getElementById('error')!;
+const refreshNoteEl = document.getElementById('refresh-note')!;
+
+const expandedCurlHosts = new Set<string>();
+
+function syncExpandedCurlFromDom(): void {
+  hostsEl.querySelectorAll<HTMLDetailsElement>('details.curl-details').forEach((details) => {
+    const id = details.closest('[data-host-id]')?.getAttribute('data-host-id');
+    if (!id) {
+      return;
+    }
+    if (details.open) {
+      expandedCurlHosts.add(id);
+    } else {
+      expandedCurlHosts.delete(id);
+    }
+  });
+}
+
+function detectOrigin(): string {
+  return window.location.origin;
+}
+
+function buildCurlExample(hostId: string, modelId: string): string {
+  const origin = detectOrigin();
+
+  if (modelId === 'smolvlm-500m') {
+    return `curl -X POST '${origin}/v1/describe' \\
+  -H 'Content-Type: application/json' \\
+  -d '{
+    "host": "${hostId}",
+    "image_url": "${SAMPLE_IMAGE_URL}",
+    "instruction": "What do you see?",
+    "max_new_tokens": 100
+  }'`;
+  }
+
+  return `curl -X POST '${origin}/v1/detect' \\
+  -H 'Content-Type: application/json' \\
+  -d '{
+    "host": "${hostId}",
+    "image_url": "${SAMPLE_IMAGE_URL}",
+    "threshold": 0.5
+  }'`;
+}
+
+function formatTime(ms: number): string {
+  return new Date(ms).toLocaleString();
+}
+
+function relativeTime(ms: number): string {
+  const seconds = Math.floor((Date.now() - ms) / 1000);
+  if (seconds < 60) {
+    return `${seconds}s ago`;
+  }
+  const minutes = Math.floor(seconds / 60);
+  if (minutes < 60) {
+    return `${minutes}m ago`;
+  }
+  const hours = Math.floor(minutes / 60);
+  return `${hours}h ago`;
+}
+
+function showError(message: string): void {
+  errorEl.textContent = message;
+  errorEl.style.display = 'block';
+}
+
+function clearError(): void {
+  errorEl.style.display = 'none';
+}
+
+async function fetchBrokerHealth(): Promise<boolean> {
+  const res = await fetch(`${API_BASE}/health`);
+  if (!res.ok) {
+    return false;
+  }
+  const json = (await res.json()) as {status?: string};
+  return json.status === 'ok';
+}
+
+async function fetchHosts(): Promise<HostInfo[]> {
+  const res = await fetch(`${API_BASE}/v1/hosts`);
+  if (!res.ok) {
+    throw new Error(`GET /v1/hosts failed (${res.status})`);
+  }
+  const json = (await res.json()) as {hosts: HostInfo[]};
+  return json.hosts ?? [];
+}
+
+function renderHosts(hosts: HostInfo[]): void {
+  const online = hosts.filter((h) => h.online).length;
+  onlineCountEl.textContent = String(online);
+  onlineCountEl.className = `stat-value ${online > 0 ? 'ok' : 'warn'}`;
+  totalCountEl.textContent = String(hosts.length);
+
+  if (hosts.length === 0) {
+    hostsEl.innerHTML = `
+      <div class="empty">
+        <p>No hosts registered yet.</p>
+        <p><a href="/public">Open the host page</a>, pick an id, and click “Start hosting”.</p>
+      </div>`;
+    return;
+  }
+
+  const sorted = [...hosts].sort((a, b) => {
+    if (a.online !== b.online) {
+      return a.online ? -1 : 1;
+    }
+    return b.lastHeartbeat - a.lastHeartbeat;
+  });
+
+  syncExpandedCurlFromDom();
+
+  hostsEl.innerHTML = sorted
+    .map((host) => {
+      const curl = buildCurlExample(host.id, host.model);
+      const escapedCurl = curl.replace(/'/g, '&#39;');
+      return `
+        <article class="host-card" data-host-id="${escapeHtml(host.id)}">
+          <div class="host-header">
+            <span class="host-id">${escapeHtml(host.id)}</span>
+            <span class="badge ${host.online ? 'online' : 'offline'}">
+              ${host.online ? 'Online' : 'Offline'}
+            </span>
+          </div>
+          <div class="host-meta">
+            <span>Model: ${escapeHtml(host.model)}</span>
+            <span>Registered: ${formatTime(host.registeredAt)}</span>
+            <span>Last heartbeat: ${relativeTime(host.lastHeartbeat)}</span>
+          </div>
+          <details class="curl-details"${expandedCurlHosts.has(host.id) ? ' open' : ''}>
+            <summary>
+              <span class="summary-show">Show curl example</span>
+              <span class="summary-hide">Hide curl example</span>
+            </summary>
+            <div class="curl-body">
+              <button type="button" class="copy" data-curl="${escapedCurl}">Copy curl</button>
+              <pre>${escapeHtml(curl)}</pre>
+            </div>
+          </details>
+        </article>`;
+    })
+    .join('');
+
+  hostsEl.querySelectorAll('button.copy').forEach((btn) => {
+    btn.addEventListener('click', async () => {
+      const curl = btn.getAttribute('data-curl')?.replace(/&#39;/g, "'") ?? '';
+      await navigator.clipboard.writeText(curl);
+      const label = btn.textContent;
+      btn.textContent = 'Copied!';
+      setTimeout(() => {
+        btn.textContent = label;
+      }, 1500);
+    });
+  });
+}
+
+function escapeHtml(text: string): string {
+  return text
+    .replace(/&/g, '&amp;')
+    .replace(/</g, '&lt;')
+    .replace(/>/g, '&gt;')
+    .replace(/"/g, '&quot;');
+}
+
+async function refresh(): Promise<void> {
+  try {
+    const healthy = await fetchBrokerHealth();
+    brokerStatusEl.textContent = healthy ? 'Healthy' : 'Unreachable';
+    brokerStatusEl.className = `stat-value ${healthy ? 'ok' : 'err'}`;
+
+    if (!healthy) {
+      showError(
+        `Broker at ${detectOrigin()} is not responding. Run \`npm run dev\` or \`npm run dev:api\`.`,
+      );
+      hostsEl.innerHTML = '';
+      onlineCountEl.textContent = '—';
+      totalCountEl.textContent = '—';
+      return;
+    }
+
+    clearError();
+    const hosts = await fetchHosts();
+    renderHosts(hosts);
+    refreshNoteEl.textContent = `Last updated ${new Date().toLocaleTimeString()} · refreshing every ${REFRESH_MS / 1000}s`;
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    showError(message);
+    brokerStatusEl.textContent = 'Error';
+    brokerStatusEl.className = 'stat-value err';
+  }
+}
+
+hostsEl.addEventListener('toggle', (event) => {
+  const details = event.target;
+  if (!(details instanceof HTMLDetailsElement) || !details.classList.contains('curl-details')) {
+    return;
+  }
+  const id = details.closest('[data-host-id]')?.getAttribute('data-host-id');
+  if (!id) {
+    return;
+  }
+  if (details.open) {
+    expandedCurlHosts.add(id);
+  } else {
+    expandedCurlHosts.delete(id);
+  }
+});
+
+void refresh();
+setInterval(() => {
+  void refresh();
+}, REFRESH_MS);

src/pages/detectionNodeHost.ts

@@ -0,0 +1,338 @@
+import {
+  CLUSTER_MODELS,
+  DEFAULT_CLUSTER_MODEL_ID,
+  getClusterModel,
+  isImplementedClusterModel,
+} from '../../shared/clusterModels';
+import type {ObjectDetector} from '../detection/ObjectDetector';
+import type {DetectionResult} from '../detection/workerMessages';
+import type {VideoDescriber} from '../videodescription/VideoDescriber';
+
+const API_BASE = import.meta.env.VITE_API_BASE ?? '';
+
+const OBJECT_DETECTION_MODEL_ID = 'rfdetr-medium';
+const VIDEO_DESCRIPTION_MODEL_ID = 'smolvlm-500m';
+
+type ClusterTaskKind = 'detect' | 'describe';
+
+interface HostTaskMessage {
+  id: string;
+  kind: ClusterTaskKind;
+  image_base64: string;
+  mime_type: string;
+  threshold?: number;
+  instruction?: string;
+  max_new_tokens?: number;
+}
+
+const statusEl = document.getElementById('status')!;
+const hostIdInput = document.getElementById('host-id') as HTMLInputElement;
+const modelSelect = document.getElementById('model-select') as HTMLSelectElement;
+const registerBtn = document.getElementById('register-btn') as HTMLButtonElement;
+const curlExample = document.getElementById('curl-example')!;
+
+let detector: ObjectDetector | null = null;
+let describer: VideoDescriber | null = null;
+let hostId = '';
+let selectedModelId = DEFAULT_CLUSTER_MODEL_ID;
+let processing = false;
+let eventSource: EventSource | null = null;
+const taskQueue: HostTaskMessage[] = [];
+
+function setStatus(message: string): void {
+  statusEl.textContent = message;
+  console.log('[detection-host]', message);
+}
+
+function populateModelSelect(): void {
+  modelSelect.replaceChildren(
+    ...CLUSTER_MODELS.map((model) => {
+      const option = document.createElement('option');
+      option.value = model.id;
+      option.textContent = model.implemented
+        ? model.label
+        : `${model.label} (coming soon)`;
+      option.disabled = !model.implemented;
+      option.title = model.description;
+      return option;
+    }),
+  );
+  modelSelect.value = DEFAULT_CLUSTER_MODEL_ID;
+}
+
+function updateCurlExample(): void {
+  const origin = window.location.origin;
+
+  if (selectedModelId === OBJECT_DETECTION_MODEL_ID) {
+    curlExample.textContent = `curl -X POST '${origin}/v1/detect' \\
+  -H 'Content-Type: application/json' \\
+  -d '{
+    "host": "${hostId}",
+    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
+    "threshold": 0.5
+  }'`;
+    return;
+  }
+
+  if (selectedModelId === VIDEO_DESCRIPTION_MODEL_ID) {
+    curlExample.textContent = `curl -X POST '${origin}/v1/describe' \\
+  -H 'Content-Type: application/json' \\
+  -d '{
+    "host": "${hostId}",
+    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
+    "instruction": "What do you see?",
+    "max_new_tokens": 100
+  }'`;
+    return;
+  }
+
+  const model = getClusterModel(selectedModelId);
+  curlExample.textContent = model
+    ? `# ${model.label}\n# API endpoint for this model is not wired yet.`
+    : '';
+}
+
+async function ensureModelLoaded(modelId: string): Promise<void> {
+  if (modelId === OBJECT_DETECTION_MODEL_ID) {
+    if (!detector) {
+      setStatus('Loading RF-DETR (WebGPU)…');
+      const {ObjectDetector: Detector} = await import('../detection/ObjectDetector');
+      detector = await Detector.create(setStatus);
+    }
+    return;
+  }
+
+  if (modelId === VIDEO_DESCRIPTION_MODEL_ID) {
+    if (!describer) {
+      setStatus('Loading SmolVLM-500M (WebGPU)…');
+      const {VideoDescriber: Describer} = await import('../videodescription/VideoDescriber');
+      describer = await Describer.create(setStatus);
+    }
+    return;
+  }
+
+  const model = getClusterModel(modelId);
+  throw new Error(
+    model
+      ? `${model.label} is not available on this host yet.`
+      : `Unknown model "${modelId}".`,
+  );
+}
+
+async function base64ToVideoFrame(base64: string, mimeType: string): Promise<VideoFrame> {
+  const binary = atob(base64);
+  const bytes = new Uint8Array(binary.length);
+  for (let i = 0; i < binary.length; i++) {
+    bytes[i] = binary.charCodeAt(i);
+  }
+
+  const blob = new Blob([bytes], {type: mimeType});
+  const bitmap = await createImageBitmap(blob);
+  const frame = new VideoFrame(bitmap, {timestamp: 0});
+  bitmap.close();
+  return frame;
+}
+
+async function processDetectTask(task: HostTaskMessage): Promise<void> {
+  if (!detector) {
+    return;
+  }
+
+  let frame: VideoFrame | null = null;
+
+  try {
+    frame = await base64ToVideoFrame(task.image_base64, task.mime_type);
+    const results = await detector.detect(frame, {threshold: task.threshold ?? 0.5});
+
+    await fetch(`${API_BASE}/api/tasks/${task.id}/complete`, {
+      method: 'POST',
+      headers: {'Content-Type': 'application/json'},
+      body: JSON.stringify({
+        status: 'done',
+        results: serializeDetectionResults(results),
+      }),
+    });
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    await fetch(`${API_BASE}/api/tasks/${task.id}/complete`, {
+      method: 'POST',
+      headers: {'Content-Type': 'application/json'},
+      body: JSON.stringify({status: 'error', error: message}),
+    });
+    throw error;
+  } finally {
+    frame?.close();
+  }
+}
+
+async function processDescribeTask(task: HostTaskMessage): Promise<void> {
+  if (!describer) {
+    return;
+  }
+
+  let frame: VideoFrame | null = null;
+
+  try {
+    frame = await base64ToVideoFrame(task.image_base64, task.mime_type);
+    const description = await describer.describe(frame, {
+      instruction: task.instruction ?? 'What do you see?',
+      maxNewTokens: task.max_new_tokens ?? 100,
+    });
+
+    await fetch(`${API_BASE}/api/tasks/${task.id}/complete`, {
+      method: 'POST',
+      headers: {'Content-Type': 'application/json'},
+      body: JSON.stringify({
+        status: 'done',
+        description,
+      }),
+    });
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    await fetch(`${API_BASE}/api/tasks/${task.id}/complete`, {
+      method: 'POST',
+      headers: {'Content-Type': 'application/json'},
+      body: JSON.stringify({status: 'error', error: message}),
+    });
+    throw error;
+  } finally {
+    frame?.close();
+  }
+}
+
+async function processTask(task: HostTaskMessage): Promise<void> {
+  processing = true;
+  setStatus(`Processing task ${task.id.slice(0, 8)}…`);
+
+  try {
+    if (task.kind === 'describe') {
+      await processDescribeTask(task);
+    } else {
+      await processDetectTask(task);
+    }
+    setStatus(`Task ${task.id.slice(0, 8)} complete. Waiting for requests…`);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    setStatus(`Task failed: ${message}`);
+  } finally {
+    processing = false;
+    void drainTaskQueue();
+  }
+}
+
+async function drainTaskQueue(): Promise<void> {
+  if (processing || taskQueue.length === 0) {
+    return;
+  }
+
+  const task = taskQueue.shift();
+  if (task) {
+    await processTask(task);
+  }
+}
+
+function enqueueTask(task: HostTaskMessage): void {
+  taskQueue.push(task);
+  void drainTaskQueue();
+}
+
+function serializeDetectionResults(results: DetectionResult[]): unknown[] {
+  return results.map((result) => ({
+    label: result.label,
+    score: result.score,
+    box: {
+      xmin: result.box.xmin,
+      ymin: result.box.ymin,
+      xmax: result.box.xmax,
+      ymax: result.box.ymax,
+    },
+  }));
+}
+
+function connectStream(): void {
+  eventSource?.close();
+
+  const url = `${API_BASE}/api/hosts/stream?host_id=${encodeURIComponent(hostId)}`;
+  eventSource = new EventSource(url);
+
+  eventSource.addEventListener('ready', () => {
+    const model = getClusterModel(selectedModelId);
+    const modelLabel = model?.label ?? selectedModelId;
+    setStatus(`Hosting "${hostId}" (${modelLabel}). Waiting for requests…`);
+  });
+
+  eventSource.addEventListener('task', (event) => {
+    const raw = JSON.parse(event.data) as HostTaskMessage & {kind?: ClusterTaskKind};
+    const task: HostTaskMessage = {
+      ...raw,
+      kind: raw.kind ?? 'detect',
+    };
+    enqueueTask(task);
+  });
+
+  eventSource.onerror = () => {
+    if (eventSource?.readyState === EventSource.CLOSED) {
+      setStatus('Broker connection lost. Reconnecting in 2s…');
+      setTimeout(connectStream, 2000);
+    }
+  };
+}
+
+async function registerHost(): Promise<void> {
+  hostId = hostIdInput.value.trim();
+  selectedModelId = modelSelect.value;
+
+  if (!hostId) {
+    setStatus('Enter a host id (e.g. my-gpu-node).');
+    return;
+  }
+
+  if (!isImplementedClusterModel(selectedModelId)) {
+    setStatus('Choose an available model from the list.');
+    return;
+  }
+
+  registerBtn.disabled = true;
+  modelSelect.disabled = true;
+  hostIdInput.disabled = true;
+
+  try {
+    await ensureModelLoaded(selectedModelId);
+
+    const res = await fetch(`${API_BASE}/api/hosts/register`, {
+      method: 'POST',
+      headers: {'Content-Type': 'application/json'},
+      body: JSON.stringify({id: hostId, model: selectedModelId}),
+    });
+
+    if (!res.ok) {
+      const json = await res.json().catch(() => ({}));
+      throw new Error(json.error ?? `Register failed (${res.status})`);
+    }
+
+    updateCurlExample();
+    connectStream();
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    setStatus(`Failed to start host:\n${message}`);
+    registerBtn.disabled = false;
+    modelSelect.disabled = false;
+    hostIdInput.disabled = false;
+  }
+}
+
+registerBtn.addEventListener('click', () => {
+  void registerHost();
+});
+
+modelSelect.addEventListener('change', () => {
+  selectedModelId = modelSelect.value;
+  updateCurlExample();
+});
+
+populateModelSelect();
+hostIdInput.value = `node-${Math.random().toString(36).slice(2, 8)}`;
+selectedModelId = modelSelect.value;
+updateCurlExample();
+
+setStatus('Pick a model, enter a host id, and click “Start hosting”.');

src/videodescription/VideoDescriber.ts

@@ -0,0 +1,127 @@
+import type {WorkerRequest, WorkerResponse} from './workerMessages';
+
+type PendingRequest = {
+  resolve: (description: string) => void;
+  reject: (error: Error) => void;
+};
+
+export class VideoDescriber {
+  private readonly worker: Worker;
+  private readonly pending = new Map<number, PendingRequest>();
+  private nextId = 0;
+  private readonly ready: Promise<void>;
+  private readySettled = false;
+  private resolveReady!: () => void;
+  private rejectReady!: (error: Error) => void;
+  private onStatus?: (message: string) => void;
+
+  private constructor(worker: Worker, onStatus?: (message: string) => void) {
+    this.worker = worker;
+    this.onStatus = onStatus;
+
+    this.ready = new Promise<void>((resolve, reject) => {
+      this.resolveReady = resolve;
+      this.rejectReady = reject;
+    });
+
+    this.worker.addEventListener('message', (event: MessageEvent<WorkerResponse>) => {
+      this.handleMessage(event.data);
+    });
+    this.worker.addEventListener('error', (event) => {
+      this.failReady(new Error(event.message || 'Video description worker failed'));
+    });
+
+    this.worker.postMessage({type: 'init'} satisfies WorkerRequest);
+  }
+
+  static async create(onStatus?: (message: string) => void): Promise<VideoDescriber> {
+    const worker = new Worker(
+      new URL('./videodescription.worker.ts', import.meta.url),
+      {type: 'module'},
+    );
+    const describer = new VideoDescriber(worker, onStatus);
+    await describer.ready;
+    return describer;
+  }
+
+  async warmup(frame: VideoFrame, instruction = 'What do you see?'): Promise<void> {
+    await this.describe(frame, {instruction});
+  }
+
+  async describe(
+    frame: VideoFrame,
+    options: {instruction: string; maxNewTokens?: number},
+  ): Promise<string> {
+    await this.ready;
+
+    const id = this.nextId++;
+
+    return new Promise<string>((resolve, reject) => {
+      this.pending.set(id, {resolve, reject});
+      this.worker.postMessage(
+        {
+          type: 'describe',
+          id,
+          instruction: options.instruction,
+          maxNewTokens: options.maxNewTokens ?? 100,
+          frame,
+        } satisfies WorkerRequest,
+        [frame],
+      );
+    });
+  }
+
+  dispose(): void {
+    for (const {reject} of this.pending.values()) {
+      reject(new Error('Video description worker terminated'));
+    }
+    this.pending.clear();
+    this.worker.terminate();
+  }
+
+  private handleMessage(message: WorkerResponse): void {
+    if (message.type === 'status') {
+      this.onStatus?.(message.message);
+      return;
+    }
+
+    if (message.type === 'ready') {
+      if (!this.readySettled) {
+        this.readySettled = true;
+        this.resolveReady();
+      }
+      return;
+    }
+
+    if (message.type === 'describe-result') {
+      const pending = this.pending.get(message.id);
+      if (pending) {
+        this.pending.delete(message.id);
+        pending.resolve(message.description);
+      }
+      return;
+    }
+
+    if (message.type === 'error') {
+      const error = new Error(message.error);
+
+      if (message.id !== undefined) {
+        const pending = this.pending.get(message.id);
+        if (pending) {
+          this.pending.delete(message.id);
+          pending.reject(error);
+        }
+        return;
+      }
+
+      this.failReady(error);
+    }
+  }
+
+  private failReady(error: Error): void {
+    if (!this.readySettled) {
+      this.readySettled = true;
+      this.rejectReady(error);
+    }
+  }
+}

src/videodescription/videodescription.worker.ts

@@ -0,0 +1,158 @@
+/// <reference lib="webworker" />
+
+import {
+  AutoModelForVision2Seq,
+  AutoProcessor,
+  RawImage,
+  Tensor,
+  type PreTrainedModel,
+  type Processor,
+} from '@huggingface/transformers';
+import type {WorkerRequest, WorkerResponse} from './workerMessages';
+
+const MODEL_ID = 'HuggingFaceTB/SmolVLM-500M-Instruct';
+
+const DTYPE = {
+  embed_tokens: 'fp16',
+  vision_encoder: 'q4',
+  decoder_model_merged: 'q4',
+} as const;
+
+let processor: Processor | null = null;
+let model: PreTrainedModel | null = null;
+let loadPromise: Promise<void> | null = null;
+
+function post(message: WorkerResponse): void {
+  self.postMessage(message);
+}
+
+function reportDownloadProgress(progress: {
+  status: string;
+  file?: string;
+  loaded?: number;
+  total?: number;
+}): void {
+  if (progress.status === 'progress' && progress.file) {
+    const percent =
+      progress.total && progress.total > 0
+        ? Math.round(((progress.loaded ?? 0) / progress.total) * 100)
+        : null;
+    post({
+      type: 'status',
+      message: percent
+        ? `Downloading ${progress.file} (${percent}%)…`
+        : `Downloading ${progress.file}…`,
+    });
+  }
+}
+
+async function loadPipeline(): Promise<{processor: Processor; model: PreTrainedModel}> {
+  if (processor && model) {
+    return {processor, model};
+  }
+
+  if (!loadPromise) {
+    loadPromise = (async () => {
+      post({type: 'status', message: 'Loading SmolVLM processor…'});
+      processor = await AutoProcessor.from_pretrained(MODEL_ID, {
+        progress_callback: reportDownloadProgress,
+      });
+
+      post({type: 'status', message: 'Loading SmolVLM model (WebGPU)…'});
+      model = await AutoModelForVision2Seq.from_pretrained(MODEL_ID, {
+        dtype: DTYPE,
+        device: 'webgpu',
+        progress_callback: reportDownloadProgress,
+      });
+
+      post({type: 'status', message: 'Compiling SmolVLM shaders…'});
+    })();
+  }
+
+  await loadPromise;
+  return {processor: processor!, model: model!};
+}
+
+async function frameToRawImage(frame: VideoFrame): Promise<RawImage> {
+  const bitmap = await createImageBitmap(frame);
+  try {
+    const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);
+    const ctx = canvas.getContext('2d', {willReadFrequently: true});
+    if (!ctx) {
+      throw new Error('OffscreenCanvas 2D context not available in worker');
+    }
+    ctx.drawImage(bitmap, 0, 0);
+    return RawImage.fromCanvas(canvas);
+  } finally {
+    bitmap.close();
+    frame.close();
+  }
+}
+
+async function runDescription(
+  frame: VideoFrame,
+  instruction: string,
+  maxNewTokens: number,
+): Promise<string> {
+  const {processor: proc, model: vlm} = await loadPipeline();
+  const image = await frameToRawImage(frame);
+
+  const messages = [
+    {
+      role: 'user',
+      content: [{type: 'image'}, {type: 'text', text: instruction}],
+    },
+  ];
+
+  const text = proc.apply_chat_template(messages, {
+    add_generation_prompt: true,
+  });
+
+  const inputs = await proc(text, [image], {
+    do_image_splitting: false,
+  });
+
+  const generatedIds = (await vlm.generate({
+    ...inputs,
+    max_new_tokens: maxNewTokens,
+  })) as Tensor;
+
+  const promptLength = inputs.input_ids.dims.at(-1) ?? 0;
+  const output = proc.batch_decode(generatedIds.slice(null, [promptLength, null]), {
+    skip_special_tokens: true,
+  });
+
+  return output[0]?.trim() ?? '';
+}
+
+self.onmessage = async (event: MessageEvent<WorkerRequest>) => {
+  const message = event.data;
+
+  try {
+    if (message.type === 'init') {
+      await loadPipeline();
+      post({type: 'ready'});
+      return;
+    }
+
+    if (message.type === 'describe') {
+      const description = await runDescription(
+        message.frame,
+        message.instruction,
+        message.maxNewTokens,
+      );
+      post({type: 'describe-result', id: message.id, description});
+      return;
+    }
+  } catch (error) {
+    if (message.type === 'describe') {
+      message.frame.close();
+    }
+
+    post({
+      type: 'error',
+      error: error instanceof Error ? error.message : String(error),
+      id: message.type === 'describe' ? message.id : undefined,
+    });
+  }
+};

src/videodescription/workerMessages.ts

@@ -0,0 +1,15 @@
+export type WorkerRequest =
+  | {type: 'init'}
+  | {
+      type: 'describe';
+      id: number;
+      frame: VideoFrame;
+      instruction: string;
+      maxNewTokens: number;
+    };
+
+export type WorkerResponse =
+  | {type: 'status'; message: string}
+  | {type: 'ready'}
+  | {type: 'describe-result'; id: number; description: string}
+  | {type: 'error'; error: string; id?: number};

src/vite-env.d.ts

@@ -0,0 +1 @@
+/// <reference types="vite/client" />

tsconfig.json

@@ -0,0 +1,14 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "lib": ["ES2022", "DOM", "DOM.Iterable"],
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "strict": true,
+    "skipLibCheck": true,
+    "noEmit": true,
+    "types": ["@webgpu/types"]
+  },
+  "include": ["src", "shared"],
+  "exclude": ["server"]
+}

tsconfig.server.json

@@ -0,0 +1,14 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "lib": ["ES2022"],
+    "module": "NodeNext",
+    "moduleResolution": "NodeNext",
+    "strict": true,
+    "skipLibCheck": true,
+    "noEmit": true,
+    "esModuleInterop": true,
+    "resolveJsonModule": true
+  },
+  "include": ["server", "shared"]
+}

vite.config.ts

@@ -0,0 +1,48 @@
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { defineConfig } from 'vite';
+
+const API_PORT = 8787;
+const rootDir = path.dirname(fileURLToPath(import.meta.url));
+
+export default defineConfig(({ command, mode }) => ({
+  base: mode === 'space' ? '/' : command === 'build' ? '/webgpu-video-ai/' : '/',
+  optimizeDeps: {
+    exclude: ['@huggingface/transformers'],
+  },
+  worker: {
+    format: 'es',
+  },
+  build: {
+    outDir: 'docs',
+    rollupOptions: {
+      input: {
+        main: path.resolve(rootDir, 'index.html'),
+        monitor: path.resolve(rootDir, 'monitor.html'),
+      },
+    },
+  },
+  server: {
+    port: 5180,
+    headers: {
+      'Cross-Origin-Opener-Policy': 'same-origin',
+      'Cross-Origin-Embedder-Policy': 'require-corp',
+    },
+    proxy: {
+      '/api': `http://localhost:${API_PORT}`,
+      '/v1': `http://localhost:${API_PORT}`,
+      '/health': `http://localhost:${API_PORT}`,
+    },
+  },
+  preview: {
+    headers: {
+      'Cross-Origin-Opener-Policy': 'same-origin',
+      'Cross-Origin-Embedder-Policy': 'require-corp',
+    },
+    proxy: {
+      '/api': `http://localhost:${API_PORT}`,
+      '/v1': `http://localhost:${API_PORT}`,
+      '/health': `http://localhost:${API_PORT}`,
+    },
+  },
+}));