README.md

---
title: GPU Detection Cluster
emoji: 🎮
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
models:
  - onnx-community/rfdetr_medium-ONNX
  - HuggingFaceTB/SmolVLM-500M-Instruct
---

# WebGPU Cluster

A **distributed inference grid** that turns browsers with WebGPU into cluster nodes. Host **RF-DETR** object detection or **SmolVLM** image description in a Web Worker; a **Node broker** queues tasks and exposes them over HTTP so any client (`curl`, Python, Node, etc.) can call your GPU.

**Repository:** [github.com/apssouza22/webgpu-video-cluster](https://github.com/apssouza22/webgpu-video-cluster)  
**Live demo (Hugging Face Space):** [apssouza22-webgpu-cluster.hf.space](https://apssouza22-webgpu-cluster.hf.space/) · [Space repo](https://huggingface.co/spaces/apssouza22/webgpu-cluster)

```text
curl / Python / app  →  Node broker (task queue)  →  SSE  →  browser host (WebGPU)
                                                              ↓
                                                    RF-DETR · SmolVLM
```

Inference runs **in the host’s browser** on their hardware — not on the broker machine. The broker only coordinates tasks and fetches remote images.

## Quick start

```bash
npm install
npm run dev
```

1. Open **http://localhost:5180** (landing page)
2. Click **Join the grid** (or open **http://localhost:5180/host.html**), choose a **model to share**, pick a host id (e.g. `my-gpu-node`), and click **Start hosting** (loads the model on WebGPU; keep the tab open)
3. Open **http://localhost:5180/monitor.html** to see registered hosts and copy curl examples
4. From another terminal:

```bash
curl -X POST 'http://localhost:5180/v1/detect' \
  -H 'Content-Type: application/json' \
  -d '{
    "host": "my-gpu-node",
    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
    "threshold": 0.5
  }'
```

Response:

```json
{
  "task_id": "...",
  "host": "my-gpu-node",
  "threshold": 0.5,
  "detections": [
    { "label": "cat", "score": 0.92, "box": { "xmin": 10, "ymin": 20, "xmax": 200, "ymax": 180 } }
  ]
}
```

`npm run dev` runs **Vite** (port 5180) and the **API broker** (port 8787). Vite proxies `/v1`, `/api`, and `/health` to the broker.

## Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/v1/detect` | Object detection with RF-DETR (waits for host) |
| `POST` | `/v1/describe` | Image description with SmolVLM (waits for host) |
| `GET` | `/v1/hosts` | List registered hosts and online status |
| `GET` | `/v1/models` | Cluster models (id, label, implementation status) |
| `GET` | `/v1/tasks/:id` | Task status / results |
| `GET` | `/health` | Broker health check |
| — | `/` | Landing page — join or view the grid |
| — | `/host.html` | Browser host — register and share GPU |
| — | `/monitor.html` | Dashboard — live host list and curl examples |
| `POST` | `/api/hosts/register` | Called by the browser host |
| `GET` | `/api/hosts/stream?host_id=` | SSE — browser host receives tasks |

**`POST /v1/detect` body:**

- `host` (required) — id from the browser host page
- `image_url` or `image_base64` (required) — broker fetches URLs server-side
- `threshold` (optional, default `0.5`)

**`POST /v1/describe` body:**

- `host` (required)
- `image_url` or `image_base64` (required)
- `instruction` (optional, default `"What do you see?"`)
- `max_new_tokens` (optional, default `100`)

## Scripts

| Command | Description |
|---------|-------------|
| `npm run dev` | Vite (5180) + API broker (8787) |
| `npm run dev:api` | API broker only |
| `npm run dev:web` | Vite only (proxies API when broker is running) |
| `npm run build` | Typecheck + production bundle to `docs/` (base `/webgpu-video-ai/` for GitHub Pages) |
| `npm run build:space` | Same bundle with base `/` for Hugging Face Spaces |
| `npm run preview` | Serve the production build locally |
| `npm run start` | Broker + static UI (`SERVE_STATIC=1`, `PORT` default 8787) |
| `npm run start:api` | Run broker without watch |

## Cluster models

Models are defined in `shared/clusterModels.ts`. The host page dropdown and `GET /v1/models` both read from that list. To add a model: add an entry there, wire loading in `src/pages/addNode.ts` (`ensureModelLoaded`), add a task handler under `src/tasks/`, and register broker routes in `server/`.

| Model id | Endpoint | Worker |
|----------|----------|--------|
| `rfdetr-medium` | `POST /v1/detect` | `src/detection/detection.worker.ts` |
| `smolvlm-500m` | `POST /v1/describe` | `src/videodescription/videodescription.worker.ts` |

- **RF-DETR:** `onnx-community/rfdetr_medium-ONNX` via `@huggingface/transformers` with `device: 'webgpu'`
- **SmolVLM:** `HuggingFaceTB/SmolVLM-500M-Instruct` with quantized vision/decoder weights (same approach as the [SmolVLM realtime WebGPU demo](https://huggingface.co/spaces/webml-community/smolvlm-realtime-webgpu))

Models download from Hugging Face on first load. Inference runs in a **Web Worker**; the broker sends base64 images and the host converts them to `VideoFrame` for the worker.

## Requirements

- Browser with **WebGPU** (Chrome or Edge desktop recommended)
- Dev server COOP/COEP headers in `vite.config.ts` (required for Transformers.js / WASM)

## Project layout

| Path | Role |
|------|------|
| `server/` | Express broker — task queue, host registry, SSE |
| `src/pages/addNode.ts` | Browser host — register, pull tasks, run inference |
| `src/pages/clusterMonitor.ts` | Monitor dashboard |
| `src/detection/` | RF-DETR worker and main-thread API |
| `src/videodescription/` | SmolVLM worker and main-thread API |
| `shared/clusterModels.ts` | Model catalog for UI and API |

## Hugging Face Space

Hosted at **[apssouza22/webgpu-cluster](https://huggingface.co/spaces/apssouza22/webgpu-cluster)** (Docker SDK, port 7860).

| | URL |
|--|-----|
| Landing | [apssouza22-webgpu-cluster.hf.space](https://apssouza22-webgpu-cluster.hf.space/) |
| Host UI | [apssouza22-webgpu-cluster.hf.space/host.html](https://apssouza22-webgpu-cluster.hf.space/host.html) |
| Monitor | [apssouza22-webgpu-cluster.hf.space/monitor.html](https://apssouza22-webgpu-cluster.hf.space/monitor.html) |

1. Open the host UI in **Chrome or Edge** (WebGPU required).
2. Choose a host id and model, then click **Start hosting** — keep the tab open.
3. Call the API on the same origin:

```bash
curl -X POST 'https://apssouza22-webgpu-cluster.hf.space/v1/detect' \
  -H 'Content-Type: application/json' \
  -d '{
    "host": "my-gpu-node",
    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
    "threshold": 0.5
  }'
```

The Space container serves the broker and static files only; **inference runs in the visitor’s browser**.

### Deploy a new version

```bash
npm run build:space
hf upload apssouza22/webgpu-cluster . . \
  --repo-type space \
  --exclude ".git/*" \
  --exclude "node_modules/*" \
  --commit-message "Your change summary"
```

Wait until the Space shows **Running**, then check `curl https://apssouza22-webgpu-cluster.hf.space/health`. Full steps: [SPACES.md](./SPACES.md#deploy-a-new-version).

## License

MIT