---
title: Vision Lab — Multimodal AI In-Browser
emoji: 🔬
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
license: mit
models:
  - Xenova/vit-gpt2-image-captioning
  - Xenova/owlvit-base-patch32
  - Xenova/clip-vit-base-patch32
  - Xenova/trocr-small-printed
---

# Vision Lab

A multimodal computer-vision + language playground that runs **entirely in
the browser** — no backend, no API keys, no server costs, and nothing you
upload ever leaves your machine. Built on
[🤗 Transformers.js](https://huggingface.co/docs/transformers.js), which
executes real transformer models client-side via WebAssembly, with
[WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API)
acceleration when the browser supports it.

## Capabilities

| Tab | Task | Model |
|---|---|---|
| Caption | Image → natural-language description | `Xenova/vit-gpt2-image-captioning` |
| Detect Objects | Draws bounding boxes for any labels you define, even ones the model wasn't explicitly trained to detect | `Xenova/owlvit-base-patch32` |
| Classify | Zero-shot classification against any labels you define | `Xenova/clip-vit-base-patch32` |
| Read Text | OCR on printed text in an image | `Xenova/trocr-small-printed` |
| Semantic Search | Rank a gallery of images against a natural-language query | `Xenova/clip-vit-base-patch32` (shared) |

## How it works

- **Lazy loading.** Each model downloads and initializes only the first time
  its tab is used — not all four load up front. Progress is streamed live to
  the system log at the bottom of the page.
- **WebGPU / WASM fallback.** On load, the app probes for a WebGPU adapter.
  If present, inference runs hardware-accelerated; otherwise it falls back to
  WASM (CPU) automatically.
- **Semantic search without embeddings math.** Rather than manually computing
  and comparing CLIP image/text embeddings, the search tab reuses the
  zero-shot classification pipeline with a single candidate label (the
  query) per gallery image — the resulting confidence score is a valid
  relevance signal for ranking.
- **Browser caching.** Once a model has been downloaded, the browser's own
  cache keeps it available on repeat visits without re-downloading.

## Why this shape

This is deliberately built as a **static** Hugging Face Space (`sdk: static`)
rather than a Gradio Space, so it can be hosted for free with no compute
plan required — all inference happens on the visitor's own device.

## Running locally

No build step. Just open `index.html` in a browser, or serve the folder:

```bash
python -m http.server 8000
```

## Extending it

Ideas for going further: add a webcam capture option alongside file upload,
swap in a larger captioning model for better quality (trading off download
size), or add an "export gallery as JSON" button to persist search results
across sessions.

## A note on transformers.js task coverage

Not every task in the Python `transformers` library has a browser
equivalent. Visual question answering, for instance, isn't currently a
supported transformers.js pipeline — only its listed NLP, vision, audio,
and multimodal tasks (embeddings, zero-shot classification/detection) run
client-side. Object detection was used here instead of VQA for exactly that
reason: it's a confirmed, supported multimodal task that still combines
free-form text labels with an image.