Spaces:
Running
Running
File size: 3,364 Bytes
a527039 8605ade a527039 8605ade b8bc0f0 8605ade a527039 8605ade b8bc0f0 8605ade b8bc0f0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | ---
title: Vision Lab — Multimodal AI In-Browser
emoji: 🔬
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
license: mit
models:
- Xenova/vit-gpt2-image-captioning
- Xenova/owlvit-base-patch32
- Xenova/clip-vit-base-patch32
- Xenova/trocr-small-printed
---
# Vision Lab
A multimodal computer-vision + language playground that runs **entirely in
the browser** — no backend, no API keys, no server costs, and nothing you
upload ever leaves your machine. Built on
[🤗 Transformers.js](https://huggingface.co/docs/transformers.js), which
executes real transformer models client-side via WebAssembly, with
[WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API)
acceleration when the browser supports it.
## Capabilities
| Tab | Task | Model |
|---|---|---|
| Caption | Image → natural-language description | `Xenova/vit-gpt2-image-captioning` |
| Detect Objects | Draws bounding boxes for any labels you define, even ones the model wasn't explicitly trained to detect | `Xenova/owlvit-base-patch32` |
| Classify | Zero-shot classification against any labels you define | `Xenova/clip-vit-base-patch32` |
| Read Text | OCR on printed text in an image | `Xenova/trocr-small-printed` |
| Semantic Search | Rank a gallery of images against a natural-language query | `Xenova/clip-vit-base-patch32` (shared) |
## How it works
- **Lazy loading.** Each model downloads and initializes only the first time
its tab is used — not all four load up front. Progress is streamed live to
the system log at the bottom of the page.
- **WebGPU / WASM fallback.** On load, the app probes for a WebGPU adapter.
If present, inference runs hardware-accelerated; otherwise it falls back to
WASM (CPU) automatically.
- **Semantic search without embeddings math.** Rather than manually computing
and comparing CLIP image/text embeddings, the search tab reuses the
zero-shot classification pipeline with a single candidate label (the
query) per gallery image — the resulting confidence score is a valid
relevance signal for ranking.
- **Browser caching.** Once a model has been downloaded, the browser's own
cache keeps it available on repeat visits without re-downloading.
## Why this shape
This is deliberately built as a **static** Hugging Face Space (`sdk: static`)
rather than a Gradio Space, so it can be hosted for free with no compute
plan required — all inference happens on the visitor's own device.
## Running locally
No build step. Just open `index.html` in a browser, or serve the folder:
```bash
python -m http.server 8000
```
## Extending it
Ideas for going further: add a webcam capture option alongside file upload,
swap in a larger captioning model for better quality (trading off download
size), or add an "export gallery as JSON" button to persist search results
across sessions.
## A note on transformers.js task coverage
Not every task in the Python `transformers` library has a browser
equivalent. Visual question answering, for instance, isn't currently a
supported transformers.js pipeline — only its listed NLP, vision, audio,
and multimodal tasks (embeddings, zero-shot classification/detection) run
client-side. Object detection was used here instead of VQA for exactly that
reason: it's a confirmed, supported multimodal task that still combines
free-form text labels with an image.
|