Spaces:
Running
Running
| title: Vision Lab — Multimodal AI In-Browser | |
| emoji: 🔬 | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: static | |
| pinned: false | |
| license: mit | |
| models: | |
| - Xenova/vit-gpt2-image-captioning | |
| - Xenova/owlvit-base-patch32 | |
| - Xenova/clip-vit-base-patch32 | |
| - Xenova/trocr-small-printed | |
| # Vision Lab | |
| A multimodal computer-vision + language playground that runs **entirely in | |
| the browser** — no backend, no API keys, no server costs, and nothing you | |
| upload ever leaves your machine. Built on | |
| [🤗 Transformers.js](https://huggingface.co/docs/transformers.js), which | |
| executes real transformer models client-side via WebAssembly, with | |
| [WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API) | |
| acceleration when the browser supports it. | |
| ## Capabilities | |
| | Tab | Task | Model | | |
| |---|---|---| | |
| | Caption | Image → natural-language description | `Xenova/vit-gpt2-image-captioning` | | |
| | Detect Objects | Draws bounding boxes for any labels you define, even ones the model wasn't explicitly trained to detect | `Xenova/owlvit-base-patch32` | | |
| | Classify | Zero-shot classification against any labels you define | `Xenova/clip-vit-base-patch32` | | |
| | Read Text | OCR on printed text in an image | `Xenova/trocr-small-printed` | | |
| | Semantic Search | Rank a gallery of images against a natural-language query | `Xenova/clip-vit-base-patch32` (shared) | | |
| ## How it works | |
| - **Lazy loading.** Each model downloads and initializes only the first time | |
| its tab is used — not all four load up front. Progress is streamed live to | |
| the system log at the bottom of the page. | |
| - **WebGPU / WASM fallback.** On load, the app probes for a WebGPU adapter. | |
| If present, inference runs hardware-accelerated; otherwise it falls back to | |
| WASM (CPU) automatically. | |
| - **Semantic search without embeddings math.** Rather than manually computing | |
| and comparing CLIP image/text embeddings, the search tab reuses the | |
| zero-shot classification pipeline with a single candidate label (the | |
| query) per gallery image — the resulting confidence score is a valid | |
| relevance signal for ranking. | |
| - **Browser caching.** Once a model has been downloaded, the browser's own | |
| cache keeps it available on repeat visits without re-downloading. | |
| ## Why this shape | |
| This is deliberately built as a **static** Hugging Face Space (`sdk: static`) | |
| rather than a Gradio Space, so it can be hosted for free with no compute | |
| plan required — all inference happens on the visitor's own device. | |
| ## Running locally | |
| No build step. Just open `index.html` in a browser, or serve the folder: | |
| ```bash | |
| python -m http.server 8000 | |
| ``` | |
| ## Extending it | |
| Ideas for going further: add a webcam capture option alongside file upload, | |
| swap in a larger captioning model for better quality (trading off download | |
| size), or add an "export gallery as JSON" button to persist search results | |
| across sessions. | |
| ## A note on transformers.js task coverage | |
| Not every task in the Python `transformers` library has a browser | |
| equivalent. Visual question answering, for instance, isn't currently a | |
| supported transformers.js pipeline — only its listed NLP, vision, audio, | |
| and multimodal tasks (embeddings, zero-shot classification/detection) run | |
| client-side. Object detection was used here instead of VQA for exactly that | |
| reason: it's a confirmed, supported multimodal task that still combines | |
| free-form text labels with an image. | |