| # BUILD PROMPT — paste everything below this line into Codex / OpenCode | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| You are a senior Python + Gradio engineer. Build a Hugging Face Space called **"Third Eye"**: a | |
| fully voice-driven accessibility app for blind / low-vision users. The user points a webcam at | |
| something (menu, medicine label, sign, scene), speaks a question, and hears the answer back in | |
| their language. **Zero typing required** on the happy path. | |
| Work in the current project folder. Build **incrementally** and **verify at every checkpoint**. | |
| Do not write the whole app at once. Do not invent APIs. Follow these rules exactly. | |
| ─────────────────────────────────────────────────────────────────────────────── | |
| ## ABSOLUTE RULES (do not break these) | |
| 1. **Verify before coding inference.** Model load/call APIs differ by version. For every model, | |
| open its Hugging Face model card and confirm the exact model ID, load call, and inference call | |
| BEFORE writing that stage. If a tool to fetch docs is available (e.g. context7), use it. If a | |
| model ID does not resolve, **STOP and report it** — never silently substitute another model. | |
| 2. **Sponsor models ONLY** (table below). No OpenAI/Whisper/Google/etc. anywhere. | |
| 3. **Build in phases. Stop at each CHECKPOINT and confirm it passes before continuing.** | |
| 4. **Never show a raw traceback to the user.** Every stage is wrapped; failures become | |
| `gr.Warning("friendly message")` and a graceful fallback. | |
| 5. **MUST-HAVE before SHOULD-HAVE before NICE-TO-HAVE.** A working minimal app beats a broken | |
| complete one. If you run low on time, ship the vertical slice. | |
| 6. When unsure about an API after reading the card, write the smallest possible test script and run | |
| it before wiring that stage into the UI. | |
| ─────────────────────────────────────────────────────────────────────────────── | |
| ## MODELS (only these — verify each card before use) | |
| | Role | Model ID | Params | Sponsor | | |
| |---|---|---|---| | |
| | Vision + OCR (PRIMARY) | `openbmb/MiniCPM-V-2_0` | 2.8B | OpenBMB | | |
| | Vision + OCR (FALLBACK only if quality unacceptable) | `openbmb/MiniCPM-V-4_5` | 8B | OpenBMB | | |
| | Speech-to-text | `CohereLabs/cohere-transcribe-03-2026` | 2B | Cohere | | |
| | Text-to-speech | `openbmb/VoxCPM2` | 2B | OpenBMB | | |
| Primary param budget = 2.8B (≤ 4B → qualifies for Tiny Titan). If you must use the 8B fallback, | |
| write in the README: "fallback used, Tiny Titan badge forfeited." Never swap silently. | |
| ─────────────────────────────────────────────────────────────────────────────── | |
| ## TECH STACK | |
| - Gradio 5.x (`gr.Blocks`) on a Hugging Face Space (`sdk: gradio`). | |
| - Modal serverless GPU (A10G) runs vision + TTS + STT. | |
| - Python 3.11. No other cloud APIs. | |
| ─────────────────────────────────────────────────────────────────────────────── | |
| ## TARGET FILE TREE | |
| ``` | |
| app.py # Gradio UI + orchestration | |
| modal_backend.py # Modal app: describe_scene(), speak(), transcribe_audio() | |
| cohere_stt.py # Cohere Transcribe wrapper (imported by modal_backend) | |
| utils.py # image<->bytes, bytes<->wav, safe_call wrapper | |
| requirements.txt | |
| .env.example # MODAL_TOKEN_ID, MODAL_TOKEN_SECRET, HF_TOKEN | |
| README.md # HF frontmatter + story + edge section | |
| BLOG.md # Field Notes draft | |
| DEMO_SCRIPT.md # 45s shot list | |
| assets/ | |
| custom.css # "Iris" design system | |
| sample_menu.jpg sample_label.jpg sample_sign.jpg (use 3 royalty-free / your own photos) | |
| ``` | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 0 — VERIFY REALITY (do this first, write NO inference code yet) | |
| For each model (`MiniCPM-V-2_0`, `VoxCPM2`, `cohere-transcribe-03-2026`): | |
| - Confirm the model ID resolves on Hugging Face. | |
| - Read the card's usage example. Record the EXACT: import, `from_pretrained` args | |
| (`trust_remote_code`, dtype, etc.), and the inference call signature. | |
| - Note especially: MiniCPM-V's `model.chat(...)` signature varies — some versions take | |
| `image=<PIL>, msgs=[{"role":"user","content":<str>}]`; others take | |
| `image=None, msgs=[{"role":"user","content":[<PIL>, <str>]}]`. Use whatever THIS card shows. | |
| - For VoxCPM2: find the real synthesis call (it may need a reference voice / a `generate` method, | |
| not `model.synthesize`). For Cohere Transcribe: confirm whether it loads via `transformers` | |
| `pipeline("automatic-speech-recognition", ...)` or needs a custom call. | |
| **CHECKPOINT 0:** Output a short table of the verified API for each model (load call + infer call). | |
| If anything can't be verified, list it explicitly and propose the smallest fix. Then continue. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 1 — SCAFFOLD (UI renders with fake data) | |
| Create all files. Use STUB functions that return canned data so the UI runs with no GPU. | |
| `requirements.txt`: | |
| ``` | |
| gradio>=5.0 | |
| modal | |
| pillow | |
| soundfile | |
| numpy | |
| ``` | |
| (Add `transformers`, `torch`, `accelerate`, `sentencepiece`, `timm` to the **Modal image**, not the | |
| Space requirements — the Space does not run the models locally.) | |
| `utils.py` — implement: | |
| ```python | |
| import io, base64, tempfile, numpy as np | |
| from PIL import Image | |
| def image_to_bytes(image) -> bytes: | |
| if isinstance(image, np.ndarray): | |
| image = Image.fromarray(image) | |
| buf = io.BytesIO(); image.convert("RGB").save(buf, format="JPEG"); return buf.getvalue() | |
| def bytes_to_wav(audio_bytes: bytes) -> str: | |
| f = tempfile.NamedTemporaryFile(delete=False, suffix=".wav") | |
| f.write(audio_bytes); f.close(); return f.name | |
| def safe_call(fn, *args, fallback=None, warn="Something went wrong.", **kwargs): | |
| import gradio as gr | |
| try: | |
| return fn(*args, **kwargs) | |
| except Exception as e: | |
| gr.Warning(f"{warn} ({type(e).__name__})") | |
| return fallback | |
| ``` | |
| `app.py` — build `gr.Blocks(css=open("assets/custom.css").read())` with: | |
| - A header with the **Iris orb** (a `gr.HTML` div, class `iris idle`) + an ARIA live status line. | |
| - A language `gr.Dropdown` (English, Hindi, German, Tamil, Telugu, Kannada), default English. | |
| - Three `gr.Tab`s: **Describe**, **Ask**, **Read Text**. Each has a `gr.Image(sources=["webcam","upload"])`, | |
| Ask also has `gr.Audio(sources=["microphone"])`, plus a large primary button, a `gr.Audio` output | |
| (set `autoplay=True`), and a large-font `gr.Textbox` output for the transcript. | |
| - The 3 sample images wired as `gr.Examples` so judges can test with no webcam. | |
| - For now, button click calls a STUB `run_pipeline(...)` that returns a placeholder wav path + text. | |
| **CHECKPOINT 1:** `python app.py` launches locally; UI loads; clicking a button shows placeholder | |
| text and the Iris orb is visible. No GPU involved yet. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 2 — MODAL VISION (real description) | |
| `modal_backend.py`: create the Modal app + GPU image, implement `describe_scene` using the | |
| **verified** MiniCPM-V API from Phase 0. | |
| ```python | |
| import modal | |
| app = modal.App("third-eye-backend") | |
| vision_image = modal.Image.debian_slim().pip_install( | |
| "transformers>=4.40","torch","pillow","accelerate","sentencepiece","timm","soundfile") | |
| @app.function(gpu="A10G", image=vision_image, timeout=180) | |
| def describe_scene(image_bytes: bytes, question: str, lang: str = "en") -> str: | |
| import io, torch | |
| from PIL import Image | |
| from transformers import AutoModel, AutoTokenizer | |
| model_id = "openbmb/MiniCPM-V-2_0" | |
| model = AutoModel.from_pretrained(model_id, trust_remote_code=True, | |
| torch_dtype=torch.float16).cuda().eval() | |
| tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) | |
| image = Image.open(io.BytesIO(image_bytes)).convert("RGB") | |
| prompt = question.strip() or "Describe everything you see in detail." | |
| # >>> USE THE EXACT .chat() SIGNATURE YOU VERIFIED IN PHASE 0 <<< | |
| return model.chat(image=image, msgs=[{"role":"user","content":prompt}], tokenizer=tok) | |
| ``` | |
| Deploy: `modal deploy modal_backend.py`. Then write `test_vision.py` that reads `sample_menu.jpg`, | |
| calls `describe_scene.remote(...)`, prints the answer. | |
| **CHECKPOINT 2:** Real, sensible text comes back for the menu image. If quality is unusable, switch | |
| to `MiniCPM-V-4_5` and note the forfeit in README (do not do this lightly). | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 3 — MODAL TTS (real speech) | |
| Add `speak(text, lang)` to `modal_backend.py` using the **verified** VoxCPM2 API. Return WAV bytes. | |
| Write `test_tts.py` that synthesizes "Hello, this is Third Eye." and saves `out.wav`. | |
| **CHECKPOINT 3:** `out.wav` plays intelligible speech. (If multilingual needs a lang/voice arg, wire | |
| `lang` through now.) | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 4 — WIRE "DESCRIBE" END-TO-END ← MINIMUM VALID SUBMISSION | |
| Replace the stub `run_pipeline` in `app.py`: | |
| ```python | |
| def run_pipeline(image, audio_path, mode, lang): | |
| if image is None: | |
| gr.Warning("No image captured. Point the camera and try again.") | |
| return None, "No image captured.", "" | |
| img_bytes = image_to_bytes(image) | |
| if mode == "Ask" and audio_path: | |
| question = safe_call(transcribe_audio.remote, audio_path, | |
| warn="Couldn't hear you — type your question instead.", fallback="") | |
| elif mode == "Read Text": | |
| question = "Read all text visible in this image, word by word, exactly as written." | |
| else: | |
| question = "Describe everything in this image in detail for a blind user." | |
| answer = safe_call(describe_scene.remote, img_bytes, question, lang, | |
| warn="Vision model is waking up — try once more.", fallback="") | |
| if not answer: | |
| return None, "Could not analyze the image.", question | |
| audio_bytes = safe_call(speak.remote, answer, lang, warn="Voice unavailable — showing text.", | |
| fallback=None) | |
| audio_out = bytes_to_wav(audio_bytes) if audio_bytes else None | |
| return audio_out, answer, question | |
| ``` | |
| Show `gr.Progress` with "Loading AI models (first run: ~30s)…" around the first heavy call. | |
| **CHECKPOINT 4:** In the running Space/app, pick `sample_menu.jpg` in the Describe tab → audio | |
| auto-plays a description + the transcript shows. THIS IS THE MINIMUM VALID SUBMISSION. Commit here. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 5 — STT + "ASK" (zero-typing loop) | |
| Implement `transcribe_audio(audio_path)` in `modal_backend.py` (delegates to `cohere_stt.py`) using | |
| the verified Cohere Transcribe API. Wire the Ask tab: mic → transcribe → describe → speak. | |
| **CHECKPOINT 5:** Record a spoken question about the sample image → hear a spoken answer. No typing. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 6 — "READ TEXT" + LANGUAGE + CSS POLISH | |
| - Read Text tab uses the fixed OCR prompt (already in run_pipeline). | |
| - Confirm the language dropdown changes TTS output language (test English + Hindi minimum). | |
| - Build the real **Iris** `assets/custom.css` (see DESIGN SPEC below). Drive orb state from `app.py` | |
| by updating the orb HTML's class (idle / listening / seeing / thinking / speaking) at each stage. | |
| **CHECKPOINT 6:** All three tabs work; Hindi TTS works; UI matches the Iris spec. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 7 — HARDENING | |
| - Cold-start progress shown on first call. | |
| - Mic failure → reveal a `gr.Textbox` typed-question fallback (never block). | |
| - TTS failure → large-font text output only, with a `gr.Warning`. | |
| - Every stage wrapped in `safe_call`; no traceback ever reaches the user. | |
| - Confirm 3 examples load and run with no webcam. | |
| **CHECKPOINT 7:** Manually break the mic and break TTS — the app degrades gracefully, never crashes. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 8 — SUBMISSION ASSETS | |
| `README.md` — start with this frontmatter VERBATIM: | |
| ```yaml | |
| --- | |
| title: Third Eye | |
| emoji: 👁️ | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: gradio | |
| sdk_version: "5.0" | |
| app_file: app.py | |
| pinned: false | |
| tags: | |
| - hackathon | |
| - build-small | |
| - backyard-ai | |
| - accessibility | |
| - blind | |
| - openbmb/MiniCPM-V-2_0 | |
| - openbmb/VoxCPM2 | |
| - cohere/cohere-transcribe-03-2026 | |
| - tiny-titan | |
| - off-brand | |
| --- | |
| ``` | |
| Then write: what it is, who it's for, how to use, models+sizes table (call out 2.8B Tiny Titan), | |
| architecture paragraph, **on-device/edge section** (honest claim + roadmap: these models quantize to | |
| int4 GGUF and can run offline on a phone via llama.cpp — framed as roadmap, not a shipped phone | |
| build), accessibility & Iris design, run-it-yourself (env vars + `modal deploy` + Space secrets: | |
| MODAL_TOKEN_ID / MODAL_TOKEN_SECRET / HF_TOKEN), credits (OpenBMB, Cohere, Modal, HF). | |
| `BLOG.md` — a Field Notes draft: "What VLM quality really feels like at 2.8B" (what worked, where | |
| MiniCPM-V-2 struggled vs the 8B fallback, OCR accuracy notes). `DEMO_SCRIPT.md` — the 45–60s shot | |
| list (eye opens → blindfolded menu read aloud in Hindi → label/sign cuts → tagline). | |
| **CHECKPOINT 8:** Space builds clean, loads on a cold visit, all assets present. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # PHASE 9 — NICE-TO-HAVE (only if time remains, in this order) | |
| 1. Bounding-box "Zoom & Read" tab via `gr.ImageEditor`: user draws a rectangle, the crop is sent to | |
| MiniCPM-V with "Read the text in this image exactly as written." | |
| 2. Cache model weights on a `modal.Volume` to cut cold-start time. | |
| 3. A small GGUF int4 on-device proof + a benchmark table in README (params, int4 size, target device). | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # DESIGN SPEC — "Iris" custom CSS (Off-Brand track) | |
| Accessibility constraints rendered as a futuristic aesthetic. Pure CSS so it's reliable to build. | |
| - **Background** `#06070A` + faint radial vignette; optional subtle grain. | |
| - **Accent gradient** `#5B7CFA → #3DE0FF`; glows via layered `box-shadow`. | |
| - **Text** `#F5F7FA`; base font 20px, output text 24px+, line-height 1.7; contrast ≥ WCAG AA. | |
| - **Iris orb**: a centered circular `div` (~140px) with the accent radial gradient and a soft outer | |
| glow. Define keyframe animations per state class: | |
| - `.iris.idle` slow breathing scale 1.0↔1.04 (~4s). | |
| - `.iris.listening` pulsing ring. | |
| - `.iris.seeing` a scan-line sweep. | |
| - `.iris.thinking` faster, tighter pulse. | |
| - `.iris.speaking` waveform-like glow pulse synced loosely to playback. | |
| - **Primary button**: large pill / circle, min 96px hit target, accent gradient, thick cyan focus ring. | |
| - **Surfaces**: glass panels — `backdrop-filter: blur(12px)`, 1px hairline border, subtle inner glow. | |
| - **Motion**: wrap ALL animations in `@media (prefers-reduced-motion: reduce)` to disable them. | |
| - **Focus**: visible thick cyan focus ring on every interactive element (serves keyboard + the look). | |
| - The ARIA `live=polite` status line mirrors the orb state in words for screen-reader users. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| # DONE = ALL OF THIS TRUE | |
| - [ ] Describe / Ask / Read Text all work end-to-end, zero typing on the happy path. | |
| - [ ] Audio answers auto-play; transcript shown large. | |
| - [ ] Language dropdown drives multilingual TTS (English + Hindi verified). | |
| - [ ] 3 bundled examples run with no webcam. | |
| - [ ] Cold-start progress + mic/TTS fallbacks + `gr.Warning` everywhere; no raw tracebacks. | |
| - [ ] Iris custom CSS live (Off-Brand); WCAG-AA contrast. | |
| - [ ] README frontmatter verbatim; edge/on-device section honest; BLOG.md + DEMO_SCRIPT.md present. | |
| - [ ] Primary model 2.8B ≤ 4B (Tiny Titan) — stated in README. | |
| - [ ] Space builds clean and loads cold. | |
| Report which checkpoints passed and paste the final file tree when done. | |
| ═══════════════════════════════════════════════════════════════════════════════ | |
| ``` | |