AGENT_PROMPT.md

BUILD PROMPT — paste everything below this line into Codex / OpenCode

═══════════════════════════════════════════════════════════════════════════════

You are a senior Python + Gradio engineer. Build a Hugging Face Space called "Third Eye": a fully voice-driven accessibility app for blind / low-vision users. The user points a webcam at something (menu, medicine label, sign, scene), speaks a question, and hears the answer back in their language. Zero typing required on the happy path.

Work in the current project folder. Build incrementally and verify at every checkpoint. Do not write the whole app at once. Do not invent APIs. Follow these rules exactly.

───────────────────────────────────────────────────────────────────────────────

ABSOLUTE RULES (do not break these)

1. Verify before coding inference. Model load/call APIs differ by version. For every model, open its Hugging Face model card and confirm the exact model ID, load call, and inference call BEFORE writing that stage. If a tool to fetch docs is available (e.g. context7), use it. If a model ID does not resolve, STOP and report it — never silently substitute another model.
2. Sponsor models ONLY (table below). No OpenAI/Whisper/Google/etc. anywhere.
3. Build in phases. Stop at each CHECKPOINT and confirm it passes before continuing.
4. Never show a raw traceback to the user. Every stage is wrapped; failures become gr.Warning("friendly message") and a graceful fallback.
5. MUST-HAVE before SHOULD-HAVE before NICE-TO-HAVE. A working minimal app beats a broken complete one. If you run low on time, ship the vertical slice.
6. When unsure about an API after reading the card, write the smallest possible test script and run it before wiring that stage into the UI.

───────────────────────────────────────────────────────────────────────────────

MODELS (only these — verify each card before use)

Role	Model ID	Params	Sponsor
Vision + OCR (PRIMARY)	openbmb/MiniCPM-V-2_0	2.8B	OpenBMB
Vision + OCR (FALLBACK only if quality unacceptable)	openbmb/MiniCPM-V-4_5	8B	OpenBMB
Speech-to-text	CohereLabs/cohere-transcribe-03-2026	2B	Cohere
Text-to-speech	openbmb/VoxCPM2	2B	OpenBMB

Primary param budget = 2.8B (≤ 4B → qualifies for Tiny Titan). If you must use the 8B fallback, write in the README: "fallback used, Tiny Titan badge forfeited." Never swap silently.

───────────────────────────────────────────────────────────────────────────────

TECH STACK

• Gradio 5.x (gr.Blocks) on a Hugging Face Space (sdk: gradio).
• Modal serverless GPU (A10G) runs vision + TTS + STT.
• Python 3.11. No other cloud APIs.

───────────────────────────────────────────────────────────────────────────────

TARGET FILE TREE

app.py              # Gradio UI + orchestration
modal_backend.py    # Modal app: describe_scene(), speak(), transcribe_audio()
cohere_stt.py       # Cohere Transcribe wrapper (imported by modal_backend)
utils.py            # image<->bytes, bytes<->wav, safe_call wrapper
requirements.txt
.env.example        # MODAL_TOKEN_ID, MODAL_TOKEN_SECRET, HF_TOKEN
README.md           # HF frontmatter + story + edge section
BLOG.md             # Field Notes draft
DEMO_SCRIPT.md      # 45s shot list
assets/
  custom.css        # "Iris" design system
  sample_menu.jpg   sample_label.jpg   sample_sign.jpg   (use 3 royalty-free / your own photos)

═══════════════════════════════════════════════════════════════════════════════

PHASE 0 — VERIFY REALITY (do this first, write NO inference code yet)

For each model (MiniCPM-V-2_0, VoxCPM2, cohere-transcribe-03-2026):

• Confirm the model ID resolves on Hugging Face.
• Read the card's usage example. Record the EXACT: import, from_pretrained args (trust_remote_code, dtype, etc.), and the inference call signature.
• Note especially: MiniCPM-V's model.chat(...) signature varies — some versions take image=<PIL>, msgs=[{"role":"user","content":<str>}]; others take image=None, msgs=[{"role":"user","content":[<PIL>, <str>]}]. Use whatever THIS card shows.
• For VoxCPM2: find the real synthesis call (it may need a reference voice / a generate method, not model.synthesize). For Cohere Transcribe: confirm whether it loads via transformers pipeline("automatic-speech-recognition", ...) or needs a custom call.

CHECKPOINT 0: Output a short table of the verified API for each model (load call + infer call). If anything can't be verified, list it explicitly and propose the smallest fix. Then continue.

═══════════════════════════════════════════════════════════════════════════════

PHASE 1 — SCAFFOLD (UI renders with fake data)

Create all files. Use STUB functions that return canned data so the UI runs with no GPU.

requirements.txt:

gradio>=5.0
modal
pillow
soundfile
numpy

(Add transformers, torch, accelerate, sentencepiece, timm to the Modal image, not the Space requirements — the Space does not run the models locally.)

utils.py — implement:

import io, base64, tempfile, numpy as np
from PIL import Image

def image_to_bytes(image) -> bytes:
    if isinstance(image, np.ndarray):
        image = Image.fromarray(image)
    buf = io.BytesIO(); image.convert("RGB").save(buf, format="JPEG"); return buf.getvalue()

def bytes_to_wav(audio_bytes: bytes) -> str:
    f = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
    f.write(audio_bytes); f.close(); return f.name

def safe_call(fn, *args, fallback=None, warn="Something went wrong.", **kwargs):
    import gradio as gr
    try:
        return fn(*args, **kwargs)
    except Exception as e:
        gr.Warning(f"{warn} ({type(e).__name__})")
        return fallback

app.py — build gr.Blocks(css=open("assets/custom.css").read()) with:

• A header with the Iris orb (a gr.HTML div, class iris idle) + an ARIA live status line.
• A language gr.Dropdown (English, Hindi, German, Tamil, Telugu, Kannada), default English.
• Three gr.Tabs: Describe, Ask, Read Text. Each has a gr.Image(sources=["webcam","upload"]), Ask also has gr.Audio(sources=["microphone"]), plus a large primary button, a gr.Audio output (set autoplay=True), and a large-font gr.Textbox output for the transcript.
• The 3 sample images wired as gr.Examples so judges can test with no webcam.
• For now, button click calls a STUB run_pipeline(...) that returns a placeholder wav path + text.

CHECKPOINT 1: python app.py launches locally; UI loads; clicking a button shows placeholder text and the Iris orb is visible. No GPU involved yet.

═══════════════════════════════════════════════════════════════════════════════

PHASE 2 — MODAL VISION (real description)

modal_backend.py: create the Modal app + GPU image, implement describe_scene using the verified MiniCPM-V API from Phase 0.

import modal
app = modal.App("third-eye-backend")
vision_image = modal.Image.debian_slim().pip_install(
    "transformers>=4.40","torch","pillow","accelerate","sentencepiece","timm","soundfile")

@app.function(gpu="A10G", image=vision_image, timeout=180)
def describe_scene(image_bytes: bytes, question: str, lang: str = "en") -> str:
    import io, torch
    from PIL import Image
    from transformers import AutoModel, AutoTokenizer
    model_id = "openbmb/MiniCPM-V-2_0"
    model = AutoModel.from_pretrained(model_id, trust_remote_code=True,
              torch_dtype=torch.float16).cuda().eval()
    tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
    image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
    prompt = question.strip() or "Describe everything you see in detail."
    # >>> USE THE EXACT .chat() SIGNATURE YOU VERIFIED IN PHASE 0 <<<
    return model.chat(image=image, msgs=[{"role":"user","content":prompt}], tokenizer=tok)

Deploy: modal deploy modal_backend.py. Then write test_vision.py that reads sample_menu.jpg, calls describe_scene.remote(...), prints the answer.

CHECKPOINT 2: Real, sensible text comes back for the menu image. If quality is unusable, switch to MiniCPM-V-4_5 and note the forfeit in README (do not do this lightly).

═══════════════════════════════════════════════════════════════════════════════

PHASE 3 — MODAL TTS (real speech)

Add speak(text, lang) to modal_backend.py using the verified VoxCPM2 API. Return WAV bytes. Write test_tts.py that synthesizes "Hello, this is Third Eye." and saves out.wav.

CHECKPOINT 3: out.wav plays intelligible speech. (If multilingual needs a lang/voice arg, wire lang through now.)

═══════════════════════════════════════════════════════════════════════════════

PHASE 4 — WIRE "DESCRIBE" END-TO-END ← MINIMUM VALID SUBMISSION

Replace the stub run_pipeline in app.py:

def run_pipeline(image, audio_path, mode, lang):
    if image is None:
        gr.Warning("No image captured. Point the camera and try again.")
        return None, "No image captured.", ""
    img_bytes = image_to_bytes(image)
    if mode == "Ask" and audio_path:
        question = safe_call(transcribe_audio.remote, audio_path,
                             warn="Couldn't hear you — type your question instead.", fallback="")
    elif mode == "Read Text":
        question = "Read all text visible in this image, word by word, exactly as written."
    else:
        question = "Describe everything in this image in detail for a blind user."
    answer = safe_call(describe_scene.remote, img_bytes, question, lang,
                       warn="Vision model is waking up — try once more.", fallback="")
    if not answer:
        return None, "Could not analyze the image.", question
    audio_bytes = safe_call(speak.remote, answer, lang, warn="Voice unavailable — showing text.",
                            fallback=None)
    audio_out = bytes_to_wav(audio_bytes) if audio_bytes else None
    return audio_out, answer, question

Show gr.Progress with "Loading AI models (first run: ~30s)…" around the first heavy call.

CHECKPOINT 4: In the running Space/app, pick sample_menu.jpg in the Describe tab → audio auto-plays a description + the transcript shows. THIS IS THE MINIMUM VALID SUBMISSION. Commit here.

═══════════════════════════════════════════════════════════════════════════════

PHASE 5 — STT + "ASK" (zero-typing loop)

Implement transcribe_audio(audio_path) in modal_backend.py (delegates to cohere_stt.py) using the verified Cohere Transcribe API. Wire the Ask tab: mic → transcribe → describe → speak.

CHECKPOINT 5: Record a spoken question about the sample image → hear a spoken answer. No typing.

═══════════════════════════════════════════════════════════════════════════════

PHASE 6 — "READ TEXT" + LANGUAGE + CSS POLISH

• Read Text tab uses the fixed OCR prompt (already in run_pipeline).
• Confirm the language dropdown changes TTS output language (test English + Hindi minimum).
• Build the real Iris assets/custom.css (see DESIGN SPEC below). Drive orb state from app.py by updating the orb HTML's class (idle / listening / seeing / thinking / speaking) at each stage.

CHECKPOINT 6: All three tabs work; Hindi TTS works; UI matches the Iris spec.

═══════════════════════════════════════════════════════════════════════════════

PHASE 7 — HARDENING

• Cold-start progress shown on first call.
• Mic failure → reveal a gr.Textbox typed-question fallback (never block).
• TTS failure → large-font text output only, with a gr.Warning.
• Every stage wrapped in safe_call; no traceback ever reaches the user.
• Confirm 3 examples load and run with no webcam.

CHECKPOINT 7: Manually break the mic and break TTS — the app degrades gracefully, never crashes.

═══════════════════════════════════════════════════════════════════════════════ # PHASE 8 — SUBMISSION ASSETS README.md — start with this frontmatter VERBATIM: ```yaml

title: Third Eye emoji: 👁️ colorFrom: indigo colorTo: blue sdk: gradio sdk_version: "5.0" app_file: app.py pinned: false tags: - hackathon - build-small - backyard-ai - accessibility - blind - openbmb/MiniCPM-V-2_0 - openbmb/VoxCPM2 - cohere/cohere-transcribe-03-2026 - tiny-titan - off-brand

Then write: what it is, who it's for, how to use, models+sizes table (call out 2.8B Tiny Titan),
architecture paragraph, **on-device/edge section** (honest claim + roadmap: these models quantize to
int4 GGUF and can run offline on a phone via llama.cpp — framed as roadmap, not a shipped phone
build), accessibility & Iris design, run-it-yourself (env vars + `modal deploy` + Space secrets:
MODAL_TOKEN_ID / MODAL_TOKEN_SECRET / HF_TOKEN), credits (OpenBMB, Cohere, Modal, HF).

`BLOG.md` — a Field Notes draft: "What VLM quality really feels like at 2.8B" (what worked, where
MiniCPM-V-2 struggled vs the 8B fallback, OCR accuracy notes). `DEMO_SCRIPT.md` — the 45–60s shot
list (eye opens → blindfolded menu read aloud in Hindi → label/sign cuts → tagline).

**CHECKPOINT 8:** Space builds clean, loads on a cold visit, all assets present.

═══════════════════════════════════════════════════════════════════════════════
# PHASE 9 — NICE-TO-HAVE (only if time remains, in this order)
1. Bounding-box "Zoom & Read" tab via `gr.ImageEditor`: user draws a rectangle, the crop is sent to
   MiniCPM-V with "Read the text in this image exactly as written."
2. Cache model weights on a `modal.Volume` to cut cold-start time.
3. A small GGUF int4 on-device proof + a benchmark table in README (params, int4 size, target device).

═══════════════════════════════════════════════════════════════════════════════
# DESIGN SPEC — "Iris" custom CSS (Off-Brand track)
Accessibility constraints rendered as a futuristic aesthetic. Pure CSS so it's reliable to build.

- **Background** `#06070A` + faint radial vignette; optional subtle grain.
- **Accent gradient** `#5B7CFA → #3DE0FF`; glows via layered `box-shadow`.
- **Text** `#F5F7FA`; base font 20px, output text 24px+, line-height 1.7; contrast ≥ WCAG AA.
- **Iris orb**: a centered circular `div` (~140px) with the accent radial gradient and a soft outer
  glow. Define keyframe animations per state class:
  - `.iris.idle` slow breathing scale 1.0↔1.04 (~4s).
  - `.iris.listening` pulsing ring.
  - `.iris.seeing` a scan-line sweep.
  - `.iris.thinking` faster, tighter pulse.
  - `.iris.speaking` waveform-like glow pulse synced loosely to playback.
- **Primary button**: large pill / circle, min 96px hit target, accent gradient, thick cyan focus ring.
- **Surfaces**: glass panels — `backdrop-filter: blur(12px)`, 1px hairline border, subtle inner glow.
- **Motion**: wrap ALL animations in `@media (prefers-reduced-motion: reduce)` to disable them.
- **Focus**: visible thick cyan focus ring on every interactive element (serves keyboard + the look).
- The ARIA `live=polite` status line mirrors the orb state in words for screen-reader users.

═══════════════════════════════════════════════════════════════════════════════
# DONE = ALL OF THIS TRUE
- [ ] Describe / Ask / Read Text all work end-to-end, zero typing on the happy path.
- [ ] Audio answers auto-play; transcript shown large.
- [ ] Language dropdown drives multilingual TTS (English + Hindi verified).
- [ ] 3 bundled examples run with no webcam.
- [ ] Cold-start progress + mic/TTS fallbacks + `gr.Warning` everywhere; no raw tracebacks.
- [ ] Iris custom CSS live (Off-Brand); WCAG-AA contrast.
- [ ] README frontmatter verbatim; edge/on-device section honest; BLOG.md + DEMO_SCRIPT.md present.
- [ ] Primary model 2.8B ≤ 4B (Tiny Titan) — stated in README.
- [ ] Space builds clean and loads cold.

Report which checkpoints passed and paste the final file tree when done.
═══════════════════════════════════════════════════════════════════════════════