FIXES_REPORT.md

Third Eye — Fixes Report (root causes + verification)

TL;DR

• STT was NOT inherently slow. The Cohere card advertises RTFx 500 ("blazing fast"). The ~5-minute transcriptions were a configuration bug: float32 + device_map="auto" ran the model on CPU / partially offloaded (1s per forward pass). Keep Cohere — it wins the Cohere prize. Do NOT switch to Whisper.
• Vision hallucinations ($20 for $18) came from sampling; switched to greedy + bf16.
• All other reported bugs were already fixed; verified in code. 3 minor issues closed.
• 38/38 unit tests pass (CPU). GPU fixes MEASURED on Modal A10G: STT 5 min → 0.6s warm; vision now reads $18 (not $20) and mango lassi (not MANGOLAISSI). See "Verification" below.

---

Root cause analysis

BUG 1 — STT slow (~5 min). FIXED (config) — cohere_stt.py

Evidence: logs show 1.01s/it per forward pass. A 2B model on an A10G does a forward pass in ~10–30 ms, not ~1000 ms. The model card benchmarks RTFx ~524 (a 6 s clip ≈ 0.01 s of compute). The app was ~25,000× slower than the model's own benchmark → not the model, the config.

Three compounding causes, all fixed:

1. from_pretrained(..., device_map="auto") with no torch_dtype → loaded in float32. On a single-GPU container this both doubles memory and pushes layers toward CPU offload, which streams weights per token. → Now: torch_dtype=bfloat16 and an explicit .to("cuda").
2. No decoding flags → if generation_config defaulted to beam search, every token costs N forward passes (the repeated 6/6 progress bars). → Now: num_beams=1, do_sample=False (greedy).
3. max_new_tokens=256 → trimmed to 128 (speech answers are short).

Why this is the fix and not a guess: greedy + half-precision + single-GPU pin is the standard cure for "transformers generate is mysteriously 100×+ too slow," and it directly removes every mechanism the logs pointed at (per-forward CPU cost, per-token beam multiplier).

BUG 1b — "Ask" mode answered the wrong question. FIXED — cohere_stt.py

Symptom: spoken questions weren't answered correctly. Not a wiring bug — the vision model answers fine. The STT was hallucinating a tail over trailing silence:

SPOKEN: 'What is the cheapest item on the menu and how much does it cost?'
HEARD:  'What is the cheapest item on the menu and how much does it cost? Yes, sir. It was a clue.'  ← fake

The fake tail corrupts the question handed to the vision model (here it survived because the real text came first; with a shorter/noisier real-mic clip the tail dominates and the answer goes wrong). Fix: trim leading/trailing silence with librosa.effects.trim(top_db=30) before transcription — removes the silent padding the model hallucinates over. Verified:

HEARD: 'What is the cheapest item on the menu and how much does it cost?'   (exact match, 39->29 tokens)
HEARD: 'How often should I take this medication?' -> ANSWER: 'take one capsule every 8 hours.'  (correct)
clean-clip regression: 'Hello, this is a test...' still transcribes perfectly.

Vision hallucination ($20≠$18). FIXED — modal_backend.py

sampling=True, temperature=0.2 still lets the model invent plausible-but-wrong numbers. Switched to sampling=False (greedy) — the faithful, repeatable choice for an accessibility/OCR tool — and bf16 (what the MiniCPM-V-2 card recommends; more stable than fp16 on small text).

Temp-file leak. FIXED — utils.py + app.py

bytes_to_wav() left WAVs in temp forever. Added prune_old_wavs() (removes only Third-Eye WAVs older than 10 min, so a clip still being served is never deleted), called at each pipeline start.

Previously-reported bugs — verified fixed in code

• READ "s" output → improved verbatim prompt (app.py: resolve_question). ✓
• repetition_penalty/max_new_tokens removed from model.chat (they caused refusals). ✓
• STT gated-model 403 → specific, actionable error message. ✓
• safe_call now surfaces exception type+message (truncated). ✓

---

Read Text OCR distortion (MANGOLAISSI). FIXED (mitigated) — modal_backend.py

Root cause: small text on a full 1024x1536 image exceeds a 2.8B VLM's effective OCR resolution, so it merged MANGO LASSI -> MANGOLAISSI. Prompt engineering did NOT help (tested: still MANGOLIASSI + a hallucinated price). Validated cause: cropping to the text region fixed it (Mango Lassi $5), proving it's a resolution problem, not a knowledge problem.

Fix: automatic tiled OCR for Read Text mode (describe_scene(..., tile=True)): split the image into two overlapping top/bottom bands, OCR each (enlarges relative text), and stitch the results with word-overlap dedup (stitch_overlapping_text, unit-tested). Automatic — no box-drawing, which matters because the user is blind and can't draw a selection box. Measured:

label:  AMOXICILLIN 500 mg Take one capsule every 8 hours Finish all medication EXP 12/2027   (perfect)
sign:   CENTRAL STATION 250 m PLATFORM 1-6                                                     (perfect)
menu:   ...GRILLED FISH $18 MANGOS LASSI $5    (space restored & readable; was "MANGOLAISSI")

Residual honest limit: the menu still shows a minor glyph error (MANGOS), the genuine 2.8B ceiling. It is now legible speech, not gibberish. Cost: Read Text does 2 model calls (~6s).

Faster-model question

Within Cohere's family this transcribe model is the ASR model, and once configured correctly it is fast (RTFx ~500). No turbo variant is needed. Keeping Cohere keeps the Cohere award eligible.

---

Verification — MEASURED on Modal GPU (A10G), 2026-06-14

Local (CPU, this machine):

• 38/38 unit tests pass (python -m pytest -q); ruff check clean; both modules import.

STT benchmark (modal run modal_backend.py::stt_benchmark):

[third-eye STT] loaded CohereLabs/cohere-transcribe-03-2026 | cuda_available=True | device=cuda:0 | dtype=torch.bfloat16
[COLD] total round-trip: 58.8s          (one-time container + model load)
[third-eye STT] generate: 1.56s for 26 tokens   (first call, includes CUDA warmup)
[third-eye STT] generate: 0.16s for 26 tokens   (warm: ~160 tok/s — correct for 2B on A10G)
[WARM] total round-trip: 0.6s
transcript: 'Hello, this is a test of the third eye text to speech system.'   (correct)

Result: 5 minutes → 0.6s warm round-trip (500-1000x). STT is now fast enough to demo live.

Vision/TTS smoke test (modal run modal_backend.py::smoke_test):

[third-eye VISION] loaded openbmb/MiniCPM-V-2 | device=cuda:0 | dtype=torch.bfloat16
[third-eye VISION] chat: 3.85s
"...Prices range from $6 for a soup to $18 for grilled fish, with mango lassi priced at $5."

Result: greedy sampling=False works (no refusal); $18 read correctly (was $20); mango lassi spelled correctly (was MANGOLAISSI); all three models confirmed on cuda + bf16.

Diagnostic safety net: if device=cpu ever appears in the STT log, the STT image installed a CPU-only torch — fix by pinning a CUDA wheel: pip_install("torch>=2.5", index_url="https://download.pytorch.org/whl/cu124"). (Not needed currently — verified cuda:0.)