# Third Eye — Fixes Report (root causes + verification)

## TL;DR
- **STT was NOT inherently slow.** The Cohere card advertises RTFx ~500 ("blazing fast").
  The ~5-minute transcriptions were a **configuration bug**: float32 + `device_map="auto"`
  ran the model on CPU / partially offloaded (~1s per forward pass). **Keep Cohere — it wins the
  Cohere prize. Do NOT switch to Whisper.**
- Vision hallucinations (`$20` for `$18`) came from **sampling**; switched to **greedy** + **bf16**.
- All other reported bugs were already fixed; verified in code. 3 minor issues closed.
- **38/38 unit tests pass** (CPU). **GPU fixes MEASURED on Modal A10G:** STT 5 min → 0.6s warm;
  vision now reads `$18` (not `$20`) and `mango lassi` (not `MANGOLAISSI`). See "Verification" below.

---

## Root cause analysis

### BUG 1 — STT slow (~5 min). FIXED (config) — `cohere_stt.py`
**Evidence:** logs show `1.01s/it` per forward pass. A 2B model on an A10G does a forward pass in
~10–30 ms, not ~1000 ms. The model card benchmarks RTFx ~524 (a 6 s clip ≈ 0.01 s of compute).
The app was ~25,000× slower than the model's own benchmark → not the model, the config.

**Three compounding causes, all fixed:**
1. `from_pretrained(..., device_map="auto")` with **no `torch_dtype`** → loaded in **float32**.
   On a single-GPU container this both doubles memory and pushes layers toward CPU offload, which
   streams weights per token. → Now: `torch_dtype=bfloat16` and an explicit `.to("cuda")`.
2. No decoding flags → if `generation_config` defaulted to beam search, every token costs N forward
   passes (the repeated `6/6` progress bars). → Now: `num_beams=1, do_sample=False` (greedy).
3. `max_new_tokens=256` → trimmed to `128` (speech answers are short).

**Why this is the fix and not a guess:** greedy + half-precision + single-GPU pin is the standard
cure for "transformers `generate` is mysteriously 100×+ too slow," and it directly removes every
mechanism the logs pointed at (per-forward CPU cost, per-token beam multiplier).

### BUG 1b — "Ask" mode answered the wrong question. FIXED — `cohere_stt.py`
**Symptom:** spoken questions weren't answered correctly. **Not a wiring bug** — the vision model
answers fine. The STT was **hallucinating a tail over trailing silence**:
```
SPOKEN: 'What is the cheapest item on the menu and how much does it cost?'
HEARD:  'What is the cheapest item on the menu and how much does it cost? Yes, sir. It was a clue.'  ← fake
```
The fake tail corrupts the question handed to the vision model (here it survived because the real
text came first; with a shorter/noisier real-mic clip the tail dominates and the answer goes wrong).
**Fix:** trim leading/trailing silence with `librosa.effects.trim(top_db=30)` before transcription —
removes the silent padding the model hallucinates over. **Verified:**
```
HEARD: 'What is the cheapest item on the menu and how much does it cost?'   (exact match, 39->29 tokens)
HEARD: 'How often should I take this medication?' -> ANSWER: 'take one capsule every 8 hours.'  (correct)
clean-clip regression: 'Hello, this is a test...' still transcribes perfectly.
```

### Vision hallucination (`$20`≠`$18`). FIXED — `modal_backend.py`
`sampling=True, temperature=0.2` still lets the model invent plausible-but-wrong numbers. Switched
to **`sampling=False` (greedy)** — the faithful, repeatable choice for an accessibility/OCR tool —
and **bf16** (what the MiniCPM-V-2 card recommends; more stable than fp16 on small text).

### Temp-file leak. FIXED — `utils.py` + `app.py`
`bytes_to_wav()` left WAVs in temp forever. Added `prune_old_wavs()` (removes only Third-Eye WAVs
older than 10 min, so a clip still being served is never deleted), called at each pipeline start.

### Previously-reported bugs — verified fixed in code
- READ "s" output → improved verbatim prompt (`app.py: resolve_question`). ✓
- `repetition_penalty`/`max_new_tokens` removed from `model.chat` (they caused refusals). ✓
- STT gated-model 403 → specific, actionable error message. ✓
- `safe_call` now surfaces exception type+message (truncated). ✓

---

### Read Text OCR distortion (`MANGOLAISSI`). FIXED (mitigated) — `modal_backend.py`
**Root cause:** small text on a full 1024x1536 image exceeds a 2.8B VLM's effective OCR
resolution, so it merged `MANGO LASSI` -> `MANGOLAISSI`. Prompt engineering did NOT help
(tested: still `MANGOLIASSI` + a hallucinated price). **Validated cause:** cropping to the text
region fixed it (`Mango Lassi $5`), proving it's a resolution problem, not a knowledge problem.

**Fix:** automatic **tiled OCR** for Read Text mode (`describe_scene(..., tile=True)`): split the
image into two overlapping top/bottom bands, OCR each (enlarges relative text), and stitch the
results with word-overlap dedup (`stitch_overlapping_text`, unit-tested). **Automatic — no
box-drawing**, which matters because the user is blind and can't draw a selection box. Measured:
```
label:  AMOXICILLIN 500 mg Take one capsule every 8 hours Finish all medication EXP 12/2027   (perfect)
sign:   CENTRAL STATION 250 m PLATFORM 1-6                                                     (perfect)
menu:   ...GRILLED FISH $18 MANGOS LASSI $5    (space restored & readable; was "MANGOLAISSI")
```
**Residual honest limit:** the menu still shows a minor glyph error (`MANGOS`), the genuine 2.8B
ceiling. It is now legible speech, not gibberish. Cost: Read Text does 2 model calls (~6s).

## Faster-model question
Within Cohere's family this transcribe model **is** the ASR model, and once configured correctly it
*is* fast (RTFx ~500). No turbo variant is needed. Keeping Cohere keeps the Cohere award eligible.

---

## Verification — MEASURED on Modal GPU (A10G), 2026-06-14
**Local (CPU, this machine):**
- `38/38` unit tests pass (`python -m pytest -q`); `ruff check` clean; both modules import.

**STT benchmark** (`modal run modal_backend.py::stt_benchmark`):
```
[third-eye STT] loaded CohereLabs/cohere-transcribe-03-2026 | cuda_available=True | device=cuda:0 | dtype=torch.bfloat16
[COLD] total round-trip: 58.8s          (one-time container + model load)
[third-eye STT] generate: 1.56s for 26 tokens   (first call, includes CUDA warmup)
[third-eye STT] generate: 0.16s for 26 tokens   (warm: ~160 tok/s — correct for 2B on A10G)
[WARM] total round-trip: 0.6s
transcript: 'Hello, this is a test of the third eye text to speech system.'   (correct)
```
**Result: ~5 minutes → 0.6s warm round-trip (~500-1000x). STT is now fast enough to demo live.**

**Vision/TTS smoke test** (`modal run modal_backend.py::smoke_test`):
```
[third-eye VISION] loaded openbmb/MiniCPM-V-2 | device=cuda:0 | dtype=torch.bfloat16
[third-eye VISION] chat: 3.85s
"...Prices range from $6 for a soup to $18 for grilled fish, with mango lassi priced at $5."
```
**Result:** greedy `sampling=False` works (no refusal); `$18` read correctly (was `$20`);
`mango lassi` spelled correctly (was `MANGOLAISSI`); all three models confirmed on cuda + bf16.

**Diagnostic safety net:** if `device=cpu` ever appears in the STT log, the STT image installed a
CPU-only torch — fix by pinning a CUDA wheel: `pip_install("torch>=2.5",
index_url="https://download.pytorch.org/whl/cu124")`. (Not needed currently — verified cuda:0.)
```