Publish README.md from marimo

README.md

@@ -21,10 +21,10 @@ tags:
 No cloud API, no API keys, no telemetry — and a core that imports nothing outside the
 Python standard library.
 
-> **Status 0.2.0: it runs, on pixels.** The screen is captured for real, OCR turns it
-> into a short labelled menu, a **256M** vision model picks from that menu, and the
-> harness resolves the choice to a real pixel and (behind two locks) clicks it.
-> What is *not* solved is model accuracy — see [measured results](#measured-results).
+> **Status 0.3.0: it drives a real desktop.** Screen capture, an OCR element index with
+> real coordinates, a real mouse click, and a *verified* outcome — all through the
+> harness, in one episode. What is not solved is model reliability: see
+> [measured results](#measured-results).
 
 ## The idea, in one loop
 
@@ -45,59 +45,65 @@ Python standard library.
 2. **Think** — the model is shown a *short numbered menu* built from that index and
    answers with one number (plus a verb). It never emits coordinates.
 3. **Act** — the harness maps the chosen label back to pixels and executes it.
-4. **Verify** — re-read the screen, and stop if the same action repeats against an
-   unchanged screen, because that is a stuck agent, not a persistent one.
+4. **Verify** — re-read the screen; stop if the same action repeats against an unchanged
+   screen, and only call the episode a success if a verifier can see it worked.
 
 ## Measured results
 
-Everything below was run from this repo, **CPU only, dry-run**, against the synthetic
-screenshot (`--image example`), because the build machine has no desktop. Nothing here
-is aspirational.
+Everything below was run from this repo on CPU, dry-run unless stated. Nothing is
+aspirational.
 
-**A rule-based planner, end to end:** OCR read five labels off the screenshot, matched
-the goal, and resolved it to a pixel.
+### On a live desktop, with a real click
+
+A real Tk window (a Save dialog with Save/Cancel buttons) on a **real X server**
+(`Xvfb :99`), driven with real mouse input. The button sits at window `+120+120`,
+`x=20, y=155`, `110x40`, so its centre is `(195, 295)`:
 
 ```
-$ python -m tiny_computer_agent --image example --brain planner
+$ python -m tiny_computer_agent --live --real-input --verify-text Saved "click 'Save' to save the document"
+mode: REAL INPUT (mouse and keyboard live)  |  screen: real desktop (monitor 1)  |  brain: planner
 index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
-elements detected: 5
 
-  step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (295, 284)
-  step 1 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+  step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: clicked (195, 296)
+
+result: success - verified after 1 action(s)
 ```
 
-**A 256M vision model — SmolVLM-256M-Instruct, CPU — read the button off the pixels:**
+The click landed on the button, the app changed its own window to "Saved", and the
+verifier confirmed it before the episode was called a success. **Pixels in, a real click
+out, and evidence that it worked.**
+
+### On a replayed screenshot
 
 ```
-$ python -m tiny_computer_agent --image example --brain vlm
+$ python -m tiny_computer_agent --image example --brain planner
 elements detected: 5
-
-  step 0 | click label='Save' reason="the reply names 'Save'" | ok: (dry-run) would click -> pixel (295, 284)
-  step 1 | click label='Save' ...
-  step 2 | ... ERROR no progress: repeating click label='Save' against an unchanged screen
+  step 0 | click label='Save' ... -> pixel (295, 284)          # OCR found the button
 ```
 
-It named the correct target from pixels alone. The episode is recorded as a *failure*
-only because the replayed screenshot never changes, so the no-progress guard stops a
-loop that a real desktop would have moved past.
-
-**A 0.5B text planner over the OCR index — Qwen2.5-0.5B-Instruct — is not reliable:**
+**SmolVLM-256M-Instruct** (256M parameters, CPU) read the button off the pixels and
+named the right target:
 
 ```
-$ python -m tiny_computer_agent --image example --brain text-planner
-  step 0 | click label='Cancel' reason='menu index 5'
-  step 1 | click label='Files'  reason="recovered 'Files' from the model's reply: {"kind": "click", "index": 0, ...}"
-  step 2 | click label='Save'   reason='menu index 4'
-result: failure - step budget of 3 exhausted
+  step 0 | click label='Save' reason="the reply names 'Save'" -> pixel (295, 284)
 ```
 
-It clicked Cancel before Save, then wandered. The harness held up regardless: every
-action was validated against the closed vocabulary, a garbled JSON reply was recovered
-into a usable action, and the budget stopped it. **A tiny VLM beats a tiny text model
-here** — which is why the vision path is the recommended one.
+### What did *not* work, and what caught it
+
+- **The 256M VLM echoes the prompt on a busy live screen.** Asked to click Save on the
+  live Tk window it replied with the prompt text back. The harness's echo detection
+  refused it (`BrainError: the reply mentions 4 menu entries, so it looks like an echo of
+  the prompt`) and stopped — **no blind click**. On the cleaner synthetic screenshot the
+  same model chose correctly, so this is a prompt/robustness problem, not a wiring one.
+- **Qwen2.5-0.5B text planner over the OCR index is not reliable**: it clicked Cancel
+  before Save, then recovered a garbled JSON reply, then found Save — budget exhausted.
+- **The grid fallback announced itself.** After the dialog closed, OCR found no text, the
+  index degraded to 96 numbered cells, and the run printed
+  `warning: no text was found on screen, so the index is 96 numbered grid cells and the
+  agent is choosing blind` — instead of silently guessing.
 
-**116 tests pass**, including real OCR over a real image asserting the `Save` button's
-coordinates, and a screenshot replay that runs the whole pipeline with no display.
+**123 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+that click a real button.
 
 ## Install
 
@@ -105,7 +111,7 @@ coordinates, and a screenshot replay that runs the whole pipeline with no displa
 pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
 pip install -e ".[desktop,ocr]"      # + real capture and real input
 pip install -e ".[vision,ocr]"       # + the tiny model brains
-pytest                               # 116 tests, no display or network needed
+pytest                               # 123 tests, no display or network needed
 ```
 
 ## Run it on your machine
@@ -114,18 +120,37 @@ pytest                               # 116 tests, no display or network needed
 python -m tiny_computer_agent --doctor                 # what can this machine do?
 python -m tiny_computer_agent --image example          # replay a screenshot, dry run
 python -m tiny_computer_agent --live                   # your real screen, dry run
-python -m tiny_computer_agent --live --real-input "click 'OK'"   # needs the env var too
+python -m tiny_computer_agent --live --real-input "click 'OK'"
+python -m tiny_computer_agent --live --real-input --verify-text "Saved" "click 'Save'"
 ```
 
 `--doctor` is the first thing to run on a new machine: it checks the display session,
-capture backend, pyautogui, the OCR engine, the model runtime, and the kill switch, and
-prints the exact fix for anything missing. Real input needs **two** deliberate acts:
+capture backend, pyautogui, the OCR engine, the model runtime and the kill switch, and
+prints the fix for anything missing. Real input needs **two** deliberate acts:
 
 ```bash
 export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
-python -m tiny_computer_agent --live --real-input "click 'Save'"
 ```
 
+**Pass `--verify-text`.** Without it, a successful action that removes its own target
+(clicking Save closes the dialog) is indistinguishable from a failure, and the episode is
+recorded as one. With it, the harness only reports success when it can see the text it
+was given.
+
+### No desktop? Use a virtual one
+
+Xvfb is enough for everything except a real model's hallucinations:
+
+```bash
+Xvfb :99 -screen 0 1280x800x24 &
+export DISPLAY=:99
+TINY_COMPUTER_AGENT_LIVE_TEST=1 python -m pytest tests/test_live_desktop.py
+```
+
+That test starts its own Tk window, asserts OCR finds the button at the coordinates it
+was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in
+because it moves your mouse — run it on a throwaway desktop.
+
 ## How it works
 
 | Module | Job |
@@ -133,57 +158,60 @@ python -m tiny_computer_agent --live --real-input "click 'Save'"
 | `actions.py` | The closed vocabulary — `click` `type` `key` `scroll` `wait` `done`, validated before anything executes |
 | `observe.py` | `Element`/`Observation`, `FakeScreen`, `ImageScreen`, `GrabScreen` (mss → Pillow fallback) |
 | `perception.py` | Detectors: `OcrDetector` (RapidOCR), `GridDetector`, `AutoDetector`, `IndexedScreen` |
-| `brain.py` | `DecisionMenu` (the numbered menu), `ModelBrain`, lenient `parse_decision`, `MockBrain`, `PlannerBrain` |
+| `brain.py` | `DecisionMenu` (the numbered menu), `ModelBrain`, lenient `parse_decision`, echo detection, `MockBrain`, `PlannerBrain` |
 | `models.py` | `VLMBrain` (SmolVLM-256M) and `TextPlannerBrain` (Qwen2.5-0.5B), lazily loaded |
 | `act.py` | `DryRunExecutor` (the default) and `PyAutoGuiExecutor` |
 | `safety.py` | `Policy`: deny lists, kill switch, step budget, the blind-grid-click guard |
-| `loop.py` | `AgentLoop`, `Episode`, `Step`, and the no-progress guard |
+| `loop.py` | `AgentLoop`, `Episode`, `Step`, the no-progress guard and the verifier |
 | `desktop.py` | Preflight checks, `doctor`, and `build_agent` in one call |
 | `demo.py` | The synthetic desktop used by tests and `--image example` |
 
 ## Why a tiny model can do this at all
 
-The harness does the heavy lifting, so the model never has to be clever:
-
 | Job | Who does it |
 |---|---|
 | Finding *where* a button is | OCR / accessibility tree — **not** the model |
 | Choosing *what* to do | the model, but only by picking a number from a short menu |
-| Knowing whether it worked | the next frame, plus the no-progress guard |
+| Knowing whether it worked | the verifier and the no-progress guard, not the model's word |
 | Recovering from a bad step | the retry policy in `loop.py`, not the weights |
 
-The menu is the trick: a 256M model cannot emit `(295, 284)` reliably, but it can pick
-"4" out of five options — and the harness owns the coordinate lookup.
+A 256M model cannot emit `(295, 284)` reliably, but it can pick "4" out of five options —
+and the harness owns the coordinate lookup, the vocabulary check and the definition of
+success.
 
 ## Safety, because this thing has your mouse
 
 - Run it in a **VM, container, or throwaway desktop account**, not your real session.
 - **Dry run is the default**, so you can read a whole episode before anything clicks.
-- Deny lists for typing (`sudo`, `rm -rf`, "password"), for widgets (terminal, keychain,
-  wallet, bank) and for key chords.
+- Deny lists for typing (`sudo`, `rm -rf`, "password"), widgets (terminal, keychain,
+  wallet, bank) and key chords.
 - A **kill-switch file** checked before every action, plus a hard step budget.
-- **Blind grid clicks are refused with real input** unless you pass `--allow-grid`. When
-  OCR finds nothing on screen, the fallback index is a numbered grid — coordinates with
-  no meaning — and a model choosing one is guessing about your desktop.
+- **Blind grid clicks are refused with real input** unless you pass `--allow-grid`: when
+  OCR finds nothing, the fallback index is coordinates with no meaning, and a model
+  choosing one is guessing about your desktop.
+- **A brain's "done" is a claim, not evidence**: pass `--verify-text` and the harness
+  decides success itself.
 - **Local-first, for real:** pixels and keystrokes never leave the machine.
 
 ## What works, and what does not
 
-**Works:** real capture, a real OCR index with real coordinates, a 256M VLM choosing the
-right labelled target, the closed vocabulary, the policy gates, the no-progress guard,
-the doctor, the CLI, 116 tests.
+**Works:** real capture, a real OCR index with coordinates that land on the right pixel,
+a real click that an app reacts to, verified success, the closed vocabulary, the policy
+gates, the no-progress guard, echo detection, the doctor, the CLI, 123 tests — and a
+256M VLM that reads a button label from pixels on a clean screenshot.
 
-**Does not work yet:** accuracy. A 0.5B text planner picks wrong actions; a 256M VLM can
-answer "done" before doing anything, and the harness currently trusts that; the
-accessibility-tree detector (AT-SPI / UI Automation) is not implemented; and nothing here
-has been run against a live multi-window desktop, only against a synthetic screenshot.
+**Does not work yet:** reliability on a *live* screen. The 256M VLM echoed the prompt
+there; the 0.5B text planner picks wrong actions; the accessibility-tree detector is not
+implemented, so labels come from OCR only; and the harness still trusts a model's `done`
+when no verifier is given.
 
 ## Roadmap
 
 - [x] Capture, OCR index, closed vocabulary, dry-run default, policy gates, doctor
-- [x] A 256M VLM that picks a labelled target from pixels
-- [ ] Make the harness distrust a `done` that arrives before any action has been taken
-- [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows) for exact widgets
+- [x] A real click on a real window, with a verified outcome
+- [x] Blind-grid-click guard, echo detection, no-progress guard
+- [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
+- [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
 - [ ] Score episodes: replay a trace and count how often the choice was right
 - [ ] marimo console: live screen preview, action log, pause / resume / step