Publish tiny_computer_agent_app.py from marimo

tiny_computer_agent_app.py

@@ -44,10 +44,10 @@ def readme():
     No cloud API, no API keys, no telemetry — and a core that imports nothing outside the
     Python standard library.
 
-    > **Status 0.2.0: it runs, on pixels.** The screen is captured for real, OCR turns it
-    > into a short labelled menu, a **256M** vision model picks from that menu, and the
-    > harness resolves the choice to a real pixel and (behind two locks) clicks it.
-    > What is *not* solved is model accuracy — see [measured results](#measured-results).
+    > **Status 0.3.0: it drives a real desktop.** Screen capture, an OCR element index with
+    > real coordinates, a real mouse click, and a *verified* outcome — all through the
+    > harness, in one episode. What is not solved is model reliability: see
+    > [measured results](#measured-results).
 
     ## The idea, in one loop
 
@@ -68,59 +68,65 @@ def readme():
     2. **Think** — the model is shown a *short numbered menu* built from that index and
        answers with one number (plus a verb). It never emits coordinates.
     3. **Act** — the harness maps the chosen label back to pixels and executes it.
-    4. **Verify** — re-read the screen, and stop if the same action repeats against an
-       unchanged screen, because that is a stuck agent, not a persistent one.
+    4. **Verify** — re-read the screen; stop if the same action repeats against an unchanged
+       screen, and only call the episode a success if a verifier can see it worked.
 
     ## Measured results
 
-    Everything below was run from this repo, **CPU only, dry-run**, against the synthetic
-    screenshot (`--image example`), because the build machine has no desktop. Nothing here
-    is aspirational.
+    Everything below was run from this repo on CPU, dry-run unless stated. Nothing is
+    aspirational.
 
-    **A rule-based planner, end to end:** OCR read five labels off the screenshot, matched
-    the goal, and resolved it to a pixel.
+    ### On a live desktop, with a real click
+
+    A real Tk window (a Save dialog with Save/Cancel buttons) on a **real X server**
+    (`Xvfb :99`), driven with real mouse input. The button sits at window `+120+120`,
+    `x=20, y=155`, `110x40`, so its centre is `(195, 295)`:
 
     ```
-    $ python -m tiny_computer_agent --image example --brain planner
+    $ python -m tiny_computer_agent --live --real-input --verify-text Saved "click 'Save' to save the document"
+    mode: REAL INPUT (mouse and keyboard live)  |  screen: real desktop (monitor 1)  |  brain: planner
     index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
-    elements detected: 5
 
-      step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (295, 284)
-      step 1 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+      step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: clicked (195, 296)
+
+    result: success - verified after 1 action(s)
     ```
 
-    **A 256M vision model — SmolVLM-256M-Instruct, CPU — read the button off the pixels:**
+    The click landed on the button, the app changed its own window to "Saved", and the
+    verifier confirmed it before the episode was called a success. **Pixels in, a real click
+    out, and evidence that it worked.**
+
+    ### On a replayed screenshot
 
     ```
-    $ python -m tiny_computer_agent --image example --brain vlm
+    $ python -m tiny_computer_agent --image example --brain planner
     elements detected: 5
-
-      step 0 | click label='Save' reason="the reply names 'Save'" | ok: (dry-run) would click -> pixel (295, 284)
-      step 1 | click label='Save' ...
-      step 2 | ... ERROR no progress: repeating click label='Save' against an unchanged screen
+      step 0 | click label='Save' ... -> pixel (295, 284)          # OCR found the button
     ```
 
-    It named the correct target from pixels alone. The episode is recorded as a *failure*
-    only because the replayed screenshot never changes, so the no-progress guard stops a
-    loop that a real desktop would have moved past.
-
-    **A 0.5B text planner over the OCR index — Qwen2.5-0.5B-Instruct — is not reliable:**
+    **SmolVLM-256M-Instruct** (256M parameters, CPU) read the button off the pixels and
+    named the right target:
 
     ```
-    $ python -m tiny_computer_agent --image example --brain text-planner
-      step 0 | click label='Cancel' reason='menu index 5'
-      step 1 | click label='Files'  reason="recovered 'Files' from the model's reply: {\"kind\": \"click\", \"index\": 0, ...}"
-      step 2 | click label='Save'   reason='menu index 4'
-    result: failure - step budget of 3 exhausted
+      step 0 | click label='Save' reason="the reply names 'Save'" -> pixel (295, 284)
     ```
 
-    It clicked Cancel before Save, then wandered. The harness held up regardless: every
-    action was validated against the closed vocabulary, a garbled JSON reply was recovered
-    into a usable action, and the budget stopped it. **A tiny VLM beats a tiny text model
-    here** — which is why the vision path is the recommended one.
+    ### What did *not* work, and what caught it
+
+    - **The 256M VLM echoes the prompt on a busy live screen.** Asked to click Save on the
+      live Tk window it replied with the prompt text back. The harness's echo detection
+      refused it (`BrainError: the reply mentions 4 menu entries, so it looks like an echo of
+      the prompt`) and stopped — **no blind click**. On the cleaner synthetic screenshot the
+      same model chose correctly, so this is a prompt/robustness problem, not a wiring one.
+    - **Qwen2.5-0.5B text planner over the OCR index is not reliable**: it clicked Cancel
+      before Save, then recovered a garbled JSON reply, then found Save — budget exhausted.
+    - **The grid fallback announced itself.** After the dialog closed, OCR found no text, the
+      index degraded to 96 numbered cells, and the run printed
+      `warning: no text was found on screen, so the index is 96 numbered grid cells and the
+      agent is choosing blind` — instead of silently guessing.
 
-    **116 tests pass**, including real OCR over a real image asserting the `Save` button's
-    coordinates, and a screenshot replay that runs the whole pipeline with no display.
+    **123 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+    that click a real button.
 
     ## Install
 
@@ -128,7 +134,7 @@ def readme():
     pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
     pip install -e ".[desktop,ocr]"      # + real capture and real input
     pip install -e ".[vision,ocr]"       # + the tiny model brains
-    pytest                               # 116 tests, no display or network needed
+    pytest                               # 123 tests, no display or network needed
     ```
 
     ## Run it on your machine
@@ -137,18 +143,37 @@ def readme():
     python -m tiny_computer_agent --doctor                 # what can this machine do?
     python -m tiny_computer_agent --image example          # replay a screenshot, dry run
     python -m tiny_computer_agent --live                   # your real screen, dry run
-    python -m tiny_computer_agent --live --real-input "click 'OK'"   # needs the env var too
+    python -m tiny_computer_agent --live --real-input "click 'OK'"
+    python -m tiny_computer_agent --live --real-input --verify-text "Saved" "click 'Save'"
     ```
 
     `--doctor` is the first thing to run on a new machine: it checks the display session,
-    capture backend, pyautogui, the OCR engine, the model runtime, and the kill switch, and
-    prints the exact fix for anything missing. Real input needs **two** deliberate acts:
+    capture backend, pyautogui, the OCR engine, the model runtime and the kill switch, and
+    prints the fix for anything missing. Real input needs **two** deliberate acts:
 
     ```bash
     export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
-    python -m tiny_computer_agent --live --real-input "click 'Save'"
     ```
 
+    **Pass `--verify-text`.** Without it, a successful action that removes its own target
+    (clicking Save closes the dialog) is indistinguishable from a failure, and the episode is
+    recorded as one. With it, the harness only reports success when it can see the text it
+    was given.
+
+    ### No desktop? Use a virtual one
+
+    Xvfb is enough for everything except a real model's hallucinations:
+
+    ```bash
+    Xvfb :99 -screen 0 1280x800x24 &
+    export DISPLAY=:99
+    TINY_COMPUTER_AGENT_LIVE_TEST=1 python -m pytest tests/test_live_desktop.py
+    ```
+
+    That test starts its own Tk window, asserts OCR finds the button at the coordinates it
+    was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in
+    because it moves your mouse — run it on a throwaway desktop.
+
     ## How it works
 
     | Module | Job |
@@ -156,57 +181,60 @@ def readme():
     | `actions.py` | The closed vocabulary — `click` `type` `key` `scroll` `wait` `done`, validated before anything executes |
     | `observe.py` | `Element`/`Observation`, `FakeScreen`, `ImageScreen`, `GrabScreen` (mss → Pillow fallback) |
     | `perception.py` | Detectors: `OcrDetector` (RapidOCR), `GridDetector`, `AutoDetector`, `IndexedScreen` |
-    | `brain.py` | `DecisionMenu` (the numbered menu), `ModelBrain`, lenient `parse_decision`, `MockBrain`, `PlannerBrain` |
+    | `brain.py` | `DecisionMenu` (the numbered menu), `ModelBrain`, lenient `parse_decision`, echo detection, `MockBrain`, `PlannerBrain` |
     | `models.py` | `VLMBrain` (SmolVLM-256M) and `TextPlannerBrain` (Qwen2.5-0.5B), lazily loaded |
     | `act.py` | `DryRunExecutor` (the default) and `PyAutoGuiExecutor` |
     | `safety.py` | `Policy`: deny lists, kill switch, step budget, the blind-grid-click guard |
-    | `loop.py` | `AgentLoop`, `Episode`, `Step`, and the no-progress guard |
+    | `loop.py` | `AgentLoop`, `Episode`, `Step`, the no-progress guard and the verifier |
     | `desktop.py` | Preflight checks, `doctor`, and `build_agent` in one call |
     | `demo.py` | The synthetic desktop used by tests and `--image example` |
 
     ## Why a tiny model can do this at all
 
-    The harness does the heavy lifting, so the model never has to be clever:
-
     | Job | Who does it |
     |---|---|
     | Finding *where* a button is | OCR / accessibility tree — **not** the model |
     | Choosing *what* to do | the model, but only by picking a number from a short menu |
-    | Knowing whether it worked | the next frame, plus the no-progress guard |
+    | Knowing whether it worked | the verifier and the no-progress guard, not the model's word |
     | Recovering from a bad step | the retry policy in `loop.py`, not the weights |
 
-    The menu is the trick: a 256M model cannot emit `(295, 284)` reliably, but it can pick
-    "4" out of five options — and the harness owns the coordinate lookup.
+    A 256M model cannot emit `(295, 284)` reliably, but it can pick "4" out of five options —
+    and the harness owns the coordinate lookup, the vocabulary check and the definition of
+    success.
 
     ## Safety, because this thing has your mouse
 
     - Run it in a **VM, container, or throwaway desktop account**, not your real session.
     - **Dry run is the default**, so you can read a whole episode before anything clicks.
-    - Deny lists for typing (`sudo`, `rm -rf`, "password"), for widgets (terminal, keychain,
-      wallet, bank) and for key chords.
+    - Deny lists for typing (`sudo`, `rm -rf`, "password"), widgets (terminal, keychain,
+      wallet, bank) and key chords.
     - A **kill-switch file** checked before every action, plus a hard step budget.
-    - **Blind grid clicks are refused with real input** unless you pass `--allow-grid`. When
-      OCR finds nothing on screen, the fallback index is a numbered grid — coordinates with
-      no meaning — and a model choosing one is guessing about your desktop.
+    - **Blind grid clicks are refused with real input** unless you pass `--allow-grid`: when
+      OCR finds nothing, the fallback index is coordinates with no meaning, and a model
+      choosing one is guessing about your desktop.
+    - **A brain's "done" is a claim, not evidence**: pass `--verify-text` and the harness
+      decides success itself.
     - **Local-first, for real:** pixels and keystrokes never leave the machine.
 
     ## What works, and what does not
 
-    **Works:** real capture, a real OCR index with real coordinates, a 256M VLM choosing the
-    right labelled target, the closed vocabulary, the policy gates, the no-progress guard,
-    the doctor, the CLI, 116 tests.
+    **Works:** real capture, a real OCR index with coordinates that land on the right pixel,
+    a real click that an app reacts to, verified success, the closed vocabulary, the policy
+    gates, the no-progress guard, echo detection, the doctor, the CLI, 123 tests — and a
+    256M VLM that reads a button label from pixels on a clean screenshot.
 
-    **Does not work yet:** accuracy. A 0.5B text planner picks wrong actions; a 256M VLM can
-    answer "done" before doing anything, and the harness currently trusts that; the
-    accessibility-tree detector (AT-SPI / UI Automation) is not implemented; and nothing here
-    has been run against a live multi-window desktop, only against a synthetic screenshot.
+    **Does not work yet:** reliability on a *live* screen. The 256M VLM echoed the prompt
+    there; the 0.5B text planner picks wrong actions; the accessibility-tree detector is not
+    implemented, so labels come from OCR only; and the harness still trusts a model's `done`
+    when no verifier is given.
 
     ## Roadmap
 
     - [x] Capture, OCR index, closed vocabulary, dry-run default, policy gates, doctor
-    - [x] A 256M VLM that picks a labelled target from pixels
-    - [ ] Make the harness distrust a `done` that arrives before any action has been taken
-    - [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows) for exact widgets
+    - [x] A real click on a real window, with a verified outcome
+    - [x] Blind-grid-click guard, echo detection, no-progress guard
+    - [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
+    - [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
     - [ ] Score episodes: replay a trace and count how often the choice was right
     - [ ] marimo console: live screen preview, action log, pause / resume / step
 
@@ -229,7 +257,7 @@ def repo_files():
 
     [project]
     name = "tiny-computer-agent"
-    version = "0.2.0"
+    version = "0.3.0"
     description = "An agentic harness that drives a real computer with a tiny, fully local model - no cloud API."
     readme = "README.md"
     requires-python = ">=3.10"