Publish README.md from marimo

README.md

@@ -18,12 +18,13 @@ tags:
 # 🖱️ Tiny Computer Agent
 
 **An agentic harness that drives a real computer with a tiny, fully local model.**
-No cloud API, no API keys, no telemetry — and a core that imports nothing outside
-the Python standard library.
+No cloud API, no API keys, no telemetry — and a core that imports nothing outside the
+Python standard library.
 
-> **Status: early alpha.** The loop runs end to end today, with a scripted screen,
-> dry-run execution, a CLI and 56 offline tests. The vision backend is a
-> documented seam, not a feature — see [what works](#-what-works-today).
+> **Status 0.2.0: it runs, on pixels.** The screen is captured for real, OCR turns it
+> into a short labelled menu, a **256M** vision model picks from that menu, and the
+> harness resolves the choice to a real pixel and (behind two locks) clicks it.
+> What is *not* solved is model accuracy — see [measured results](#measured-results).
 
 ## The idea, in one loop
 
@@ -38,143 +39,153 @@ the Python standard library.
         repeat until done, or the step budget runs out
 ```
 
-1. **Observe** — grab the screen, downscale it, optionally attach OCR text or the
-   accessibility tree so the model never has to guess where things are.
-2. **Think** — a tiny (≤ 3B, quantised) vision-language model emits **exactly one
-   action** from a closed vocabulary.
-3. **Act** — the harness executes it with OS-level input (mouse, keyboard, scroll).
-4. **Verify** — the next frame shows whether it worked. Repeat, bounded by a step
-   budget and a kill switch.
+1. **Observe** — capture the screen, then build an **element index**: OCR (and,
+   optionally, the OS accessibility tree) locates text and widgets and gives each one a
+   label and a real screen coordinate.
+2. **Think** — the model is shown a *short numbered menu* built from that index and
+   answers with one number (plus a verb). It never emits coordinates.
+3. **Act** — the harness maps the chosen label back to pixels and executes it.
+4. **Verify** — re-read the screen, and stop if the same action repeats against an
+   unchanged screen, because that is a stuck agent, not a persistent one.
 
-## What works today
+## Measured results
 
+Everything below was run from this repo, **CPU only, dry-run**, against the synthetic
+screenshot (`--image example`), because the build machine has no desktop. Nothing here
+is aspirational.
+
+**A rule-based planner, end to end:** OCR read five labels off the screenshot, matched
+the goal, and resolved it to a pixel.
+
+```
+$ python -m tiny_computer_agent --image example --brain planner
+index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
+elements detected: 5
+
+  step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (295, 284)
+  step 1 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+```
+
+**A 256M vision model — SmolVLM-256M-Instruct, CPU — read the button off the pixels:**
+
+```
+$ python -m tiny_computer_agent --image example --brain vlm
+elements detected: 5
+
+  step 0 | click label='Save' reason="the reply names 'Save'" | ok: (dry-run) would click -> pixel (295, 284)
+  step 1 | click label='Save' ...
+  step 2 | ... ERROR no progress: repeating click label='Save' against an unchanged screen
 ```
-$ python -m tiny_computer_agent
-goal: click 'Save' to save the document
-mode: dry-run  |  screen: scripted demo  |  kill switch: create /tmp/tiny-computer-agent.STOP to stop
 
-  step 0 | wait seconds=0.5 reason="'Save' is not on screen yet" | ok: (dry-run) would wait 0.5s
-  step 1 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (360, 132)
-  step 2 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+It named the correct target from pixels alone. The episode is recorded as a *failure*
+only because the replayed screenshot never changes, so the no-progress guard stops a
+loop that a real desktop would have moved past.
 
-result: success - brain reported done: already clicked 'Save'; goal assumed met
+**A 0.5B text planner over the OCR index — Qwen2.5-0.5B-Instruct — is not reliable:**
+
+```
+$ python -m tiny_computer_agent --image example --brain text-planner
+  step 0 | click label='Cancel' reason='menu index 5'
+  step 1 | click label='Files'  reason="recovered 'Files' from the model's reply: {"kind": "click", "index": 0, ...}"
+  step 2 | click label='Save'   reason='menu index 4'
+result: failure - step budget of 3 exhausted
 ```
 
-Shipped and tested:
-
-- **the loop** — `AgentLoop` runs observe → think → act → verify, **re-reads the
-  screen after every action**, and stops on done, on a step budget, on the kill
-  switch, on a policy refusal, on a failed action, or on a brain error. Every step
-  is recorded in an `Episode` you can render or turn into rows.
-- **a closed action vocabulary** — `click` `type` `key` `scroll` `wait` `done`,
-  validated *before* anything executes. A model that invents a verb gets a recorded
-  error, not a click.
-- **three brains** — `MockBrain` replays a script; `PlannerBrain` is rule-based and
-  genuinely clicks the element matching the goal (it waits for a dialog instead of
-  flailing, and refuses to click the same label twice); `VLMBrain` is the seam —
-  prompt assembly and reply validation are done, `_generate()` is yours.
-- **two executors** — `DryRunExecutor` is the default: it logs the plan and resolves
-  every label to a pixel. `PyAutoGuiExecutor` does real input and **refuses to click
-  when a label matches no element**, because a tiny model guessing pixels is how you
-  click "Delete account". Dry-run fails exactly where live input refuses, so a
-  rehearsal never gives false confidence.
-- **policy gates** — deny-listed text (`sudo`, `rm -rf`, "password", "api_key"),
-  deny-listed widgets (terminal, keychain, password manager, wallet, bank), four
-  deny-listed key chords, a kill-switch file checked before every action, and a step
-  budget.
-- **real input needs two deliberate flags** — `--real-input` *and*
-  `TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1`. Default is a dry run.
-- **56 tests** needing no display, no model and no network.
-
-What does **not** work yet: seeing a real screen *usefully*. `GrabScreen` captures
-pixels through `mss`, but nothing yet turns pixels into an element index (OCR or
-AT-SPI), so a live screen currently yields zero elements — which is exactly what the
-label path depends on. And no model is wired up.
-
-## Install and run
+It clicked Cancel before Save, then wandered. The harness held up regardless: every
+action was validated against the closed vocabulary, a garbled JSON reply was recovered
+into a usable action, and the budget stopped it. **A tiny VLM beats a tiny text model
+here** — which is why the vision path is the recommended one.
+
+**116 tests pass**, including real OCR over a real image asserting the `Save` button's
+coordinates, and a screenshot replay that runs the whole pipeline with no display.
+
+## Install
 
 ```bash
-pip install -e .                          # core: standard library only, no deps
-pytest                                    # 56 offline tests
-python -m tiny_computer_agent             # scripted demo, dry run
-python -m tiny_computer_agent --live      # your real screen, still dry run
+pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
+pip install -e ".[desktop,ocr]"      # + real capture and real input
+pip install -e ".[vision,ocr]"       # + the tiny model brains
+pytest                               # 116 tests, no display or network needed
 ```
 
-## Put it on your own task
+## Run it on your machine
 
-```python
-from tiny_computer_agent import AgentLoop, Element, FakeScreen, MockBrain
+```bash
+python -m tiny_computer_agent --doctor                 # what can this machine do?
+python -m tiny_computer_agent --image example          # replay a screenshot, dry run
+python -m tiny_computer_agent --live                   # your real screen, dry run
+python -m tiny_computer_agent --live --real-input "click 'OK'"   # needs the env var too
+```
 
-screen = FakeScreen([(Element("Save", x=320, y=120, width=80, height=24),)])
-brain = MockBrain([{"kind": "click", "label": "Save"},
-                   {"kind": "done", "reason": "saved"}])
+`--doctor` is the first thing to run on a new machine: it checks the display session,
+capture backend, pyautogui, the OCR engine, the model runtime, and the kill switch, and
+prints the exact fix for anything missing. Real input needs **two** deliberate acts:
 
-episode = AgentLoop(screen=screen, brain=brain).run("save the document")
-print(episode.render())
-episode.to_rows()          # rows for a dataframe or notebook table
+```bash
+export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
+python -m tiny_computer_agent --live --real-input "click 'Save'"
 ```
 
-Swap `FakeScreen` for `GrabScreen()` and `MockBrain` for your `VLMBrain` subclass and
-that is the whole distance between the tests and a live agent.
+## How it works
+
+| Module | Job |
+|---|---|
+| `actions.py` | The closed vocabulary — `click` `type` `key` `scroll` `wait` `done`, validated before anything executes |
+| `observe.py` | `Element`/`Observation`, `FakeScreen`, `ImageScreen`, `GrabScreen` (mss → Pillow fallback) |
+| `perception.py` | Detectors: `OcrDetector` (RapidOCR), `GridDetector`, `AutoDetector`, `IndexedScreen` |
+| `brain.py` | `DecisionMenu` (the numbered menu), `ModelBrain`, lenient `parse_decision`, `MockBrain`, `PlannerBrain` |
+| `models.py` | `VLMBrain` (SmolVLM-256M) and `TextPlannerBrain` (Qwen2.5-0.5B), lazily loaded |
+| `act.py` | `DryRunExecutor` (the default) and `PyAutoGuiExecutor` |
+| `safety.py` | `Policy`: deny lists, kill switch, step budget, the blind-grid-click guard |
+| `loop.py` | `AgentLoop`, `Episode`, `Step`, and the no-progress guard |
+| `desktop.py` | Preflight checks, `doctor`, and `build_agent` in one call |
+| `demo.py` | The synthetic desktop used by tests and `--image example` |
 
-## Why "tiny" is the whole point
+## Why a tiny model can do this at all
 
-The harness does the heavy lifting, so the model does not have to be clever:
+The harness does the heavy lifting, so the model never has to be clever:
 
 | Job | Who does it |
 |---|---|
-| Finding *where* a button is | OCR / accessibility tree / element index — not the model |
-| Choosing *what* to do | a tiny VLM over a **closed action menu** (click, type, key, scroll, wait, done) |
-| Knowing whether it worked | the next frame, plus harness-side checks |
+| Finding *where* a button is | OCR / accessibility tree — **not** the model |
+| Choosing *what* to do | the model, but only by picking a number from a short menu |
+| Knowing whether it worked | the next frame, plus the no-progress guard |
 | Recovering from a bad step | the retry policy in `loop.py`, not the weights |
 
-That keeps the agent offline, fast to start, and small enough that a laptop GPU
-(or plain CPU) can run it on the same box it is controlling.
+The menu is the trick: a 256M model cannot emit `(295, 284)` reliably, but it can pick
+"4" out of five options — and the harness owns the coordinate lookup.
 
-## Candidate backends
-
-Pluggable behind one interface; nothing here needs a network call at run time.
-
-- **Moondream 2** / **SmolVLM-256M / 500M** — a few hundred MB, safe default
-- **Qwen2-VL-2B-Instruct**, **Florence-2-base** — stronger grounding, still tiny
-- **OCR + text-only planner** — no VLM at all; the cheapest credible baseline
-- **mock brain** — deterministic scripted actions, used to test the harness itself
+## Safety, because this thing has your mouse
 
-## Layout
+- Run it in a **VM, container, or throwaway desktop account**, not your real session.
+- **Dry run is the default**, so you can read a whole episode before anything clicks.
+- Deny lists for typing (`sudo`, `rm -rf`, "password"), for widgets (terminal, keychain,
+  wallet, bank) and for key chords.
+- A **kill-switch file** checked before every action, plus a hard step budget.
+- **Blind grid clicks are refused with real input** unless you pass `--allow-grid`. When
+  OCR finds nothing on screen, the fallback index is a numbered grid — coordinates with
+  no meaning — and a model choosing one is guessing about your desktop.
+- **Local-first, for real:** pixels and keystrokes never leave the machine.
 
-```
-tiny-computer-agent/
-├── README.md
-├── LICENSE
-├── pyproject.toml
-├── tiny_computer_agent_app.py   # marimo console: run an episode, publish this card
-├── tiny_computer_agent/
-│   ├── actions.py    # the closed action vocabulary
-│   ├── observe.py    # Observation + element index; FakeScreen, GrabScreen
-│   ├── brain.py      # MockBrain, PlannerBrain, VLMBrain (the seam)
-│   ├── act.py        # DryRunExecutor (default) + PyAutoGuiExecutor
-│   ├── safety.py     # Policy: deny lists, kill switch, step budget
-│   ├── loop.py       # AgentLoop, Episode, Step
-│   └── __main__.py   # CLI
-└── tests/            # 56 tests: no display, no model, no network
-```
+## What works, and what does not
 
-## Safety, because this thing has your mouse
+**Works:** real capture, a real OCR index with real coordinates, a 256M VLM choosing the
+right labelled target, the closed vocabulary, the policy gates, the no-progress guard,
+the doctor, the CLI, 116 tests.
 
-- Run it in a **VM, container, or throwaway desktop account** — not your real session.
-- Dry run is the default; read the whole plan before letting it near your pointer.
-- Deny-list for shells, terminals, payment flows and password managers.
-- A step budget plus a kill-switch file checked before every action.
-- **Local-first, for real:** screen pixels and keystrokes never leave the machine.
+**Does not work yet:** accuracy. A 0.5B text planner picks wrong actions; a 256M VLM can
+answer "done" before doing anything, and the harness currently trusts that; the
+accessibility-tree detector (AT-SPI / UI Automation) is not implemented; and nothing here
+has been run against a live multi-window desktop, only against a synthetic screenshot.
 
 ## Roadmap
 
-- [x] `loop.py` + `act.py` with the mock brain — the harness is testable before any model download
-- [x] policy gates: deny lists, kill switch, step budget, dry run by default
-- [ ] OCR / AT-SPI element index, so a live screen has clickable labels
-- [ ] tiny-VLM grounding backend, behind `VLMBrain._generate()`
+- [x] Capture, OCR index, closed vocabulary, dry-run default, policy gates, doctor
+- [x] A 256M VLM that picks a labelled target from pixels
+- [ ] Make the harness distrust a `done` that arrives before any action has been taken
+- [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows) for exact widgets
+- [ ] Score episodes: replay a trace and count how often the choice was right
 - [ ] marimo console: live screen preview, action log, pause / resume / step
-- [ ] persist episodes, so failed runs can be replayed and scored
 
 ## License