Publish tiny_computer_agent_app.py from marimo

tiny_computer_agent_app.py

@@ -41,12 +41,13 @@ def readme():
     # 🖱️ Tiny Computer Agent
 
     **An agentic harness that drives a real computer with a tiny, fully local model.**
-    No cloud API, no API keys, no telemetry — and a core that imports nothing outside
-    the Python standard library.
+    No cloud API, no API keys, no telemetry — and a core that imports nothing outside the
+    Python standard library.
 
-    > **Status: early alpha.** The loop runs end to end today, with a scripted screen,
-    > dry-run execution, a CLI and 56 offline tests. The vision backend is a
-    > documented seam, not a feature — see [what works](#-what-works-today).
+    > **Status 0.2.0: it runs, on pixels.** The screen is captured for real, OCR turns it
+    > into a short labelled menu, a **256M** vision model picks from that menu, and the
+    > harness resolves the choice to a real pixel and (behind two locks) clicks it.
+    > What is *not* solved is model accuracy — see [measured results](#measured-results).
 
     ## The idea, in one loop
 
@@ -61,143 +62,153 @@ def readme():
             repeat until done, or the step budget runs out
     ```
 
-    1. **Observe** — grab the screen, downscale it, optionally attach OCR text or the
-       accessibility tree so the model never has to guess where things are.
-    2. **Think** — a tiny (≤ 3B, quantised) vision-language model emits **exactly one
-       action** from a closed vocabulary.
-    3. **Act** — the harness executes it with OS-level input (mouse, keyboard, scroll).
-    4. **Verify** — the next frame shows whether it worked. Repeat, bounded by a step
-       budget and a kill switch.
+    1. **Observe** — capture the screen, then build an **element index**: OCR (and,
+       optionally, the OS accessibility tree) locates text and widgets and gives each one a
+       label and a real screen coordinate.
+    2. **Think** — the model is shown a *short numbered menu* built from that index and
+       answers with one number (plus a verb). It never emits coordinates.
+    3. **Act** — the harness maps the chosen label back to pixels and executes it.
+    4. **Verify** — re-read the screen, and stop if the same action repeats against an
+       unchanged screen, because that is a stuck agent, not a persistent one.
 
-    ## What works today
+    ## Measured results
 
+    Everything below was run from this repo, **CPU only, dry-run**, against the synthetic
+    screenshot (`--image example`), because the build machine has no desktop. Nothing here
+    is aspirational.
+
+    **A rule-based planner, end to end:** OCR read five labels off the screenshot, matched
+    the goal, and resolved it to a pixel.
+
+    ```
+    $ python -m tiny_computer_agent --image example --brain planner
+    index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
+    elements detected: 5
+
+      step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (295, 284)
+      step 1 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+    ```
+
+    **A 256M vision model — SmolVLM-256M-Instruct, CPU — read the button off the pixels:**
+
+    ```
+    $ python -m tiny_computer_agent --image example --brain vlm
+    elements detected: 5
+
+      step 0 | click label='Save' reason="the reply names 'Save'" | ok: (dry-run) would click -> pixel (295, 284)
+      step 1 | click label='Save' ...
+      step 2 | ... ERROR no progress: repeating click label='Save' against an unchanged screen
     ```
-    $ python -m tiny_computer_agent
-    goal: click 'Save' to save the document
-    mode: dry-run  |  screen: scripted demo  |  kill switch: create /tmp/tiny-computer-agent.STOP to stop
 
-      step 0 | wait seconds=0.5 reason="'Save' is not on screen yet" | ok: (dry-run) would wait 0.5s
-      step 1 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (360, 132)
-      step 2 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+    It named the correct target from pixels alone. The episode is recorded as a *failure*
+    only because the replayed screenshot never changes, so the no-progress guard stops a
+    loop that a real desktop would have moved past.
 
-    result: success - brain reported done: already clicked 'Save'; goal assumed met
+    **A 0.5B text planner over the OCR index — Qwen2.5-0.5B-Instruct — is not reliable:**
+
+    ```
+    $ python -m tiny_computer_agent --image example --brain text-planner
+      step 0 | click label='Cancel' reason='menu index 5'
+      step 1 | click label='Files'  reason="recovered 'Files' from the model's reply: {\"kind\": \"click\", \"index\": 0, ...}"
+      step 2 | click label='Save'   reason='menu index 4'
+    result: failure - step budget of 3 exhausted
     ```
 
-    Shipped and tested:
-
-    - **the loop** — `AgentLoop` runs observe → think → act → verify, **re-reads the
-      screen after every action**, and stops on done, on a step budget, on the kill
-      switch, on a policy refusal, on a failed action, or on a brain error. Every step
-      is recorded in an `Episode` you can render or turn into rows.
-    - **a closed action vocabulary** — `click` `type` `key` `scroll` `wait` `done`,
-      validated *before* anything executes. A model that invents a verb gets a recorded
-      error, not a click.
-    - **three brains** — `MockBrain` replays a script; `PlannerBrain` is rule-based and
-      genuinely clicks the element matching the goal (it waits for a dialog instead of
-      flailing, and refuses to click the same label twice); `VLMBrain` is the seam —
-      prompt assembly and reply validation are done, `_generate()` is yours.
-    - **two executors** — `DryRunExecutor` is the default: it logs the plan and resolves
-      every label to a pixel. `PyAutoGuiExecutor` does real input and **refuses to click
-      when a label matches no element**, because a tiny model guessing pixels is how you
-      click "Delete account". Dry-run fails exactly where live input refuses, so a
-      rehearsal never gives false confidence.
-    - **policy gates** — deny-listed text (`sudo`, `rm -rf`, "password", "api_key"),
-      deny-listed widgets (terminal, keychain, password manager, wallet, bank), four
-      deny-listed key chords, a kill-switch file checked before every action, and a step
-      budget.
-    - **real input needs two deliberate flags** — `--real-input` *and*
-      `TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1`. Default is a dry run.
-    - **56 tests** needing no display, no model and no network.
-
-    What does **not** work yet: seeing a real screen *usefully*. `GrabScreen` captures
-    pixels through `mss`, but nothing yet turns pixels into an element index (OCR or
-    AT-SPI), so a live screen currently yields zero elements — which is exactly what the
-    label path depends on. And no model is wired up.
-
-    ## Install and run
+    It clicked Cancel before Save, then wandered. The harness held up regardless: every
+    action was validated against the closed vocabulary, a garbled JSON reply was recovered
+    into a usable action, and the budget stopped it. **A tiny VLM beats a tiny text model
+    here** — which is why the vision path is the recommended one.
+
+    **116 tests pass**, including real OCR over a real image asserting the `Save` button's
+    coordinates, and a screenshot replay that runs the whole pipeline with no display.
+
+    ## Install
 
     ```bash
-    pip install -e .                          # core: standard library only, no deps
-    pytest                                    # 56 offline tests
-    python -m tiny_computer_agent             # scripted demo, dry run
-    python -m tiny_computer_agent --live      # your real screen, still dry run
+    pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
+    pip install -e ".[desktop,ocr]"      # + real capture and real input
+    pip install -e ".[vision,ocr]"       # + the tiny model brains
+    pytest                               # 116 tests, no display or network needed
     ```
 
-    ## Put it on your own task
+    ## Run it on your machine
 
-    ```python
-    from tiny_computer_agent import AgentLoop, Element, FakeScreen, MockBrain
+    ```bash
+    python -m tiny_computer_agent --doctor                 # what can this machine do?
+    python -m tiny_computer_agent --image example          # replay a screenshot, dry run
+    python -m tiny_computer_agent --live                   # your real screen, dry run
+    python -m tiny_computer_agent --live --real-input "click 'OK'"   # needs the env var too
+    ```
 
-    screen = FakeScreen([(Element("Save", x=320, y=120, width=80, height=24),)])
-    brain = MockBrain([{"kind": "click", "label": "Save"},
-                       {"kind": "done", "reason": "saved"}])
+    `--doctor` is the first thing to run on a new machine: it checks the display session,
+    capture backend, pyautogui, the OCR engine, the model runtime, and the kill switch, and
+    prints the exact fix for anything missing. Real input needs **two** deliberate acts:
 
-    episode = AgentLoop(screen=screen, brain=brain).run("save the document")
-    print(episode.render())
-    episode.to_rows()          # rows for a dataframe or notebook table
+    ```bash
+    export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
+    python -m tiny_computer_agent --live --real-input "click 'Save'"
     ```
 
-    Swap `FakeScreen` for `GrabScreen()` and `MockBrain` for your `VLMBrain` subclass and
-    that is the whole distance between the tests and a live agent.
+    ## How it works
+
+    | Module | Job |
+    |---|---|
+    | `actions.py` | The closed vocabulary — `click` `type` `key` `scroll` `wait` `done`, validated before anything executes |
+    | `observe.py` | `Element`/`Observation`, `FakeScreen`, `ImageScreen`, `GrabScreen` (mss → Pillow fallback) |
+    | `perception.py` | Detectors: `OcrDetector` (RapidOCR), `GridDetector`, `AutoDetector`, `IndexedScreen` |
+    | `brain.py` | `DecisionMenu` (the numbered menu), `ModelBrain`, lenient `parse_decision`, `MockBrain`, `PlannerBrain` |
+    | `models.py` | `VLMBrain` (SmolVLM-256M) and `TextPlannerBrain` (Qwen2.5-0.5B), lazily loaded |
+    | `act.py` | `DryRunExecutor` (the default) and `PyAutoGuiExecutor` |
+    | `safety.py` | `Policy`: deny lists, kill switch, step budget, the blind-grid-click guard |
+    | `loop.py` | `AgentLoop`, `Episode`, `Step`, and the no-progress guard |
+    | `desktop.py` | Preflight checks, `doctor`, and `build_agent` in one call |
+    | `demo.py` | The synthetic desktop used by tests and `--image example` |
 
-    ## Why "tiny" is the whole point
+    ## Why a tiny model can do this at all
 
-    The harness does the heavy lifting, so the model does not have to be clever:
+    The harness does the heavy lifting, so the model never has to be clever:
 
     | Job | Who does it |
     |---|---|
-    | Finding *where* a button is | OCR / accessibility tree / element index — not the model |
-    | Choosing *what* to do | a tiny VLM over a **closed action menu** (click, type, key, scroll, wait, done) |
-    | Knowing whether it worked | the next frame, plus harness-side checks |
+    | Finding *where* a button is | OCR / accessibility tree — **not** the model |
+    | Choosing *what* to do | the model, but only by picking a number from a short menu |
+    | Knowing whether it worked | the next frame, plus the no-progress guard |
     | Recovering from a bad step | the retry policy in `loop.py`, not the weights |
 
-    That keeps the agent offline, fast to start, and small enough that a laptop GPU
-    (or plain CPU) can run it on the same box it is controlling.
+    The menu is the trick: a 256M model cannot emit `(295, 284)` reliably, but it can pick
+    "4" out of five options — and the harness owns the coordinate lookup.
 
-    ## Candidate backends
-
-    Pluggable behind one interface; nothing here needs a network call at run time.
-
-    - **Moondream 2** / **SmolVLM-256M / 500M** — a few hundred MB, safe default
-    - **Qwen2-VL-2B-Instruct**, **Florence-2-base** — stronger grounding, still tiny
-    - **OCR + text-only planner** — no VLM at all; the cheapest credible baseline
-    - **mock brain** — deterministic scripted actions, used to test the harness itself
+    ## Safety, because this thing has your mouse
 
-    ## Layout
+    - Run it in a **VM, container, or throwaway desktop account**, not your real session.
+    - **Dry run is the default**, so you can read a whole episode before anything clicks.
+    - Deny lists for typing (`sudo`, `rm -rf`, "password"), for widgets (terminal, keychain,
+      wallet, bank) and for key chords.
+    - A **kill-switch file** checked before every action, plus a hard step budget.
+    - **Blind grid clicks are refused with real input** unless you pass `--allow-grid`. When
+      OCR finds nothing on screen, the fallback index is a numbered grid — coordinates with
+      no meaning — and a model choosing one is guessing about your desktop.
+    - **Local-first, for real:** pixels and keystrokes never leave the machine.
 
-    ```
-    tiny-computer-agent/
-    ├── README.md
-    ├── LICENSE
-    ├── pyproject.toml
-    ├── tiny_computer_agent_app.py   # marimo console: run an episode, publish this card
-    ├── tiny_computer_agent/
-    │   ├── actions.py    # the closed action vocabulary
-    │   ├── observe.py    # Observation + element index; FakeScreen, GrabScreen
-    │   ├── brain.py      # MockBrain, PlannerBrain, VLMBrain (the seam)
-    │   ├── act.py        # DryRunExecutor (default) + PyAutoGuiExecutor
-    │   ├── safety.py     # Policy: deny lists, kill switch, step budget
-    │   ├── loop.py       # AgentLoop, Episode, Step
-    │   └── __main__.py   # CLI
-    └── tests/            # 56 tests: no display, no model, no network
-    ```
+    ## What works, and what does not
 
-    ## Safety, because this thing has your mouse
+    **Works:** real capture, a real OCR index with real coordinates, a 256M VLM choosing the
+    right labelled target, the closed vocabulary, the policy gates, the no-progress guard,
+    the doctor, the CLI, 116 tests.
 
-    - Run it in a **VM, container, or throwaway desktop account** — not your real session.
-    - Dry run is the default; read the whole plan before letting it near your pointer.
-    - Deny-list for shells, terminals, payment flows and password managers.
-    - A step budget plus a kill-switch file checked before every action.
-    - **Local-first, for real:** screen pixels and keystrokes never leave the machine.
+    **Does not work yet:** accuracy. A 0.5B text planner picks wrong actions; a 256M VLM can
+    answer "done" before doing anything, and the harness currently trusts that; the
+    accessibility-tree detector (AT-SPI / UI Automation) is not implemented; and nothing here
+    has been run against a live multi-window desktop, only against a synthetic screenshot.
 
     ## Roadmap
 
-    - [x] `loop.py` + `act.py` with the mock brain — the harness is testable before any model download
-    - [x] policy gates: deny lists, kill switch, step budget, dry run by default
-    - [ ] OCR / AT-SPI element index, so a live screen has clickable labels
-    - [ ] tiny-VLM grounding backend, behind `VLMBrain._generate()`
+    - [x] Capture, OCR index, closed vocabulary, dry-run default, policy gates, doctor
+    - [x] A 256M VLM that picks a labelled target from pixels
+    - [ ] Make the harness distrust a `done` that arrives before any action has been taken
+    - [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows) for exact widgets
+    - [ ] Score episodes: replay a trace and count how often the choice was right
     - [ ] marimo console: live screen preview, action log, pause / resume / step
-    - [ ] persist episodes, so failed runs can be replayed and scored
 
     ## License