Update project card from marimo

README.md

@@ -18,12 +18,12 @@ tags:
 # 🖱️ Tiny Computer Agent
 
 **An agentic harness that drives a real computer with a tiny, fully local model.**
-No cloud API, no API keys, no telemetry — the model is small enough to run on the
-same machine it is controlling.
+No cloud API, no API keys, no telemetry — and a core that imports nothing outside
+the Python standard library.
 
-> **Status: project scaffold.** This card pins down the harness contract
-> (observe → think → act → verify), the intended layout, and the safety model.
-> Implementation lands next; see the [roadmap](#-roadmap).
+> **Status: early alpha.** The loop runs end to end today, with a scripted screen,
+> dry-run execution, a CLI and 56 offline tests. The vision backend is a
+> documented seam, not a feature — see [what works](#-what-works-today).
 
 ## The idea, in one loop
 
@@ -46,6 +46,77 @@ same machine it is controlling.
 4. **Verify** — the next frame shows whether it worked. Repeat, bounded by a step
    budget and a kill switch.
 
+## What works today
+
+```
+$ python -m tiny_computer_agent
+goal: click 'Save' to save the document
+mode: dry-run  |  screen: scripted demo  |  kill switch: create /tmp/tiny-computer-agent.STOP to stop
+
+  step 0 | wait seconds=0.5 reason="'Save' is not on screen yet" | ok: (dry-run) would wait 0.5s
+  step 1 | click label='Save' reason="matched 'Save' in the element index" | ok: (dry-run) would click -> pixel (360, 132)
+  step 2 | done reason="already clicked 'Save'; goal assumed met" | ok: goal reported met
+
+result: success - brain reported done: already clicked 'Save'; goal assumed met
+```
+
+Shipped and tested:
+
+- **the loop** — `AgentLoop` runs observe → think → act → verify, **re-reads the
+  screen after every action**, and stops on done, on a step budget, on the kill
+  switch, on a policy refusal, on a failed action, or on a brain error. Every step
+  is recorded in an `Episode` you can render or turn into rows.
+- **a closed action vocabulary** — `click` `type` `key` `scroll` `wait` `done`,
+  validated *before* anything executes. A model that invents a verb gets a recorded
+  error, not a click.
+- **three brains** — `MockBrain` replays a script; `PlannerBrain` is rule-based and
+  genuinely clicks the element matching the goal (it waits for a dialog instead of
+  flailing, and refuses to click the same label twice); `VLMBrain` is the seam —
+  prompt assembly and reply validation are done, `_generate()` is yours.
+- **two executors** — `DryRunExecutor` is the default: it logs the plan and resolves
+  every label to a pixel. `PyAutoGuiExecutor` does real input and **refuses to click
+  when a label matches no element**, because a tiny model guessing pixels is how you
+  click "Delete account". Dry-run fails exactly where live input refuses, so a
+  rehearsal never gives false confidence.
+- **policy gates** — deny-listed text (`sudo`, `rm -rf`, "password", "api_key"),
+  deny-listed widgets (terminal, keychain, password manager, wallet, bank), four
+  deny-listed key chords, a kill-switch file checked before every action, and a step
+  budget.
+- **real input needs two deliberate flags** — `--real-input` *and*
+  `TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1`. Default is a dry run.
+- **56 tests** needing no display, no model and no network.
+
+What does **not** work yet: seeing a real screen *usefully*. `GrabScreen` captures
+pixels through `mss`, but nothing yet turns pixels into an element index (OCR or
+AT-SPI), so a live screen currently yields zero elements — which is exactly what the
+label path depends on. And no model is wired up.
+
+## Install and run
+
+```bash
+pip install -e .                          # core: standard library only, no deps
+pytest                                    # 56 offline tests
+python -m tiny_computer_agent             # scripted demo, dry run
+python -m tiny_computer_agent --live      # your real screen, still dry run
+```
+
+## Put it on your own task
+
+```python
+from tiny_computer_agent import AgentLoop, Element, FakeScreen, MockBrain
+
+screen = FakeScreen([(Element("Save", x=320, y=120, width=80, height=24),)])
+brain = MockBrain([{"kind": "click", "label": "Save"},
+                   {"kind": "done", "reason": "saved"}])
+
+episode = AgentLoop(screen=screen, brain=brain).run("save the document")
+print(episode.render())
+episode.to_rows()          # rows for a dataframe or notebook table
+```
+
+Swap `FakeScreen` for `GrabScreen()` and `MockBrain` for your `VLMBrain` subclass and
+that is the whole distance between the tests and a live agent.
+
 ## Why "tiny" is the whole point
 
 The harness does the heavy lifting, so the model does not have to be clever:
@@ -69,36 +140,41 @@ Pluggable behind one interface; nothing here needs a network call at run time.
 - **OCR + text-only planner** — no VLM at all; the cheapest credible baseline
 - **mock brain** — deterministic scripted actions, used to test the harness itself
 
-## Intended layout
+## Layout
 
 ```
 tiny-computer-agent/
 ├── README.md
-├── tiny_computer_agent.py          # marimo notebook: drive / watch the loop
-├── src/tiny_computer_agent/
-│   ├── observe.py   # screen capture + OCR / accessibility tree
-│   ├── brain.py     # model backends behind one interface
-│   ├── act.py       # action vocabulary + pyautogui / xdotool executor
-│   ├── loop.py      # observe → think → act → verify
-│   └── safety.py    # dry-run, allow/deny lists, kill switch, step budget
-└── LICENSE
+├── LICENSE
+├── pyproject.toml
+├── tiny_computer_agent_app.py   # marimo console: run an episode, publish this card
+├── tiny_computer_agent/
+│   ├── actions.py    # the closed action vocabulary
+│   ├── observe.py    # Observation + element index; FakeScreen, GrabScreen
+│   ├── brain.py      # MockBrain, PlannerBrain, VLMBrain (the seam)
+│   ├── act.py        # DryRunExecutor (default) + PyAutoGuiExecutor
+│   ├── safety.py     # Policy: deny lists, kill switch, step budget
+│   ├── loop.py       # AgentLoop, Episode, Step
+│   └── __main__.py   # CLI
+└── tests/            # 56 tests: no display, no model, no network
 ```
 
 ## Safety, because this thing has your mouse
 
 - Run it in a **VM, container, or throwaway desktop account** — not your real session.
-- `--dry-run` logs every action without executing it, so you can read the plan first.
+- Dry run is the default; read the whole plan before letting it near your pointer.
 - Deny-list for shells, terminals, payment flows and password managers.
-- A step budget plus a global hotkey kill switch before anything touches the pointer.
+- A step budget plus a kill-switch file checked before every action.
 - **Local-first, for real:** screen pixels and keystrokes never leave the machine.
 
 ## Roadmap
 
-- [ ] `loop.py` + `act.py` with the mock brain — the harness is testable before any model download
-- [ ] OCR / AT-SPI element index, and click-by-element instead of click-by-pixel
-- [ ] tiny-VLM grounding backend (transformers + llama.cpp GGUF)
-- [ ] marimo UI: live screen preview, action log, pause / resume / step
-- [ ] append-only run traces, so failed episodes can be replayed and scored
+- [x] `loop.py` + `act.py` with the mock brain — the harness is testable before any model download
+- [x] policy gates: deny lists, kill switch, step budget, dry run by default
+- [ ] OCR / AT-SPI element index, so a live screen has clickable labels
+- [ ] tiny-VLM grounding backend, behind `VLMBrain._generate()`
+- [ ] marimo console: live screen preview, action log, pause / resume / step
+- [ ] persist episodes, so failed runs can be replayed and scored
 
 ## License