Publish README.md from marimo

README.md

@@ -74,6 +74,69 @@ The click landed on the button, the app changed its own window to "Saved", and t
 verifier confirmed it before the episode was called a success. **Pixels in, a real click
 out, and evidence that it worked.**
 
+### Re-confirmed end to end on a headless Linux box
+
+The whole path above was re-run from scratch on a headless Debian container (gVisor
+sandbox, 20 vCPU, no GPU, no physical display, no network at run time) behind a virtual
+X server. First the suite, headless:
+
+```bash
+python -m pip install -e ".[desktop,ocr,dev]"
+pytest tests -q
+# 123 passed, 2 skipped in 18.6s
+```
+
+The two skips are the live-desktop tests. Give them a display and they run for real:
+
+```bash
+Xvfb :99 -screen 0 1280x800x24 &
+export DISPLAY=:99
+TINY_COMPUTER_AGENT_LIVE_TEST=1 python -m pytest tests/test_live_desktop.py -q
+# tests/test_live_desktop.py::test_ocr_finds_a_real_button_at_the_right_pixels PASSED
+# tests/test_live_desktop.py::test_a_real_click_lands_on_the_button PASSED
+# 2 passed
+```
+
+Then the same desktop, driven through the CLI with the mouse actually live:
+
+```bash
+export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
+python -m tiny_computer_agent --live --real-input --verify-text Saved     "click 'Save' to save the document"
+```
+
+```
+goal: click 'Save' to save the document
+mode: REAL INPUT (mouse and keyboard live)  |  screen: real desktop (monitor 1)  |  brain: planner  |  kill switch: create /tmp/tiny-computer-agent.STOP to stop
+index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
+
+  step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: clicked (194, 296)
+
+result: success - verified after 1 action(s)
+```
+
+The window was drawn at `+120+120` with its Save button at `x=20, y=155, 110x40`, so the
+button centre is `(195, 295)`. The click at `(194, 296)` is *on the button*, not near it —
+and the app wrote `Save` back to its own marker file. Pixels in, a real click out, and the
+app's own side effect as the proof.
+
+`--doctor` on that box, with the virtual display attached:
+
+```
+tiny-computer-agent preflight (Linux x86_64, python 3.13.11)
+
+  [ok  ] display server: DISPLAY=:99
+  [ok  ] pillow: installed
+  [ok  ] screen capture: mss (fast, multi-monitor)
+  [ok  ] input backend (pyautogui): installed
+  [ok  ] element detector (auto): OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
+  [ok  ] model runtime: not needed: the planner is rule-based (no weights, no download)
+  [ok  ] execution mode: dry run (nothing is clicked)
+  [ok  ] kill switch: create /tmp/tiny-computer-agent.STOP to abort the next action
+
+READY: the harness can drive this desktop.
+```
+
+
 ### On a replayed screenshot
 
 ```
@@ -89,6 +152,23 @@ named the right target:
   step 0 | click label='Save' reason="the reply names 'Save'" -> pixel (295, 284)
 ```
 
+### Which brain drove which run
+
+The harness and the brain are separable, and it matters which one a result came from.
+None of the runs below used a cloud API; the two model brains run on CPU:
+
+| Run | Brain | Weights | Outcome |
+|---|---|---|---|
+| Live Linux desktop, real click | `PlannerBrain` (rule-based) | none | **success, verified** |
+| `/tmp/saveapp.py` window live | `PlannerBrain` | none | clicked `Save` at `(194, 296)` |
+| `--image example` | `PlannerBrain` | none | picked `Save`, pixel `(295, 284)` |
+| `--image example` | `VLMBrain` (SmolVLM-256M-Instruct) | 256M, CPU | picked `Save` |
+| Live Linux desktop | `VLMBrain` (SmolVLM-256M-Instruct) | 256M, CPU | echoed the prompt; **refused** by echo detection |
+| Live Linux desktop | `TextPlannerBrain` (Qwen2.5-0.5B) | 0.5B, CPU | clicked Cancel, then garbled JSON; budget exhausted |
+
+The rule-based planner is what carries the verified end-to-end runs. The tiny model is the
+open problem: it is reliable on a clean screenshot and unreliable on a busy live screen.
+
 ### What did *not* work, and what caught it
 
 - **The 256M VLM echoes the prompt on a busy live screen.** Asked to click Save on the
@@ -103,16 +183,42 @@ named the right target:
   `warning: no text was found on screen, so the index is 96 numbered grid cells and the
   agent is choosing blind` — instead of silently guessing.
 
-**123 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+**125 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
 that click a real button.
 
+## The test suite, in full
+
+`pytest tests` collects **125 tests**. 123 run headless — no display, no network, no
+weights. The last 2 are the opt-in live-desktop tests, skipped unless
+`TINY_COMPUTER_AGENT_LIVE_TEST=1`.
+
+| File | Tests | What it proves |
+|---|---:|---|
+| `tests/test_actions.py` | 13 | the closed vocabulary — every action is validated before anything executes |
+| `tests/test_act.py` | 8 | `DryRunExecutor` is inert; the real executor refuses until explicitly armed |
+| `tests/test_observe.py` | 8 | element/observation geometry, `FakeScreen`, `ImageScreen`, the capture fallback |
+| `tests/test_perception.py` | 13 | OCR labels a real image with real coordinates; the grid fallback; `AutoDetector` |
+| `tests/test_brain.py` | 18 | the numbered `DecisionMenu`, lenient parsing, echo detection, `MockBrain`, `PlannerBrain` |
+| `tests/test_loop.py` | 14 | `AgentLoop` / `Episode` / `Step`, the no-progress guard, the step budget |
+| `tests/test_verify.py` | 7 | success is only claimed when a verifier can actually see it |
+| `tests/test_safety.py` | 11 | deny lists, the kill switch, the blind-grid-click guard |
+| `tests/test_grid_fallback.py` | 11 | the fallback index announces itself instead of guessing |
+| `tests/test_desktop.py` | 10 | display detection, the preflight checks, `doctor` |
+| `tests/test_cli.py` | 10 | the command line: `--doctor`, `--image`, `--live`, argument handling |
+| `tests/test_live_desktop.py` | 2 | **needs a display, opt-in**: OCR finds a real button at the pixels it was drawn at, and a real click makes the app react |
+
+```bash
+pytest tests -q                                  # 123 passed, 2 skipped
+TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 125 passed, on a desktop
+```
+
 ## Install
 
 ```bash
 pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
 pip install -e ".[desktop,ocr]"      # + real capture and real input
 pip install -e ".[vision,ocr]"       # + the tiny model brains
-pytest                               # 123 tests, no display or network needed
+pytest                               # 125 tests, no display or network needed
 ```
 
 ## Run it on your machine
@@ -152,20 +258,50 @@ That test starts its own Tk window, asserts OCR finds the button at the coordina
 was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in
 because it moves your mouse — run it on a throwaway desktop.
 
-### Testing on Windows (or macOS)
+### Testing on Windows - three ways, and which to use
+
+Nothing in the harness is Linux-specific: `pyautogui` uses `SendInput` on Windows and
+`GrabScreen` already falls back to Pillow's `ImageGrab` there. Two things can only fail on
+a real Windows desktop, which is why these runs exist: **DPI scaling** (that is what
+`--scale` is for) and the native input backend. At 125% or 150% display scaling, expect the
+coordinate assertion in `test_live_desktop.py` to be the first thing that fails - that is
+the signal, not a nuisance.
+
+| Method | Real Windows desktop? | Cost | Status |
+|---|---|---|---|
+| **GitHub Actions `windows-latest`** | yes, an interactive session | free for public repos, ~5 min | workflow committed: `.github/workflows/live-desktop.yml` |
+| **QEMU/TCG VM on a Linux host** | yes, but fully emulated | hours of wall-clock | possible; measured below |
+| **A cloud Windows dev box over RDP** | yes, native | paid by the hour | untested here |
 
-This repo's live test was run on Linux/X11, because that is what the build container
-had. For real Windows coverage, mirror the repo to GitHub: `.github/workflows/live-desktop.yml`
-runs the unit suite **and** the live desktop test on `windows-latest` — a real Windows
-desktop session, free for public repositories — plus the same test under Xvfb on Linux,
-and a preflight on macOS. Trigger it from the Actions tab.
+**1. GitHub Actions - the recommended path.** `.github/workflows/live-desktop.yml` runs
+the unit suite and the live desktop test on `windows-latest`, the same live test under
+Xvfb on Linux, and a `--doctor` preflight on macOS. The Windows job is
+`continue-on-error` on purpose: the first run tells us whether a hosted runner's session
+accepts injected input. Trigger it from the Actions tab. The repo needs a GitHub mirror -
+Hugging Face does not run GitHub Actions.
+
+**2. A local QEMU VM without KVM.** Measured on the headless container above:
+
+```
+/dev/kvm absent                     -> no hardware virtualization
+qemu-system-x86_64 -accel help      -> tcg, kvm
+qemu-system-x86_64 -accel kvm       -> Could not access KVM kernel module: No such file or directory
+qemu-system-x86_64 -accel tcg       -> boots (reached "No bootable device.")
+```
+
+So a Windows guest is *possible* with no KVM at all, but entirely under software
+emulation. TCG runs roughly 10-50x slower than native, which turns a 20-minute Windows
+install into many hours - and the harness's OCR would then run on an emulated CPU inside
+the guest. Use this only when no CI runner and no cloud box is available:
+
+```bash
+qemu-img create -f qcow2 win.qcow2 64G
+qemu-system-x86_64 -accel tcg -m 8G -smp 4   -drive file=win.qcow2,format=qcow2,if=virtio   -cdrom Win11_Eval_x64.iso -boot d   -device virtio-vga -vnc :1     # then RDP/VNC in, pip install the harness inside
+```
 
-Nothing in the harness is Linux-specific: pyautogui uses `SendInput` on Windows, and
-`GrabScreen` already falls back to Pillow's `ImageGrab` there. Two things can only fail
-on a real Windows desktop, which is exactly why the job exists: **DPI scaling** (that is
-what `--scale` is for) and the native input backend. At 125% or 150% display scaling,
-expect the coordinate assertion in `test_live_desktop.py` to be the first thing that
-fails — that is the signal, not a nuisance.
+**3. A cloud Windows box.** Any Windows VM reachable over RDP gives a native desktop, a
+real `SendInput` path and real DPI scaling. Untested here; it is the closest thing to
+"a user's actual laptop".
 
 ## How it works
 
@@ -213,7 +349,7 @@ success.
 
 **Works:** real capture, a real OCR index with coordinates that land on the right pixel,
 a real click that an app reacts to, verified success, the closed vocabulary, the policy
-gates, the no-progress guard, echo detection, the doctor, the CLI, 123 tests — and a
+gates, the no-progress guard, echo detection, the doctor, the CLI, 125 tests — and a
 256M VLM that reads a button label from pixels on a clean screenshot.
 
 **Does not work yet:** reliability on a *live* screen. The 256M VLM echoed the prompt
@@ -227,7 +363,7 @@ when no verifier is given.
 - [x] A real click on a real window, with a verified outcome
 - [x] Blind-grid-click guard, echo detection, no-progress guard
 - [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
-- [ ] Windows run of the live test (workflow included; needs a GitHub mirror)
+- [ ] Windows run of the live test - workflow committed; needs a GitHub mirror with Actions on
 - [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
 - [ ] Score episodes: replay a trace and count how often the choice was right
 - [ ] marimo console: live screen preview, action log, pause / resume / step