Publish tiny_computer_agent_app.py from marimo

tiny_computer_agent_app.py

@@ -18,7 +18,6 @@ with app.setup:
     from huggingface_hub import HfApi
 
 
-
 @app.cell(hide_code=True)
 def readme():
     readme_md = '''
@@ -97,6 +96,70 @@ def readme():
     verifier confirmed it before the episode was called a success. **Pixels in, a real click
     out, and evidence that it worked.**
 
+    ### Re-confirmed end to end on a headless Linux box
+
+    The whole path above was re-run from scratch on a headless Debian container (gVisor
+    sandbox, 20 vCPU, no GPU, no physical display, no network at run time) behind a virtual
+    X server. First the suite, headless:
+
+    ```bash
+    python -m pip install -e ".[desktop,ocr,dev]"
+    pytest tests -q
+    # 123 passed, 2 skipped in 18.6s
+    ```
+
+    The two skips are the live-desktop tests. Give them a display and they run for real:
+
+    ```bash
+    Xvfb :99 -screen 0 1280x800x24 &
+    export DISPLAY=:99
+    TINY_COMPUTER_AGENT_LIVE_TEST=1 python -m pytest tests/test_live_desktop.py -q
+    # tests/test_live_desktop.py::test_ocr_finds_a_real_button_at_the_right_pixels PASSED
+    # tests/test_live_desktop.py::test_a_real_click_lands_on_the_button PASSED
+    # 2 passed
+    ```
+
+    Then the same desktop, driven through the CLI with the mouse actually live:
+
+    ```bash
+    export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
+    python -m tiny_computer_agent --live --real-input --verify-text Saved \
+        "click 'Save' to save the document"
+    ```
+
+    ```
+    goal: click 'Save' to save the document
+    mode: REAL INPUT (mouse and keyboard live)  |  screen: real desktop (monitor 1)  |  brain: planner  |  kill switch: create /tmp/tiny-computer-agent.STOP to stop
+    index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
+
+      step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: clicked (194, 296)
+
+    result: success - verified after 1 action(s)
+    ```
+
+    The window was drawn at `+120+120` with its Save button at `x=20, y=155, 110x40`, so the
+    button centre is `(195, 295)`. The click at `(194, 296)` is *on the button*, not near it —
+    and the app wrote `Save` back to its own marker file. Pixels in, a real click out, and the
+    app's own side effect as the proof.
+
+    `--doctor` on that box, with the virtual display attached:
+
+    ```
+    tiny-computer-agent preflight (Linux x86_64, python 3.13.11)
+
+      [ok  ] display server: DISPLAY=:99
+      [ok  ] pillow: installed
+      [ok  ] screen capture: mss (fast, multi-monitor)
+      [ok  ] input backend (pyautogui): installed
+      [ok  ] element detector (auto): OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
+      [ok  ] model runtime: not needed: the planner is rule-based (no weights, no download)
+      [ok  ] execution mode: dry run (nothing is clicked)
+      [ok  ] kill switch: create /tmp/tiny-computer-agent.STOP to abort the next action
+
+    READY: the harness can drive this desktop.
+    ```
+
+
     ### On a replayed screenshot
 
     ```
@@ -112,6 +175,23 @@ def readme():
       step 0 | click label='Save' reason="the reply names 'Save'" -> pixel (295, 284)
     ```
 
+    ### Which brain drove which run
+
+    The harness and the brain are separable, and it matters which one a result came from.
+    None of the runs below used a cloud API; the two model brains run on CPU:
+
+    | Run | Brain | Weights | Outcome |
+    |---|---|---|---|
+    | Live Linux desktop, real click | `PlannerBrain` (rule-based) | none | **success, verified** |
+    | `/tmp/saveapp.py` window live | `PlannerBrain` | none | clicked `Save` at `(194, 296)` |
+    | `--image example` | `PlannerBrain` | none | picked `Save`, pixel `(295, 284)` |
+    | `--image example` | `VLMBrain` (SmolVLM-256M-Instruct) | 256M, CPU | picked `Save` |
+    | Live Linux desktop | `VLMBrain` (SmolVLM-256M-Instruct) | 256M, CPU | echoed the prompt; **refused** by echo detection |
+    | Live Linux desktop | `TextPlannerBrain` (Qwen2.5-0.5B) | 0.5B, CPU | clicked Cancel, then garbled JSON; budget exhausted |
+
+    The rule-based planner is what carries the verified end-to-end runs. The tiny model is the
+    open problem: it is reliable on a clean screenshot and unreliable on a busy live screen.
+
     ### What did *not* work, and what caught it
 
     - **The 256M VLM echoes the prompt on a busy live screen.** Asked to click Save on the
@@ -126,16 +206,42 @@ def readme():
       `warning: no text was found on screen, so the index is 96 numbered grid cells and the
       agent is choosing blind` — instead of silently guessing.
 
-    **123 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+    **125 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
     that click a real button.
 
+    ## The test suite, in full
+
+    `pytest tests` collects **125 tests**. 123 run headless — no display, no network, no
+    weights. The last 2 are the opt-in live-desktop tests, skipped unless
+    `TINY_COMPUTER_AGENT_LIVE_TEST=1`.
+
+    | File | Tests | What it proves |
+    |---|---:|---|
+    | `tests/test_actions.py` | 13 | the closed vocabulary — every action is validated before anything executes |
+    | `tests/test_act.py` | 8 | `DryRunExecutor` is inert; the real executor refuses until explicitly armed |
+    | `tests/test_observe.py` | 8 | element/observation geometry, `FakeScreen`, `ImageScreen`, the capture fallback |
+    | `tests/test_perception.py` | 13 | OCR labels a real image with real coordinates; the grid fallback; `AutoDetector` |
+    | `tests/test_brain.py` | 18 | the numbered `DecisionMenu`, lenient parsing, echo detection, `MockBrain`, `PlannerBrain` |
+    | `tests/test_loop.py` | 14 | `AgentLoop` / `Episode` / `Step`, the no-progress guard, the step budget |
+    | `tests/test_verify.py` | 7 | success is only claimed when a verifier can actually see it |
+    | `tests/test_safety.py` | 11 | deny lists, the kill switch, the blind-grid-click guard |
+    | `tests/test_grid_fallback.py` | 11 | the fallback index announces itself instead of guessing |
+    | `tests/test_desktop.py` | 10 | display detection, the preflight checks, `doctor` |
+    | `tests/test_cli.py` | 10 | the command line: `--doctor`, `--image`, `--live`, argument handling |
+    | `tests/test_live_desktop.py` | 2 | **needs a display, opt-in**: OCR finds a real button at the pixels it was drawn at, and a real click makes the app react |
+
+    ```bash
+    pytest tests -q                                  # 123 passed, 2 skipped
+    TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 125 passed, on a desktop
+    ```
+
     ## Install
 
     ```bash
     pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
     pip install -e ".[desktop,ocr]"      # + real capture and real input
     pip install -e ".[vision,ocr]"       # + the tiny model brains
-    pytest                               # 123 tests, no display or network needed
+    pytest                               # 125 tests, no display or network needed
     ```
 
     ## Run it on your machine
@@ -175,20 +281,53 @@ def readme():
     was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in
     because it moves your mouse — run it on a throwaway desktop.
 
-    ### Testing on Windows (or macOS)
+    ### Testing on Windows - three ways, and which to use
+
+    Nothing in the harness is Linux-specific: `pyautogui` uses `SendInput` on Windows and
+    `GrabScreen` already falls back to Pillow's `ImageGrab` there. Two things can only fail on
+    a real Windows desktop, which is why these runs exist: **DPI scaling** (that is what
+    `--scale` is for) and the native input backend. At 125% or 150% display scaling, expect the
+    coordinate assertion in `test_live_desktop.py` to be the first thing that fails - that is
+    the signal, not a nuisance.
+
+    | Method | Real Windows desktop? | Cost | Status |
+    |---|---|---|---|
+    | **GitHub Actions `windows-latest`** | yes, an interactive session | free for public repos, ~5 min | workflow committed: `.github/workflows/live-desktop.yml` |
+    | **QEMU/TCG VM on a Linux host** | yes, but fully emulated | hours of wall-clock | possible; measured below |
+    | **A cloud Windows dev box over RDP** | yes, native | paid by the hour | untested here |
 
-    This repo's live test was run on Linux/X11, because that is what the build container
-    had. For real Windows coverage, mirror the repo to GitHub: `.github/workflows/live-desktop.yml`
-    runs the unit suite **and** the live desktop test on `windows-latest` — a real Windows
-    desktop session, free for public repositories — plus the same test under Xvfb on Linux,
-    and a preflight on macOS. Trigger it from the Actions tab.
+    **1. GitHub Actions - the recommended path.** `.github/workflows/live-desktop.yml` runs
+    the unit suite and the live desktop test on `windows-latest`, the same live test under
+    Xvfb on Linux, and a `--doctor` preflight on macOS. The Windows job is
+    `continue-on-error` on purpose: the first run tells us whether a hosted runner's session
+    accepts injected input. Trigger it from the Actions tab. The repo needs a GitHub mirror -
+    Hugging Face does not run GitHub Actions.
 
-    Nothing in the harness is Linux-specific: pyautogui uses `SendInput` on Windows, and
-    `GrabScreen` already falls back to Pillow's `ImageGrab` there. Two things can only fail
-    on a real Windows desktop, which is exactly why the job exists: **DPI scaling** (that is
-    what `--scale` is for) and the native input backend. At 125% or 150% display scaling,
-    expect the coordinate assertion in `test_live_desktop.py` to be the first thing that
-    fails — that is the signal, not a nuisance.
+    **2. A local QEMU VM without KVM.** Measured on the headless container above:
+
+    ```
+    /dev/kvm absent                     -> no hardware virtualization
+    qemu-system-x86_64 -accel help      -> tcg, kvm
+    qemu-system-x86_64 -accel kvm       -> Could not access KVM kernel module: No such file or directory
+    qemu-system-x86_64 -accel tcg       -> boots (reached "No bootable device.")
+    ```
+
+    So a Windows guest is *possible* with no KVM at all, but entirely under software
+    emulation. TCG runs roughly 10-50x slower than native, which turns a 20-minute Windows
+    install into many hours - and the harness's OCR would then run on an emulated CPU inside
+    the guest. Use this only when no CI runner and no cloud box is available:
+
+    ```bash
+    qemu-img create -f qcow2 win.qcow2 64G
+    qemu-system-x86_64 -accel tcg -m 8G -smp 4 \
+      -drive file=win.qcow2,format=qcow2,if=virtio \
+      -cdrom Win11_Eval_x64.iso -boot d \
+      -device virtio-vga -vnc :1     # then RDP/VNC in, pip install the harness inside
+    ```
+
+    **3. A cloud Windows box.** Any Windows VM reachable over RDP gives a native desktop, a
+    real `SendInput` path and real DPI scaling. Untested here; it is the closest thing to
+    "a user's actual laptop".
 
     ## How it works
 
@@ -236,7 +375,7 @@ def readme():
 
     **Works:** real capture, a real OCR index with coordinates that land on the right pixel,
     a real click that an app reacts to, verified success, the closed vocabulary, the policy
-    gates, the no-progress guard, echo detection, the doctor, the CLI, 123 tests — and a
+    gates, the no-progress guard, echo detection, the doctor, the CLI, 125 tests — and a
     256M VLM that reads a button label from pixels on a clean screenshot.
 
     **Does not work yet:** reliability on a *live* screen. The 256M VLM echoed the prompt
@@ -250,7 +389,7 @@ def readme():
     - [x] A real click on a real window, with a verified outcome
     - [x] Blind-grid-click guard, echo detection, no-progress guard
     - [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
-    - [ ] Windows run of the live test (workflow included; needs a GitHub mirror)
+    - [ ] Windows run of the live test - workflow committed; needs a GitHub mirror with Actions on
     - [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
     - [ ] Score episodes: replay a trace and count how often the choice was right
     - [ ] marimo console: live screen preview, action log, pause / resume / step
@@ -261,7 +400,6 @@ def readme():
     '''
 
     mo.md(readme_md)
-
     return (readme_md,)
 
 
@@ -346,7 +484,6 @@ def repo_files():
 
     mo.md(f"packaging: {len(pyproject_toml) + len(license_txt) + len(gitignore_txt)} chars, "
           f"v0.2.0 with ocr/desktop/vision extras")
-
     return gitignore_txt, license_txt, pyproject_toml
 
 
@@ -361,7 +498,6 @@ def publish():
     )
 
     publish_button
-
     return REPO_ID, publish_button
 
 
@@ -430,7 +566,6 @@ def publish_status(
                     _status = mo.md(f"Publish failed: `{type(_e).__name__}: {_e}`").callout(kind="danger")
 
     _status
-
     return
 
 
@@ -471,7 +606,6 @@ def demo():
             mo.md(f"`{perception_episode.render().splitlines()[1]}`"),
         ]
     )
-
     return