Publish README.md from marimo
Browse files
README.md
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
|
|
|
|
| 2 |
---
|
| 3 |
title: Tiny Computer Agent
|
| 4 |
emoji: 🖱️
|
|
@@ -151,6 +152,21 @@ That test starts its own Tk window, asserts OCR finds the button at the coordina
|
|
| 151 |
was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in
|
| 152 |
because it moves your mouse — run it on a throwaway desktop.
|
| 153 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
## How it works
|
| 155 |
|
| 156 |
| Module | Job |
|
|
@@ -211,6 +227,7 @@ when no verifier is given.
|
|
| 211 |
- [x] A real click on a real window, with a verified outcome
|
| 212 |
- [x] Blind-grid-click guard, echo detection, no-progress guard
|
| 213 |
- [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
|
|
|
|
| 214 |
- [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
|
| 215 |
- [ ] Score episodes: replay a trace and count how often the choice was right
|
| 216 |
- [ ] marimo console: live screen preview, action log, pause / resume / step
|
|
|
|
| 1 |
|
| 2 |
+
|
| 3 |
---
|
| 4 |
title: Tiny Computer Agent
|
| 5 |
emoji: 🖱️
|
|
|
|
| 152 |
was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in
|
| 153 |
because it moves your mouse — run it on a throwaway desktop.
|
| 154 |
|
| 155 |
+
### Testing on Windows (or macOS)
|
| 156 |
+
|
| 157 |
+
This repo's live test was run on Linux/X11, because that is what the build container
|
| 158 |
+
had. For real Windows coverage, mirror the repo to GitHub: `.github/workflows/live-desktop.yml`
|
| 159 |
+
runs the unit suite **and** the live desktop test on `windows-latest` — a real Windows
|
| 160 |
+
desktop session, free for public repositories — plus the same test under Xvfb on Linux,
|
| 161 |
+
and a preflight on macOS. Trigger it from the Actions tab.
|
| 162 |
+
|
| 163 |
+
Nothing in the harness is Linux-specific: pyautogui uses `SendInput` on Windows, and
|
| 164 |
+
`GrabScreen` already falls back to Pillow's `ImageGrab` there. Two things can only fail
|
| 165 |
+
on a real Windows desktop, which is exactly why the job exists: **DPI scaling** (that is
|
| 166 |
+
what `--scale` is for) and the native input backend. At 125% or 150% display scaling,
|
| 167 |
+
expect the coordinate assertion in `test_live_desktop.py` to be the first thing that
|
| 168 |
+
fails — that is the signal, not a nuisance.
|
| 169 |
+
|
| 170 |
## How it works
|
| 171 |
|
| 172 |
| Module | Job |
|
|
|
|
| 227 |
- [x] A real click on a real window, with a verified outcome
|
| 228 |
- [x] Blind-grid-click guard, echo detection, no-progress guard
|
| 229 |
- [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
|
| 230 |
+
- [ ] Windows run of the live test (workflow included; needs a GitHub mirror)
|
| 231 |
- [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
|
| 232 |
- [ ] Score episodes: replay a trace and count how often the choice was right
|
| 233 |
- [ ] marimo console: live screen preview, action log, pause / resume / step
|