Publish README.md from marimo

README.md

@@ -24,7 +24,9 @@ Python standard library.
 
 > **Status 0.3.0: it drives a real desktop — on Linux and on Windows.** Screen capture,
 > an OCR element index with real coordinates, a real mouse click, and a *verified*
-> outcome. The suite passes on a live Linux desktop and inside a real Windows Server
+> outcome. A run can be saved as a portable *trace*, replayed offline and **scored**, so
+> "which brain is right, and how often?" is a number in a table.
+> The suite passes on a live Linux desktop and inside a real Windows Server
 > 2022 guest; what is not solved is model reliability — see
 > [measured results](#measured-results).
 
@@ -84,7 +86,7 @@ X server. First the suite, headless:
 ```bash
 python -m pip install -e ".[desktop,ocr,dev]"
 pytest tests -q
-# 123 passed, 2 skipped in 18.6s
+# 151 passed, 2 skipped in 18.7s
 ```
 
 The two skips are the live-desktop tests. Give them a display and they run for real:
@@ -102,7 +104,8 @@ Then the same desktop, driven through the CLI with the mouse actually live:
 
 ```bash
 export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
-python -m tiny_computer_agent --live --real-input --verify-text Saved     "click 'Save' to save the document"
+python -m tiny_computer_agent --live --real-input --verify-text Saved \
+    "click 'Save' to save the document"
 ```
 
 ```
@@ -153,6 +156,37 @@ named the right target:
   step 0 | click label='Save' reason="the reply names 'Save'" -> pixel (295, 284)
 ```
 
+### Scoring a brain, offline
+
+"Which brain is right, and how often?" needs no desktop, no model and no network,
+because a **trace** keeps every frame's element index and throws the pixels away. Record a
+run once; replay it against any brain, as often as you like.
+
+```
+$ python -m tiny_computer_agent --image example --trace runs/save.json --expected Save
+...
+trace: wrote runs/save.json (2 frames)
+score: correct   click 'Save' to save the document: expected 'Save', chose 'Save'
+```
+
+A trace is JSON, so it diffs in review, commits next to the tests, and replays on a machine
+with nothing installed. Scoring a whole suite is a table:
+
+```
+scoreboard: 7/9 correct (78%)
+  planner (rule-based): 100%
+  first-item (baseline): 33%
+  oracle (always right): 100%
+  correct   save the document: expected 'Save', chose 'Save'
+  incorrect save the document: expected 'Save', chose 'Files' (chose 'Files', not 'Save')
+  ...
+```
+
+The number is the brain's **decision** - what it chose when it had to commit. Whether the
+harness *allowed* that decision to run is a separate axis: a click the policy refused is
+still the choice it was, and it is flagged `(harness refused it)` rather than passed off as
+a clean success. Keeping those two apart is the point.
+
 ### Which brain drove which run
 
 The harness and the brain are separable, and it matters which one a result came from.
@@ -184,7 +218,7 @@ open problem: it is reliable on a clean screenshot and unreliable on a busy live
   `warning: no text was found on screen, so the index is 96 numbered grid cells and the
   agent is choosing blind` — instead of silently guessing.
 
-**125 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+**153 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
 that click a real button.
 
 ## Verified on real Windows
@@ -210,7 +244,7 @@ desktop was up a few minutes after that.
 **What was installed inside it.**
 
 ```
-Python 3.12.8 (embeddable distribution, unpacked to C:	iny\py)
+Python 3.12.8 (embeddable distribution, unpacked to C:\tiny\py)
 pytest, pillow 13.0.0, mss 10.2.0
 numpy 2.5.4, onnxruntime 1.31.1, opencv-python 5.0.0.93, rapidocr-onnxruntime 1.4.4
 the harness itself, installed into site-packages
@@ -226,7 +260,7 @@ the harness itself, installed into site-packages
 
 None of the three failures in the first row is a harness defect: two are
 `ModuleNotFoundError: numpy` and the third is the guard bug documented below. With the
-OCR engine installed, the same suite exits **0**.
+OCR engine installed, the same suite exits **0**. (Those Windows counts predate `trace.py`; the harness suite has since grown to 153 tests.)
 
 **The emulated-OCR caveat.** onnxruntime requires AVX2, and QEMU's software emulation of
 AVX2 is roughly two orders of magnitude slower than native. A suite that finishes in 17
@@ -252,7 +286,7 @@ exactly the kind of thing that only surfaces on a machine without an OCR engine.
 
 ## The test suite, in full
 
-`pytest tests` collects **125 tests**. 123 run headless — no display, no network, no
+`pytest tests` collects **153 tests**. 151 run headless — no display, no network, no
 weights. The last 2 are the opt-in live-desktop tests, skipped unless
 `TINY_COMPUTER_AGENT_LIVE_TEST=1`.
 
@@ -269,11 +303,12 @@ weights. The last 2 are the opt-in live-desktop tests, skipped unless
 | `tests/test_grid_fallback.py` | 11 | the fallback index announces itself instead of guessing |
 | `tests/test_desktop.py` | 10 | display detection, the preflight checks, `doctor` |
 | `tests/test_cli.py` | 10 | the command line: `--doctor`, `--image`, `--live`, argument handling |
+| `tests/test_trace.py` | 28 | traces: a run recorded with no pixels, replayed against another brain, and scored against ground truth |
 | `tests/test_live_desktop.py` | 2 | **needs a display, opt-in**: OCR finds a real button at the pixels it was drawn at, and a real click makes the app react |
 
 ```bash
-pytest tests -q                                  # 123 passed, 2 skipped
-TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 125 passed, on a desktop
+pytest tests -q                                  # 151 passed, 2 skipped
+TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 153 passed, on a desktop
 ```
 
 ## Install
@@ -282,7 +317,7 @@ TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 125 passed, on a desktop
 pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
 pip install -e ".[desktop,ocr]"      # + real capture and real input
 pip install -e ".[vision,ocr]"       # + the tiny model brains
-pytest                               # 125 tests, no display or network needed
+pytest                               # 153 tests, no display or network needed
 ```
 
 ## Run it on your machine
@@ -346,11 +381,20 @@ demands:
 ```bash
 # 1. install media, plus an answer file on removable media
 qemu-img create -f qcow2 win.qcow2 64G
-qemu-system-x86_64 -accel tcg,thread=multi -m 8G -smp 8 -machine pc   -cpu Skylake-Client   -drive file=win.qcow2,if=none,id=hd0,format=qcow2,cache=writeback   -device ide-hd,drive=hd0,bus=ide.0   -drive file=win2022.iso,if=none,id=cd0,media=cdrom,readonly=on   -device ide-cd,drive=cd0,bus=ide.1   -drive if=none,id=usb,format=raw,file=unattend.img   -device usb-ehci,id=ehci -device usb-storage,bus=ehci.0,drive=usb   -netdev user,id=n0 -device e1000,netdev=n0   -vga std -display none -vnc :1 -boot order=d,menu=off
+qemu-system-x86_64 -accel tcg,thread=multi -m 8G -smp 8 -machine pc \
+  -cpu Skylake-Client \
+  -drive file=win.qcow2,if=none,id=hd0,format=qcow2,cache=writeback \
+  -device ide-hd,drive=hd0,bus=ide.0 \
+  -drive file=win2022.iso,if=none,id=cd0,media=cdrom,readonly=on \
+  -device ide-cd,drive=cd0,bus=ide.1 \
+  -drive if=none,id=usb,format=raw,file=unattend.img \
+  -device usb-ehci,id=ehci -device usb-storage,bus=ehci.0,drive=usb \
+  -netdev user,id=n0 -device e1000,netdev=n0 \
+  -vga std -display none -vnc :1 -boot order=d,menu=off
 
 # 2. inside the guest, after autologon
 #    vc_redist.x64.exe is required before onnxruntime will load
-curl.exe -s -o C:	inyc.exe <vc_redist-url> && C:	inyc.exe /install /quiet /norestart
+curl.exe -s -o C:\tiny\vc_redist.x64.exe <vc_redist-url> && C:\tiny\vc_redist.x64.exe /install /quiet /norestart
 ```
 
 Serve files to the guest over QEMU's user-mode networking (the host is `10.0.2.2`), which
@@ -374,6 +418,7 @@ this: an answer file with non-canonical XML namespaces is **silently ignored** b
 | `loop.py` | `AgentLoop`, `Episode`, `Step`, the no-progress guard and the verifier |
 | `desktop.py` | Preflight checks, `doctor`, and `build_agent` in one call |
 | `demo.py` | The synthetic desktop used by tests and `--image example` |
+| `trace.py` | Portable traces: record a run with no pixels, replay it offline, score the choice against ground truth |
 
 ## Why a tiny model can do this at all
 
@@ -406,7 +451,7 @@ success.
 
 **Works:** real capture, a real OCR index with coordinates that land on the right pixel,
 a real click an app reacts to, verified success, the closed vocabulary, the policy gates,
-the no-progress guard, echo detection, the doctor, the CLI, 125 tests, a 256M VLM that
+the no-progress guard, echo detection, the doctor, the CLI, portable traces that score a brain's choices offline, 153 tests, a 256M VLM that
 reads a button label from pixels on a clean screenshot — and the same suite passing inside
 a real Windows Server 2022 guest with no KVM (109 passed, exit 0, with the OCR-inference
 tests excluded as impractically slow under emulation).
@@ -427,7 +472,7 @@ when no verifier is given.
 - [ ] Make the OCR guard in `test_cli.py` skip, rather than fail, when no engine is installed
 - [ ] Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
 - [ ] Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
-- [ ] Score episodes: replay a trace and count how often the choice was right
+- [x] Score episodes: replay a trace and count how often the choice was right - see [scoring a brain, offline](#scoring-a-brain-offline)
 - [ ] marimo console: live screen preview, action log, pause / resume / step
 
 ## License