preview/card.md

metadata

title: Tiny Computer Agent
emoji: 🖱️
colorFrom: indigo
colorTo: green
license: mit
tags:
  - computer-use
  - gui-agent
  - agentic-harness
  - local-first
  - tiny-models
  - offline
  - marimo

🖱️ Tiny Computer Agent

An agentic harness that drives a real computer with a tiny, fully local model. No cloud API, no API keys, no telemetry — and a core that imports nothing outside the Python standard library.

Status 0.3.0: it drives a real desktop — on Linux and on Windows. Screen capture, an OCR element index with real coordinates, a real mouse click, and a verified outcome. Every capability has a real screenshot in the capability gallery, and a run can be saved as a portable trace, replayed offline and scored, so "which brain is right, and how often?" is a number in a table. The suite passes on a live Linux desktop and inside a real Windows Server 2022 guest; what is not solved is model reliability — see measured results.

The idea, in one loop

    observe         think          act           verify
  +---------+    +---------+    +---------+    +---------+
  | screen  |--->|  tiny   |--->| mouse / |--->|  next   |
  |  frame  |    |  model  |    | keyboard|    |  frame  |
  +---------+    +---------+    +---------+    +----+----+
                                                    |
  <-------------------------------------------------+
        repeat until done, or the step budget runs out

1. Observe — capture the screen, then build an element index: OCR (and, optionally, the OS accessibility tree) locates text and widgets and gives each one a label and a real screen coordinate.
2. Think — the model is shown a short numbered menu built from that index and answers with one number (plus a verb). It never emits coordinates.
3. Act — the harness maps the chosen label back to pixels and executes it.
4. Verify — re-read the screen; stop if the same action repeats against an unchanged screen, and only call the episode a success if a verifier can see it worked.

Measured results

Everything below was run from this repo on CPU, dry-run unless stated. Nothing is aspirational.

On a live desktop, with a real click

A real Tk window (a Save dialog with Save/Cancel buttons) on a real X server (Xvfb :99), driven with real mouse input. The button sits at window +120+120, x=20, y=155, 110x40, so its centre is (195, 295):

$ python -m tiny_computer_agent --live --real-input --verify-text Saved "click 'Save' to save the document"
mode: REAL INPUT (mouse and keyboard live)  |  screen: real desktop (monitor 1)  |  brain: planner
index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))

  step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: clicked (195, 296)

result: success - verified after 1 action(s)

The click landed on the button, the app changed its own window to "Saved", and the verifier confirmed it before the episode was called a success. Pixels in, a real click out, and evidence that it worked.

Re-confirmed end to end on a headless Linux box

The whole path above was re-run from scratch on a headless Debian container (gVisor sandbox, 20 vCPU, no GPU, no physical display, no network at run time) behind a virtual X server. First the suite, headless:

python -m pip install -e ".[desktop,ocr,dev]"
pytest tests -q
# 162 passed, 2 skipped in 18.7s

The two skips are the live-desktop tests. Give them a display and they run for real:

Xvfb :99 -screen 0 1280x800x24 &
export DISPLAY=:99
TINY_COMPUTER_AGENT_LIVE_TEST=1 python -m pytest tests/test_live_desktop.py -q
# tests/test_live_desktop.py::test_ocr_finds_a_real_button_at_the_right_pixels PASSED
# tests/test_live_desktop.py::test_a_real_click_lands_on_the_button PASSED
# 2 passed

Then the same desktop, driven through the CLI with the mouse actually live:

export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1
python -m tiny_computer_agent --live --real-input --verify-text Saved \
    "click 'Save' to save the document"

goal: click 'Save' to save the document
mode: REAL INPUT (mouse and keyboard live)  |  screen: real desktop (monitor 1)  |  brain: planner  |  kill switch: create /tmp/tiny-computer-agent.STOP to stop
index: auto - OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))

  step 0 | click label='Save' reason="matched 'Save' in the element index" | ok: clicked (194, 296)

result: success - verified after 1 action(s)

The window was drawn at +120+120 with its Save button at x=20, y=155, 110x40, so the button centre is (195, 295). The click at (194, 296) is on the button, not near it — and the app wrote Save back to its own marker file. Pixels in, a real click out, and the app's own side effect as the proof.

--doctor on that box, with the virtual display attached:

tiny-computer-agent preflight (Linux x86_64, python 3.13.11)

  [ok  ] display server: DISPLAY=:99
  [ok  ] pillow: installed
  [ok  ] screen capture: mss (fast, multi-monitor)
  [ok  ] input backend (pyautogui): installed
  [ok  ] element detector (auto): OCR when it finds text, grid otherwise (rapidocr-onnxruntime ready (CPU, offline))
  [ok  ] model runtime: not needed: the planner is rule-based (no weights, no download)
  [ok  ] execution mode: dry run (nothing is clicked)
  [ok  ] kill switch: create /tmp/tiny-computer-agent.STOP to abort the next action

READY: the harness can drive this desktop.

On a replayed screenshot

$ python -m tiny_computer_agent --image example --brain planner
elements detected: 5
  step 0 | click label='Save' ... -> pixel (295, 284)          # OCR found the button

SmolVLM-256M-Instruct (256M parameters, CPU) read the button off the pixels and named the right target:

  step 0 | click label='Save' reason="the reply names 'Save'" -> pixel (295, 284)

Scoring a brain, offline

"Which brain is right, and how often?" needs no desktop, no model and no network, because a trace keeps every frame's element index and throws the pixels away. Record a run once; replay it against any brain, as often as you like.

$ python -m tiny_computer_agent --image example --trace runs/save.json --expected Save
...
trace: wrote runs/save.json (2 frames)
score: correct   click 'Save' to save the document: expected 'Save', chose 'Save'

A trace is JSON, so it diffs in review, commits next to the tests, and replays on a machine with nothing installed. Scoring a whole suite is a table:

scoreboard: 7/9 correct (78%)
  planner (rule-based): 100%
  first-item (baseline): 33%
  oracle (always right): 100%
  correct   save the document: expected 'Save', chose 'Save'
  incorrect save the document: expected 'Save', chose 'Files' (chose 'Files', not 'Save')
  ...

The number is the brain's decision - what it chose when it had to commit. Whether the harness allowed that decision to run is a separate axis: a click the policy refused is still the choice it was, and it is flagged (harness refused it) rather than passed off as a clean success. Keeping those two apart is the point.

Which brain drove which run

The harness and the brain are separable, and it matters which one a result came from. None of the runs below used a cloud API; the two model brains run on CPU:

Run	Brain	Weights	Outcome
Live Linux desktop, real click	PlannerBrain (rule-based)	none	success, verified
/tmp/saveapp.py window live	PlannerBrain	none	clicked Save at (194, 296)
--image example	PlannerBrain	none	picked Save, pixel (295, 284)
--image example	VLMBrain (SmolVLM-256M-Instruct)	256M, CPU	picked Save
Live Linux desktop	VLMBrain (SmolVLM-256M-Instruct)	256M, CPU	echoed the prompt; refused by echo detection
Live Linux desktop	TextPlannerBrain (Qwen2.5-0.5B)	0.5B, CPU	clicked Cancel, then garbled JSON; budget exhausted

The rule-based planner is what carries the verified end-to-end runs. The tiny model is the open problem: it is reliable on a clean screenshot and unreliable on a busy live screen.

What did not work, and what caught it

• The 256M VLM echoes the prompt on a busy live screen. Asked to click Save on the live Tk window it replied with the prompt text back. The harness's echo detection refused it (BrainError: the reply mentions 4 menu entries, so it looks like an echo of the prompt) and stopped — no blind click. On the cleaner synthetic screenshot the same model chose correctly, so this is a prompt/robustness problem, not a wiring one.
• Qwen2.5-0.5B text planner over the OCR index is not reliable: it clicked Cancel before Save, then recovered a garbled JSON reply, then found Save — budget exhausted.
• The grid fallback announced itself. After the dialog closed, OCR found no text, the index degraded to 96 numbered cells, and the run printed warning: no text was found on screen, so the index is 96 numbered grid cells and the agent is choosing blind — instead of silently guessing.

164 tests pass, including real OCR over a real image and (opt-in) live-desktop tests that click a real button.

Verified on real Windows

The Windows claims in this card are not taken on faith. A full Windows guest was built and the harness installed inside it, on a headless Linux host with no KVM — the same kind of sandbox the Linux runs above were produced in.

What was built.

host        headless Linux container (no /dev/kvm; software emulation only)
hypervisor  QEMU 10.0.13  -accel tcg,thread=multi  -m 8G  -smp 8  -cpu Skylake-Client
firmware    SeaBIOS (the OVMF/UEFI build would not auto-boot the installation media)
guest       Windows Server 2022 Standard Evaluation (Desktop Experience), x86_64
install     unattended from the public evaluation ISO, install.wim index 2,
            answer file on removable media, autologon as Administrator

The image-apply phase moved roughly 9 GB to disk in about thirteen minutes, and the desktop was up a few minutes after that.

What was installed inside it.

Python 3.12.8 (embeddable distribution, unpacked to C:\tiny\py)
pytest, pillow 13.0.0, mss 10.2.0
numpy 2.5.4, onnxruntime 1.31.1, opencv-python 5.0.0.93, rapidocr-onnxruntime 1.4.4
the harness itself, installed into site-packages

Results.

Configuration	Result	Wall clock
Harness suite, no OCR engine	118 passed, 3 failed, 4 skipped	7.8 s
Harness suite, OCR engine installed, test_perception.py excluded	109 passed, 3 skipped, exit 0	21 min
test_perception.py (OCR inference)	runs; roughly ten minutes per OCR pass	—

None of the three failures in the first row is a harness defect: two are ModuleNotFoundError: numpy and the third is the guard bug documented below. With the OCR engine installed, the same suite exits 0. (Those Windows counts predate trace.py; the harness suite has since grown to 164 tests.)

The emulated-OCR caveat. onnxruntime requires AVX2, and QEMU's software emulation of AVX2 is roughly two orders of magnitude slower than native. A suite that finishes in 17 seconds on Linux takes over half an hour on this guest, and the time is spent entirely inside onnxruntime_inference_collection.py. That is a property of emulation, not of the harness: on real hardware — including a GitHub windows-latest runner — it does not apply.

What running on Windows caught

Four things surfaced by actually running there. Three are genuine deployment requirements that no Linux-only test run can reveal:

Finding	Symptom	Fix
onnxruntime needs AVX2	Windows fatal exception: code 0xc000001d (STATUS_ILLEGAL_INSTRUCTION) the moment onnxruntime is imported, because QEMU's default qemu64 CPU advertises no AVX	boot the guest with -cpu Skylake-Client (any AVX2 model will do)
onnxruntime needs the MSVC runtime	ImportError: DLL load failed while importing onnxruntime_pybind11_state on a clean Server 2022	install vc_redist.x64.exe in the guest
pyautogui ships as an sdist only	pip install pytest pillow mss pyautogui fails as a whole, so pytest never arrives and the suite reports No module named pytest	install pyautogui in a separate step from the wheel-only packages
A guard that never fires	test_screenshot_replay_needs_no_display fails instead of skipping when no OCR engine is present: it guards on detector_available()[0], which is True for the grid fallback	guard on the OCR detector specifically

The last entry is a defect in this repository rather than in the environment, and it is exactly the kind of thing that only surfaces on a machine without an OCR engine.

The test suite, in full

pytest tests collects 164 tests. 162 run headless — no display, no network, no weights. The last 2 are the opt-in live-desktop tests, skipped unless TINY_COMPUTER_AGENT_LIVE_TEST=1.

File	Tests	What it proves
tests/test_actions.py	13	the closed vocabulary — every action is validated before anything executes
tests/test_act.py	8	DryRunExecutor is inert; the real executor refuses until explicitly armed
tests/test_observe.py	8	element/observation geometry, FakeScreen, ImageScreen, the capture fallback
tests/test_perception.py	13	OCR labels a real image with real coordinates; the grid fallback; AutoDetector
tests/test_brain.py	18	the numbered DecisionMenu, lenient parsing, echo detection, MockBrain, PlannerBrain
tests/test_loop.py	14	AgentLoop / Episode / Step, the no-progress guard, the step budget
tests/test_verify.py	7	success is only claimed when a verifier can actually see it
tests/test_safety.py	11	deny lists, the kill switch, the blind-grid-click guard
tests/test_grid_fallback.py	11	the fallback index announces itself instead of guessing
tests/test_desktop.py	10	display detection, the preflight checks, doctor
tests/test_cli.py	10	the command line: --doctor, --image, --live, argument handling
tests/test_trace.py	28	traces: a run recorded with no pixels, replayed against another brain, and scored against ground truth
tests/test_models.py	7	the model backends: lazy loading, device choice, honest refusals
tests/test_api_surface.py	4	the public API is a promise: everything advertised is importable, and nothing public is left out
tests/test_live_desktop.py	2	needs a display, opt-in: OCR finds a real button at the pixels it was drawn at, and a real click makes the app react

pytest tests -q                                  # 162 passed, 2 skipped
TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 164 passed, on a desktop

Install

pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
pip install -e ".[desktop,ocr]"      # + real capture and real input
pip install -e ".[vision,ocr]"       # + the tiny model brains
pytest                               # 164 tests, no display or network needed

Run it on your machine

python -m tiny_computer_agent --doctor                 # what can this machine do?
python -m tiny_computer_agent --image example          # replay a screenshot, dry run
python -m tiny_computer_agent --live                   # your real screen, dry run
python -m tiny_computer_agent --live --real-input "click 'OK'"
python -m tiny_computer_agent --live --real-input --verify-text "Saved" "click 'Save'"

--doctor is the first thing to run on a new machine: it checks the display session, capture backend, pyautogui, the OCR engine, the model runtime and the kill switch, and prints the fix for anything missing. Real input needs two deliberate acts:

export TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1

Pass --verify-text. Without it, a successful action that removes its own target (clicking Save closes the dialog) is indistinguishable from a failure, and the episode is recorded as one. With it, the harness only reports success when it can see the text it was given.

No desktop? Use a virtual one

Xvfb is enough for everything except a real model's hallucinations:

Xvfb :99 -screen 0 1280x800x24 &
export DISPLAY=:99
TINY_COMPUTER_AGENT_LIVE_TEST=1 python -m pytest tests/test_live_desktop.py

That test starts its own Tk window, asserts OCR finds the button at the coordinates it was actually drawn at, clicks it for real, and checks the app reacted. It is opt-in because it moves your mouse — run it on a throwaway desktop.

Testing on Windows - three ways, and which to use

The verified Windows run above used method 2. All three remain valid:

Method	Real Windows desktop?	Cost	Status
GitHub Actions windows-latest	yes, interactive session	free for public repos, minutes	workflow committed at .github/workflows/live-desktop.yml
QEMU/TCG VM on a Linux host	yes, fully emulated	~25 min to install, hours to run OCR	used for the results above
A cloud Windows box over RDP	yes, native	paid by the hour	untested here

Nothing in the harness is Linux-specific: pyautogui uses SendInput on Windows and GrabScreen already falls back to Pillow's ImageGrab there. Two things can only fail on a real Windows desktop, which is why these runs exist: DPI scaling (that is what --scale is for) and the native input backend. At 125% or 150% display scaling, expect the coordinate assertion in test_live_desktop.py to be the first thing to fail - that is the signal, not a nuisance.

Reproducing the local VM. With no KVM, a Windows guest is still perfectly possible - it is simply slow. The working recipe, including the two ingredients that onnxruntime demands:

# 1. install media, plus an answer file on removable media
qemu-img create -f qcow2 win.qcow2 64G
qemu-system-x86_64 -accel tcg,thread=multi -m 8G -smp 8 -machine pc \
  -cpu Skylake-Client \
  -drive file=win.qcow2,if=none,id=hd0,format=qcow2,cache=writeback \
  -device ide-hd,drive=hd0,bus=ide.0 \
  -drive file=win2022.iso,if=none,id=cd0,media=cdrom,readonly=on \
  -device ide-cd,drive=cd0,bus=ide.1 \
  -drive if=none,id=usb,format=raw,file=unattend.img \
  -device usb-ehci,id=ehci -device usb-storage,bus=ehci.0,drive=usb \
  -netdev user,id=n0 -device e1000,netdev=n0 \
  -vga std -display none -vnc :1 -boot order=d,menu=off

# 2. inside the guest, after autologon
#    vc_redist.x64.exe is required before onnxruntime will load
curl.exe -s -o C:\tiny\vc_redist.x64.exe <vc_redist-url> && C:\tiny\vc_redist.x64.exe /install /quiet /norestart

Serve files to the guest over QEMU's user-mode networking (the host is 10.0.2.2), which avoids typing long commands into the emulated console. Two practical notes from doing this: an answer file with non-canonical XML namespaces is silently ignored by Setup (you land on the language page), and keyboard input over VNC drops the shift modifier (_ becomes -, : becomes ;) - the QEMU monitor's sendkey handles shifts correctly.

How it works

Module	Job
actions.py	The closed vocabulary — click type key scroll wait done, validated before anything executes
observe.py	Element/Observation, FakeScreen, ImageScreen, GrabScreen (mss → Pillow fallback)
perception.py	Detectors: OcrDetector (RapidOCR), GridDetector, AutoDetector, IndexedScreen
brain.py	DecisionMenu (the numbered menu), ModelBrain, lenient parse_decision, echo detection, MockBrain, PlannerBrain
models.py	VLMBrain (SmolVLM-256M) and TextPlannerBrain (Qwen2.5-0.5B), lazily loaded
act.py	DryRunExecutor (the default) and PyAutoGuiExecutor
safety.py	Policy: deny lists, kill switch, step budget, the blind-grid-click guard
loop.py	AgentLoop, Episode, Step, the no-progress guard and the verifier
desktop.py	Preflight checks, doctor, and build_agent in one call
demo.py	The synthetic desktop used by tests and --image example
trace.py	Portable traces: record a run with no pixels, replay it offline, score the choice against ground truth
showcase.py	The capability gallery: one real screenshot per capability, checked against the public API

Capability gallery

68 capabilities, one screenshot each - and every one of the 141 public callables in the package (functions, methods, classes and protocols) appears in at least one of them, together with 6 of its public constants.

Nothing here is a mock-up. Each picture is produced by showcase.py: a command or a call is run for real, and its actual output - or the real desktop it acted on - is what you see. The desktop shots are real screen captures on a real X server (Xvfb, 1280x800), including a real mouse click verified by the target application's own reaction.

An honest note on the platform. These were captured on Linux, because the sandbox that builds this card has no Windows: no KVM, no QEMU and no Windows image. The harness is OS-agnostic (pyautogui uses SendInput on Windows, and GrabScreen falls back to Pillow's ImageGrab), so these are the same code paths - but they are not Windows pixels. The Windows twins are generated by the same script in .github/workflows/live-desktop.yml.

Regenerate it with python showcase.py (--list to see the ids, --only <id> for a subset, --card to rewrite this section). It runs the calls for real, so a broken capability fails loudly instead of shipping a stale picture.

area	capabilities
Action vocabulary	3
Decisions	7
Execution	2
The loop	7
Safety	4
Screens	6
Perception	6
Preflight	4
Models	5
Traces & scoring	7
Command line	9
Real desktop	7
Extensibility	1

---

Action vocabulary

The six verbs, and only six

Every action is validated on construction, before anything touches the machine.

shows: ACTION_KINDS, ACTION_MENU, Action, Action.describe, Action.to_dict, parse_action

Malformed actions are refused, not guessed at

An unknown verb or a missing target raises before the machine is touched.

shows: ActionError, Action, parse_action

The contract handed to a model

The prompt-side menu: one JSON object, six shapes, no invented verbs.

shows: ACTION_MENU

Decisions

A short numbered menu, in reading order

A 256M model cannot emit coordinates; it can pick a number from twelve.

shows: DecisionMenu, DecisionMenu.render, MenuItem, MenuItem.label

Menu lookup tolerates what models actually send

1, "2", 2.0 and "Save." all resolve; a stray word resolves to nothing.

shows: DecisionMenu.item, DecisionMenu.find, DecisionMenu.mentioned_in

Reading a decision out of a small model's reply

The LAST JSON object wins, because a model that echoes its prompt answers last.

shows: parse_decision, extract_json_object, extract_json_objects, DecisionMenu

An echoed prompt is refused; a right answer in the wrong field is recovered

Real 256M/0.5B failure modes, both caught: no blind click either way.

shows: parse_decision, BrainError, DecisionMenu.mentioned_in

A garbled verb with a valid label is still usable

A 256M model really does emit {"kind": "file", "label": "Save"}.

shows: parse_decision, BrainError

MockBrain: replay a fixed script, exactly once

Deterministic and offline - the workhorse for tests, demos and replays.

shows: MockBrain, MockBrain.decide, MockBrain.remaining

PlannerBrain: no weights, no download, and it gives up honestly

The proof the harness works end to end, and a real fallback when no model is wanted.

shows: PlannerBrain, PlannerBrain.decide, GoalUnreachable

Execution

Dry run resolves every click to pixels - and touches nothing

The default everywhere: read a whole episode before anything clicks.

shows: DryRunExecutor, DryRunExecutor.execute, resolve_point, ActionResult

Real input refuses to guess a pixel for a label it cannot find

This is the gate between a hallucinated label and 'Delete account'.

shows: ActionResult.summary, InputUnavailable, PyAutoGuiExecutor, PyAutoGuiExecutor.execute

The loop

observe -> think -> act -> verify, traced end to end

One call runs a whole episode; every step is kept for audit.

shows: AgentLoop, AgentLoop.run, Episode, Episode.render, Episode.to_rows, Step, Step.line, Step.ok

A hard step budget, so a run cannot wander

The budget ends the episode and says so; nothing executes past it.

shows: Policy, Policy.step_budget, AgentLoop.run

Repeating one action against an unchanged screen is stopped

A stuck agent is not a persistent one - it is stopped and told why.

shows: AgentLoop.run, Step, Step.error

A kill-switch file aborts before the next action

Checked before every single action: create the file, the run stops.

shows: Policy.kill_switch_path, Policy.should_stop, AgentLoop.run

A dead screen or a broken brain is an episode, not a traceback

Every failure path is recorded as a failed step with a reason.

shows: AgentLoop.run, Episode.stop_reason, GoalUnreachable, BrainError

Success is evidence, not the model's word

Without a verifier, a successful action that removes its own target looks like a failure.

shows: AgentLoop.run, Episode.success, Observation.text

Every step can be streamed out as it happens

This is the hook a live console, a progress bar or a websocket uses.

shows: AgentLoop.run, Step.line, on_step

Safety

Typing from the deny list never reaches the keyboard

Checked before execution; a refusal ends the episode and clicks nothing.

shows: Policy, Policy.check, PolicyViolation, DEFAULT_DENY_TEXT

Clicking a denied widget is refused, however the label is spelled

Terminal, keychain, wallet, password manager, bank - matched on the label.

shows: Policy.deny_labels, Policy.check, Observation.find

Key chords on the deny list are never pressed

ctrl+alt+t, ctrl+alt+delete and ctrl+shift+escape are refused by default.

shows: Policy.deny_keys, Policy.check, PolicyViolation

A numbered grid cell is a blind click: allowed to plan, refused for real

With real input the blind guess is refused unless you pass --allow-grid.

shows: Policy.allow_grid_clicks, Policy.check, GridDetector

Screens

A scripted screen, for testing the loop with no display

The final frame repeats forever, so a runaway agent cannot exhaust the script.

shows: FakeScreen, FakeScreen.capture, Observation

A still image (or a sequence) replayed as a screen

Lets the whole pipeline be exercised with no display, and a failure be replayed.

shows: ImageScreen, ImageScreen.capture, fake_desktop

Real capture: monitors, downscaling, and HiDPI mapping

Frame pixel -> screen pixel is preserved, so a click cannot drift.

shows: GrabScreen, GrabScreen.capture, GrabScreen.map_to_screen, ScreenUnavailable

Labels are compared on their alphanumeric core

OCR loses spaces, models add punctuation - so nothing matches by exact string.

shows: Element, Element.center, Element.matches, Observation.find, normalise_label

The compact text a model is shown

This is the whole world as the brain sees it - capped, and honest when empty.

shows: Observation.summary

The built-in synthetic desktop

Draws a screenshot-like desktop offline - the same image --image example replays.

shows: fake_desktop, write_fake_desktop, OcrDetector

Perception

Probing for a detector, and building one by name

The doctor and the CLI both ask this before promising anything.

shows: detector_available, make_detector, DetectorUnavailable, Detector

OCR turns a screenshot into labelled, clickable elements

The model never has to guess where anything is: this is the grounding.

shows: OcrDetector, OcrDetector.detect, Element

Three knobs keep dialog prose out of the menu

Every extra line of noise is one more chance for a small model to click prose.

shows: OcrDetector.min_score, OcrDetector.max_label_chars, OcrDetector.min_label_chars

A numbered grid, for screens with no text at all

An agent with an empty index has no legal moves; a grid always has moves.

shows: GridDetector, GridDetector.detect

OCR when there is text, a grid when there is none

The default: useful with nothing installed, better the moment OCR appears.

shows: AutoDetector, AutoDetector.detect

IndexedScreen: capture + detect, composed

One capture() call gives the loop an index it can act on.

shows: IndexedScreen, IndexedScreen.capture, Observation.summary

Preflight

Is there a desktop session to drive at all?

The first question the CLI and the doctor both ask.

shows: display_available

Every precondition, one list, each with the fix

Eight checks; a required one that fails is a blocking problem.

shows: run_checks, Check, Check.render

python -m tiny_computer_agent --doctor

The first thing to run on a new machine. Exits non-zero when not ready.

shows: detector_available, display_available, doctor, report

One call wires screen + detector + brain + executor + policy

Defaults are the safe ones: dry run, a step budget, and a kill-switch file.

shows: build_agent, Policy, AgentLoop

Models

Which model runtime is present, and are the weights cached?

Weights are loaded lazily, so asking is free.

shows: model_dependencies, model_cached, MODEL_BRAINS, make_model_brain, DEFAULT_TEXT_MODEL, DEFAULT_VISION_MODEL, BrainError

Device and dtype selection

CUDA if it is there, CPU otherwise; the harness never needs a GPU.

shows: resolve_device, pick_dtype

The prompt a model actually receives - and how it learns from a dead end

Placeholders, not worked examples: a weak model echoes the prompt, and an echo fails validation.

shows: ModelBrain, ModelBrain.build_prompt, ModelBrain.decide

A second decision is told what already failed

'you already tried this and the screen did NOT change' is the most useful line a small model gets.

shows: ModelBrain, ModelBrain.build_prompt, ModelBrain.history

The two real model backends, and lazy loading

Constructing a brain costs nothing; the weights arrive on the first decide().

shows: TextPlannerBrain, VLMBrain, TextPlannerBrain.unload, VLMBrain.unload, ModelBrain.loaded, make_model_brain

Traces & scoring

A task with one right answer, and the frames to show

Ground truth lives with the task, so a run can be scored at all.

shows: Scenario, Scenario.label, Scenario.screen, Trace.render, run_scenario

An episode becomes a portable, pixel-free trace

JSON, so it diffs in review and replays on a machine with nothing installed.

shows: Trace, Trace.blocked, Trace.choice, Trace.decided, Trace.decision_step, Trace.frames, Trace.load, Trace.name, Trace.record, Trace.save, Trace.to_dict, Trace.to_json, TraceStep, TraceStep.observation, TraceStep.to_dict

Replay a recorded trace against a different brain

The frames are the fixture; the brain is the only variable. Nothing is captured or clicked.

shows: Trace.replay, Trace.observations, score_trace

Counting how often the choice was right

Scored on normalised text, so OCR spacing is forgiven - the same bargain the index makes.

shows: score_trace, score, score_replay, Verdict, Verdict.line

Verdicts collected into an accuracy

The number the whole module exists for: which brain is right, and how often.

shows: Scoreboard, Scoreboard.accuracy, Scoreboard.accuracy_for, Scoreboard.add, Scoreboard.blocked, Scoreboard.brains, Scoreboard.by_brain, Scoreboard.confusion, Scoreboard.earned, Scoreboard.render, Scoreboard.to_rows, Scoreboard.total

Every scenario through every brain, in one table

A fresh brain per task, so state like 'already clicked this' cannot leak between them.

shows: benchmark, Scoreboard.to_rows

Replay a whole recorded suite against one brain

The literal roadmap step: the frames are fixed, nothing new is captured.

shows: benchmark_traces, load_traces, Trace.save

Command line

--image example (replay a screenshot, dry run)

No display needed: the whole pipeline over a still image.

shows: main, build_parser, ImageScreen, detector_available

--script (replay a fixed action list instead of a brain)

The script is validated exactly like a model's answer: nothing skips the gates.

shows: MockBrain, demo_screen, parse_action

--verify-text (the harness decides success, not the brain)

Without it, an action that removes its own target looks like a failure.

shows: AgentLoop.run, Episode.success

--trace / --expected (record the run, then score it)

Writes JSON next to the code, and prints the score straight away.

shows: Trace.record, Trace.save, score_trace

--no-ocr (index a numbered grid instead)

Useful on a machine with no OCR engine, or for a canvas app.

shows: GridDetector, make_detector, detector_available

Exit codes: 0 success, 1 failure, 2 bad usage, 3 no screen

A shell can tell 'it worked' from 'it gave up' without parsing any text.

shows: main

The two deliberate acts before the mouse can move

--real-input without the env var, and --brain vlm with no pixels: both refuse.

shows: main, ALLOW_REAL_INPUT_ENV

--live with no display is refused, with the fix

It tells you what to do instead of dying inside an episode.

shows: display_available, main

--live (your real screen, dry run)

Real capture, real OCR, a real element index - and not one click.

shows: GrabScreen, IndexedScreen, AutoDetector, main

Real desktop

GrabScreen: capture a real desktop

1280x800 of a real X server. mss first, Pillow's ImageGrab as the fallback.

shows: GrabScreen, GrabScreen.capture, ScreenUnavailable

The element index, drawn on the pixels it came from

Every box is a real OCR word with a real screen coordinate: the model picks a label, not a pixel.

shows: OcrDetector, OcrDetector.detect, IndexedScreen, Element.center

A real click, and the window's own reaction

Pixels in, a label chosen, a pixel resolved, a button pressed - and the dialog closed itself.

shows: PyAutoGuiExecutor, AgentLoop.run, Policy.dry_run, resolve_point

Success only when the harness can see it

The app changed its own title and wrote a marker file; the verifier saw the change.

shows: AgentLoop.run, Episode.success, Episode.stop_reason

A screen with no text still has legal moves

OCR finds nothing on a canvas, so the index degrades to numbered cells - and says so.

shows: AutoDetector, GridDetector, OcrDetector

The safety gate, on a real button, in a real window

The Terminal button is visible and labelled - and the policy refuses to press it.

shows: Policy.check, Policy.deny_labels, PolicyViolation, Observation.find

The same click, driven end to end from the CLI

--live --real-input --verify-text Saved: capture, OCR, click, verify, exit code.

shows: main, PyAutoGuiExecutor, AgentLoop.run

Extensibility

Every stage is a Protocol: bring your own screen, brain, detector, executor

The loop depends on four one-method contracts, not on the shipped implementations.

shows: Brain, Brain.decide, ScreenSource, ScreenSource.capture, Detector, Detector.detect, Executor, Executor.execute, ActionResult

Why a tiny model can do this at all

Job	Who does it
Finding where a button is	OCR / accessibility tree — not the model
Choosing what to do	the model, but only by picking a number from a short menu
Knowing whether it worked	the verifier and the no-progress guard, not the model's word
Recovering from a bad step	the retry policy in loop.py, not the weights

A 256M model cannot emit (295, 284) reliably, but it can pick "4" out of five options — and the harness owns the coordinate lookup, the vocabulary check and the definition of success.

Safety, because this thing has your mouse

• Run it in a VM, container, or throwaway desktop account, not your real session.
• Dry run is the default, so you can read a whole episode before anything clicks.
• Deny lists for typing (sudo, rm -rf, "password"), widgets (terminal, keychain, wallet, bank) and key chords.
• A kill-switch file checked before every action, plus a hard step budget.
• Blind grid clicks are refused with real input unless you pass --allow-grid: when OCR finds nothing, the fallback index is coordinates with no meaning, and a model choosing one is guessing about your desktop.
• A brain's "done" is a claim, not evidence: pass --verify-text and the harness decides success itself.
• Local-first, for real: pixels and keystrokes never leave the machine.

What works, and what does not

Works: real capture, a real OCR index with coordinates that land on the right pixel, a real click an app reacts to, verified success, the closed vocabulary, the policy gates, the no-progress guard, echo detection, the doctor, the CLI, portable traces that score a brain's choices offline, 153 tests, a 256M VLM that reads a button label from pixels on a clean screenshot — and the same suite passing inside a real Windows Server 2022 guest with no KVM (109 passed, exit 0, with the OCR-inference tests excluded as impractically slow under emulation).

Does not work yet: reliability on a live screen. The 256M VLM echoed the prompt there; the 0.5B text planner picks wrong actions; the accessibility-tree detector is not implemented, so labels come from OCR only; and the harness still trusts a model's done when no verifier is given.

Roadmap

• Capture, OCR index, closed vocabulary, dry-run default, policy gates, doctor
• A real click on a real window, with a verified outcome
• Blind-grid-click guard, echo detection, no-progress guard
• The harness suite running on real Windows (Server 2022 guest, no KVM — see above)
• Windows run of the live desktop test: the workflow is committed, but it needs a GitHub mirror with Actions enabled — Hugging Face does not run GitHub Actions
• Make the OCR guard in test_cli.py skip, rather than fail, when no engine is installed
• Make the VLM reliable on a live screen (it echoes the prompt instead of choosing)
• Accessibility-tree detector (AT-SPI on Linux, UI Automation on Windows)
• Score episodes: replay a trace and count how often the choice was right - see scoring a brain, offline
• A capability gallery: one real screenshot per capability, generated by showcase.py
• A Windows run of the gallery: the workflow builds it on windows-latest, but it needs a GitHub mirror with Actions enabled - Hugging Face does not run GitHub Actions
• marimo console: live screen preview, action log, pause / resume / step

License

MIT — see LICENSE.