Blame for gallery/README.md

---
title: Capability gallery
---

## Capability gallery

**74 capabilities, one screenshot each - and every one of the 147 public callables in the package (functions, methods, classes and protocols) appears in at least one of them**, together with 6 of its public constants.

Nothing here is a mock-up. Each picture is produced by [`showcase.py`](showcase.py): a command or a call is run for real, and its actual output - or the real desktop it acted on - is what you see. The desktop shots are real screen captures on a real X server (Xvfb, 1280x800), including a real mouse click verified by the target application's own reaction.

> **An honest note on the platform.** These were captured on Linux, because the sandbox that builds this card has no Windows: no KVM, no QEMU and no Windows image. The harness is OS-agnostic (pyautogui uses `SendInput` on Windows, and `GrabScreen` falls back to Pillow's `ImageGrab`), so these are the same code paths - but they are not Windows pixels. The Windows twins are generated by the same script in [`.github/workflows/live-desktop.yml`](.github/workflows/live-desktop.yml).

Regenerate it with `python showcase.py` (`--list` to see the ids, `--only <id>` for a subset, `--card` to rewrite this section). It runs the calls for real, so a broken capability fails loudly instead of shipping a stale picture.

| area | capabilities |
|---|---:|
| Action vocabulary | 3 |
| Decisions | 7 |
| Execution | 2 |
| The loop | 8 |
| Safety | 4 |
| Screens | 6 |
| Perception | 8 |
| Preflight | 4 |
| Models | 7 |
| Traces & scoring | 7 |
| Command line | 9 |
| Real desktop | 8 |
| Extensibility | 1 |

---

### Action vocabulary

**The six verbs, and only six**

*Every action is validated on construction, before anything touches the machine.*

<img src="gallery/vocab-verbs.png" width="860">

`shows:` `ACTION_KINDS`, `ACTION_MENU`, `Action`, `Action.describe`, `Action.to_dict`, `parse_action`

**Malformed actions are refused, not guessed at**

*An unknown verb or a missing target raises before the machine is touched.*

<img src="gallery/vocab-refusals.png" width="860">

`shows:` `ActionError`, `Action`, `parse_action`

**The contract handed to a model**

*The prompt-side menu: one JSON object, six shapes, no invented verbs.*

<img src="gallery/vocab-menu.png" width="860">

`shows:` `ACTION_MENU`

### Decisions

**A short numbered menu, in reading order**

*A 256M model cannot emit coordinates; it can pick a number from twelve.*

<img src="gallery/menu-numbered.png" width="860">

`shows:` `DecisionMenu`, `DecisionMenu.render`, `MenuItem`, `MenuItem.label`

**Menu lookup tolerates what models actually send**

*1, "2", 2.0 and "Save." all resolve; a stray word resolves to nothing.*

<img src="gallery/menu-lookup.png" width="860">

`shows:` `DecisionMenu.item`, `DecisionMenu.find`, `DecisionMenu.mentioned_in`

**Reading a decision out of a small model's reply**

*The LAST JSON object wins, because a model that echoes its prompt answers last.*

<img src="gallery/decision-json.png" width="860">

`shows:` `parse_decision`, `extract_json_object`, `extract_json_objects`, `DecisionMenu`

**An echoed prompt is refused; a right answer in the wrong field is recovered**

*Real 256M/0.5B failure modes, both caught: no blind click either way.*

<img src="gallery/decision-echo-guard.png" width="860">

`shows:` `parse_decision`, `BrainError`, `DecisionMenu.mentioned_in`

**A garbled verb with a valid label is still usable**

*A 256M model really does emit {"kind": "file", "label": "Save"}.*

<img src="gallery/decision-garbled.png" width="860">

`shows:` `parse_decision`, `BrainError`

**MockBrain: replay a fixed script, exactly once**

*Deterministic and offline - the workhorse for tests, demos and replays.*

<img src="gallery/brain-mock.png" width="860">

`shows:` `MockBrain`, `MockBrain.decide`, `MockBrain.remaining`

**PlannerBrain: no weights, no download, and it gives up honestly**

*The proof the harness works end to end, and a real fallback when no model is wanted.*

<img src="gallery/brain-planner.png" width="860">

`shows:` `PlannerBrain`, `PlannerBrain.decide`, `GoalUnreachable`

### Execution

**Dry run resolves every click to pixels - and touches nothing**

*The default everywhere: read a whole episode before anything clicks.*

<img src="gallery/exec-dry-run.png" width="860">

`shows:` `DryRunExecutor`, `DryRunExecutor.execute`, `resolve_point`, `ActionResult`

**Real input refuses to guess a pixel for a label it cannot find**

*This is the gate between a hallucinated label and 'Delete account'.*

<img src="gallery/exec-refuses-unresolved.png" width="860">

`shows:` `ActionResult.summary`, `InputUnavailable`, `PyAutoGuiExecutor`, `PyAutoGuiExecutor.execute`

### The loop

**observe -> think -> act -> verify, traced end to end**

*One call runs a whole episode; every step is kept for audit.*

<img src="gallery/loop-end-to-end.png" width="860">

`shows:` `AgentLoop`, `AgentLoop.run`, `Episode`, `Episode.render`, `Episode.to_rows`, `Step`, `Step.line`, `Step.ok`

**A hard step budget, so a run cannot wander**

*The budget ends the episode and says so; nothing executes past it.*

<img src="gallery/loop-budget.png" width="860">

`shows:` `Policy`, `Policy.step_budget`, `AgentLoop.run`

**Repeating one action against an unchanged screen is stopped**

*A stuck agent is not a persistent one - it is stopped and told why.*

<img src="gallery/loop-no-progress-guard.png" width="860">

`shows:` `AgentLoop.run`, `Step`, `Step.error`

**A kill-switch file aborts before the next action**

*Checked before every single action: create the file, the run stops.*

<img src="gallery/loop-kill-switch.png" width="860">

`shows:` `Policy.kill_switch_path`, `Policy.should_stop`, `AgentLoop.run`

**A dead screen or a broken brain is an episode, not a traceback**

*Every failure path is recorded as a failed step with a reason.*

<img src="gallery/loop-survives-failures.png" width="860">

`shows:` `AgentLoop.run`, `Episode.stop_reason`, `GoalUnreachable`, `BrainError`

**Success is evidence, not the model's word**

*Without a verifier, a successful action that removes its own target looks like a failure.*

<img src="gallery/loop-verifier.png" width="860">

`shows:` `AgentLoop.run`, `Episode.success`, `Observation.text`

**Every step can be streamed out as it happens**

*This is the hook a live console, a progress bar or a websocket uses.*

<img src="gallery/loop-on-step.png" width="860">

`shows:` `AgentLoop.run`, `Step.line`, `on_step`

**The same loop, one turn at a time, with pause and resume**

*run() is batch; a console needs the caller to own the loop. These are the same rules.*

<img src="gallery/stepwise.png" width="860">

`shows:` `AgentLoop`, `AgentLoop.begin`, `AgentLoop.step_once`, `AgentLoop.pause`, `AgentLoop.resume`, `AgentLoop.finished`, `AgentLoop.run`, `Step.line`

### Safety

**Typing from the deny list never reaches the keyboard**

*Checked before execution; a refusal ends the episode and clicks nothing.*

<img src="gallery/safety-deny-text.png" width="860">

`shows:` `Policy`, `Policy.check`, `PolicyViolation`, `DEFAULT_DENY_TEXT`

**Clicking a denied widget is refused, however the label is spelled**

*Terminal, keychain, wallet, password manager, bank - matched on the label.*

<img src="gallery/safety-deny-label.png" width="860">

`shows:` `Policy.deny_labels`, `Policy.check`, `Observation.find`

**Key chords on the deny list are never pressed**

*ctrl+alt+t, ctrl+alt+delete and ctrl+shift+escape are refused by default.*

<img src="gallery/safety-deny-keys.png" width="860">

`shows:` `Policy.deny_keys`, `Policy.check`, `PolicyViolation`

**A numbered grid cell is a blind click: allowed to plan, refused for real**

*With real input the blind guess is refused unless you pass --allow-grid.*

<img src="gallery/safety-blind-grid.png" width="860">

`shows:` `Policy.allow_grid_clicks`, `Policy.check`, `GridDetector`

### Screens

**A scripted screen, for testing the loop with no display**

*The final frame repeats forever, so a runaway agent cannot exhaust the script.*

<img src="gallery/screen-fake.png" width="860">

`shows:` `FakeScreen`, `FakeScreen.capture`, `Observation`

**A still image (or a sequence) replayed as a screen**

*Lets the whole pipeline be exercised with no display, and a failure be replayed.*

<img src="gallery/screen-image.png" width="860">

`shows:` `ImageScreen`, `ImageScreen.capture`, `fake_desktop`

**Real capture: monitors, downscaling, and HiDPI mapping**

*Frame pixel -> screen pixel is preserved, so a click cannot drift.*

<img src="gallery/capture-options.png" width="860">

`shows:` `GrabScreen`, `GrabScreen.capture`, `GrabScreen.map_to_screen`, `ScreenUnavailable`

**Labels are compared on their alphanumeric core**

*OCR loses spaces, models add punctuation - so nothing matches by exact string.*

<img src="gallery/element-matching.png" width="860">

`shows:` `Element`, `Element.center`, `Element.matches`, `Observation.find`, `normalise_label`

**The compact text a model is shown**

*This is the whole world as the brain sees it - capped, and honest when empty.*

<img src="gallery/observation-summary.png" width="860">

`shows:` `Observation.summary`

**The built-in synthetic desktop**

*Draws a screenshot-like desktop offline - the same image --image example replays.*

<img src="gallery/fake-desktop.png" width="860">

`shows:` `fake_desktop`, `write_fake_desktop`, `OcrDetector`

### Perception

**Probing for a detector, and building one by name**

*The doctor and the CLI both ask this before promising anything.*

<img src="gallery/detector-available.png" width="860">

`shows:` `detector_available`, `make_detector`, `DetectorUnavailable`, `Detector`

**OCR turns a screenshot into labelled, clickable elements**

*The model never has to guess where anything is: this is the grounding.*

<img src="gallery/ocr-labels.png" width="860">

`shows:` `OcrDetector`, `OcrDetector.detect`, `Element`

**Three knobs keep dialog prose out of the menu**

*Every extra line of noise is one more chance for a small model to click prose.*

<img src="gallery/ocr-filters.png" width="860">

`shows:` `OcrDetector.min_score`, `OcrDetector.max_label_chars`, `OcrDetector.min_label_chars`

**A numbered grid, for screens with no text at all**

*An agent with an empty index has no legal moves; a grid always has moves.*

<img src="gallery/grid-detector.png" width="860">

`shows:` `GridDetector`, `GridDetector.detect`

**OCR when there is text, a grid when there is none**

*The default: useful with nothing installed, better the moment OCR appears.*

<img src="gallery/auto-detector.png" width="860">

`shows:` `AutoDetector`, `AutoDetector.detect`, `AutoDetector.coordinates_of_last_detect`

**IndexedScreen: capture + detect, composed**

*One capture() call gives the loop an index it can act on.*

<img src="gallery/indexed-screen.png" width="860">

`shows:` `IndexedScreen`, `IndexedScreen.capture`, `Observation.summary`

**The accessibility tree: elements the OS already knows about**

*No pixels are read: GTK publishes every button's name and screen extents.*

<img src="gallery/atspi-index.png" width="860">

`shows:` `AtspiDetector`, `AtspiDetector.detect`, `AtspiDetector.text`, `AtspiDetector.coordinates_of_last_detect`, `AtspiReader`, `AtspiReader.read`, `AccessibleNode`, `TreeReader`, `TreeReader.read`, `accessibility_available`, `INTERACTIVE_ROLES`, `TEXT_ROLES`

**The Windows backend: the same index, from UI Automation**

*Windows-only, so the mapping is exercised here against a synthetic control tree.*

<img src="gallery/uia-mapping.png" width="860">

`shows:` `UiaReader`, `UiaReader.read`, `UiaDetector`, `UiaDetector.detect`, `UiaDetector.text`, `UiaDetector.coordinates_of_last_detect`

### Preflight

**Is there a desktop session to drive at all?**

*The first question the CLI and the doctor both ask.*

<img src="gallery/display-available.png" width="860">

`shows:` `display_available`

**Every precondition, one list, each with the fix**

*Eight checks; a required one that fails is a blocking problem.*

<img src="gallery/run-checks.png" width="860">

`shows:` `run_checks`, `Check`, `Check.render`

**python -m tiny_computer_agent --doctor**

*The first thing to run on a new machine. Exits non-zero when not ready.*

<img src="gallery/cli-doctor.png" width="860">

`shows:` `detector_available`, `display_available`, `doctor`, `report`

**One call wires screen + detector + brain + executor + policy**

*Defaults are the safe ones: dry run, a step budget, and a kill-switch file.*

<img src="gallery/build-agent.png" width="860">

`shows:` `build_agent`, `Policy`, `AgentLoop`

### Models

**Which model runtime is present, and are the weights cached?**

*Weights are loaded lazily, so asking is free.*

<img src="gallery/models-deps.png" width="860">

`shows:` `model_dependencies`, `model_cached`, `MODEL_BRAINS`, `make_model_brain`, `DEFAULT_TEXT_MODEL`, `DEFAULT_VISION_MODEL`, `BrainError`

**Device and dtype selection**

*CUDA if it is there, CPU otherwise; the harness never needs a GPU.*

<img src="gallery/models-device.png" width="860">

`shows:` `resolve_device`, `pick_dtype`

**The prompt a model actually receives - and how it learns from a dead end**

*Placeholders, not worked examples: a weak model echoes the prompt, and an echo fails validation.*

<img src="gallery/models-prompt.png" width="860">

`shows:` `ModelBrain`, `ModelBrain.build_prompt`, `ModelBrain.decide`

**A second decision is told what already failed**

*'you already tried this and the screen did NOT change' is the most useful line a small model gets.*

<img src="gallery/models-prompt-history.png" width="860">

`shows:` `ModelBrain`, `ModelBrain.build_prompt`, `ModelBrain.history`

**The two real model backends, and lazy loading**

*Constructing a brain costs nothing; the weights arrive on the first decide().*

<img src="gallery/models-swap.png" width="860">

`shows:` `TextPlannerBrain`, `VLMBrain`, `TextPlannerBrain.unload`, `VLMBrain.unload`, `ModelBrain.loaded`, `make_model_brain`

**Constrained decoding: make an echo impossible, not merely detected**

*The 256M model's failure was answering with the prompt. A mask removes that option.*

<img src="gallery/decoding.png" width="860">

`shows:` `MenuMask`, `digit_token_ids`, `choice_token_ids`, `pair_token_ids`, `read_choice`, `read_verdict`, `brevity_ladder`, `majority_choice`, `DIGITS`

**Four ways to ask a tiny model, and what each one costs**

*json=free text, choice=one masked number, scan=yes/no per item, rank=log-odds order.*

<img src="gallery/models-decision-modes.png" width="860">

`shows:` `ModelBrain`, `ModelBrain.loaded`, `TextPlannerBrain`, `VLMBrain`, `make_model_brain`

### Traces & scoring

**A task with one right answer, and the frames to show**

*Ground truth lives with the task, so a run can be scored at all.*

<img src="gallery/scenario.png" width="860">

`shows:` `Scenario`, `Scenario.label`, `Scenario.screen`, `Trace.render`, `run_scenario`

**An episode becomes a portable, pixel-free trace**

*JSON, so it diffs in review and replays on a machine with nothing installed.*

<img src="gallery/trace-record.png" width="860">

`shows:` `Trace`, `Trace.blocked`, `Trace.choice`, `Trace.decided`, `Trace.decision_step`, `Trace.frames`, `Trace.load`, `Trace.name`, `Trace.record`, `Trace.save`, `Trace.to_dict`, `Trace.to_json`, `TraceStep`, `TraceStep.observation`, `TraceStep.to_dict`

**Replay a recorded trace against a different brain**

*The frames are the fixture; the brain is the only variable. Nothing is captured or clicked.*

<img src="gallery/trace-replay.png" width="860">

`shows:` `Trace.replay`, `Trace.observations`, `score_trace`

**Counting how often the choice was right**

*Scored on normalised text, so OCR spacing is forgiven - the same bargain the index makes.*

<img src="gallery/score-trace.png" width="860">

`shows:` `score_trace`, `score`, `score_replay`, `Verdict`, `Verdict.line`

**Verdicts collected into an accuracy**

*The number the whole module exists for: which brain is right, and how often.*

<img src="gallery/scoreboard.png" width="860">

`shows:` `Scoreboard`, `Scoreboard.accuracy`, `Scoreboard.accuracy_for`, `Scoreboard.add`, `Scoreboard.blocked`, `Scoreboard.brains`, `Scoreboard.by_brain`, `Scoreboard.confusion`, `Scoreboard.earned`, `Scoreboard.render`, `Scoreboard.to_rows`, `Scoreboard.total`

**Every scenario through every brain, in one table**

*A fresh brain per task, so state like 'already clicked this' cannot leak between them.*

<img src="gallery/benchmark.png" width="860">

`shows:` `benchmark`, `Scoreboard.to_rows`

**Replay a whole recorded suite against one brain**

*The literal roadmap step: the frames are fixed, nothing new is captured.*

<img src="gallery/benchmark-traces.png" width="860">

`shows:` `benchmark_traces`, `load_traces`, `Trace.save`

### Command line

**--image example  (replay a screenshot, dry run)**

*No display needed: the whole pipeline over a still image.*

<img src="gallery/cli-image.png" width="860">

`shows:` `main`, `build_parser`, `ImageScreen`, `detector_available`

**--script  (replay a fixed action list instead of a brain)**

*The script is validated exactly like a model's answer: nothing skips the gates.*

<img src="gallery/cli-script.png" width="860">

`shows:` `MockBrain`, `demo_screen`, `parse_action`

**--verify-text  (the harness decides success, not the brain)**

*Without it, an action that removes its own target looks like a failure.*

<img src="gallery/cli-verify-text.png" width="860">

`shows:` `AgentLoop.run`, `Episode.success`

**--trace / --expected  (record the run, then score it)**

*Writes JSON next to the code, and prints the score straight away.*

<img src="gallery/cli-trace.png" width="860">

`shows:` `Trace.record`, `Trace.save`, `score_trace`

**--no-ocr  (index a numbered grid instead)**

*Useful on a machine with no OCR engine, or for a canvas app.*

<img src="gallery/cli-detector.png" width="860">

`shows:` `GridDetector`, `make_detector`, `detector_available`

**Exit codes: 0 success, 1 failure, 2 bad usage, 3 no screen**

*A shell can tell 'it worked' from 'it gave up' without parsing any text.*

<img src="gallery/cli-exit-codes.png" width="860">

`shows:` `main`

**The two deliberate acts before the mouse can move**

*--real-input without the env var, and --brain vlm with no pixels: both refuse.*

<img src="gallery/cli-refusals.png" width="860">

`shows:` `main`, `ALLOW_REAL_INPUT_ENV`

**--live with no display is refused, with the fix**

*It tells you what to do instead of dying inside an episode.*

<img src="gallery/cli-live-headless.png" width="860">

`shows:` `display_available`, `main`

**--live  (your real screen, dry run)**

*Real capture, real OCR, a real element index - and not one click.*

<img src="gallery/cli-live-dry-run.png" width="860">

`shows:` `GrabScreen`, `IndexedScreen`, `AutoDetector`, `main`

### Real desktop

**GrabScreen: capture a real desktop**

*1280x800 of a real X server. mss first, Pillow's ImageGrab as the fallback.*

<img src="gallery/desk-capture.png" width="860">

`shows:` `GrabScreen`, `GrabScreen.capture`, `ScreenUnavailable`

**The element index, drawn on the pixels it came from**

*Every box is a real OCR word with a real screen coordinate: the model picks a label, not a pixel.*

<img src="gallery/desk-ocr-index.png" width="860">

`shows:` `OcrDetector`, `OcrDetector.detect`, `IndexedScreen`, `Element.center`

**A real click, and the window's own reaction**

*Pixels in, a label chosen, a pixel resolved, a button pressed - and the dialog closed itself.*

<img src="gallery/desk-click.png" width="860">

`shows:` `PyAutoGuiExecutor`, `AgentLoop.run`, `Policy.dry_run`, `resolve_point`

**Success only when the harness can see it**

*The app changed its own title and wrote a marker file; the verifier saw the change.*

<img src="gallery/desk-verify.png" width="860">

`shows:` `AgentLoop.run`, `Episode.success`, `Episode.stop_reason`

**A screen with no text still has legal moves**

*OCR finds nothing on a canvas, so the index degrades to numbered cells - and says so.*

<img src="gallery/desk-grid-fallback.png" width="860">

`shows:` `AutoDetector`, `GridDetector`, `OcrDetector`

**The safety gate, on a real button, in a real window**

*The Terminal button is visible and labelled - and the policy refuses to press it.*

<img src="gallery/desk-policy-refusal.png" width="860">

`shows:` `Policy.check`, `Policy.deny_labels`, `PolicyViolation`, `Observation.find`

**The same click, driven end to end from the CLI**

*--live --real-input --verify-text Saved: capture, OCR, click, verify, exit code.*

<img src="gallery/desk-cli-real-input.png" width="860">

`shows:` `main`, `PyAutoGuiExecutor`, `AgentLoop.run`

**Indexed by the accessibility tree, acted on with the real mouse**

*Roles and extents come from GTK itself; the click is a real one and the app reacts.*

<img src="gallery/desk-atspi.png" width="860">

`shows:` `AtspiDetector`, `AtspiDetector.detect`, `AtspiReader`, `IndexedScreen`, `PyAutoGuiExecutor`, `PlannerBrain`

### Extensibility

**Every stage is a Protocol: bring your own screen, brain, detector, executor**

*The loop depends on four one-method contracts, not on the shipped implementations.*

<img src="gallery/protocols.png" width="860">

`shows:` `Brain`, `Brain.decide`, `ScreenSource`, `ScreenSource.capture`, `Detector`, `Detector.detect`, `Executor`, `Executor.execute`, `ActionResult`