| --- |
| title: Capability gallery |
| --- |
| |
| ## Capability gallery |
|
|
| **74 capabilities, one screenshot each - and every one of the 147 public callables in the package (functions, methods, classes and protocols) appears in at least one of them**, together with 6 of its public constants. |
|
|
| Nothing here is a mock-up. Each picture is produced by [`showcase.py`](showcase.py): a command or a call is run for real, and its actual output - or the real desktop it acted on - is what you see. The desktop shots are real screen captures on a real X server (Xvfb, 1280x800), including a real mouse click verified by the target application's own reaction. |
|
|
| > **An honest note on the platform.** These were captured on Linux, because the sandbox that builds this card has no Windows: no KVM, no QEMU and no Windows image. The harness is OS-agnostic (pyautogui uses `SendInput` on Windows, and `GrabScreen` falls back to Pillow's `ImageGrab`), so these are the same code paths - but they are not Windows pixels. The Windows twins are generated by the same script in [`.github/workflows/live-desktop.yml`](.github/workflows/live-desktop.yml). |
|
|
| Regenerate it with `python showcase.py` (`--list` to see the ids, `--only <id>` for a subset, `--card` to rewrite this section). It runs the calls for real, so a broken capability fails loudly instead of shipping a stale picture. |
|
|
| | area | capabilities | |
| |---|---:| |
| | Action vocabulary | 3 | |
| | Decisions | 7 | |
| | Execution | 2 | |
| | The loop | 8 | |
| | Safety | 4 | |
| | Screens | 6 | |
| | Perception | 8 | |
| | Preflight | 4 | |
| | Models | 7 | |
| | Traces & scoring | 7 | |
| | Command line | 9 | |
| | Real desktop | 8 | |
| | Extensibility | 1 | |
|
|
| --- |
|
|
| ### Action vocabulary |
|
|
| **The six verbs, and only six** |
|
|
| *Every action is validated on construction, before anything touches the machine.* |
|
|
| <img src="gallery/vocab-verbs.png" width="860"> |
|
|
| `shows:` `ACTION_KINDS`, `ACTION_MENU`, `Action`, `Action.describe`, `Action.to_dict`, `parse_action` |
|
|
| **Malformed actions are refused, not guessed at** |
|
|
| *An unknown verb or a missing target raises before the machine is touched.* |
|
|
| <img src="gallery/vocab-refusals.png" width="860"> |
|
|
| `shows:` `ActionError`, `Action`, `parse_action` |
|
|
| **The contract handed to a model** |
|
|
| *The prompt-side menu: one JSON object, six shapes, no invented verbs.* |
|
|
| <img src="gallery/vocab-menu.png" width="860"> |
|
|
| `shows:` `ACTION_MENU` |
|
|
| ### Decisions |
|
|
| **A short numbered menu, in reading order** |
|
|
| *A 256M model cannot emit coordinates; it can pick a number from twelve.* |
|
|
| <img src="gallery/menu-numbered.png" width="860"> |
|
|
| `shows:` `DecisionMenu`, `DecisionMenu.render`, `MenuItem`, `MenuItem.label` |
|
|
| **Menu lookup tolerates what models actually send** |
|
|
| *1, "2", 2.0 and "Save." all resolve; a stray word resolves to nothing.* |
|
|
| <img src="gallery/menu-lookup.png" width="860"> |
|
|
| `shows:` `DecisionMenu.item`, `DecisionMenu.find`, `DecisionMenu.mentioned_in` |
|
|
| **Reading a decision out of a small model's reply** |
|
|
| *The LAST JSON object wins, because a model that echoes its prompt answers last.* |
|
|
| <img src="gallery/decision-json.png" width="860"> |
|
|
| `shows:` `parse_decision`, `extract_json_object`, `extract_json_objects`, `DecisionMenu` |
|
|
| **An echoed prompt is refused; a right answer in the wrong field is recovered** |
|
|
| *Real 256M/0.5B failure modes, both caught: no blind click either way.* |
|
|
| <img src="gallery/decision-echo-guard.png" width="860"> |
|
|
| `shows:` `parse_decision`, `BrainError`, `DecisionMenu.mentioned_in` |
|
|
| **A garbled verb with a valid label is still usable** |
|
|
| *A 256M model really does emit {"kind": "file", "label": "Save"}.* |
|
|
| <img src="gallery/decision-garbled.png" width="860"> |
|
|
| `shows:` `parse_decision`, `BrainError` |
|
|
| **MockBrain: replay a fixed script, exactly once** |
|
|
| *Deterministic and offline - the workhorse for tests, demos and replays.* |
|
|
| <img src="gallery/brain-mock.png" width="860"> |
|
|
| `shows:` `MockBrain`, `MockBrain.decide`, `MockBrain.remaining` |
|
|
| **PlannerBrain: no weights, no download, and it gives up honestly** |
|
|
| *The proof the harness works end to end, and a real fallback when no model is wanted.* |
|
|
| <img src="gallery/brain-planner.png" width="860"> |
|
|
| `shows:` `PlannerBrain`, `PlannerBrain.decide`, `GoalUnreachable` |
|
|
| ### Execution |
|
|
| **Dry run resolves every click to pixels - and touches nothing** |
|
|
| *The default everywhere: read a whole episode before anything clicks.* |
|
|
| <img src="gallery/exec-dry-run.png" width="860"> |
|
|
| `shows:` `DryRunExecutor`, `DryRunExecutor.execute`, `resolve_point`, `ActionResult` |
|
|
| **Real input refuses to guess a pixel for a label it cannot find** |
|
|
| *This is the gate between a hallucinated label and 'Delete account'.* |
|
|
| <img src="gallery/exec-refuses-unresolved.png" width="860"> |
|
|
| `shows:` `ActionResult.summary`, `InputUnavailable`, `PyAutoGuiExecutor`, `PyAutoGuiExecutor.execute` |
|
|
| ### The loop |
|
|
| **observe -> think -> act -> verify, traced end to end** |
|
|
| *One call runs a whole episode; every step is kept for audit.* |
|
|
| <img src="gallery/loop-end-to-end.png" width="860"> |
|
|
| `shows:` `AgentLoop`, `AgentLoop.run`, `Episode`, `Episode.render`, `Episode.to_rows`, `Step`, `Step.line`, `Step.ok` |
|
|
| **A hard step budget, so a run cannot wander** |
|
|
| *The budget ends the episode and says so; nothing executes past it.* |
|
|
| <img src="gallery/loop-budget.png" width="860"> |
|
|
| `shows:` `Policy`, `Policy.step_budget`, `AgentLoop.run` |
|
|
| **Repeating one action against an unchanged screen is stopped** |
|
|
| *A stuck agent is not a persistent one - it is stopped and told why.* |
|
|
| <img src="gallery/loop-no-progress-guard.png" width="860"> |
|
|
| `shows:` `AgentLoop.run`, `Step`, `Step.error` |
|
|
| **A kill-switch file aborts before the next action** |
|
|
| *Checked before every single action: create the file, the run stops.* |
|
|
| <img src="gallery/loop-kill-switch.png" width="860"> |
|
|
| `shows:` `Policy.kill_switch_path`, `Policy.should_stop`, `AgentLoop.run` |
|
|
| **A dead screen or a broken brain is an episode, not a traceback** |
|
|
| *Every failure path is recorded as a failed step with a reason.* |
|
|
| <img src="gallery/loop-survives-failures.png" width="860"> |
|
|
| `shows:` `AgentLoop.run`, `Episode.stop_reason`, `GoalUnreachable`, `BrainError` |
|
|
| **Success is evidence, not the model's word** |
|
|
| *Without a verifier, a successful action that removes its own target looks like a failure.* |
|
|
| <img src="gallery/loop-verifier.png" width="860"> |
|
|
| `shows:` `AgentLoop.run`, `Episode.success`, `Observation.text` |
|
|
| **Every step can be streamed out as it happens** |
|
|
| *This is the hook a live console, a progress bar or a websocket uses.* |
|
|
| <img src="gallery/loop-on-step.png" width="860"> |
|
|
| `shows:` `AgentLoop.run`, `Step.line`, `on_step` |
|
|
| **The same loop, one turn at a time, with pause and resume** |
|
|
| *run() is batch; a console needs the caller to own the loop. These are the same rules.* |
|
|
| <img src="gallery/stepwise.png" width="860"> |
|
|
| `shows:` `AgentLoop`, `AgentLoop.begin`, `AgentLoop.step_once`, `AgentLoop.pause`, `AgentLoop.resume`, `AgentLoop.finished`, `AgentLoop.run`, `Step.line` |
|
|
| ### Safety |
|
|
| **Typing from the deny list never reaches the keyboard** |
|
|
| *Checked before execution; a refusal ends the episode and clicks nothing.* |
|
|
| <img src="gallery/safety-deny-text.png" width="860"> |
|
|
| `shows:` `Policy`, `Policy.check`, `PolicyViolation`, `DEFAULT_DENY_TEXT` |
|
|
| **Clicking a denied widget is refused, however the label is spelled** |
|
|
| *Terminal, keychain, wallet, password manager, bank - matched on the label.* |
|
|
| <img src="gallery/safety-deny-label.png" width="860"> |
|
|
| `shows:` `Policy.deny_labels`, `Policy.check`, `Observation.find` |
|
|
| **Key chords on the deny list are never pressed** |
|
|
| *ctrl+alt+t, ctrl+alt+delete and ctrl+shift+escape are refused by default.* |
|
|
| <img src="gallery/safety-deny-keys.png" width="860"> |
|
|
| `shows:` `Policy.deny_keys`, `Policy.check`, `PolicyViolation` |
|
|
| **A numbered grid cell is a blind click: allowed to plan, refused for real** |
|
|
| *With real input the blind guess is refused unless you pass --allow-grid.* |
|
|
| <img src="gallery/safety-blind-grid.png" width="860"> |
|
|
| `shows:` `Policy.allow_grid_clicks`, `Policy.check`, `GridDetector` |
|
|
| ### Screens |
|
|
| **A scripted screen, for testing the loop with no display** |
|
|
| *The final frame repeats forever, so a runaway agent cannot exhaust the script.* |
|
|
| <img src="gallery/screen-fake.png" width="860"> |
|
|
| `shows:` `FakeScreen`, `FakeScreen.capture`, `Observation` |
|
|
| **A still image (or a sequence) replayed as a screen** |
|
|
| *Lets the whole pipeline be exercised with no display, and a failure be replayed.* |
|
|
| <img src="gallery/screen-image.png" width="860"> |
|
|
| `shows:` `ImageScreen`, `ImageScreen.capture`, `fake_desktop` |
|
|
| **Real capture: monitors, downscaling, and HiDPI mapping** |
|
|
| *Frame pixel -> screen pixel is preserved, so a click cannot drift.* |
|
|
| <img src="gallery/capture-options.png" width="860"> |
|
|
| `shows:` `GrabScreen`, `GrabScreen.capture`, `GrabScreen.map_to_screen`, `ScreenUnavailable` |
|
|
| **Labels are compared on their alphanumeric core** |
|
|
| *OCR loses spaces, models add punctuation - so nothing matches by exact string.* |
|
|
| <img src="gallery/element-matching.png" width="860"> |
|
|
| `shows:` `Element`, `Element.center`, `Element.matches`, `Observation.find`, `normalise_label` |
|
|
| **The compact text a model is shown** |
|
|
| *This is the whole world as the brain sees it - capped, and honest when empty.* |
|
|
| <img src="gallery/observation-summary.png" width="860"> |
|
|
| `shows:` `Observation.summary` |
|
|
| **The built-in synthetic desktop** |
|
|
| *Draws a screenshot-like desktop offline - the same image --image example replays.* |
|
|
| <img src="gallery/fake-desktop.png" width="860"> |
|
|
| `shows:` `fake_desktop`, `write_fake_desktop`, `OcrDetector` |
|
|
| ### Perception |
|
|
| **Probing for a detector, and building one by name** |
|
|
| *The doctor and the CLI both ask this before promising anything.* |
|
|
| <img src="gallery/detector-available.png" width="860"> |
|
|
| `shows:` `detector_available`, `make_detector`, `DetectorUnavailable`, `Detector` |
|
|
| **OCR turns a screenshot into labelled, clickable elements** |
|
|
| *The model never has to guess where anything is: this is the grounding.* |
|
|
| <img src="gallery/ocr-labels.png" width="860"> |
|
|
| `shows:` `OcrDetector`, `OcrDetector.detect`, `Element` |
|
|
| **Three knobs keep dialog prose out of the menu** |
|
|
| *Every extra line of noise is one more chance for a small model to click prose.* |
|
|
| <img src="gallery/ocr-filters.png" width="860"> |
|
|
| `shows:` `OcrDetector.min_score`, `OcrDetector.max_label_chars`, `OcrDetector.min_label_chars` |
|
|
| **A numbered grid, for screens with no text at all** |
|
|
| *An agent with an empty index has no legal moves; a grid always has moves.* |
|
|
| <img src="gallery/grid-detector.png" width="860"> |
|
|
| `shows:` `GridDetector`, `GridDetector.detect` |
|
|
| **OCR when there is text, a grid when there is none** |
|
|
| *The default: useful with nothing installed, better the moment OCR appears.* |
|
|
| <img src="gallery/auto-detector.png" width="860"> |
|
|
| `shows:` `AutoDetector`, `AutoDetector.detect`, `AutoDetector.coordinates_of_last_detect` |
|
|
| **IndexedScreen: capture + detect, composed** |
|
|
| *One capture() call gives the loop an index it can act on.* |
|
|
| <img src="gallery/indexed-screen.png" width="860"> |
|
|
| `shows:` `IndexedScreen`, `IndexedScreen.capture`, `Observation.summary` |
|
|
| **The accessibility tree: elements the OS already knows about** |
|
|
| *No pixels are read: GTK publishes every button's name and screen extents.* |
|
|
| <img src="gallery/atspi-index.png" width="860"> |
|
|
| `shows:` `AtspiDetector`, `AtspiDetector.detect`, `AtspiDetector.text`, `AtspiDetector.coordinates_of_last_detect`, `AtspiReader`, `AtspiReader.read`, `AccessibleNode`, `TreeReader`, `TreeReader.read`, `accessibility_available`, `INTERACTIVE_ROLES`, `TEXT_ROLES` |
|
|
| **The Windows backend: the same index, from UI Automation** |
|
|
| *Windows-only, so the mapping is exercised here against a synthetic control tree.* |
|
|
| <img src="gallery/uia-mapping.png" width="860"> |
|
|
| `shows:` `UiaReader`, `UiaReader.read`, `UiaDetector`, `UiaDetector.detect`, `UiaDetector.text`, `UiaDetector.coordinates_of_last_detect` |
|
|
| ### Preflight |
|
|
| **Is there a desktop session to drive at all?** |
|
|
| *The first question the CLI and the doctor both ask.* |
|
|
| <img src="gallery/display-available.png" width="860"> |
|
|
| `shows:` `display_available` |
|
|
| **Every precondition, one list, each with the fix** |
|
|
| *Eight checks; a required one that fails is a blocking problem.* |
|
|
| <img src="gallery/run-checks.png" width="860"> |
|
|
| `shows:` `run_checks`, `Check`, `Check.render` |
|
|
| **python -m tiny_computer_agent --doctor** |
|
|
| *The first thing to run on a new machine. Exits non-zero when not ready.* |
|
|
| <img src="gallery/cli-doctor.png" width="860"> |
|
|
| `shows:` `detector_available`, `display_available`, `doctor`, `report` |
|
|
| **One call wires screen + detector + brain + executor + policy** |
|
|
| *Defaults are the safe ones: dry run, a step budget, and a kill-switch file.* |
|
|
| <img src="gallery/build-agent.png" width="860"> |
|
|
| `shows:` `build_agent`, `Policy`, `AgentLoop` |
|
|
| ### Models |
|
|
| **Which model runtime is present, and are the weights cached?** |
|
|
| *Weights are loaded lazily, so asking is free.* |
|
|
| <img src="gallery/models-deps.png" width="860"> |
|
|
| `shows:` `model_dependencies`, `model_cached`, `MODEL_BRAINS`, `make_model_brain`, `DEFAULT_TEXT_MODEL`, `DEFAULT_VISION_MODEL`, `BrainError` |
|
|
| **Device and dtype selection** |
|
|
| *CUDA if it is there, CPU otherwise; the harness never needs a GPU.* |
|
|
| <img src="gallery/models-device.png" width="860"> |
|
|
| `shows:` `resolve_device`, `pick_dtype` |
|
|
| **The prompt a model actually receives - and how it learns from a dead end** |
|
|
| *Placeholders, not worked examples: a weak model echoes the prompt, and an echo fails validation.* |
|
|
| <img src="gallery/models-prompt.png" width="860"> |
|
|
| `shows:` `ModelBrain`, `ModelBrain.build_prompt`, `ModelBrain.decide` |
|
|
| **A second decision is told what already failed** |
|
|
| *'you already tried this and the screen did NOT change' is the most useful line a small model gets.* |
|
|
| <img src="gallery/models-prompt-history.png" width="860"> |
|
|
| `shows:` `ModelBrain`, `ModelBrain.build_prompt`, `ModelBrain.history` |
|
|
| **The two real model backends, and lazy loading** |
|
|
| *Constructing a brain costs nothing; the weights arrive on the first decide().* |
|
|
| <img src="gallery/models-swap.png" width="860"> |
|
|
| `shows:` `TextPlannerBrain`, `VLMBrain`, `TextPlannerBrain.unload`, `VLMBrain.unload`, `ModelBrain.loaded`, `make_model_brain` |
|
|
| **Constrained decoding: make an echo impossible, not merely detected** |
|
|
| *The 256M model's failure was answering with the prompt. A mask removes that option.* |
|
|
| <img src="gallery/decoding.png" width="860"> |
|
|
| `shows:` `MenuMask`, `digit_token_ids`, `choice_token_ids`, `pair_token_ids`, `read_choice`, `read_verdict`, `brevity_ladder`, `majority_choice`, `DIGITS` |
|
|
| **Four ways to ask a tiny model, and what each one costs** |
|
|
| *json=free text, choice=one masked number, scan=yes/no per item, rank=log-odds order.* |
|
|
| <img src="gallery/models-decision-modes.png" width="860"> |
|
|
| `shows:` `ModelBrain`, `ModelBrain.loaded`, `TextPlannerBrain`, `VLMBrain`, `make_model_brain` |
|
|
| ### Traces & scoring |
|
|
| **A task with one right answer, and the frames to show** |
|
|
| *Ground truth lives with the task, so a run can be scored at all.* |
|
|
| <img src="gallery/scenario.png" width="860"> |
|
|
| `shows:` `Scenario`, `Scenario.label`, `Scenario.screen`, `Trace.render`, `run_scenario` |
|
|
| **An episode becomes a portable, pixel-free trace** |
|
|
| *JSON, so it diffs in review and replays on a machine with nothing installed.* |
|
|
| <img src="gallery/trace-record.png" width="860"> |
|
|
| `shows:` `Trace`, `Trace.blocked`, `Trace.choice`, `Trace.decided`, `Trace.decision_step`, `Trace.frames`, `Trace.load`, `Trace.name`, `Trace.record`, `Trace.save`, `Trace.to_dict`, `Trace.to_json`, `TraceStep`, `TraceStep.observation`, `TraceStep.to_dict` |
|
|
| **Replay a recorded trace against a different brain** |
|
|
| *The frames are the fixture; the brain is the only variable. Nothing is captured or clicked.* |
|
|
| <img src="gallery/trace-replay.png" width="860"> |
|
|
| `shows:` `Trace.replay`, `Trace.observations`, `score_trace` |
|
|
| **Counting how often the choice was right** |
|
|
| *Scored on normalised text, so OCR spacing is forgiven - the same bargain the index makes.* |
|
|
| <img src="gallery/score-trace.png" width="860"> |
|
|
| `shows:` `score_trace`, `score`, `score_replay`, `Verdict`, `Verdict.line` |
|
|
| **Verdicts collected into an accuracy** |
|
|
| *The number the whole module exists for: which brain is right, and how often.* |
|
|
| <img src="gallery/scoreboard.png" width="860"> |
|
|
| `shows:` `Scoreboard`, `Scoreboard.accuracy`, `Scoreboard.accuracy_for`, `Scoreboard.add`, `Scoreboard.blocked`, `Scoreboard.brains`, `Scoreboard.by_brain`, `Scoreboard.confusion`, `Scoreboard.earned`, `Scoreboard.render`, `Scoreboard.to_rows`, `Scoreboard.total` |
|
|
| **Every scenario through every brain, in one table** |
|
|
| *A fresh brain per task, so state like 'already clicked this' cannot leak between them.* |
|
|
| <img src="gallery/benchmark.png" width="860"> |
|
|
| `shows:` `benchmark`, `Scoreboard.to_rows` |
|
|
| **Replay a whole recorded suite against one brain** |
|
|
| *The literal roadmap step: the frames are fixed, nothing new is captured.* |
|
|
| <img src="gallery/benchmark-traces.png" width="860"> |
|
|
| `shows:` `benchmark_traces`, `load_traces`, `Trace.save` |
|
|
| ### Command line |
|
|
| **--image example (replay a screenshot, dry run)** |
|
|
| *No display needed: the whole pipeline over a still image.* |
|
|
| <img src="gallery/cli-image.png" width="860"> |
|
|
| `shows:` `main`, `build_parser`, `ImageScreen`, `detector_available` |
|
|
| **--script (replay a fixed action list instead of a brain)** |
|
|
| *The script is validated exactly like a model's answer: nothing skips the gates.* |
|
|
| <img src="gallery/cli-script.png" width="860"> |
|
|
| `shows:` `MockBrain`, `demo_screen`, `parse_action` |
|
|
| **--verify-text (the harness decides success, not the brain)** |
|
|
| *Without it, an action that removes its own target looks like a failure.* |
|
|
| <img src="gallery/cli-verify-text.png" width="860"> |
|
|
| `shows:` `AgentLoop.run`, `Episode.success` |
|
|
| **--trace / --expected (record the run, then score it)** |
|
|
| *Writes JSON next to the code, and prints the score straight away.* |
|
|
| <img src="gallery/cli-trace.png" width="860"> |
|
|
| `shows:` `Trace.record`, `Trace.save`, `score_trace` |
|
|
| **--no-ocr (index a numbered grid instead)** |
|
|
| *Useful on a machine with no OCR engine, or for a canvas app.* |
|
|
| <img src="gallery/cli-detector.png" width="860"> |
|
|
| `shows:` `GridDetector`, `make_detector`, `detector_available` |
|
|
| **Exit codes: 0 success, 1 failure, 2 bad usage, 3 no screen** |
|
|
| *A shell can tell 'it worked' from 'it gave up' without parsing any text.* |
|
|
| <img src="gallery/cli-exit-codes.png" width="860"> |
|
|
| `shows:` `main` |
|
|
| **The two deliberate acts before the mouse can move** |
|
|
| *--real-input without the env var, and --brain vlm with no pixels: both refuse.* |
|
|
| <img src="gallery/cli-refusals.png" width="860"> |
|
|
| `shows:` `main`, `ALLOW_REAL_INPUT_ENV` |
|
|
| **--live with no display is refused, with the fix** |
|
|
| *It tells you what to do instead of dying inside an episode.* |
|
|
| <img src="gallery/cli-live-headless.png" width="860"> |
|
|
| `shows:` `display_available`, `main` |
|
|
| **--live (your real screen, dry run)** |
|
|
| *Real capture, real OCR, a real element index - and not one click.* |
|
|
| <img src="gallery/cli-live-dry-run.png" width="860"> |
|
|
| `shows:` `GrabScreen`, `IndexedScreen`, `AutoDetector`, `main` |
|
|
| ### Real desktop |
|
|
| **GrabScreen: capture a real desktop** |
|
|
| *1280x800 of a real X server. mss first, Pillow's ImageGrab as the fallback.* |
|
|
| <img src="gallery/desk-capture.png" width="860"> |
|
|
| `shows:` `GrabScreen`, `GrabScreen.capture`, `ScreenUnavailable` |
|
|
| **The element index, drawn on the pixels it came from** |
|
|
| *Every box is a real OCR word with a real screen coordinate: the model picks a label, not a pixel.* |
|
|
| <img src="gallery/desk-ocr-index.png" width="860"> |
|
|
| `shows:` `OcrDetector`, `OcrDetector.detect`, `IndexedScreen`, `Element.center` |
|
|
| **A real click, and the window's own reaction** |
|
|
| *Pixels in, a label chosen, a pixel resolved, a button pressed - and the dialog closed itself.* |
|
|
| <img src="gallery/desk-click.png" width="860"> |
|
|
| `shows:` `PyAutoGuiExecutor`, `AgentLoop.run`, `Policy.dry_run`, `resolve_point` |
|
|
| **Success only when the harness can see it** |
|
|
| *The app changed its own title and wrote a marker file; the verifier saw the change.* |
|
|
| <img src="gallery/desk-verify.png" width="860"> |
|
|
| `shows:` `AgentLoop.run`, `Episode.success`, `Episode.stop_reason` |
|
|
| **A screen with no text still has legal moves** |
|
|
| *OCR finds nothing on a canvas, so the index degrades to numbered cells - and says so.* |
|
|
| <img src="gallery/desk-grid-fallback.png" width="860"> |
|
|
| `shows:` `AutoDetector`, `GridDetector`, `OcrDetector` |
|
|
| **The safety gate, on a real button, in a real window** |
|
|
| *The Terminal button is visible and labelled - and the policy refuses to press it.* |
|
|
| <img src="gallery/desk-policy-refusal.png" width="860"> |
|
|
| `shows:` `Policy.check`, `Policy.deny_labels`, `PolicyViolation`, `Observation.find` |
|
|
| **The same click, driven end to end from the CLI** |
|
|
| *--live --real-input --verify-text Saved: capture, OCR, click, verify, exit code.* |
|
|
| <img src="gallery/desk-cli-real-input.png" width="860"> |
|
|
| `shows:` `main`, `PyAutoGuiExecutor`, `AgentLoop.run` |
|
|
| **Indexed by the accessibility tree, acted on with the real mouse** |
|
|
| *Roles and extents come from GTK itself; the click is a real one and the app reacts.* |
|
|
| <img src="gallery/desk-atspi.png" width="860"> |
|
|
| `shows:` `AtspiDetector`, `AtspiDetector.detect`, `AtspiReader`, `IndexedScreen`, `PyAutoGuiExecutor`, `PlannerBrain` |
|
|
| ### Extensibility |
|
|
| **Every stage is a Protocol: bring your own screen, brain, detector, executor** |
|
|
| *The loop depends on four one-method contracts, not on the shipped implementations.* |
|
|
| <img src="gallery/protocols.png" width="860"> |
|
|
| `shows:` `Brain`, `Brain.decide`, `ScreenSource`, `ScreenSource.capture`, `Detector`, `Detector.detect`, `Executor`, `Executor.execute`, `ActionResult` |
|
|
|
|
