2.0: skills, approval gates, a resumable task runner, structural verification

README.md

@@ -22,9 +22,13 @@ tags:
 No cloud API, no API keys, no telemetry — and a core that imports nothing outside the
 Python standard library.
 
-> **Status 0.3.0: it drives a real desktop — on Linux and on Windows.** Screen capture,
-> an OCR element index with real coordinates, a real mouse click, and a *verified*
-> outcome. **All 74 capabilities have a real screenshot** in the [capability gallery](#capability-gallery) (Linux captures).
+> **Status 2.0.0: it drives a real desktop — on Linux and on Windows — and it learns.**
+> Screen capture, an OCR element index with real coordinates, a real mouse click, and a
+> *verified* outcome. A verified run can be written down as a **skill** and replayed with no
+> model at all; a queue of goals can be run, resumed after a kill, and reported on; an
+> **approval gate** can stand in front of every action; and what changed on screen is a
+> **measured** structural diff rather than a guess.
+> **All 74 capabilities have a real screenshot** in the [capability gallery](#capability-gallery) (Linux captures).
 > The loop can be driven from a [marimo console](#the-marimo-console) with a live preview, an
 > action log and pause / resume / step; elements can come from the
 > [accessibility tree](#what-the-machine-already-knows-the-accessibility-tree) instead of OCR;
@@ -90,7 +94,7 @@ X server. First the suite, headless:
 ```bash
 python -m pip install -e ".[desktop,ocr,dev]"
 pytest tests -q
-# 227 passed, 2 skipped in 18.7s
+# 227 passed, 2 skipped in 18.7s          (that run was 0.3.0; 2.0 has grown this to 284)
 ```
 
 The two skips are the live-desktop tests. Give them a display and they run for real:
@@ -262,7 +266,7 @@ open problem: it is reliable on a clean screenshot and unreliable on a busy live
   `warning: no text was found on screen, so the index is 96 numbered grid cells and the
   agent is choosing blind` — instead of silently guessing.
 
-**229 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+**284 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
 that click a real button.
 
 ## Verified on real Windows
@@ -310,7 +314,7 @@ the harness itself, installed into site-packages
 
 None of the three failures in the first row is a harness defect: two are
 `ModuleNotFoundError: numpy` and the third is the guard bug documented below. With the
-OCR engine installed, the same suite exits **0**. (Those Windows counts predate `trace.py`; the harness suite has since grown to 229 tests.)
+OCR engine installed, the same suite exits **0**. (Those Windows counts predate `trace.py` and 2.0; the harness suite has since grown to 284 tests.)
 
 **The emulated-OCR caveat.** onnxruntime requires AVX2, and QEMU's software emulation of
 AVX2 is roughly two orders of magnitude slower than native. A suite that finishes in 17
@@ -336,9 +340,10 @@ exactly the kind of thing that only surfaces on a machine without an OCR engine.
 
 ## The test suite, in full
 
-`pytest tests` collects **229 tests**. 227 run headless — no display, no network, no
+`pytest tests` collects **284 tests**. 282 run headless — no display, no network, no
 weights. The last 2 are the opt-in live-desktop tests, skipped unless
-`TINY_COMPUTER_AGENT_LIVE_TEST=1`.
+`TINY_COMPUTER_AGENT_LIVE_TEST=1`. On a box without PyGObject, two of the 282 *fail* rather
+than skip: they assert the AT-SPI reader is reachable, and that is the honest outcome.
 
 | File | Tests | What it proves |
 |---|---:|---|
@@ -352,19 +357,24 @@ weights. The last 2 are the opt-in live-desktop tests, skipped unless
 | `tests/test_safety.py` | 11 | deny lists, the kill switch, the blind-grid-click guard |
 | `tests/test_grid_fallback.py` | 11 | the fallback index announces itself instead of guessing |
 | `tests/test_desktop.py` | 10 | display detection, the preflight checks, `doctor` |
-| `tests/test_cli.py` | 10 | the command line: `--doctor`, `--image`, `--live`, argument handling |
+| `tests/test_cli.py` | 16 | the command line: `--doctor`, `--image`, `--live`, and the 2.0 `skills` / `batch` subcommands |
 | `tests/test_trace.py` | 28 | traces: a run recorded with no pixels, replayed against another brain, and scored against ground truth |
 | `tests/test_models.py` | 7 | the model backends: lazy loading, device choice, honest refusals |
 | `tests/test_api_surface.py` | 4 | the public API is a promise: everything advertised is importable, and nothing public is left out |
 | `tests/test_decoding.py` | 24 | the mask, the answer space, the retry ladder, the vote - and that an echo is refused |
-| `tests/test_accessibility.py` | 16 | role filtering, the coordinate contract, the fallback chain, and real AT-SPI |
+| `tests/test_accessibility.py` | 26 | role filtering, the coordinate contract, the fallback chain, and real AT-SPI |
+| `tests/test_diff.py` | 8 | what changed between two frames: added, removed, moved, and the verifier that remembers |
 | `tests/test_stepwise.py` | 11 | begin / step / pause / resume, and that it equals the batch loop |
 | `tests/test_ocr_guard.py` | 4 | the OCR guard really skips with no engine - the guard that never fired |
+| `tests/test_gates.py` | 8 | the operator's yes: whitelists, ask-once gates, and a refusal that is a step |
+| `tests/test_skills.py` | 17 | only a verified run becomes a skill; re-grounding by label; aiming at nothing raises |
+| `tests/test_runner.py` | 10 | a queue, a JSONL resume file, and a report counts what was verified |
 | `tests/test_live_desktop.py` | 2 | **needs a display, opt-in**: OCR finds a real button at the pixels it was drawn at, and a real click makes the app react |
 
 ```bash
-pytest tests -q                                  # 227 passed, 2 skipped
-TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 229 passed, on a desktop
+pytest tests -q                                  # 274 passed, 8 skipped, 2 failed only where
+                                                 # PyGObject/AT-SPI is missing (gir1.2-atspi-2.0)
+TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # + the 2 live-desktop tests, on a desktop
 ```
 
 ## Install
@@ -373,7 +383,7 @@ TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 229 passed, on a desktop
 pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
 pip install -e ".[desktop,ocr]"      # + real capture and real input
 pip install -e ".[vision,ocr]"       # + the tiny model brains
-pytest                               # 229 tests, no display or network needed
+pytest                               # 284 tests, no display or network needed
 ```
 
 The accessibility detector needs system packages rather than an extra:
@@ -485,6 +495,10 @@ this: an answer file with non-canonical XML namespaces is **silently ignored** b
 | `trace.py` | Portable traces: record a run with no pixels, replay it offline, score the choice against ground truth |
 | `decoding.py` | Constrained decoding: mask the logits to a menu number or a yes/no, so an echo is impossible |
 | `accessibility.py` | The accessibility tree: `AtspiReader` and `AtspiDetector`, roles and real extents |
+| `diff.py` | What changed between two frames: `Delta` (added / removed / moved) and `ChangeVerifier`, so "the dialog closed" is checkable |
+| `skills.py` | **2.0** - Skills: a *verified* run written down once, replayed by `SkillBrain` with no model. Labels, never coordinates |
+| `gates.py` | **2.0** - Approval gates: `AllowOnlyLabels`, `InteractiveGate` (yes / no / always / never), `AllOf` / `AnyOf`. A refusal is a step |
+| `runner.py` | **2.0** - `TaskRunner`: a queue of goals, a JSONL resume file, and a `RunReport` that counts what was *verified* |
 | `showcase.py` | The capability gallery: one real screenshot per capability, checked against the public API |
 | `measure_vlm.py` | Measures the four ways to ask a model, on a real screen, and prints the table |
 | `evaluate_daily.py` | A day's tasks against a real app, verified by the app's own log |
@@ -591,7 +605,7 @@ eleven tasks are a sample, not a survey.
 
 ## Capability gallery
 
-**74 capabilities, one screenshot each - and every one of the 141 public callables in the package (functions, methods, classes and protocols) appears in at least one of them**, together with 6 of its public constants.
+**74 capabilities, one screenshot each - and 173 of the package's 180 pre-2.0 public callables (functions, methods, classes and protocols) appear in at least one of them**, together with 10 of its public constants. The 2.0 additions are not pictured yet: **84 new public callables** (skills, approval gates, the batch runner and the structural diff) plus four new CLI entry points. The counts are read off `gallery/index.json`, which records what each picture shows.
 
 Nothing here is a mock-up. Each picture is produced by [`showcase.py`](showcase.py): a command or a call is run for real, and its actual output - or the real desktop it acted on - is what you see. The desktop shots are real screen captures on a real X server (Xvfb, 1280x800), including a real mouse click verified by the target application's own reaction.
 
@@ -1218,7 +1232,8 @@ success.
 
 **Works:** real capture, a real OCR index with coordinates that land on the right pixel,
 a real click an app reacts to, verified success, the closed vocabulary, the policy gates,
-the no-progress guard, echo detection, the doctor, the CLI, portable traces that score a brain's choices offline, 164 tests, a 256M VLM that
+the no-progress guard, echo detection, the doctor, the CLI, portable traces that score a brain's choices offline, skills learned from verified runs,
+a resumable task runner, operator approval gates, 284 tests, a 256M VLM that
 reads a button label from pixels on a clean screenshot — and the same suite passing inside
 a real Windows Server 2022 guest with no KVM (109 passed, exit 0, with the OCR-inference
 tests excluded as impractically slow under emulation).
@@ -1247,6 +1262,11 @@ when no verifier is given.
 - [x] A daily-usage evaluation: 11 tasks (menu, scroll, typing, shortcut, second window,
       verify, refuse, kill switch) against a real GTK app, checked against the application's
       own log - see [can it do a day's work](#can-it-do-a-days-work)
+- [x] **2.0** Skills: a verified run is written down once and replayed with no model at all
+- [x] **2.0** Operator approval gates - a label whitelist, or ask before every action
+- [x] **2.0** A resumable task queue with a report, and structural verification of "it worked"
+- [x] **2.0** `skills` and `batch` subcommands, and an example queue at `tasks/day.json`
+- [ ] A gallery run that pictures the 2.0 additions: the 84 new public callables have no screenshot yet
 - [ ] **Verify UI Automation** (Windows) against a real Windows desktop. The reader and the
       detector are implemented and unit-tested against a synthetic control tree, and
       `accessibility_available("uia")` reports the truth on any machine - but no Windows

pyproject.toml

@@ -5,8 +5,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tiny-computer-agent"
-version = "0.3.0"
-description = "An agentic harness that drives a real computer with a tiny, fully local model - no cloud API."
+version = "2.0.0"
+description = "An agentic harness that drives a real computer with a tiny, fully local model - no cloud API. It learns from verified runs."
 readme = "README.md"
 requires-python = ">=3.10"
 license = { text = "MIT" }

tasks/day.json

@@ -0,0 +1,12 @@
+[
+  {
+    "goal": "click 'Save' to save the document",
+    "name": "save",
+    "verify_gone": "Name"
+  },
+  {
+    "goal": "click 'Files' to open the file manager",
+    "name": "open-files",
+    "verify_text": "untitled"
+  }
+]

tests/test_api_surface.py

@@ -19,7 +19,9 @@ def test_all_is_sorted_and_free_of_duplicates():
 def test_no_public_callable_in_a_module_is_missing_from_all():
     """A capability a user cannot import is a capability they cannot use."""
     modules = ["actions", "act", "observe", "perception", "brain", "models", "loop",
-               "safety", "desktop", "demo", "trace"]
+               "safety", "desktop", "demo", "trace",
+               # 2.0: the operator layer, all of which must stay importable from the root
+               "diff", "gates", "runner", "skills"]
     unexported = []
     for name in modules:
         module = importlib.import_module(f"tiny_computer_agent.{name}")

tests/test_cli.py

@@ -1,3 +1,5 @@
+import json
+
 import pytest
 
 from tiny_computer_agent import __main__ as cli
@@ -69,3 +71,85 @@ def test_a_missing_image_is_reported_not_crashed(capsys):
 def test_vlm_brain_requires_a_live_screen(capsys):
     assert cli.main(["--brain", "vlm"]) == 2
     assert "needs --live" in capsys.readouterr().err
+
+
+# --- 2.0: the subcommands, which are how a run becomes a habit --------------- #
+
+def test_skills_list_on_an_empty_library_says_how_to_start(tmp_path, capsys):
+    assert cli.main(["skills", "list", "--path", str(tmp_path / "none.json")]) == 0
+    assert "no skills" in capsys.readouterr().out
+
+
+def test_learning_refuses_without_evidence(tmp_path, capsys):
+    code = cli.main(["skills", "learn", "click 'Save'",
+                     "--script", '[{"kind": "click", "label": "Save"}]',
+                     "--path", str(tmp_path / "skills.json")])
+    assert code == 2
+    assert "refusing to learn" in capsys.readouterr().err
+
+
+def test_learn_then_list_then_show_then_remove(tmp_path, capsys):
+    path = str(tmp_path / "skills.json")
+    script = '[{"kind": "click", "label": "Files"}, {"kind": "click", "label": "Save"}]'
+
+    assert cli.main(["skills", "learn", "click 'Save' to save the document",
+                     "--script", script, "--verify-gone", "Name", "--path", path]) == 0
+    assert "learned 'save-document'" in capsys.readouterr().out
+
+    assert cli.main(["skills", "list", "--path", path]) == 0
+    out = capsys.readouterr().out
+    assert "save-document" in out and "verified" in out
+
+    assert cli.main(["skills", "show", "save-document", "--path", path]) == 0
+    assert "'Name' goes away" in capsys.readouterr().out
+
+    assert cli.main(["skills", "remove", "save-document", "--path", path]) == 0
+    assert "removed" in capsys.readouterr().out
+    assert cli.main(["skills", "show", "save-document", "--path", path]) == 1
+
+
+def test_a_learned_skill_can_be_replayed_by_the_brain(tmp_path):
+    """The point of the whole feature: the second run needs no script."""
+    path = tmp_path / "skills.json"
+    script = '[{"kind": "click", "label": "Files"}, {"kind": "click", "label": "Save"}]'
+    assert cli.main(["skills", "learn", "click 'Save' to save the document",
+                     "--script", script, "--verify-gone", "Name", "--path", str(path)]) == 0
+
+    from tiny_computer_agent import SkillBrain, SkillLibrary
+    from tiny_computer_agent.__main__ import demo_screen
+    from tiny_computer_agent.loop import AgentLoop
+
+    library = SkillLibrary.load(path)
+    loop = AgentLoop(screen=demo_screen(), brain=SkillBrain(library))
+    episode = loop.run("click 'Save' to save the document")
+
+    assert [s.action.label for s in episode.steps if s.action and s.action.kind == "click"] \
+        == ["Files", "Save"]
+
+
+def test_batch_runs_a_queue_and_exits_on_its_report(tmp_path, capsys):
+    tasks = tmp_path / "tasks.json"
+    tasks.write_text(json.dumps([{"goal": "click 'Save' to save the document",
+                                  "verify_gone": "Name"},
+                                 {"goal": "click 'Files' to open the file manager",
+                                  "verify_text": "untitled"}]))
+    state = tmp_path / "state.jsonl"
+    report = tmp_path / "report.json"
+
+    assert cli.main(["batch", str(tasks), "--state", str(state), "--report", str(report)]) == 0
+    out = capsys.readouterr().out
+    assert "2/2 verified" in out
+    assert json.loads(report.read_text())["passed"] == 2
+
+    assert cli.main(["batch", str(tasks), "--state", str(state)]) == 0
+    assert "already run" in capsys.readouterr().out
+
+
+def test_batch_rejects_tasks_that_are_not_tasks(tmp_path, capsys):
+    bad = tmp_path / "bad.json"
+    bad.write_text(json.dumps([{"nothing": "here"}]))
+    assert cli.main(["batch", str(bad)]) == 2
+    assert "needs at least a 'goal'" in capsys.readouterr().err
+    missing = tmp_path / "absent.json"
+    assert cli.main(["batch", str(missing)]) == 2
+    assert "cannot read" in capsys.readouterr().err

tests/test_desktop.py

@@ -92,3 +92,30 @@ def test_build_agent_wires_the_kill_switch(tmp_path):
                        kill_switch_path=str(marker))
     assert loop.policy.kill_switch_path == str(marker)
     assert DEFAULT_KILL_SWITCH.endswith("tiny-computer-agent.STOP")
+
+
+def test_the_one_call_builder_can_wire_learned_skills_and_a_gate():
+    """2.0: one call wires the desktop, what it has learned, and the operator's gate."""
+    from tiny_computer_agent import (AllowOnlyLabels, Element, FakeScreen, Policy, Skill,
+                                     SkillBrain, SkillLibrary, SkillStep, Action, build_agent)
+
+    library = SkillLibrary()
+    library.add(Skill(name="save", goal="click 'Save' to save the document", verified=True,
+                      steps=(SkillStep(Action("click", label="Save"), aimed_at="Save"),)))
+    gate = AllowOnlyLabels(["Save"])
+
+    loop = build_agent(brain="planner", screen=FakeScreen([(Element("Save", x=10, y=10, width=60, height=20),)]),
+                       detector="ocr", dry_run=True, skills=library, gate=gate)
+
+    assert isinstance(loop.brain, SkillBrain)
+    assert loop.gate is gate
+    episode = loop.run("click 'Save' to save the document")
+    assert [s.action.label for s in episode.steps if s.action and s.action.kind == "click"] == ["Save"]
+
+
+def test_without_skills_the_builder_leaves_the_brain_alone():
+    from tiny_computer_agent import Element, FakeScreen, PlannerBrain, build_agent
+
+    loop = build_agent(brain="planner", screen=FakeScreen([(Element("Save", x=1, y=1, width=9, height=9),)]),
+                       detector="ocr")
+    assert isinstance(loop.brain, PlannerBrain) and loop.gate is None

tests/test_diff.py

@@ -0,0 +1,91 @@
+"""A change you cannot measure is a change you cannot verify."""
+
+from tiny_computer_agent import ChangeVerifier, Delta, Element, Observation, diff, element_key
+
+
+def frame(elements, text=""):
+    return Observation(index=0, frame="frame", elements=tuple(elements), text=text)
+
+
+def button(label, x=10, y=10, width=60, height=20, kind="button"):
+    return Element(label, kind=kind, x=x, y=y, width=width, height=height)
+
+
+def test_nothing_changed_is_measured_rather_than_guessed():
+    picture = frame([button("Save", x=100)])
+    delta = diff(picture, picture)
+
+    assert delta.empty and not delta.changed
+    assert delta.summary() == "no change"
+    assert isinstance(delta, Delta)
+
+
+def test_a_dialog_closing_is_a_removed_element_not_a_missing_string():
+    """The whole point: verification without depending on the dialog's own words."""
+    before = frame([button("Save As"), button("Cancel", x=80)])
+    after = frame([button("Save As")])
+    delta = diff(before, after)
+
+    assert delta.closed and not delta.opened
+    assert [e.label for e in delta.removed] == ["Cancel"]
+    assert delta.matches(disappeared=["cancel"])
+    assert not delta.matches(appeared=["cancel"])
+
+
+def test_a_menu_opening_is_an_added_element():
+    delta = diff(frame([button("File")]), frame([button("File"), button("Save As", y=60)]))
+
+    assert delta.opened and not delta.closed
+    assert [e.label for e in delta.added] == ["Save As"]
+    assert delta.matches(appeared=["save as"])
+
+
+def test_a_redraw_is_not_movement_and_a_real_move_is():
+    still = diff(frame([button("Save", x=100)]), frame([button("Save", x=101)]))
+    assert still.empty, "a one-pixel re-render is not a change"
+
+    moved = diff(frame([button("Save", x=100)]), frame([button("Save", x=400)]))
+    assert moved.moved and moved.settled, "nothing appeared or vanished: it moved"
+    assert not moved.changed is False
+    assert moved.to_dict()["moved"] == ["Save"]
+
+
+def test_labels_are_compared_the_way_the_index_compares_them():
+    """OCR drops spaces and models add punctuation, so identity is the alphanumeric core."""
+    before = frame([button("Save As")])
+    after = frame([button("SaveAs.")])
+    delta = diff(before, after)
+
+    assert delta.empty, "the same label, differently spelled, is the same element"
+    assert element_key(button("Save As")) == element_key(button("SaveAs."))
+
+
+def test_added_and_removed_are_told_apart_when_counts_differ():
+    before = frame([button("Row 1"), button("Row 2")])
+    after = frame([button("Row 2"), button("Row 3"), button("Row 4")])
+    delta = diff(before, after)
+
+    assert sorted(e.label for e in delta.added) == ["Row 3", "Row 4"]
+    assert [e.label for e in delta.removed] == ["Row 1"]
+    assert delta.summary().startswith("+2")
+
+
+def test_the_verifier_remembers_the_previous_frame():
+    verify = ChangeVerifier(disappeared=["Cancel"])
+    with_cancel = frame([button("Save"), button("Cancel", x=200)])
+    without_cancel = frame([button("Save")])
+
+    assert verify(with_cancel) is False, "the first frame has nothing to compare against"
+    assert verify(without_cancel) is True
+    assert verify.last is not None and verify.last.closed, "Cancel is gone: the dialog closed"
+    assert verify(without_cancel) is False, "and the comparison advances with the frame"
+    assert verify.last is not None and verify.last.empty
+
+
+def test_the_verifier_can_be_reset_between_episodes():
+    verify = ChangeVerifier(appeared=["Saved"])
+    verify(frame([button("Save")]))
+    verify.reset()
+
+    assert verify.previous is None and verify.last is None
+    assert verify(frame([button("Saved")])) is False

tests/test_gates.py

@@ -0,0 +1,97 @@
+"""A refusal from the operator is a step, not a traceback."""
+
+from tiny_computer_agent import (
+    Action, AgentLoop, AllOf, AllowAll, AllowOnlyLabels, AnyOf, Approval, DenyAll, Element,
+    FakeScreen, InteractiveGate, MockBrain, Policy, mutating,
+)
+
+
+def survey():
+    answers = ["y", "n", "a", "v"]
+    asked = []
+
+    def reader(question: str) -> str:
+        asked.append(question)
+        return answers.pop(0) if answers else ""
+
+    return reader, asked
+
+
+def test_read_only_verbs_never_need_consent():
+    assert not mutating(Action("wait", seconds=1))
+    assert not mutating(Action("done", reason="finished"))
+    assert mutating(Action("click", label="Save"))
+    assert mutating(Action("type", text="hello"))
+
+
+def test_deny_all_kills_a_live_loop_but_lets_it_wait():
+    gate = DenyAll()
+    assert gate.approve(Action("wait", seconds=1)).approved
+    refused = gate.approve(Action("click", label="Save"))
+    assert refused.approved is False and "said no" in refused.reason
+
+
+def test_the_whitelist_gate_allows_a_renamed_button_and_refuses_the_rest():
+    gate = AllowOnlyLabels(["Save", "Cancel"])
+
+    assert gate.approve(Action("click", label="Save")).approved
+    assert gate.approve(Action("click", label="Save.")).approved, "the index's comparison applies"
+    assert not gate.approve(Action("click", label="Terminal")).approved
+    assert gate.approve(Action("type", text="rm -rf /")).approved is False, "text is not whitelisted"
+
+
+def test_the_interactive_gate_asks_and_remembers():
+    reader, asked = survey()
+    gate = InteractiveGate(reader=reader, writer=lambda line: None)
+
+    assert gate.approve(Action("click", label="Save")).approved          # y
+    assert not gate.approve(Action("click", label="Save")).approved      # n
+    assert gate.approve(Action("click", label="Save")).approved          # a  -> remembered
+    assert gate.approve(Action("click", label="Save")).approved          # ... so not asked again
+    assert len(asked) == 3
+    assert gate.approve(Action("wait", seconds=1)).approved, "and it is never asked about a wait"
+
+
+def test_the_interactive_gate_remembers_a_never():
+    reader, _ = survey()
+    gate = InteractiveGate(reader=reader, writer=lambda line: None)
+    gate.approve(Action("click", label="Save"))     # y
+    gate.approve(Action("click", label="Save"))     # n
+    gate.approve(Action("click", label="Save"))     # a
+    assert gate.approve(Action("click", label="Save")).approved
+    gate._always = False                            # a fresh session would ask again
+    answers = ["v", ""]
+    gate.reader = lambda question: answers.pop(0)
+    assert not gate.approve(Action("click", label="Save")).approved
+    assert not gate.approve(Action("click", label="Save")).approved, "never sticks"
+
+
+def test_an_absent_operator_is_a_refusal_not_a_hang():
+    def no_one(question: str) -> str:
+        raise EOFError("no stdin")
+
+    assert not InteractiveGate(reader=no_one).approve(Action("click", label="Save")).approved
+
+
+def test_all_of_is_strict_and_any_of_is_permissive():
+    allowed = AllowOnlyLabels(["Save"])
+    assert not AllOf([allowed, DenyAll()]).approve(Action("click", label="Save")).approved
+    assert AllOf([allowed, AllowAll()]).approve(Action("click", label="Save")).approved
+    assert AnyOf([DenyAll(), allowed]).approve(Action("click", label="Save")).approved
+    assert not AnyOf([DenyAll()]).approve(Action("click", label="Save")).approved
+    assert isinstance(AllowAll().approve(Action("click", label="Save")), Approval)
+
+
+def test_a_gate_refusal_stops_the_episode_and_is_recorded():
+    screen = FakeScreen([(Element("Save", x=10, y=10, width=60, height=20),)])
+    brain = MockBrain([{"kind": "click", "label": "Save"}, {"kind": "done", "reason": "saved"}])
+    loop = AgentLoop(screen=screen, brain=brain, policy=Policy(step_budget=4),
+                     gate=AllowOnlyLabels(["Cancel"]))
+
+    episode = loop.run("click 'Save' to save the document")
+
+    assert episode.success is False
+    assert "operator refused" in episode.stop_reason
+    assert episode.steps[0].error is not None and episode.steps[0].error.startswith("not approved")
+    assert episode.steps[0].result is None, "nothing was executed"
+    assert len(episode.steps) == 1, "the episode ended there"

tests/test_runner.py

@@ -0,0 +1,143 @@
+"""A queue that survives being killed, and a report that counts what was verified."""
+
+import json
+
+from tiny_computer_agent import (
+    AgentLoop, DryRunExecutor, Element, FakeScreen, MockBrain, Policy, RunReport, Task,
+    TaskOutcome, TaskRunner,
+)
+
+
+def element(label="Save", x=10, y=10):
+    return Element(label, x=x, y=y, width=60, height=20)
+
+
+def build_loop(task):
+    """Fresh loop per task, exactly as the runner's contract requires."""
+    brain = MockBrain([{"kind": "click", "label": "Save"}])
+    return AgentLoop(screen=FakeScreen([(element(),)], text="Saved"), brain=brain,
+                     executor=DryRunExecutor(), policy=Policy(step_budget=2))
+
+
+def test_a_task_names_itself_and_carries_the_harness_check():
+    task = Task("click 'Save' to save the document", verify_text="Saved")
+
+    assert task.name == "save-document"
+    assert task.line().startswith("save-document")
+    verify = task.verifier()
+    assert verify is not None and verify.__call__ is not None
+    assert verify(type("O", (), {"text": "Saved"})()) is True
+    assert Task("no check here").verifier() is None
+    assert Task("explicit", name="mine").name == "mine"
+
+
+def test_a_verified_task_is_reported_as_a_success():
+    runner = TaskRunner(build_loop)
+    report = runner.run([Task("click 'Save' to save the document", verify_text="Saved")])
+
+    assert isinstance(report, RunReport)
+    assert report.total == 1 and report.passed == 1 and report.accuracy == 1.0
+    assert report.outcomes[0].success is True
+    assert report.outcomes[0].steps >= 1
+    assert "verified" in report.render() or "1/1" in report.render()
+
+
+def test_a_task_that_never_verifies_is_a_failure_not_a_guess():
+    report = TaskRunner(build_loop).run([Task("click 'Save'", verify_text="NeverAppears", budget=1)])
+
+    assert report.failed == 1 and report.accuracy == 0.0
+    assert report.outcomes[0].success is False
+    assert "budget" in report.outcomes[0].stop_reason
+
+
+def test_a_broken_loop_is_an_outcome_not_a_traceback():
+    def exploding(task):
+        raise RuntimeError("no display here")
+
+    outcome = TaskRunner(exploding).run_one(Task("anything"))
+
+    assert outcome.success is False
+    assert outcome.error == "RuntimeError: no display here"
+    assert "caught" in outcome.stop_reason
+
+
+def test_the_state_file_lets_a_killed_run_be_resumed(tmp_path):
+    state = tmp_path / "runs" / "today.jsonl"
+    tasks = [Task("click 'Save'", name="first", verify_text="Saved"),
+             Task("click 'Cancel'", name="second", verify_text="Saved")]
+    first = TaskRunner(build_loop, state_path=state).run(tasks)
+
+    assert first.total == 2 and first.passed == 2
+    lines = [json.loads(line) for line in state.read_text().splitlines()]
+    assert [row["name"] for row in lines] == ["first", "second"]
+
+    seen = []
+    def tracks(task):
+        seen.append(task.name)
+        return build_loop(task)
+
+    second = TaskRunner(tracks, state_path=state).run(tasks)
+    assert seen == [], "a resumed run does not repeat finished work"
+    assert second.total == 2 and second.passed == 2
+    assert all("already run" in o.stop_reason for o in second.outcomes)
+
+
+def test_a_half_written_state_line_does_not_break_a_resume(tmp_path):
+    state = tmp_path / "today.jsonl"
+    state.write_text(json.dumps(TaskOutcome("done", "g", True, 1, "verified", 0.5).to_dict()) + "\n"
+                     + '{"name": "torn", "suc')
+    runner = TaskRunner(build_loop, state_path=state)
+    completed = runner.completed()
+
+    assert set(completed) == {"done"}, "a killed run leaves a torn line, and it is ignored"
+    report = runner.run([Task("g", name="done"), Task("click 'Save'", name="fresh", verify_text="Saved")])
+    assert report.passed == 2
+
+
+def test_the_report_saves_and_lists(tmp_path):
+    report = TaskRunner(build_loop).run([Task("click 'Save'", verify_text="Saved")])
+    path = report.save(tmp_path / "report.json")
+    saved = json.loads(path.read_text())
+
+    assert saved["passed"] == 1 and saved["total"] == 1
+    assert saved["outcomes"][0]["name"] == "save"
+    assert set(report.to_rows()[0]) == {"task", "goal", "success", "steps", "seconds", "reason"}
+    assert "report" in report.render() and report.steps >= 1
+
+
+def test_the_runner_tells_the_caller_about_each_task_as_it_finishes():
+    heard = []
+    runner = TaskRunner(build_loop, on_task=heard.append)
+    runner.run([Task("click 'Save'", verify_text="Saved")])
+
+    assert len(heard) == 1 and heard[0].name == "save"
+
+
+def test_a_task_can_be_verified_by_an_element_going_away():
+    """The check for a task whose success is a dialog closing, where text says nothing."""
+    task = Task("click 'Save' to save the document", verify_gone="Name")
+    verify = task.verifier()
+
+    assert "'Name' to go" in task.line()
+    assert verify is not None
+    assert verify.__call__ is not None
+
+    from tiny_computer_agent import Element, Observation
+
+    dialog = Observation(elements=(Element("Name", kind="input", x=10, y=10),
+                                   Element("Save", x=90, y=10)))
+    closed = Observation(elements=(Element("Save", x=90, y=10),))
+    assert verify(dialog) is False, "the first frame only establishes the baseline"
+    assert verify(closed) is True
+
+
+def test_a_task_can_be_verified_by_an_element_arriving():
+    task = Task("open the File menu", verify_appeared="Save As")
+    verify = task.verifier()
+    from tiny_computer_agent import Element, Observation
+
+    menu = Observation(elements=(Element("File", x=0, y=0),
+                                 Element("Save As", x=0, y=30)))
+    assert verify(Observation(elements=(Element("File", x=0, y=0),))) is False
+    assert verify(menu) is True
+    assert task.to_dict()["verify_appeared"] == "Save As"

tests/test_skills.py

@@ -0,0 +1,205 @@
+"""A skill is only as trustworthy as the run that produced it."""
+
+import pytest
+
+from tiny_computer_agent import (
+    Action, AgentLoop, BrainError, DryRunExecutor, Element, Episode, FakeScreen, GoalUnreachable,
+    MockBrain, Observation, Policy, Skill, SkillBrain, SkillLibrary, SkillStep, Step,
+    record_skill, slug,
+)
+
+
+def save_button(label="Save", x=10, y=10):
+    return Element(label, x=x, y=y, width=60, height=20)
+
+
+def screen_of(*labels, text=""):
+    return FakeScreen([tuple(save_button(l, x=10 + 80 * i) for i, l in enumerate(labels))], text=text)
+
+
+def verified_click_episode(goal="click 'Save' to save the document", label="Save"):
+    """A real, verified episode - produced by a real loop, not hand-built."""
+    loop = AgentLoop(screen=screen_of(label, text="Saved"),
+                     brain=MockBrain([{"kind": "click", "label": label},
+                                      {"kind": "done", "reason": "clicked it"}]),
+                     verify=lambda observation: "Saved" in observation.text)
+    return loop.run(goal)
+
+
+def test_slugs_are_readable_and_stable():
+    assert slug("click 'Save' to save the document") == "save-document"
+    assert slug("open the File menu and choose Save As") == "open-file-menu-choose-save-as"
+    assert slug("") == "skill"
+
+
+def test_an_unverified_run_is_never_written_down_as_a_skill():
+    """The one thing this project does not do: turn a guess into a library others trust."""
+    episode = verified_click_episode()
+    episode.success = False
+    assert record_skill(episode) is None
+
+    empty = Episode(goal="do something", success=True)
+    assert record_skill(empty) is None, "no steps is not a skill"
+
+
+def test_a_verified_run_becomes_a_reusable_skill():
+    skill = record_skill(verified_click_episode())
+
+    assert skill is not None
+    assert skill.verified and skill.name == "save-document"
+    assert len(skill.steps) == 1
+    assert skill.steps[0].action.kind == "click"
+    assert skill.steps[0].aimed_at == "Save"
+    assert skill.recorded, "a skill says when it was recorded"
+
+
+def test_a_skill_stores_labels_and_never_coordinates():
+    """Coordinates are the thing that stops working tomorrow."""
+    skill = record_skill(verified_click_episode())
+    action = skill.actions()[0]
+
+    assert action.label == "Save"
+    assert action.x is None and action.y is None
+    assert "Save" in skill.steps[0].describe()
+
+
+def test_the_entry_condition_is_the_first_label_not_every_label():
+    skill = Skill(name="menu", goal="open the File menu and choose Save As", verified=True,
+                  steps=(SkillStep(Action("click", label="File"), aimed_at="File"),
+                         SkillStep(Action("click", label="Save As"), aimed_at="Save As")))
+
+    assert skill.preconditions() == ("File",), "Save As does not exist until the menu is open"
+
+
+def test_a_library_matches_only_when_the_entry_label_is_on_screen():
+    library = SkillLibrary()
+    library.add(Skill(name="save", goal="click 'Save' to save the document", verified=True,
+                      steps=(SkillStep(Action("click", label="Save"), aimed_at="Save"),)))
+
+    goal = "click 'Save' to save the document"
+    assert library.match(goal, Observation(elements=(save_button(),))) is not None
+    assert library.match(goal, Observation(elements=())) is None, "aim at nothing, match nothing"
+    assert library.match("format the hard disk", Observation(elements=(save_button(),))) is None
+
+
+def test_a_library_round_trips_through_json(tmp_path):
+    library = SkillLibrary()
+    record_skill(verified_click_episode(), library=library)
+    path = library.save(tmp_path / "skills.json")
+
+    reloaded = SkillLibrary.load(path)
+    assert reloaded.names() == library.names()
+    skill = reloaded.get("save-document")
+    assert skill is not None and skill.steps[0].action.kind == "click"
+    assert skill.verified is True
+    assert SkillLibrary.load(tmp_path / "absent.json").names() == ()
+    assert reloaded.to_rows()[0]["needs"] == "Save"
+
+
+def test_the_brain_regrounds_on_the_label_the_screen_shows_now():
+    library = SkillLibrary()
+    library.add(Skill(name="save", goal="click 'Save' to save the document", verified=True,
+                      steps=(SkillStep(Action("click", label="Save"), aimed_at="Save"),)))
+    brain = SkillBrain(library)
+
+    action = brain.decide("click 'Save' to save the document",
+                          Observation(elements=(save_button("Save."),)))
+    assert action.kind == "click"
+    assert action.label == "Save.", "the screen's spelling wins over the recorded one"
+
+
+def test_a_skill_that_aimed_at_nothing_raises_instead_of_clicking():
+    library = SkillLibrary()
+    library.add(Skill(name="two", goal="save then confirm", verified=True,
+                      steps=(SkillStep(Action("click", label="Save"), aimed_at="Save"),
+                             SkillStep(Action("click", label="Confirm"), aimed_at="Confirm"))))
+    brain = SkillBrain(library)
+    goal = "save then confirm"
+
+    assert brain.decide(goal, Observation(elements=(save_button(),))).label == "Save"
+    with pytest.raises(BrainError, match="not on this screen"):
+        brain.decide(goal, Observation(elements=(save_button(),)))
+
+
+def test_the_brain_says_done_when_the_skill_is_finished():
+    library = SkillLibrary()
+    library.add(Skill(name="one", goal="click 'Save' to save the document", verified=True,
+                      steps=(SkillStep(Action("click", label="Save"), aimed_at="Save"),)))
+    brain = SkillBrain(library)
+    goal = "click 'Save' to save the document"
+    brain.decide(goal, Observation(elements=(save_button(),)))
+
+    finished = brain.decide(goal, Observation(elements=(save_button(),)))
+    assert finished.kind == "done" and "finished" in finished.reason
+    assert brain.progress == "1/1"
+
+
+def test_the_brain_gives_up_instead_of_inventing_an_action():
+    brain = SkillBrain(SkillLibrary())
+    with pytest.raises(GoalUnreachable, match="no skill matches"):
+        brain.decide("format the hard disk", Observation(elements=(save_button(),)))
+
+
+def test_the_brain_defers_to_a_fallback_then_reuses_the_choice():
+    library = SkillLibrary()
+    brain = SkillBrain(library, fallback=MockBrain([{"kind": "done", "reason": "fallback chose"}]))
+    action = brain.decide("click 'Save'", Observation(elements=(save_button(),)))
+
+    assert action.kind == "done" and "fallback" in action.reason
+
+
+def test_a_skill_replays_through_a_real_loop_with_no_model():
+    library = SkillLibrary()
+    record_skill(verified_click_episode(), library=library)
+    goal = "click 'Save' to save the document"
+
+    loop = AgentLoop(screen=screen_of("Save", text="Saved"), brain=SkillBrain(library),
+                     executor=DryRunExecutor(), policy=Policy(step_budget=3),
+                     verify=lambda observation: "Saved" in observation.text)
+    episode = loop.run(goal)
+
+    assert episode.success, episode.render()
+    assert [s.action.label for s in episode.steps if s.action and s.action.kind == "click"] == ["Save"]
+
+
+def test_a_skill_carries_the_check_that_made_it_trustworthy():
+    skill = Skill(name="s", goal="g", verified=True, verify_text="Saved",
+                  steps=(SkillStep(Action("click", label="Save"), aimed_at="Save"),))
+    verify = skill.verifier()
+
+    assert verify is not None and verify(Observation(text="the file was Saved")) is True
+    assert verify(Observation(text="nothing happened")) is False
+    assert Skill(name="s", goal="g", steps=skill.steps).verifier() is None
+
+
+def test_an_empty_skill_is_not_a_skill():
+    with pytest.raises(ValueError, match="no steps"):
+        Skill(name="empty", goal="nothing", steps=())
+
+
+def test_a_skill_keeps_a_structural_check_across_a_round_trip(tmp_path):
+    """A skill whose success is a dialog closing must survive being written down."""
+    episode = verified_click_episode()
+    library = SkillLibrary()
+    skill = record_skill(episode, library=library, verify_gone="Name")
+
+    assert skill is not None and skill.verified
+    assert skill.verification() == "'Name' goes away"
+    verify = skill.verifier()
+    assert verify is not None and callable(verify)
+
+    reloaded = SkillLibrary.load(library.save(tmp_path / "skills.json")).get(skill.name)
+    assert reloaded is not None
+    assert reloaded.verify_gone == "Name"
+    assert reloaded.verification() == "'Name' goes away"
+
+
+def test_recording_refuses_a_claim_that_no_verifier_backed():
+    """A brain reporting 'done' is a claim; only evidence makes a skill."""
+    from tiny_computer_agent import AgentLoop, MockBrain
+
+    loop = AgentLoop(screen=screen_of("Save", text=""), brain=MockBrain([{"kind": "done", "reason": "trust me"}]))
+    episode = loop.run("click 'Save' to save the document")
+
+    assert episode.success, "the loop takes the brain's word when there is no verifier"
+    assert record_skill(episode) is None, "but a skill library does not"

tiny_computer_agent/__init__.py

@@ -107,7 +107,30 @@ from tiny_computer_agent.perception import (
     detector_available,
     make_detector,
 )
+from tiny_computer_agent.diff import ChangeVerifier, Delta, diff, element_key
+from tiny_computer_agent.gates import (
+    READ_ONLY_KINDS,
+    AllOf,
+    AllowAll,
+    AllowOnlyLabels,
+    AnyOf,
+    Approval,
+    DenyAll,
+    Gate,
+    InteractiveGate,
+    mutating,
+)
+from tiny_computer_agent.runner import RunReport, Task, TaskOutcome, TaskRunner
 from tiny_computer_agent.safety import Policy, PolicyViolation
+from tiny_computer_agent.skills import (
+    DEFAULT_SKILLS_PATH,
+    Skill,
+    SkillBrain,
+    SkillLibrary,
+    SkillStep,
+    record_skill,
+    slug,
+)
 from tiny_computer_agent.trace import (
     Scenario,
     Scoreboard,
@@ -123,28 +146,35 @@ from tiny_computer_agent.trace import (
     score_trace,
 )
 
-__version__ = "0.3.0"
+__version__ = "2.0.0"
 
 __all__ = [
-    "ACTION_KINDS", "ACTION_MENU", "AccessibleNode", "Action",
-    "ActionError", "ActionResult", "AgentLoop", "AtspiDetector",
-    "AtspiReader", "AutoDetector", "BRAIN_NAMES", "Brain",
-    "BrainError", "Check", "DEFAULT_KILL_SWITCH", "DEFAULT_TEXT_MODEL",
-    "DEFAULT_VISION_MODEL", "DecisionMenu", "Detector", "DetectorUnavailable",
-    "DryRunExecutor", "Element", "Episode", "Executor",
-    "FakeScreen", "GoalUnreachable", "GrabScreen", "GridDetector",
-    "INTERACTIVE_ROLES", "ImageScreen", "IndexedScreen", "InputUnavailable",
-    "MODEL_BRAINS", "MenuItem", "MockBrain", "ModelBrain",
-    "Observation", "OcrDetector", "PlannerBrain", "Policy",
-    "PolicyViolation", "PyAutoGuiExecutor", "Scenario", "Scoreboard",
-    "ScreenSource", "ScreenUnavailable", "Step", "TEXT_ROLES",
-    "TextPlannerBrain", "Trace", "TraceStep", "TreeReader",
-    "UiaDetector", "UiaReader", "VLMBrain", "Verdict",
-    "__version__", "accessibility_available", "benchmark", "benchmark_traces",
-    "build_agent", "detector_available", "display_available", "doctor",
-    "extract_json_object", "extract_json_objects", "fake_desktop", "load_traces",
-    "make_detector", "make_model_brain", "model_cached", "model_dependencies",
-    "normalise_label", "parse_action", "parse_decision", "pick_dtype",
-    "resolve_device", "resolve_point", "run_checks", "run_scenario",
-    "score", "score_replay", "score_trace", "write_fake_desktop",
+    "ACTION_KINDS",             "ACTION_MENU",              "AccessibleNode",           "Action",
+    "ActionError",              "ActionResult",             "AgentLoop",                "AllOf",
+    "AllowAll",                 "AllowOnlyLabels",          "AnyOf",                    "Approval",
+    "AtspiDetector",            "AtspiReader",              "AutoDetector",             "BRAIN_NAMES",
+    "Brain",                    "BrainError",               "ChangeVerifier",           "Check",
+    "DEFAULT_KILL_SWITCH",      "DEFAULT_SKILLS_PATH",      "DEFAULT_TEXT_MODEL",       "DEFAULT_VISION_MODEL",
+    "DecisionMenu",             "Delta",                    "DenyAll",                  "Detector",
+    "DetectorUnavailable",      "DryRunExecutor",           "Element",                  "Episode",
+    "Executor",                 "FakeScreen",               "Gate",                     "GoalUnreachable",
+    "GrabScreen",               "GridDetector",             "INTERACTIVE_ROLES",        "ImageScreen",
+    "IndexedScreen",            "InputUnavailable",         "InteractiveGate",          "MODEL_BRAINS",
+    "MenuItem",                 "MockBrain",                "ModelBrain",               "Observation",
+    "OcrDetector",              "PlannerBrain",             "Policy",                   "PolicyViolation",
+    "PyAutoGuiExecutor",        "READ_ONLY_KINDS",          "RunReport",                "Scenario",
+    "Scoreboard",               "ScreenSource",             "ScreenUnavailable",        "Skill",
+    "SkillBrain",               "SkillLibrary",             "SkillStep",                "Step",
+    "TEXT_ROLES",               "Task",                     "TaskOutcome",              "TaskRunner",
+    "TextPlannerBrain",         "Trace",                    "TraceStep",                "TreeReader",
+    "UiaDetector",              "UiaReader",                "VLMBrain",                 "Verdict",
+    "__version__",              "accessibility_available",  "benchmark",                "benchmark_traces",
+    "build_agent",              "detector_available",       "diff",                     "display_available",
+    "doctor",                   "element_key",              "extract_json_object",      "extract_json_objects",
+    "fake_desktop",             "load_traces",              "make_detector",            "make_model_brain",
+    "model_cached",             "model_dependencies",       "mutating",                 "normalise_label",
+    "parse_action",             "parse_decision",           "pick_dtype",               "record_skill",
+    "resolve_device",           "resolve_point",            "run_checks",               "run_scenario",
+    "score",                    "score_replay",             "score_trace",              "slug",
+    "write_fake_desktop",
 ]

tiny_computer_agent/__main__.py

@@ -7,6 +7,12 @@
     python -m tiny_computer_agent --live              # your real screen, dry-run
     python -m tiny_computer_agent --live --real-input "click 'OK'"
 
+2.0 adds two subcommands, for everything after that first run:
+
+    python -m tiny_computer_agent skills list                  # what has been learned
+    python -m tiny_computer_agent skills learn "click 'Save'" --script '[{"kind":"click","label":"Save"}]' --verify-text Saved
+    python -m tiny_computer_agent batch tasks/day.json          # a queue, resumable, reported
+
 Real mouse and keyboard input additionally requires
 ``TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT=1``. Dry run is the default, so you can read a
 whole episode before anything touches your pointer.
@@ -21,6 +27,7 @@ import platform
 import sys
 
 from tiny_computer_agent.act import DryRunExecutor, PyAutoGuiExecutor
+from tiny_computer_agent.diff import ChangeVerifier
 from tiny_computer_agent.brain import MockBrain, PlannerBrain
 from tiny_computer_agent.demo import fake_desktop
 from tiny_computer_agent.desktop import (
@@ -32,6 +39,8 @@ from tiny_computer_agent.observe import (
 )
 from tiny_computer_agent.perception import IndexedScreen, detector_available, make_detector
 from tiny_computer_agent.safety import Policy
+from tiny_computer_agent.runner import Task, TaskRunner
+from tiny_computer_agent.skills import DEFAULT_SKILLS_PATH, SkillBrain, SkillLibrary, record_skill
 from tiny_computer_agent.trace import Trace, score_trace
 
 ALLOW_REAL_INPUT_ENV = "TINY_COMPUTER_AGENT_ALLOW_REAL_INPUT"
@@ -144,7 +153,195 @@ def _build_screen(args: argparse.Namespace):
     return demo_screen(), "scripted demo"
 
 
+SUBCOMMANDS = ("skills", "batch")
+
+
+def _screen_arguments(parser: argparse.ArgumentParser) -> None:
+    """The screen flags the subcommands share with the main parser."""
+    parser.add_argument("--image", default=None,
+                        help=f"replay a screenshot; '{EXAMPLE_IMAGE}' uses the built-in one")
+    parser.add_argument("--live", action="store_true", help="use this machine's real screen")
+    parser.add_argument("--detector", default="auto",
+                        choices=("auto", "best", "atspi", "ocr", "grid"))
+    parser.add_argument("--no-ocr", action="store_true", help="index a numbered grid instead")
+    parser.add_argument("--monitor", type=int, default=1)
+    parser.add_argument("--scale", type=float, default=None)
+    parser.add_argument("--max-width", type=int, default=1600)
+    parser.add_argument("--path", default=str(DEFAULT_SKILLS_PATH),
+                        help="the skill library to read or write")
+
+
+def build_skills_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="tiny-computer-agent skills",
+        description="Skills are verified runs, written down once, replayed forever.")
+    actions = parser.add_subparsers(dest="action", required=True)
+    listing = actions.add_parser("list", help="what the library holds")
+    show = actions.add_parser("show", help="one skill, step by step")
+    show.add_argument("name")
+    remove = actions.add_parser("remove", help="forget a skill")
+    remove.add_argument("name")
+    learn = actions.add_parser("learn", help="run a script once, and keep it if it verifies")
+    learn.add_argument("goal")
+    learn.add_argument("--script", required=True, help="JSON list of actions to perform once")
+    learn.add_argument("--verify-text", default=None, metavar="TEXT",
+                       help="what 'it worked' looks like on screen: some text appears")
+    learn.add_argument("--verify-gone", default=None, metavar="LABEL",
+                       help="or: this element must disappear (the dialog closed)")
+    learn.add_argument("--verify-appeared", default=None, metavar="LABEL",
+                       help="or: this element must appear (the menu opened)")
+    learn.add_argument("--name", default=None, help="name the skill yourself")
+    _screen_arguments(learn)                      # which also carries --path
+    for read_only in (listing, show, remove):     # these only ever need the library
+        read_only.add_argument("--path", default=str(DEFAULT_SKILLS_PATH),
+                               help="the skill library to read or write")
+    return parser
+
+
+def skills_main(argv: list[str]) -> int:
+    args = build_skills_parser().parse_args(argv)
+    library = SkillLibrary.load(args.path)
+    if args.action == "list":
+        if not library:
+            print(f"no skills in {args.path} yet. Learn one with:\n"
+                  f"  python -m tiny_computer_agent skills learn \"click 'Save'\" "
+                  f"--script '[{{\"kind\": \"click\", \"label\": \"Save\"}}]' --verify-text Saved")
+            return 0
+        rows = library.to_rows()
+        widths = {key: max(len(key), *(len(str(row[key])) for row in rows)) for key in rows[0]}
+        print(f"{len(rows)} skill(s) in {args.path}")
+        print("  " + "  ".join(key.ljust(widths[key]) for key in rows[0]))
+        for row in rows:
+            print("  " + "  ".join(str(row[key]).ljust(widths[key]) for key in row))
+        return 0
+    if args.action == "show":
+        skill = library.get(args.name)
+        if skill is None:
+            print(f"no skill named {args.name!r} in {args.path}", file=sys.stderr)
+            return 1
+        print(skill.line())
+        print(f"  verify: {skill.verification()}")
+        print(f"  entry:  {', '.join(skill.preconditions()) or '(none - no label to re-ground)'}")
+        for index, step in enumerate(skill.steps, 1):
+            print(f"  {index}. {step.describe()}")
+        return 0
+    if args.action == "remove":
+        library = SkillLibrary.load(args.path)
+        if not library.remove(args.name):
+            print(f"no skill named {args.name!r} in {args.path}", file=sys.stderr)
+            return 1
+        print(f"removed {args.name!r}; {len(library)} left in {library.save()}")
+        return 0
+
+    # learn: the first run is scripted, and only a verified one becomes a skill
+    if not (args.verify_text or args.verify_gone or args.verify_appeared):
+        print("refusing to learn: pass --verify-text (or --verify-gone / --verify-appeared) so "
+              "the run has evidence. A skill recorded from a brain's own 'done' would be a "
+              "guess with a name.", file=sys.stderr)
+        return 2
+    try:
+        screen, description = _build_screen(args)
+    except ScreenUnavailable as exc:
+        print(f"cannot see the screen: {exc}", file=sys.stderr)
+        return 3
+    try:
+        script = json.loads(args.script)
+    except json.JSONDecodeError as exc:
+        print(f"--script is not valid JSON: {exc}", file=sys.stderr)
+        return 2
+    _, index_detail = detector_available("grid" if args.no_ocr else args.detector)
+    print(f"learning {args.goal!r} on {description} ({index_detail})")
+    if args.verify_text:
+        needle = args.verify_text.lower()
+        verify: object = lambda observation: needle in observation.text.lower()
+    else:
+        verify = ChangeVerifier(appeared=(args.verify_appeared,) if args.verify_appeared else (),
+                                disappeared=(args.verify_gone,) if args.verify_gone else ())
+    loop = AgentLoop(screen=screen, brain=MockBrain(script), executor=DryRunExecutor(),
+                     policy=Policy(dry_run=True, step_budget=len(script) + 2),
+                     on_step=lambda step: print("  " + step.line()),
+                     verify=verify)
+    episode = loop.run(args.goal)
+    print()
+    print(episode.render())
+    skill = record_skill(episode, name=args.name, library=library, verify_text=args.verify_text or "",
+                         verify_gone=args.verify_gone or "",
+                         verify_appeared=args.verify_appeared or "")
+    if skill is None:
+        print("\nnot learned: the run did not end verified, so there is nothing worth keeping.",
+              file=sys.stderr)
+        return 1
+    print(f"\nlearned {skill.name!r} ({len(skill.steps)} step(s)) -> {library.save()}")
+    return 0
+
+
+def build_batch_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="tiny-computer-agent batch",
+        description="Run a queue of goals, resuming what already finished, and report on it.")
+    parser.add_argument("tasks", help="JSON: a list of {goal, name?, verify_text?, budget?}")
+    parser.add_argument("--state", default=None,
+                        help="JSONL file appended as each task finishes (resume from it)")
+    parser.add_argument("--report", default=None, help="write the report as JSON here")
+    parser.add_argument("--max-steps", type=int, default=6)
+    parser.add_argument("--real-input", action="store_true",
+                        help="actually drive the mouse (requires the allow env var)")
+    parser.add_argument("--brain", default="planner", choices=BRAIN_NAMES)
+    _screen_arguments(parser)
+    return parser
+
+
+def batch_main(argv: list[str]) -> int:
+    args = build_batch_parser().parse_args(argv)
+    try:
+        payload = json.loads(open(args.tasks).read())
+    except (OSError, json.JSONDecodeError) as exc:
+        print(f"cannot read {args.tasks}: {exc}", file=sys.stderr)
+        return 2
+    if not isinstance(payload, list) or not payload:
+        print(f"{args.tasks} should hold a non-empty list of tasks", file=sys.stderr)
+        return 2
+    try:
+        from_args = [Task(goal=entry["goal"], name=entry.get("name", ""),
+                          budget=entry.get("budget"), verify_text=entry.get("verify_text", ""),
+                          verify_gone=entry.get("verify_gone", ""),
+                          verify_appeared=entry.get("verify_appeared", ""))
+                     for entry in payload]
+    except (TypeError, KeyError) as exc:
+        print(f"a task needs at least a 'goal': {exc}", file=sys.stderr)
+        return 2
+
+    if args.real_input and os.environ.get(ALLOW_REAL_INPUT_ENV) != "1":
+        print(f"refusing --real-input: set {ALLOW_REAL_INPUT_ENV}=1 first.", file=sys.stderr)
+        return 2
+    try:
+        screen, description = _build_screen(args)
+    except ScreenUnavailable as exc:
+        print(f"cannot see the screen: {exc}", file=sys.stderr)
+        return 3
+
+    def make_loop(task: Task) -> AgentLoop:
+        from tiny_computer_agent.models import make_model_brain
+        brain = PlannerBrain() if args.brain == "planner" else make_model_brain(args.brain)
+        return AgentLoop(screen=screen, brain=brain,
+                         executor=PyAutoGuiExecutor() if args.real_input else DryRunExecutor(),
+                         policy=Policy(dry_run=not args.real_input, step_budget=task.budget or args.max_steps))
+
+    print(f"batch: {len(from_args)} task(s) on {description}, "
+          f"{'REAL INPUT' if args.real_input else 'dry-run'}")
+    report = TaskRunner(make_loop, state_path=args.state,
+                        on_task=lambda outcome: print("  " + outcome.line())).run(from_args)
+    print()
+    print(report.render())
+    if args.report:
+        print(f"report: wrote {report.save(args.report)}")
+    return 0 if report.failed == 0 else 1
+
+
 def main(argv: list[str] | None = None) -> int:
+    argv = list(sys.argv[1:] if argv is None else argv)
+    if argv and argv[0] in SUBCOMMANDS:
+        return skills_main(argv[1:]) if argv[0] == "skills" else batch_main(argv[1:])
     args = build_parser().parse_args(argv)
     detector_name = "grid" if args.no_ocr else args.detector
 

tiny_computer_agent/desktop.py

@@ -21,7 +21,9 @@ from tiny_computer_agent.observe import GrabScreen, ScreenSource
 from tiny_computer_agent.perception import (
     Detector, IndexedScreen, detector_available, make_detector,
 )
+from tiny_computer_agent.gates import Gate
 from tiny_computer_agent.safety import Policy
+from tiny_computer_agent.skills import SkillBrain, SkillLibrary
 
 DEFAULT_KILL_SWITCH = "/tmp/tiny-computer-agent.STOP"
 BRAIN_NAMES = ("planner", "text-planner", "vlm")
@@ -170,12 +172,18 @@ def build_agent(*, brain: Brain | str | None = None, screen: ScreenSource | None
                 max_steps: int = 10, kill_switch_path: str | None = DEFAULT_KILL_SWITCH,
                 allow_grid_clicks: bool = False,
                 on_step: Callable[[Step], None] | None = None,
-                detector_options: dict[str, Any] | None = None) -> AgentLoop:
+                detector_options: dict[str, Any] | None = None,
+                skills: SkillLibrary | None = None,
+                gate: Gate | None = None) -> AgentLoop:
     """Wire a real desktop into an :class:`AgentLoop` in one call.
 
     Defaults are the safe ones: dry run, a step budget of 10, and a kill-switch file
     on disk that aborts the next action if it appears. ``screen``/``detector``/
     ``brain`` can be injected, which is how the tests drive this without a desktop.
+
+    Pass ``skills`` and the brain becomes a :class:`~tiny_computer_agent.skills.SkillBrain`
+    that answers from the library first and only falls back to the model when no learned
+    skill applies; pass ``gate`` and the operator stands in front of every action.
     """
     if detector is None or isinstance(detector, str):
         detector = make_detector(detector or "ocr", **(detector_options or {}))
@@ -189,9 +197,11 @@ def build_agent(*, brain: Brain | str | None = None, screen: ScreenSource | None
             from tiny_computer_agent.models import make_model_brain
 
             brain = make_model_brain(name)
+    if skills is not None:
+        brain = SkillBrain(skills, fallback=brain)
     executor: Executor = DryRunExecutor() if dry_run else PyAutoGuiExecutor()
     policy = Policy(dry_run=dry_run, step_budget=max_steps,
                     kill_switch_path=kill_switch_path,
                     allow_grid_clicks=allow_grid_clicks)
     return AgentLoop(screen=screen, brain=brain, executor=executor, policy=policy,
-                     on_step=on_step)
+                     on_step=on_step, gate=gate)

tiny_computer_agent/diff.py

@@ -0,0 +1,154 @@
+"""What changed between two observations - in elements, not pixels.
+
+The harness already refuses to guess and insists on evidence. This is the missing half of
+that habit: a structural way to say *what changed* on a screen, so a verifier can be written
+without hard-coding a string that only today's dialog happens to contain, and so "nothing
+changed" is a measurement rather than a hunch.
+
+    before = screen.capture()
+    ... an action ...
+    delta = diff(before, screen.capture())
+    if delta.closed:            # the dialog went away
+
+    verify = ChangeVerifier(disappeared=["Save As"])   # AgentLoop(verify=verify)
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from math import hypot
+from typing import Any, Iterable
+
+from .observe import Element, Observation, normalise_label
+
+DEFAULT_TOLERANCE = 3
+"""Pixels. Below this a box has not really moved, it has been re-rendered."""
+
+
+def element_key(element: Element) -> str:
+    """Identity for matching across two frames: the label as the harness compares it, plus kind."""
+    return f"{normalise_label(element.label)}|{element.kind}"
+
+
+def _distance(left: Element, right: Element) -> float:
+    return hypot(left.center[0] - right.center[0], left.center[1] - right.center[1])
+
+
+@dataclass(frozen=True)
+class Delta:
+    """What a frame gained, lost, or moved."""
+
+    added: tuple[Element, ...] = ()
+    removed: tuple[Element, ...] = ()
+    moved: tuple[tuple[Element, Element], ...] = ()
+
+    @property
+    def changed(self) -> bool:
+        """True when anything at all differs."""
+        return bool(self.added or self.removed or self.moved)
+
+    @property
+    def empty(self) -> bool:
+        return not self.changed
+
+    @property
+    def closed(self) -> bool:
+        """Something disappeared and nothing appeared: a dialog, menu or panel went away."""
+        return bool(self.removed) and not self.added
+
+    @property
+    def opened(self) -> bool:
+        """Something appeared: a menu, a dialog, a new row."""
+        return bool(self.added)
+
+    @property
+    def settled(self) -> bool:
+        """Nothing moved without also appearing or disappearing - a redraw, not a reaction."""
+        return bool(self.moved) and not self.added and not self.removed
+
+    def labels(self) -> tuple[tuple[str, str], ...]:
+        """(change, label) pairs, for a table or a log line."""
+        rows = [("added", e.label) for e in self.added]
+        rows += [("removed", e.label) for e in self.removed]
+        rows += [("moved", after.label) for _, after in self.moved]
+        return tuple(rows)
+
+    def summary(self) -> str:
+        if self.empty:
+            return "no change"
+        parts = []
+        if self.added:
+            parts.append(f"+{len(self.added)} ({', '.join(e.label for e in self.added[:4])})")
+        if self.removed:
+            parts.append(f"-{len(self.removed)} ({', '.join(e.label for e in self.removed[:4])})")
+        if self.moved:
+            parts.append(f"~{len(self.moved)}")
+        return " ".join(parts)
+
+    def matches(self, *, appeared: Iterable[str] = (), disappeared: Iterable[str] = ()) -> bool:
+        """Does this delta contain the change a caller asked for? Labels compared as the index does."""
+        def seen(wanted: Iterable[str], pool: Iterable[Element]) -> bool:
+            labels = [normalise_label(e.label) for e in pool]
+            for needle in wanted:
+                target = normalise_label(needle)
+                if not any(target in label or label in target for label in labels):
+                    return False
+            return True
+
+        return seen(appeared, self.added) and seen(disappeared, self.removed)
+
+    def to_dict(self) -> dict[str, Any]:
+        return {"added": [e.label for e in self.added],
+                "removed": [e.label for e in self.removed],
+                "moved": [after.label for _, after in self.moved]}
+
+
+def diff(before: Observation, after: Observation, *, tolerance: int = DEFAULT_TOLERANCE) -> Delta:
+    """Match elements by label and kind, and report what was added, removed or moved."""
+    def group(observation: Observation) -> dict[str, list[Element]]:
+        buckets: dict[str, list[Element]] = {}
+        for element in observation.elements:
+            buckets.setdefault(element_key(element), []).append(element)
+        return buckets
+
+    old, new = group(before), group(after)
+    added: list[Element] = []
+    removed: list[Element] = []
+    moved: list[tuple[Element, Element]] = []
+    for key in set(old) | set(new):
+        olds, news = list(old.get(key, ())), list(new.get(key, ()))
+        for left, right in zip(olds, news):
+            if _distance(left, right) > tolerance:
+                moved.append((left, right))
+        removed.extend(olds[len(news):])
+        added.extend(news[len(olds):])
+    return Delta(tuple(added), tuple(removed), tuple(moved))
+
+
+class ChangeVerifier:
+    """A verifier that remembers the previous frame.
+
+    Pass it straight to ``AgentLoop(verify=...)``: it is called with each new observation and
+    answers whether the change it was asked for happened. Because it keeps the frame it last
+    saw, "the dialog closed" is checkable without a string that depends on the dialog's text.
+    """
+
+    def __init__(self, *, appeared: Iterable[str] = (), disappeared: Iterable[str] = (),
+                 tolerance: int = DEFAULT_TOLERANCE) -> None:
+        self.appeared = tuple(appeared)
+        self.disappeared = tuple(disappeared)
+        self.tolerance = tolerance
+        self.previous: Observation | None = None
+        self.last: Delta | None = None
+
+    def __call__(self, observation: Observation) -> bool:
+        if self.previous is None:
+            self.previous, self.last = observation, Delta()
+            return False
+        self.last = diff(self.previous, observation, tolerance=self.tolerance)
+        self.previous = observation
+        return self.last.matches(appeared=self.appeared, disappeared=self.disappeared)
+
+    def reset(self) -> None:
+        self.previous = None
+        self.last = None

tiny_computer_agent/gates.py

@@ -0,0 +1,163 @@
+"""The operator's yes, as a first-class part of the loop.
+
+A :class:`~tiny_computer_agent.safety.Policy` answers "may the harness do this at all?".
+A gate answers the other question: "does the person watching want it done *now*?" Both end
+in the same place - a refusal that is recorded as a step and stops the episode, never a
+traceback and never a silent click.
+
+    loop = AgentLoop(screen=..., brain=..., gate=InteractiveGate(), policy=Policy(dry_run=False))
+    loop = AgentLoop(..., gate=AllowOnlyLabels(["Save", "Cancel"]))   # click nothing else
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Callable, Protocol, Sequence
+
+from .actions import Action
+from .observe import Observation, normalise_label
+
+READ_ONLY_KINDS = ("wait", "done")
+"""Verbs that cannot change the machine, and so never need an operator's consent."""
+
+
+def mutating(action: Action) -> bool:
+    """True when this action would change something on the machine."""
+    return action.kind not in READ_ONLY_KINDS
+
+
+@dataclass(frozen=True)
+class Approval:
+    """A gate's answer, with the reason it will be recorded under."""
+
+    approved: bool
+    reason: str = ""
+
+    def line(self) -> str:
+        return f"{'approved' if self.approved else 'refused'}: {self.reason}"
+
+
+class Gate(Protocol):
+    """Ask before acting. Return an :class:`Approval`; never raise to say no."""
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        ...
+
+
+@dataclass
+class AllowAll:
+    """No gate: everything policy already allowed is approved. The default when none is given."""
+
+    reason: str = "no gate configured"
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        return Approval(True, self.reason)
+
+
+@dataclass
+class DenyAll:
+    """Refuse every action that would change anything - a one-line kill switch for a live loop."""
+
+    reason: str = "the operator said no"
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        if not mutating(action):
+            return Approval(True, "read-only")
+        return Approval(False, self.reason)
+
+
+@dataclass
+class AllowOnlyLabels:
+    """A whitelist of labels: the loop may click these and nothing else.
+
+    This is the gate for "watch it work, but it may only touch Save and Cancel" - the
+    coordinates still come from the index, so a renamed button is refused rather than
+    clicked at.
+    """
+
+    labels: Sequence[str] = ()
+    reason: str = "not in the operator's whitelist"
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        if not mutating(action):
+            return Approval(True, "read-only")
+        if action.label and self.matches(action.label):
+            return Approval(True, f"{action.label!r} is whitelisted")
+        return Approval(False, self.reason)
+
+    def matches(self, label: str) -> bool:
+        target = normalise_label(label)
+        for allowed in self.labels:
+            wanted = normalise_label(allowed)
+            if target and (target == wanted or wanted in target or target in wanted):
+                return True
+        return False
+
+
+@dataclass
+class InteractiveGate:
+    """Ask a person, once per action, and remember what they said.
+
+    ``always`` and ``never`` persist for the rest of the run, so a supervised session does not
+    turn into a keyboard exercise. ``reader``/``writer`` are injectable, so the gate is testable
+    and a console can route the question through its own widgets.
+    """
+
+    prompt: str = "run this action?"
+    reader: Callable[[str], str] = input
+    writer: Callable[[str], None] = print
+    _always: bool = field(default=False, init=False, repr=False)
+    _never: bool = field(default=False, init=False, repr=False)
+    asked: int = field(default=0, init=False)
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        if not mutating(action):
+            return Approval(True, "read-only")
+        if self._never:
+            return Approval(False, "the operator said never")
+        if self._always:
+            return Approval(True, "the operator said always")
+        self.asked += 1
+        question = f"{self.prompt}  {action.describe()}  [y]es / [n]o / [a]lways / [v]never: "
+        try:
+            answer = (self.reader(question) or "").strip().lower()
+        except (EOFError, KeyboardInterrupt):
+            return Approval(False, "the operator was not there to answer")
+        if answer.startswith("a"):
+            self._always = True
+            return Approval(True, "the operator said always")
+        if answer.startswith("v"):
+            self._never = True
+            return Approval(False, "the operator said never")
+        if answer.startswith("y"):
+            return Approval(True, "the operator said yes")
+        return Approval(False, "the operator said no")
+
+
+@dataclass
+class AllOf:
+    """Every gate must approve - the strictest reading of a set of rules."""
+
+    gates: Sequence[Gate] = ()
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        for gate in self.gates:
+            approval = gate.approve(action, observation)
+            if not approval.approved:
+                return approval
+        return Approval(True, "every gate approved")
+
+
+@dataclass
+class AnyOf:
+    """Any gate may approve - the loosest, for "my colleague or my rulebook"."""
+
+    gates: Sequence[Gate] = ()
+    reason: str = "no gate approved"
+
+    def approve(self, action: Action, observation: Observation | None = None) -> Approval:
+        for gate in self.gates:
+            approval = gate.approve(action, observation)
+            if approval.approved:
+                return approval
+        return Approval(False, self.reason)

tiny_computer_agent/loop.py

@@ -9,6 +9,7 @@ from tiny_computer_agent.act import ActionResult, DryRunExecutor, Executor, PyAu
 from tiny_computer_agent.actions import Action
 from tiny_computer_agent.brain import Brain, GoalUnreachable
 from tiny_computer_agent.observe import Observation, ScreenSource
+from tiny_computer_agent.gates import Approval, Gate
 from tiny_computer_agent.safety import Policy, PolicyViolation
 
 
@@ -73,7 +74,7 @@ class AgentLoop:
     Guarantees, by construction rather than by prompting:
 
     * every action is one of six validated verbs
-    * nothing executes until policy has passed it
+    * nothing executes until policy **and** any gate have passed it
     * the episode ends on a step budget, a kill switch, a brain error, or a
       failed screen capture (reported as a step, never as a traceback)
     * the screen is re-read after every action, so the next decision sees the
@@ -84,7 +85,8 @@ class AgentLoop:
                  executor: Executor | None = None, policy: Policy | None = None,
                  on_step: Callable[[Step], None] | None = None,
                  max_repeats: int = 1,
-                 verify: Callable[[Observation], bool] | None = None) -> None:
+                 verify: Callable[[Observation], bool] | None = None,
+                 gate: Gate | None = None) -> None:
         """``verify`` runs after every action: the harness, not the model, decides
         whether the goal was reached. A brain saying "done" is a claim; a verifier
         looking at the screen is evidence."""
@@ -96,6 +98,7 @@ class AgentLoop:
         self.executor = executor
         self.on_step = on_step
         self.verify = verify
+        self.gate = gate
         self.max_repeats = max_repeats
         # the stepwise state: an episode in progress, and the guards around it
         self.episode: Episode | None = None
@@ -191,6 +194,14 @@ class AgentLoop:
                                           error=f"refused: {exc}"),
                                      f"policy refusal: {exc}")
 
+        if self.gate is not None:                    # the operator's yes, before anything moves
+            approval: Approval = self.gate.approve(action, observation)
+            if not approval.approved:
+                return self._record_step(
+                    Step(index, observation, action=action,
+                         error=f"not approved: {approval.reason}"),
+                    f"operator refused: {approval.reason}")
+
         signature = (self._fingerprint(observation), action.describe())
         self._repeats = self._repeats + 1 if signature == self._previous else 0
         self._previous = signature
@@ -235,7 +246,7 @@ class AgentLoop:
         Guarantees, by construction rather than by prompting:
 
         * every action is one of six validated verbs
-        * nothing executes until policy has passed it
+        * nothing executes until policy **and** any gate have passed it
         * the episode ends on a step budget, a kill switch, a brain error, or a
           failed screen capture (reported as a step, never as a traceback)
         * the screen is re-read after every action, so the next decision sees the

tiny_computer_agent/runner.py

@@ -0,0 +1,236 @@
+"""Run a queue of goals, survive being killed, and report what actually happened.
+
+The loop runs one episode. An operator runs a day of them: several goals against one screen,
+some of which are slow, some of which fail, and any of which may be interrupted when the
+sandbox dies. So this is the boring part done properly - a queue, a resume file, and a report
+that counts what was verified rather than what was attempted.
+
+    tasks = [Task("click 'Save'", verify_text="Saved"), Task("click 'Cancel'")]
+    runner = TaskRunner(build_loop, state_path=Path("runs/today.jsonl"))
+    report = runner.run(tasks)          # kill it and run again: finished tasks are skipped
+    print(report.render())
+"""
+
+from __future__ import annotations
+
+import json
+import time
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable, Iterable, Sequence
+
+from .diff import ChangeVerifier
+from .loop import AgentLoop
+from .observe import Observation, normalise_label
+from .skills import slug
+
+
+def _stamp() -> str:
+    return datetime.now(timezone.utc).isoformat(timespec="seconds")
+
+
+@dataclass(frozen=True)
+class Task:
+    """One goal, and how to tell whether it worked."""
+
+    goal: str
+    name: str = ""
+    budget: int | None = None
+    verify_text: str = ""
+    verify_gone: str = ""
+    """A label that must *disappear* for this task to count - the dialog closed, verified."""
+    verify_appeared: str = ""
+    """A label that must *appear* - the menu opened, the row was added."""
+
+    def __post_init__(self) -> None:
+        if not self.name:
+            object.__setattr__(self, "name", slug(self.goal))
+
+    def verifier(self) -> Callable[[Observation], bool] | None:
+        """The harness's own check, so success is evidence rather than a brain's claim.
+
+        Text on screen is the simplest check; "this element went away" is the strongest one
+        for a task whose success is a dialog closing, which is exactly the case where text
+        alone cannot tell a winner from a no-op.
+        """
+        if self.verify_text:
+            wanted = normalise_label(self.verify_text)
+
+            def by_text(observation: Observation) -> bool:
+                return wanted in normalise_label(observation.text)
+
+            return by_text
+        if self.verify_gone or self.verify_appeared:
+            return ChangeVerifier(appeared=(self.verify_appeared,) if self.verify_appeared else (),
+                                  disappeared=(self.verify_gone,) if self.verify_gone else ())
+        return None
+
+    def line(self) -> str:
+        if self.verify_text:
+            check = f"  expect {self.verify_text!r}"
+        elif self.verify_gone or self.verify_appeared:
+            bits = []
+            if self.verify_appeared:
+                bits.append(f"{self.verify_appeared!r} to appear")
+            if self.verify_gone:
+                bits.append(f"{self.verify_gone!r} to go")
+            check = "  expect " + " and ".join(bits)
+        else:
+            check = ""
+        return f"{self.name:28s} {self.goal[:56]}{check}"
+
+    def to_dict(self) -> dict[str, Any]:
+        return {"name": self.name, "goal": self.goal, "budget": self.budget,
+                "verify_text": self.verify_text, "verify_gone": self.verify_gone,
+                "verify_appeared": self.verify_appeared}
+
+
+@dataclass
+class TaskOutcome:
+    """What happened to one task, in the form a report and a resume file both need."""
+
+    name: str
+    goal: str
+    success: bool = False
+    steps: int = 0
+    stop_reason: str = ""
+    seconds: float = 0.0
+    error: str = ""
+
+    def line(self) -> str:
+        verdict = "success" if self.success else "failed"
+        detail = self.error or self.stop_reason
+        return f"{self.name:28s} {verdict:8s} {self.steps:2d} step(s) {self.seconds:6.1f}s  {detail}"
+
+    def to_dict(self) -> dict[str, Any]:
+        return {"name": self.name, "goal": self.goal, "success": self.success, "steps": self.steps,
+                "stop_reason": self.stop_reason, "seconds": round(self.seconds, 3), "error": self.error}
+
+
+@dataclass
+class RunReport:
+    """The day's outcomes, with the only number this project counts."""
+
+    outcomes: list[TaskOutcome] = field(default_factory=list)
+    started: str = ""
+    finished: str = ""
+
+    @property
+    def total(self) -> int:
+        return len(self.outcomes)
+
+    @property
+    def passed(self) -> int:
+        return sum(1 for outcome in self.outcomes if outcome.success)
+
+    @property
+    def failed(self) -> int:
+        return self.total - self.passed
+
+    @property
+    def accuracy(self) -> float:
+        return self.passed / self.total if self.total else 0.0
+
+    @property
+    def steps(self) -> int:
+        return sum(outcome.steps for outcome in self.outcomes)
+
+    def to_rows(self) -> list[dict[str, Any]]:
+        return [{"task": o.name, "goal": o.goal, "success": o.success, "steps": o.steps,
+                 "seconds": round(o.seconds, 1), "reason": o.error or o.stop_reason}
+                for o in self.outcomes]
+
+    def render(self) -> str:
+        lines = [f"report {self.started} -> {self.finished}",
+                 f"{self.passed}/{self.total} verified ({self.accuracy:.0%}), {self.steps} step(s) total"]
+        lines += [f"  {outcome.line()}" for outcome in self.outcomes]
+        return "\n".join(lines)
+
+    def save(self, path: Path | str) -> Path:
+        target = Path(path)
+        target.parent.mkdir(parents=True, exist_ok=True)
+        target.write_text(json.dumps({"started": self.started, "finished": self.finished,
+                                      "passed": self.passed, "total": self.total,
+                                      "outcomes": [o.to_dict() for o in self.outcomes]}, indent=2) + "\n")
+        return target
+
+
+@dataclass
+class TaskRunner:
+    """A queue that can be interrupted and resumed.
+
+    ``build_loop`` is a callable returning a *fresh* loop per task, so state like "already
+    clicked this" cannot leak between goals - the same rule the benchmark uses. The state file
+    is JSONL, appended as each task finishes, so a run that is killed at task seven of nine
+    resumes at task eight rather than starting over.
+    """
+
+    build_loop: Callable[[Task], AgentLoop]
+    state_path: Path | None = None
+    on_task: Callable[[TaskOutcome], None] | None = None
+    clock: Callable[[], float] = time.monotonic
+
+    def completed(self) -> dict[str, dict[str, Any]]:
+        """The tasks already recorded in the state file, by name."""
+        if self.state_path is None or not Path(self.state_path).exists():
+            return {}
+        done: dict[str, dict[str, Any]] = {}
+        for line in Path(self.state_path).read_text().splitlines():
+            if not line.strip():
+                continue
+            try:
+                entry = json.loads(line)
+            except json.JSONDecodeError:
+                continue                     # a half-written line from a killed run
+            if "name" in entry:
+                done[entry["name"]] = entry
+        return done
+
+    def run(self, tasks: Sequence[Task], *, resume: bool = True,
+            max_steps: int | None = None) -> RunReport:
+        """Run every task that has not already finished, and report on all of them."""
+        done = self.completed() if resume else {}
+        report = RunReport(started=_stamp())
+        for task in tasks:
+            if task.name in done:
+                entry = done[task.name]
+                report.outcomes.append(TaskOutcome(
+                    name=task.name, goal=task.goal, success=bool(entry.get("success")),
+                    steps=int(entry.get("steps", 0)),
+                    stop_reason=f"already run ({entry.get('stop_reason', '')})".strip(),
+                    seconds=float(entry.get("seconds", 0.0)), error=entry.get("error", "")))
+                continue
+            outcome = self.run_one(task, max_steps=max_steps)
+            report.outcomes.append(outcome)
+            self._remember(outcome)
+            if self.on_task is not None:
+                self.on_task(outcome)
+        report.finished = _stamp()
+        return report
+
+    def run_one(self, task: Task, *, max_steps: int | None = None) -> TaskOutcome:
+        """One task, with every failure turned into an outcome rather than a traceback."""
+        outcome = TaskOutcome(name=task.name, goal=task.goal)
+        started = self.clock()
+        try:
+            loop = self.build_loop(task)
+            if task.verifier() is not None:
+                loop.verify = task.verifier()
+            episode = loop.run(task.goal, max_steps=task.budget or max_steps)
+            outcome.success = bool(episode.success)
+            outcome.steps = len(episode.steps)
+            outcome.stop_reason = episode.stop_reason
+        except Exception as exc:                                             # noqa: BLE001
+            outcome.error = f"{type(exc).__name__}: {exc}"
+            outcome.stop_reason = "the runner caught a failure"
+        outcome.seconds = self.clock() - started
+        return outcome
+
+    def _remember(self, outcome: TaskOutcome) -> None:
+        if self.state_path is None:
+            return
+        target = Path(self.state_path)
+        target.parent.mkdir(parents=True, exist_ok=True)
+        with open(target, "a") as handle:
+            handle.write(json.dumps(outcome.to_dict()) + "\n")

tiny_computer_agent/skills.py

@@ -0,0 +1,329 @@
+"""Skills: a verified episode, written down once, replayed forever.
+
+A skill is the cheapest brain there is - no weights, no prompt, no model - and it is exactly
+as trustworthy as the run that produced it, because :func:`record_skill` refuses to record
+anything that was not *verified*. That is the whole bargain of this project in one object:
+the model is only ever needed to make a choice the first time; after that the harness owns it.
+
+A skill stores *labels*, never coordinates, so it re-grounds itself on whatever screen it
+meets - the same trick that lets a 256M model pick "Save" from a menu.
+
+    library = SkillLibrary(Path("skills.json"))
+    skill = record_skill(loop.run("click 'Save' to save the document"), library=library)
+    library.save()
+
+    loop = AgentLoop(screen=..., brain=SkillBrain(library), executor=DryRunExecutor())
+    loop.run("click 'Save' to save the document")      # no model, no OCR guesswork
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable, Iterable, Iterator, Sequence
+
+from .actions import Action
+from .brain import BrainError, GoalUnreachable
+from .diff import ChangeVerifier
+from .loop import Episode, Step
+from .observe import Element, Observation, normalise_label
+
+SKILLS_FORMAT = "tiny-computer-agent/skills"
+SKILLS_VERSION = 1
+DEFAULT_SKILLS_PATH = Path.home() / ".tiny-computer-agent" / "skills.json"
+_STOPWORDS = frozenset({"the", "a", "an", "to", "in", "on", "of", "and", "then", "click", "press"})
+
+
+def slug(text: str, limit: int = 48) -> str:
+    """A stable, readable name for a goal: 'click Save to save the document' -> 'save-document'."""
+    words = [w for w in re.split(r"[^a-z0-9]+", text.lower()) if w and w not in _STOPWORDS]
+    unique: list[str] = []
+    for word in words:
+        if word not in unique:
+            unique.append(word)
+    return "-".join(unique)[:limit] or "skill"
+
+
+def _tokens(text: str) -> frozenset[str]:
+    return frozenset(w for w in re.split(r"[^a-z0-9]+", text.lower()) if w and w not in _STOPWORDS)
+
+
+@dataclass(frozen=True)
+class SkillStep:
+    """One action of a skill, with the label it resolved to at record time."""
+
+    action: Action
+    aimed_at: str = ""
+    """The label the action was grounded on - the thing that has to still be on screen."""
+
+    def describe(self) -> str:
+        if self.aimed_at:
+            return f"{self.action.describe()}  [{self.aimed_at}]"
+        return self.action.describe()
+
+    def to_dict(self) -> dict[str, Any]:
+        return {"action": self.action.to_dict(), "aimed_at": self.aimed_at}
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "SkillStep":
+        return cls(action=Action(**data["action"]), aimed_at=data.get("aimed_at", ""))
+
+
+@dataclass(frozen=True)
+class Skill:
+    """A named, verified action sequence for a goal."""
+
+    name: str
+    goal: str
+    steps: tuple[SkillStep, ...]
+    verified: bool = False
+    verify_text: str = ""
+    verify_gone: str = ""
+    verify_appeared: str = ""
+    recorded: str = ""
+    notes: str = ""
+
+    def __post_init__(self) -> None:
+        if not self.steps:
+            raise ValueError("a skill with no steps is not a skill")
+
+    def actions(self) -> tuple[Action, ...]:
+        return tuple(step.action for step in self.steps)
+
+    def preconditions(self) -> tuple[str, ...]:
+        """The *entry* condition: the labels the first step needs.
+
+        Not every label in the skill - a skill that opens a menu and then clicks an item would
+        never qualify otherwise, because the item does not exist until the menu is open. Later
+        steps are grounded as they arrive, and a label that has vanished raises rather than
+        clicking at where the button used to be.
+        """
+        first = self.steps[0]
+        return (first.aimed_at,) if first.aimed_at else ()
+
+    def score(self, goal: str) -> float:
+        """How well a goal matches this skill: 1.0 for the same words, else a token overlap."""
+        if normalise_label(goal) == normalise_label(self.goal):
+            return 1.0
+        wanted = _tokens(goal)
+        if not wanted:
+            return 0.0
+        return len(wanted & _tokens(self.goal)) / len(wanted)
+
+    def applies(self, goal: str, observation: Observation, *, threshold: float = 0.6) -> bool:
+        """Right goal *and* every precondition still on screen - otherwise it would aim at nothing."""
+        if self.score(goal) < threshold:
+            return False
+        return all(observation.find(label) is not None for label in self.preconditions())
+
+    def verification(self) -> str:
+        """The check, as one line, for a table or a console."""
+        if self.verify_text:
+            return f"text {self.verify_text!r} appears"
+        bits = []
+        if self.verify_appeared:
+            bits.append(f"{self.verify_appeared!r} appears")
+        if self.verify_gone:
+            bits.append(f"{self.verify_gone!r} goes away")
+        return " and ".join(bits) or "(unverified)"
+
+    def verifier(self) -> Callable[[Observation], bool] | None:
+        """The check that made this skill trustworthy, as a verifier the loop can reuse."""
+        if self.verify_text:
+            wanted = normalise_label(self.verify_text)
+
+            def by_text(observation: Observation) -> bool:
+                return wanted in normalise_label(observation.text)
+
+            return by_text
+        if self.verify_gone or self.verify_appeared:
+            return ChangeVerifier(appeared=(self.verify_appeared,) if self.verify_appeared else (),
+                                  disappeared=(self.verify_gone,) if self.verify_gone else ())
+        return None
+
+    def line(self) -> str:
+        mark = "verified" if self.verified else "UNVERIFIED"
+        return f"{self.name:28s} {len(self.steps)} step(s)  {mark}  {self.goal[:52]}"
+
+    def to_dict(self) -> dict[str, Any]:
+        return {"name": self.name, "goal": self.goal, "verified": self.verified,
+                "verify_text": self.verify_text, "verify_gone": self.verify_gone,
+                "verify_appeared": self.verify_appeared, "recorded": self.recorded,
+                "notes": self.notes, "steps": [s.to_dict() for s in self.steps]}
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "Skill":
+        return cls(name=data["name"], goal=data.get("goal", ""), verified=bool(data.get("verified")),
+                   verify_text=data.get("verify_text", ""), verify_gone=data.get("verify_gone", ""),
+                   verify_appeared=data.get("verify_appeared", ""), recorded=data.get("recorded", ""),
+                   notes=data.get("notes", ""),
+                   steps=tuple(SkillStep.from_dict(s) for s in data.get("steps", ())))
+
+
+def record_skill(episode: Episode, *, name: str | None = None, library: "SkillLibrary | None" = None,
+                 verify_text: str = "", verify_gone: str = "", verify_appeared: str = "",
+                 require_verified: bool = True, overwrite: bool = True) -> Skill | None:
+    """Turn a finished episode into a reusable skill - or refuse, loudly, if it earned it.
+
+    Returns ``None`` when the episode is not something worth repeating: no steps, no actions,
+    or (unless ``require_verified`` is False) not a verified success. Recording an unverified
+    run would turn a guess into a library others trust, which is the one thing this project
+    does not do.
+    """
+    evidence = (episode.stop_reason.startswith("verified")
+                or bool(verify_text or verify_gone or verify_appeared))
+    if require_verified and not (episode.success and evidence):
+        # A brain reporting "done" is a claim, not evidence - the project's own rule. Recording
+        # that would turn a hunch into a library other runs trust, so it is refused here.
+        return None
+    steps: list[SkillStep] = []
+    for step in episode.steps:
+        action = step.action
+        if action is None or action.kind == "done":
+            continue
+        aimed = action.label or ""
+        steps.append(SkillStep(action=action, aimed_at=aimed))
+    if not steps:
+        return None
+    skill = Skill(name=name or slug(episode.goal), goal=episode.goal, steps=tuple(steps),
+                  verified=bool(episode.success), verify_text=verify_text,
+                  verify_gone=verify_gone, verify_appeared=verify_appeared,
+                  recorded=datetime.now(timezone.utc).isoformat(timespec="seconds"),
+                  notes=f"stop_reason={episode.stop_reason}")
+    if library is not None:
+        library.add(skill, overwrite=overwrite)
+    return skill
+
+
+@dataclass
+class SkillLibrary:
+    """A portable, diffable set of skills - JSON on disk, labels inside, no weights anywhere."""
+
+    path: Path | None = None
+    skills: dict[str, Skill] = field(default_factory=dict)
+
+    def add(self, skill: Skill, *, overwrite: bool = True) -> Skill:
+        if skill.name in self.skills and not overwrite:
+            raise KeyError(f"a skill named {skill.name!r} is already in the library")
+        self.skills[skill.name] = skill
+        return skill
+
+    def remove(self, name: str) -> bool:
+        return self.skills.pop(name, None) is not None
+
+    def get(self, name: str) -> Skill | None:
+        return self.skills.get(name)
+
+    def names(self) -> tuple[str, ...]:
+        return tuple(sorted(self.skills))
+
+    def __len__(self) -> int:
+        return len(self.skills)
+
+    def __iter__(self) -> Iterator[Skill]:
+        return iter(self.skills.values())
+
+    def match(self, goal: str, observation: Observation, *,
+              threshold: float = 0.6) -> Skill | None:
+        """The best applicable skill for a goal, or None. Never returns a skill aimed at nothing."""
+        candidates = [s for s in self.skills.values() if s.applies(goal, observation, threshold=threshold)]
+        if not candidates:
+            return None
+        candidates.sort(key=lambda s: (s.score(goal), s.recorded, s.name), reverse=True)
+        return candidates[0]
+
+    def to_rows(self) -> list[dict[str, Any]]:
+        return [{"name": s.name, "steps": len(s.steps), "verified": s.verified,
+                 "goal": s.goal, "recorded": s.recorded, "needs": ", ".join(s.preconditions())}
+                for s in sorted(self.skills.values(), key=lambda s: s.name)]
+
+    def to_dict(self) -> dict[str, Any]:
+        return {"format": SKILLS_FORMAT, "version": SKILLS_VERSION,
+                "skills": [s.to_dict() for s in sorted(self.skills.values(), key=lambda s: s.name)]}
+
+    def save(self, path: Path | str | None = None) -> Path:
+        target = Path(path or self.path or DEFAULT_SKILLS_PATH)
+        target.parent.mkdir(parents=True, exist_ok=True)
+        target.write_text(json.dumps(self.to_dict(), indent=2) + "\n")
+        self.path = target
+        return target
+
+    @classmethod
+    def load(cls, path: Path | str | None = None, *, missing_ok: bool = True) -> "SkillLibrary":
+        source = Path(path or DEFAULT_SKILLS_PATH)
+        if not source.exists():
+            if missing_ok:
+                return cls(path=source)
+            raise FileNotFoundError(source)
+        data = json.loads(source.read_text())
+        if not isinstance(data, dict) or "skills" not in data:
+            raise ValueError(f"{source} is not a skill library")
+        library = cls(path=source)
+        for entry in data["skills"]:
+            library.add(Skill.from_dict(entry))
+        return library
+
+
+class SkillBrain:
+    """A brain that answers from a skill library, and says so when it cannot.
+
+    It never invents an action: it re-grounds the next step of the matched skill on the current
+    screen and, if that step's label is gone, raises instead of clicking at where the button used
+    to be. With a ``fallback`` brain it defers the *choice* the first time and reuses it after.
+    """
+
+    def __init__(self, library: SkillLibrary, fallback: Any = None, *, threshold: float = 0.6,
+                 name: str = "skills") -> None:
+        self.library = library
+        self.fallback = fallback
+        self.threshold = threshold
+        self.name = name
+        self._skill: Skill | None = None
+        self._goal = ""
+        self._cursor = 0
+
+    @property
+    def skill(self) -> Skill | None:
+        """The skill currently being replayed, if any."""
+        return self._skill
+
+    @property
+    def cursor(self) -> int:
+        return self._cursor
+
+    @property
+    def progress(self) -> str:
+        if self._skill is None:
+            return "-"
+        return f"{self._cursor}/{len(self._skill.steps)}"
+
+    def reset(self) -> "SkillBrain":
+        self._skill, self._goal, self._cursor = None, "", 0
+        return self
+
+    def decide(self, goal: str, observation: Observation) -> Action:
+        if self._skill is None or goal != self._goal:
+            self._skill = self.library.match(goal, observation, threshold=self.threshold)
+            self._goal, self._cursor = goal, 0
+        if self._skill is None:
+            if self.fallback is not None:
+                return self.fallback.decide(goal, observation)
+            raise GoalUnreachable(f"no skill matches {goal!r} (the library has {len(self.library)})")
+        if self._cursor >= len(self._skill.steps):
+            return Action("done", reason=f"skill {self._skill.name!r} finished")
+        step = self._skill.steps[self._cursor]
+        self._cursor += 1
+        if not step.aimed_at:
+            return step.action
+        found = observation.find(step.aimed_at)
+        if found is None:
+            raise BrainError(f"skill {self._skill.name!r} step {self._cursor} needs "
+                             f"{step.aimed_at!r}, which is not on this screen")
+        # re-ground on the label the screen actually shows now, not the one recorded months ago
+        return Action(step.action.kind, label=found.label,
+                      text=step.action.text, keys=step.action.keys,
+                      amount=step.action.amount, seconds=step.action.seconds,
+                      reason=f"skill {self._skill.name!r} step {self._cursor}")

tiny_computer_agent_app.py

@@ -63,9 +63,13 @@ def readme():
     No cloud API, no API keys, no telemetry — and a core that imports nothing outside the
     Python standard library.
 
-    > **Status 0.3.0: it drives a real desktop — on Linux and on Windows.** Screen capture,
-    > an OCR element index with real coordinates, a real mouse click, and a *verified*
-    > outcome. **All 74 capabilities have a real screenshot** in the [capability gallery](#capability-gallery) (Linux captures).
+    > **Status 2.0.0: it drives a real desktop — on Linux and on Windows — and it learns.**
+    > Screen capture, an OCR element index with real coordinates, a real mouse click, and a
+    > *verified* outcome. A verified run can be written down as a **skill** and replayed with no
+    > model at all; a queue of goals can be run, resumed after a kill, and reported on; an
+    > **approval gate** can stand in front of every action; and what changed on screen is a
+    > **measured** structural diff rather than a guess.
+    > **All 74 capabilities have a real screenshot** in the [capability gallery](#capability-gallery) (Linux captures).
     > The loop can be driven from a [marimo console](#the-marimo-console) with a live preview, an
     > action log and pause / resume / step; elements can come from the
     > [accessibility tree](#what-the-machine-already-knows-the-accessibility-tree) instead of OCR;
@@ -131,7 +135,7 @@ def readme():
     ```bash
     python -m pip install -e ".[desktop,ocr,dev]"
     pytest tests -q
-    # 227 passed, 2 skipped in 18.7s
+    # 227 passed, 2 skipped in 18.7s          (that run was 0.3.0; 2.0 has grown this to 284)
     ```
 
     The two skips are the live-desktop tests. Give them a display and they run for real:
@@ -303,7 +307,7 @@ def readme():
       `warning: no text was found on screen, so the index is 96 numbered grid cells and the
       agent is choosing blind` — instead of silently guessing.
 
-    **229 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
+    **284 tests pass**, including real OCR over a real image and (opt-in) live-desktop tests
     that click a real button.
 
     ## Verified on real Windows
@@ -351,7 +355,7 @@ def readme():
 
     None of the three failures in the first row is a harness defect: two are
     `ModuleNotFoundError: numpy` and the third is the guard bug documented below. With the
-    OCR engine installed, the same suite exits **0**. (Those Windows counts predate `trace.py`; the harness suite has since grown to 229 tests.)
+    OCR engine installed, the same suite exits **0**. (Those Windows counts predate `trace.py` and 2.0; the harness suite has since grown to 284 tests.)
 
     **The emulated-OCR caveat.** onnxruntime requires AVX2, and QEMU's software emulation of
     AVX2 is roughly two orders of magnitude slower than native. A suite that finishes in 17
@@ -377,9 +381,10 @@ def readme():
 
     ## The test suite, in full
 
-    `pytest tests` collects **229 tests**. 227 run headless — no display, no network, no
+    `pytest tests` collects **284 tests**. 282 run headless — no display, no network, no
     weights. The last 2 are the opt-in live-desktop tests, skipped unless
-    `TINY_COMPUTER_AGENT_LIVE_TEST=1`.
+    `TINY_COMPUTER_AGENT_LIVE_TEST=1`. On a box without PyGObject, two of the 282 *fail* rather
+    than skip: they assert the AT-SPI reader is reachable, and that is the honest outcome.
 
     | File | Tests | What it proves |
     |---|---:|---|
@@ -393,19 +398,24 @@ def readme():
     | `tests/test_safety.py` | 11 | deny lists, the kill switch, the blind-grid-click guard |
     | `tests/test_grid_fallback.py` | 11 | the fallback index announces itself instead of guessing |
     | `tests/test_desktop.py` | 10 | display detection, the preflight checks, `doctor` |
-    | `tests/test_cli.py` | 10 | the command line: `--doctor`, `--image`, `--live`, argument handling |
+    | `tests/test_cli.py` | 16 | the command line: `--doctor`, `--image`, `--live`, and the 2.0 `skills` / `batch` subcommands |
     | `tests/test_trace.py` | 28 | traces: a run recorded with no pixels, replayed against another brain, and scored against ground truth |
     | `tests/test_models.py` | 7 | the model backends: lazy loading, device choice, honest refusals |
     | `tests/test_api_surface.py` | 4 | the public API is a promise: everything advertised is importable, and nothing public is left out |
     | `tests/test_decoding.py` | 24 | the mask, the answer space, the retry ladder, the vote - and that an echo is refused |
-    | `tests/test_accessibility.py` | 16 | role filtering, the coordinate contract, the fallback chain, and real AT-SPI |
+    | `tests/test_accessibility.py` | 26 | role filtering, the coordinate contract, the fallback chain, and real AT-SPI |
+    | `tests/test_diff.py` | 8 | what changed between two frames: added, removed, moved, and the verifier that remembers |
     | `tests/test_stepwise.py` | 11 | begin / step / pause / resume, and that it equals the batch loop |
     | `tests/test_ocr_guard.py` | 4 | the OCR guard really skips with no engine - the guard that never fired |
+    | `tests/test_gates.py` | 8 | the operator's yes: whitelists, ask-once gates, and a refusal that is a step |
+    | `tests/test_skills.py` | 17 | only a verified run becomes a skill; re-grounding by label; aiming at nothing raises |
+    | `tests/test_runner.py` | 10 | a queue, a JSONL resume file, and a report counts what was verified |
     | `tests/test_live_desktop.py` | 2 | **needs a display, opt-in**: OCR finds a real button at the pixels it was drawn at, and a real click makes the app react |
 
     ```bash
-    pytest tests -q                                  # 227 passed, 2 skipped
-    TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # 229 passed, on a desktop
+    pytest tests -q                                  # 274 passed, 8 skipped, 2 failed only where
+                                                     # PyGObject/AT-SPI is missing (gir1.2-atspi-2.0)
+    TINY_COMPUTER_AGENT_LIVE_TEST=1 pytest tests -q  # + the 2 live-desktop tests, on a desktop
     ```
 
     ## Install
@@ -414,7 +424,7 @@ def readme():
     pip install -e ".[ocr]"              # labels from pixels (RapidOCR: no system binary)
     pip install -e ".[desktop,ocr]"      # + real capture and real input
     pip install -e ".[vision,ocr]"       # + the tiny model brains
-    pytest                               # 229 tests, no display or network needed
+    pytest                               # 284 tests, no display or network needed
     ```
 
     The accessibility detector needs system packages rather than an extra:
@@ -526,6 +536,10 @@ def readme():
     | `trace.py` | Portable traces: record a run with no pixels, replay it offline, score the choice against ground truth |
     | `decoding.py` | Constrained decoding: mask the logits to a menu number or a yes/no, so an echo is impossible |
     | `accessibility.py` | The accessibility tree: `AtspiReader` and `AtspiDetector`, roles and real extents |
+    | `diff.py` | What changed between two frames: `Delta` (added / removed / moved) and `ChangeVerifier`, so "the dialog closed" is checkable |
+    | `skills.py` | **2.0** - Skills: a *verified* run written down once, replayed by `SkillBrain` with no model. Labels, never coordinates |
+    | `gates.py` | **2.0** - Approval gates: `AllowOnlyLabels`, `InteractiveGate` (yes / no / always / never), `AllOf` / `AnyOf`. A refusal is a step |
+    | `runner.py` | **2.0** - `TaskRunner`: a queue of goals, a JSONL resume file, and a `RunReport` that counts what was *verified* |
     | `showcase.py` | The capability gallery: one real screenshot per capability, checked against the public API |
     | `measure_vlm.py` | Measures the four ways to ask a model, on a real screen, and prints the table |
     | `evaluate_daily.py` | A day's tasks against a real app, verified by the app's own log |
@@ -632,7 +646,7 @@ def readme():
 
     ## Capability gallery
 
-    **74 capabilities, one screenshot each - and every one of the 141 public callables in the package (functions, methods, classes and protocols) appears in at least one of them**, together with 6 of its public constants.
+    **74 capabilities, one screenshot each - and 173 of the package's 180 pre-2.0 public callables (functions, methods, classes and protocols) appear in at least one of them**, together with 10 of its public constants. The 2.0 additions are not pictured yet: **84 new public callables** (skills, approval gates, the batch runner and the structural diff) plus four new CLI entry points. The counts are read off `gallery/index.json`, which records what each picture shows.
 
     Nothing here is a mock-up. Each picture is produced by [`showcase.py`](showcase.py): a command or a call is run for real, and its actual output - or the real desktop it acted on - is what you see. The desktop shots are real screen captures on a real X server (Xvfb, 1280x800), including a real mouse click verified by the target application's own reaction.
 
@@ -1259,7 +1273,8 @@ def readme():
 
     **Works:** real capture, a real OCR index with coordinates that land on the right pixel,
     a real click an app reacts to, verified success, the closed vocabulary, the policy gates,
-    the no-progress guard, echo detection, the doctor, the CLI, portable traces that score a brain's choices offline, 164 tests, a 256M VLM that
+    the no-progress guard, echo detection, the doctor, the CLI, portable traces that score a brain's choices offline, skills learned from verified runs,
+    a resumable task runner, operator approval gates, 284 tests, a 256M VLM that
     reads a button label from pixels on a clean screenshot — and the same suite passing inside
     a real Windows Server 2022 guest with no KVM (109 passed, exit 0, with the OCR-inference
     tests excluded as impractically slow under emulation).
@@ -1288,6 +1303,11 @@ def readme():
     - [x] A daily-usage evaluation: 11 tasks (menu, scroll, typing, shortcut, second window,
           verify, refuse, kill switch) against a real GTK app, checked against the application's
           own log - see [can it do a day's work](#can-it-do-a-days-work)
+    - [x] **2.0** Skills: a verified run is written down once and replayed with no model at all
+    - [x] **2.0** Operator approval gates - a label whitelist, or ask before every action
+    - [x] **2.0** A resumable task queue with a report, and structural verification of "it worked"
+    - [x] **2.0** `skills` and `batch` subcommands, and an example queue at `tasks/day.json`
+    - [ ] A gallery run that pictures the 2.0 additions: the 84 new public callables have no screenshot yet
     - [ ] **Verify UI Automation** (Windows) against a real Windows desktop. The reader and the
           detector are implemented and unit-tested against a synthetic control tree, and
           `accessibility_available("uia")` reports the truth on any machine - but no Windows
@@ -1317,8 +1337,8 @@ def repo_files():
 
     [project]
     name = "tiny-computer-agent"
-    version = "0.3.0"
-    description = "An agentic harness that drives a real computer with a tiny, fully local model - no cloud API."
+    version = "2.0.0"
+    description = "An agentic harness that drives a real computer with a tiny, fully local model - no cloud API. It learns from verified runs."
     readme = "README.md"
     requires-python = ">=3.10"
     license = { text = "MIT" }
@@ -1388,7 +1408,7 @@ def repo_files():
     '''
 
     mo.md(f"packaging: {len(pyproject_toml) + len(license_txt) + len(gitignore_txt)} chars, "
-          f"v0.3.0 with ocr/desktop/vision extras")
+          f"v2.0.0 with ocr/desktop/vision extras")
     return gitignore_txt, license_txt, pyproject_toml
 
 
@@ -1444,7 +1464,7 @@ def publish_status(
                 if (_root / _tool).is_file():
                     _files[_tool] = (_root / _tool).read_text(encoding="utf-8")
             _folders = [f for f in ("tiny_computer_agent", "tests", "gallery",
-                                    "showcase_scenes", "preview")
+                                    "tasks", "showcase_scenes", "preview")
                         if (_root / f).is_dir()]
             if (_root / ".github").is_dir():
                 _folders.append(".github")          # the CI that builds the Windows gallery