Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules

The problem. Coding agents like Claude Code follow text that developers write for them: a CLAUDE.md, skills the agent may choose to open, and rules that load automatically when it reads matching files. Developers add to this harness every time they have to correct the agent, and almost never check whether an addition actually helps. One success proves little: the same agent can pass a task once and fail it the next time.

What Cortex does. It is three commands for Claude Code plus a small CLI that does the measuring:

• /harvest, at the end of a session, saves a fix the developer made as a replayable task (the broken commit, the request, a check that fails before the fix and passes after it), and each correction as a one-line lesson.
• /evolve finds the correction that recurs most, writes one skill or rule for it, and runs the agent on the tasks with and without it: a cheap screen (each task twice per arm), then a stricter confirm (three times). The candidate is kept only if it actually entered the agent's context, shows a net gain, and makes no task noticeably worse.
• /prune tests existing items the same way, by removing them. An item that never entered the context is UNMEASURED and stays: Cortex neither adds nor deletes text the agent never used.

The verdicts are computed by a script; no model judges them.

How it was tested. A purpose-built repository with four house rules a model tends to break: every user-visible change gets a line in CHANGELOG.md; money is integer cents, scaled only with the project's rate helpers; a new exporter needs a registry entry, a documentation row and a golden file; and code reads the time only through the project's clock. A fifth family of plain bugs has nothing to learn and serves as the control. A script plays the developer. Four independent runs with Claude Haiku 4.5 each started from no harness, and every harness was then measured on 30 held-out scenarios it had never seen (6,462 rollouts in the whole study).

What we found.

• On the four rule families, the evolved harnesses raised the held-out pass rate from 21.5% to 76.5%: +55.0 points (95% CI [+34.1, +74.1]). The control family moved by −0.8. The same harness measured twice differed by −3.3 [−7.3, 0.0], which is the noise floor.
• Rules did most of the work: 16 of the 18 items kept were rules. A rule enters the context whenever its files are read; a skill must be chosen. Candidate skills offered on every turn were opened in 0 of 198 rollouts; skills offered only with the files a task touched, in 28 of 66.
• Cortex kept none of the 20 placebo and 5 harmful candidates planted in its path, and 2 of 4 known-good ones. The agent never opened the placebos or the harmful skills, though, and the harmful rules got no verdict, so this shows that Cortex keeps no unused text, not that it detects damage.
• The value depends on the model: one run's harness gained +66.7 points under Haiku but +5.6 [−2.8, +13.9] under Claude Sonnet 5, which already followed most of the conventions. On a real open-source project (structlog), where the agent was rarely corrected, Cortex found nothing to learn.

Limits. One synthetic repository, conventions a machine can check, a scripted developer, and a gain that belongs mostly to a weaker model.

Everything is open. The tool, the lab and all the data are on GitHub. Every number in the paper is a row of NUMBERS.csv with the script that computes it, and the whole analysis rebuilds offline in about two minutes.

Happy to answer questions.