docs/decisions/0004-sqlite-durable-layer.md

0004 SQLite Durable Layer

Date: 2026-05-22

Status

Accepted

Context

Harness v0 stores all operational data in markdown files: TEST_MATRIX.md rows, HARNESS_BACKLOG.md items, decision records, and story status. This works for human reading but creates friction for agents:

• Editing markdown tables is error-prone and hard to validate.
• There is no structured way to query past intakes, traces, or friction reports.
• The harness has no observability foundation for future evolution.

Recent research on harness engineering (arXiv:2604.25850, arXiv:2605.13357, arXiv:2603.28052) identifies observability and structured traces as the foundation for harness improvement. All three approaches require queryable operational data, not prose documents.

Decision

Add a SQLite database (harness.db) and a thin CLI (scripts/bin/harness-cli) as the durable layer for operational harness data.

The database stores:

• Intake records: classification of incoming work.
• Stories: work packets and their validation proof status (replaces manual TEST_MATRIX.md rows).
• Decisions: durable records with optional verification commands.
• Backlog items: harness improvement proposals with predicted and actual impact.
• Traces: agent execution records including actions, files, errors, outcome, and harness friction.

The schema is version-controlled under scripts/schema/. The database file is .gitignored because each project instance generates its own operational data.

Policy docs (HARNESS.md, FEATURE_INTAKE.md, ARCHITECTURE.md) remain as human-readable references. The database stores what agents produce, not what they should do.

Alternatives Considered

1. Keep everything in markdown. Rejected because it prevents structured queries, makes observability impossible, and forces agents to edit fragile tables.
2. Use JSON files. Rejected because concurrent writes are unsafe and queries require custom tooling.
3. Use a full database server. Rejected because it adds deployment complexity that does not match Harness v0 scope.

Consequences

Positive:

• Agents record structured data instead of editing markdown tables.
• Intake, story, decision, backlog, and trace data is queryable.
• The harness has an observability foundation for future evolution.
• Schema migrations enable the durable layer to grow with the harness.

Tradeoffs:

• Requires sqlite3 to be available in the environment.
• The database is not version-controlled, so each instance starts empty.
• Markdown docs and the database may drift if agents use one but not the other.

Follow-Up

• Seed existing decisions (0001-0003) into the database during init.
• Add context engineering rules keyed by task type and risk lane.
• Add harness maturity ladder (H0-H4) once the durable layer proves useful.