docs/ARCHITECTURE.md

# Architecture — Veritas Ledger v4

Veritas Ledger is an **epistemic ledger runtime** that prevents *hallucination
laundering* in long-horizon LLM agents: a guess asserted at step *k* that, by
step *k+10*, sits in context indistinguishable from a tool result and is treated
as ground truth. It separates *what was observed* from *what the model inferred*,
and makes every trust decision replayable and reversible.

## Three planes

1. **Sealing plane** — the only path that mints seals. The `sealing-shim` executes
   every tool call at the boundary, canonicalizes the result, and posts it to
   `POST /seal` with a **replayable re-acquisition procedure** (M1: tool, canonical
   args, timestamp, content hash, determinism). The model never touches this plane
   (invariant §2.1). Content-addressed blobs live in R2.

2. **Epistemic plane** — the append-only, hash-chained event log (D1), folded into
   current belief state. Entries are observations (sealed), derivations, or
   conjectures. Trust tier and belief come **only** from seal + reputation +
   corroboration (invariant §2.2), never from content or lineage. The DAG is
   **downward-only** (§2.3): a parent link can get a claim invalidated sooner, never
   promoted. Retraction is a **soft, reversible** multiplicative downgrade (§2.4),
   recorded per node so replaying the log with the trigger removed restores belief
   exactly.

3. **Trust/verification plane** — the `verifier` runs two-stage contradiction
   detection (cheap embedding+lexical Stage A → LLM adjudication Stage B) and feeds
   the **M3 quorum gate**: a cascade fires only when ≥K independent, dedup-surviving
   sealed origins contradict the same target (§2.5). On a passing cascade, the
   **M2 reputation ledger** debits the stale supporting sources and credits the
   corroborating quorum — itself hash-chained, and itself quorum-gated.

## Components (this build)

| Package | Role |
|---|---|
| `ledger-core` | Pure, I/O-free: Zod schemas, JCS canonicalization, hash chain, trust calculus, event fold + soft cascade, M3 quorum/dedup primitives. |
| `ledger-worker` | Cloudflare Worker + `TenantLedger` Durable Object (single writer per tenant, §2.8). D1 / KV / R2 bindings. HTTP API + role-separated auth. Ships a **disjoint** clean-room chain verifier (`verify.ts`). |
| `llm-provider` | Pluggable `LlmProvider` / `EmbeddingProvider` — OpenAI adapter (default) + deterministic fake. |
| `sealing-shim` | Tool-boundary seal proxy + tool registry + shared `LedgerClient`. |
| `verifier` | Two-stage contradiction detection feeding the DO quorum gate. |
| `agent-harness` | Ledger-wired agent facade, tier-prefixed context rendering, `commit_claim` / `verify_conjecture`, transcript + §7.5 contrastive logging, policies. |
| `attribution/` (Python) | M4 offline pipeline: ContextCite surrogate + LOO + stability gate → measured edges, parquet + stability report, `/edges/measured` upsert. |
| `evals` | Scenario generator + arms 1–2–3 + metrics + kill-criteria report. |

## The mechanisms (M1–M4)

- **M1 Replayable seals** — each seal embeds its deterministic re-acquisition
  procedure; `POST /replay-audit/:seal_id` re-fetches (or re-hashes the stored
  blob) and diffs, emitting a staleness/tamper finding and quarantining on
  mismatch. A successful re-fetch upgrades `sealed_unverified → sealed_verified`.
- **M2 Source reputation** — Beta-Binomial per `source_id`, hash-chained; retraction
  debits supporting sources, surviving contradiction credits the quorum.
- **M3 Cross-origin quorum + dedup** — MinHash (Jaccard > 0.7) and embedding cosine
  (> 0.92) collapse near-duplicate/syndicated origins; independence requires distinct
  `source_id` and registrable domain. Corroboration is counted from ledger facts,
  never re-parsed from retrieved text.
- **M4 Measured dependency edges** — declared edges (weight 1.0) until the offline
  Python `attribution/` pipeline measures counterfactual weights (ContextCite-style
  surrogate over context entries, LOO validation, cross-run stability gate ρ ≥ 0.6);
  cascades attenuate by measured edge weight, so an over-declared dependency measured
  near zero stops propagating retractions. Below the stability gate, declared edges
  are kept. Wired into eval arm 3.

## Data flow (planted-premise trajectory)

```
shim.seal(premise) ─▶ /seal ─▶ observation entry (sealed_unverified)
agent.commit_claim(derivation, parents=[premise]) ─▶ /entry ─▶ derived entry + declared edge
shim.seal(corroboration ×2, independent domains)
verifier.check(target=premise, candidates) ─▶ Stage A ─▶ Stage B ─▶ /contradiction
  └▶ DO quorum gate (K≥2, dedup) ──pass──▶ soft cascade (premise ×0.05, dependents attenuated)
                                        └▶ reputation: debit premise source, credit quorum
GET /context ─▶ premise renders [RETRACTED] once, dependents downgraded, corrected sources dominate
```

## Rendering contract v2 (Run 3) — the working set is not the record

Run 2 established that *showing* a retracted claim is what launders it: arm 2
rendered `[RETRACTED …]` markers and still persisted at 54%, because the model's
salience heuristic re-reads the retracted text regardless of the annotation.

The v2 contract (`agent-harness/src/render.ts`, `renderContextV2`) treats the
context window as a **working set, not a record**. The ledger is the record; the
audit artifact (`GET /audit`) is where retraction history belongs. Rendering a
retraction notice into the working set re-injects the very salience we are trying
to remove. So v2:

- **silently drops** invalidated entries (no marker, no one-turn notice);
- is **corroboration-aware**: effective render tier = `max(storage tier,
  corroboration state)`. A sealed observation that served as a passing-quorum
  member is promoted to `verified` presentation. This is computed in `GET
  /context` from the event log — the stored entry's tier is never mutated
  (append-only, §2.6). This closes the Run-2 arm-2b failure where a
  quorum-corroborated corrector was mislabeled identically to poison;
- **demotes only uncorroborated sealed externals**, placed after all
  verified/derived/conjecture content with a bare `[uncorroborated]` prefix —
  placement, not warning text, is the signal;
- drops belief **numbers** (noise) and orders each block by belief descending so
  soft-cascade-downgraded survivors sink within their block.

The v1 renderer is retained behind `RENDERING=v1` for the regression arm (2v1).

### §6.3 caveat — the compaction 92% is a scripted worst-case (arm-1m, `arm-1m-results`)

Run-3b's flagship compaction number (flat 92% persistence vs ledger 0%) used a
*scripted* flat summary that dropped the correcting observation. The measured-
summarizer arm (`evals/report/arm-1m/`, n=50 live gpt-4.1-mini) removes that
scaffold: the correction is delivered before the compaction boundary and the
context is compacted by a **neutral** LLM call. Result: the neutral summarizer
**preserved the correction 100%** [92.9, 100], so flat persistence under measured
compaction is **0%** [0, 7.1]. So the 92% is the **worst-case bound** (a summarizer
that drops corrections), not the typical case. The ledger's real claim is narrower
and stronger: it makes correction-survival **independent of summarizer quality** —
flat context ranges 0%→92% depending on the retention policy, whereas the ledger
regenerates the resolved state unconditionally.

## Invariant enforcement

Every §2 invariant has at least one test that fails if it breaks:

| Invariant | Test |
|---|---|
| §2.1 model never seals; client trust fields ignored | `ledger-worker/test/worker.test.ts` (auth + agent-plane) |
| §2.2 authority = f(seal, reputation, quorum) | `ledger-core/test/trust.test.ts` |
| §2.3 downward-only (adding an edge never raises belief) | `ledger-core/test/invariants.test.ts` (property) |
| §2.4 soft/reversible (restore-replay exactness) | `ledger-core/test/invariants.test.ts` (property) |
| §2.5 cascade requires quorum | `ledger-core/test/quorum.test.ts`, `worker.test.ts` |
| §2.6 append-only hash chain; tamper detection | `ledger-core/test/chain.test.ts`, disjoint verifier |
| §2.7 sealed attests origin not truth | `THREAT-MODEL.md`, poisoned-tool suite |
| §2.8 single writer per tenant | `TenantLedger` DO routing |