/fix parsing and term extract (#21)

- [NOTICKET] fix: retire the fuzzy clustering pass; add E1c post-cluster recall (2cc21c92688d662c5cfdfeeb6784bb85b98575ca)
- [NOTICKET] feat: parsing schema 0.4.0 — asset pointers for figures and tables (e662ac089b741c90e19b9f51f4933fb97a1225bc)
- [NOTICKET] docs: sync REPO_STATUS / DEV_PLAN / contracts for pipeline v2 (c3c184d83bec4a4d42e9ad3f7265560802817e5b)
- [NOTICKET] docs: close K15 with the paid v1-vs-v2 comparison (62782fed9292f67d7b376783920a07197d6776e9)
- [NOTICKET] fix: test the caption hypothesis; ship the signal at weight 0.0 (36a166e1844d186d6cfb8fff9ea070f3dc69b220)
- [NOTICKET] fix: cap key-parameter pre-emption in the review queue (Step 2) (4e3a6aefef7024db3ea4b7946fd7771a6c269601)
- [NOTICKET] feat: assert on invariants, report a band, classify the genre (Steps 0 + 5) (f1d64b113db0289601806635a9b154201a6127fc)
- [NOTICKET] fix: gate spend in cluster units and surface the estimate (Step 6) (cdb5fe82cc5f2893ccbb5e28282e8dc46a27e061)
- [NOTICKET] fix: retire escalation, budget windows by subword (Finding 6 + the over-cap window) (1faca810ea9c17fd3f86cd388afa942f48dfef78)
- [NOTICKET] docs: sync for the open-queue pass, add the Step 8 and Step 7 proposals (7659cf0333d6d14af923d227ceff139eb359118f)
- [NOTICKET] feat: pull one linked partner chunk into the evidence window (5d5102547e162caa837d7c89839c875b4029f067)
- [NOTICKET] fix: add a fault-code label, measured; re-diagnose the Komatsu tables (366697cb4b2db593a0772bc26b891b5422cc9b31)
- [NOTICKET] feat(knowledge): readable concept keys + key-value lookup (m1, m2, m6, m8) (0ba89b18f02b46549cffbf2e216d51f8e8cc82a8)
- [NOTICKET] feat(knowledge): incremental ingest — versions, reuse, staleness (i1-i7) (72e18a592a8951e8b98d5ce84e4e673986c48b55)
- [NOTICKET] test(knowledge): cross-session dedupe (i6) (0974503b9d280ff6d059898c17c5777a86571014)
- [NOTICKET] feat(knowledge): promoted columns + withdrawal (m3, m4, m5, i7) (e2ae28f347023b01c03208aefe947f1f37b4969d)
- [NOTICKET] feat(knowledge): review diff + the rule/domain boundary (r4, o2) (5cb2e2da72d6002ef8ea96cba8faf3a768faa8dc)
- [NOTICKET] feat(knowledge): two review modes, recorded against a person (r1, r2, r3) (6ef79904de4af18660e493664d47d1c4d1bcf442)
- [NOTICKET] docs(knowledge): bring the DDL handoff current for Harry (e43356130f634eb0078ad709304a143a91b8ad52)
- [NOTICKET] docs: sync REPO_STATUS / DEV_PLAN / contract for the 8 Sep list (K24) (be227a85bb553c5376d85c76cd001f4954ca9829)

API_CONTRACT_BE_PYTHON.md

@@ -802,7 +802,9 @@ Frontend rendering guidance:
 
 ## Knowledge
 
-> Added 2026-09-07 (KNOWLEDGE_SURFACE_PLAN.md). Turns a user's uploaded documents into reviewable knowledge artifacts: glossary terms, rules, formulas and one domain context per document. **Admin-shaped, not part of the chat flow** — no chat endpoint calls any of this, and the analytics path does not yet read what it produces.
+> Added 2026-09-07 (KNOWLEDGE_SURFACE_PLAN.md). Turns a user's uploaded documents into reviewable knowledge artifacts: glossary terms, rules, formulas and one domain context per document. **Admin-shaped, not part of the chat flow** — no chat endpoint calls any of this.
+>
+> **Corrected 2026-09-14:** the line that stood here ("the analytics path does not yet read what it produces") is out of date — the consumption path was armed 2026-09-08 (DEV_PLAN §0.8 K5/K9/K11). The planner reads the company's approved terminology on every slow-path turn. **Approved entries only**, so it stays inert until an expert works the queue.
 
 **Asynchronous by necessity.** Parsing a nine-page document takes ~80 s and extracting from it ~170 s, far past any HTTP timeout, so ingestion admits a batch and returns immediately. **Polling, not SSE** (decided 2026-09-07): a job row survives a page refresh and a dropped connection, a stream does not, and a four-minute stream would hold a worker for its duration.
 
@@ -821,9 +823,18 @@ Body:
 | `user_id` | Yes | The caller. Must be a UUID (422 otherwise). Scope is derived from it. |
 | `document_ids` | No | Omit for the one-click case: every parseable document this user owns. Named ids are still filtered by owner and type. |
 | `reparse` | No | Re-parse even when the content hash matches a stored artifact. Default `false` — on by default it silently re-spends. |
+| `review_mode` | No | **Added 2026-09-14.** `expert` (default) or `llm`. Who rules on what this batch produces. |
 
 Only `pdf` and `docx` are parsed; spreadsheets and text files belong to the structured-data path and come back in `skipped` **with a reason**, never silently dropped.
 
+**`review_mode` (2026-09-14).** `expert` leaves every entry waiting for a human. `llm` approves
+them on arrival with **no reviewer step and no second opinion** — the entries were written by a
+model and are accepted as-is; there is no second model call. What the FE must surface when
+offering it: choosing `llm` records `authorised_by` against the caller, so an entry approved this
+way can always answer *who allowed this*. Default is `expert`, so nobody gets LLM approval by
+accident. Entries are written through the ordinary append-only review path, so an expert can
+still disagree with any of them afterwards.
+
 `reused: true` means a job for this company was already running and is being returned instead of a second one starting — a double click is not an error.
 
 ```json
@@ -846,6 +857,48 @@ Job state and per-document outcomes. **404** when the id is unknown — unlike `
 
 `stage` (`parsing` · `filtering` · `estimating` · `extracting` · `persisting`) is ordered by cost: everything up to and including `estimating` is free, and the ceiling is enforced between `estimating` and `extracting`.
 
+> **Updated 2026-09-11 — the estimate is now returned, and `blocked` is now reachable.**
+>
+> `estimated_cost` was on the job model from the start and was **never populated**, so the field a
+> caller would poll to answer "what will this cost, and why was that document refused" always came
+> back `null`. It now carries the pre-flight shape of the batch, per document and in total. A
+> wholly-refused batch also settles **`blocked`** rather than `failed` — previously every spend
+> refusal was reported as a generic failure, which hid the one outcome with an action attached.
+>
+> **Spend is driven by CLUSTER COUNT** — one LLM call per term cluster — not by page count, so the
+> per-document breakdown is what tells an operator which document to split. There are now **two
+> ceilings**, and `blocked_by` says which one bound:
+>
+> | `blocked_by` | Meaning | The operator's move |
+> |---|---|---|
+> | `clusters` | Too many term clusters (exact, known free before spending) | Split the document |
+> | `prompt_tokens` | Too many estimated prompt tokens | The document is simply long |
+>
+> ```json
+> {
+>   "status": "blocked",
+>   "estimated_cost": {
+>     "clusters": 2083, "total_calls": 2150, "estimated_prompt_tokens": 5100000,
+>     "per_document": {
+>       "d1": {"clusters": 83,   "total_calls": 106, "estimated_prompt_tokens": 201751},
+>       "d2": {"clusters": 2000, "total_calls": 2044, "estimated_prompt_tokens": 4898249}
+>     },
+>     "blocked": {"d2": "clusters"}
+>   },
+>   "results": [
+>     {"document_id": "d2", "status": "failed", "blocked_by": "clusters",
+>      "error": "estimated 2,000 term clusters exceeds the ceiling of 750 (267% of it) — nothing was spent; 2,044 LLM calls were declined"}
+>   ]
+> }
+> ```
+>
+> Each `results[]` entry also carries `estimate` — **on success as well as refusal**, because a
+> number a caller only ever sees when it is already too late cannot be planned against.
+>
+> `estimate.genre` is advisory document classification (`standard` · `diagnostic` · `conceptual` ·
+> `unknown`) with a `well_served` flag. **Nothing is routed on it yet**; it exists so a caller can be
+> told *before* spending that a troubleshooting manual will yield almost no definitions.
+
 ### `GET /api/v1/knowledge/jobs`
 
 Recent jobs for one `user_id`, newest first — lets a caller recover after a refresh without having stashed a `job_id`.

DEV_PLAN.md

@@ -278,6 +278,74 @@ not to set for now).
 
 ## 0.8. Knowledge pipeline — test integration (2026-08-19 → **building, 2026-08-21**)
 
+> ## Updated 2026-09-11 — the open-queue pass (K17–K22)
+>
+> Six items off `knowledge_pipeline_runs/SESSION_HANDOFF.md` §4, **zero API spend**: every number
+> below comes from replaying the six persisted paid runs or re-running the free stages on the three
+> v2 artifacts. Detail in REPO_STATUS §14; per-item rows K17–K22 in the table below.
+>
+> **The headline is Step 2**, the item everyone agreed on and nobody had done: the review queue put
+> 24 unresolved key parameters in front of Komatsu's `Engine` (108 mentions, rank 25). It is rank 1
+> now, with nothing dropped.
+>
+> **Two findings worth reading as findings, not as changes:**
+> 1. **The spend ceiling was gating on a partial number.** `estimate_cost` summed only the glossary
+>    branch under a field named `estimated_prompt_tokens`. Real undercount 8.8%–18.4%; Open Pit is at
+>    **78%** of the ceiling, not 71%.
+> 2. **Escalation has never worked.** 48 calls across three documents and two configurations,
+>    **0 definitions** recovered. Retired by constant, loop left in place.
+>
+> **What is deliberately NOT switched on:** genre routing. The classifier is right on all three
+> documents, but the label sets are unscoreable without a second gold set and branch-skipping rests
+> on a single document, so `ROUTE_LABELS` / `ROUTE_BRANCHES` are both off and the verdict is recorded
+> rather than acted on. **Flipping either is Rifqi's call and one line.**
+>
+> **Still open and unchanged:** the paid Step 0 band (needs authorised repeated runs), Phase 4's
+> vision call, Step 8, Step 7, R4, and E5 — Sofhia still has not seen the `src/knowledge_parsing/`
+> diff.
+
+> ## Updated 2026-09-11 — v2: clustering fix + parsing 0.4.0 (asset pointers)
+>
+> Two changes, measured on three documents (BUMA standard · Komatsu shop manual · Open Pit textbook
+> Ch.1) at **zero API spend** for the free stages — the parse cache fingerprint is unchanged, so
+> re-parses reuse the cached Mistral responses. Full write-up:
+> `knowledge_pipeline_runs/CHANGELOG_v2.md`.
+>
+> **The fuzzy clustering pass is RETIRED (K13).** `token_set_ratio` returns exactly 100 when one token
+> set is a subset of the other, so a head noun always absorbed its own compounds and the threshold of
+> 92 was **inert**. 131 of its 140 merges were subset matches (all wrong); 6 were plurals, now folded
+> deterministically; 3 were distinct metrics wrongly merged. Token-subset merges **15/18/97 → 0**, and
+> post-cluster gold recall **0.8293 → 0.8537** — the gold term `EWH`, which `Working Hours` had
+> absorbed, is recovered. `_fuzzy_match` is parked, not deleted.
+>
+> **Parsing `schema_version` 0.4.0 (K14).** A figure is no longer its own chunk. It was, with an EMPTY
+> `text`, which made it unreachable by construction — **13 figures across three documents, each
+> described by a vision model we pay for, and ZERO reaching any prompt.** Now the `![](asset://<id>)`
+> reference is folded into the prose chunk at its reading-order position and the figure travels as an
+> `Asset` (content-addressed, persisted bytes, verbatim caption, unwrapped description). Image chunks
+> **2/3/8 → 0/0/0**; recall unmoved (E1 and E1c both identical v1→v2). Tables deliberately stay their
+> own chunk. `adapter.py` now carries `assets`, `referenced_by` and **`table_html`**, which parsing had
+> emitted for months while the adapter dropped it.
+>
+> **New metric E1c (post-cluster recall)**, and **E2's compression kill line REMOVED** — over-merging
+> *raises* compression, so the metric was rewarding the defect it should have exposed.
+>
+> **Two settings changes measured and REJECTED:** `WINDOW_WORDS` 256 (truncates 46 of 99 Open Pit
+> windows — the "flat recall" justifying it was measured on BUMA, the one document whose truncation
+> count does not vary with window size) and `SPAN_SCORE_THRESHOLD` 0.15 (+50% clusters against a +20%
+> bar). `EVIDENCE_K` raised 3 → 5 on Rifqi's call; measured cost +1.8% to +15% evidence tokens, and on
+> the one gold-set document top-5 recovers the same 16/18 as top-3.
+>
+> **Export for the team:** `knowledge_pipeline_runs/06_export_knowledge_parsing_v0.4.0/`, verified to
+> import and validate a fixture with no repo on `sys.path`. ⚠️ `src/knowledge_parsing/` is Sofhia's
+> half — she needs the diff before it goes out.
+>
+> **Paid comparison landed 2026-09-11 (K15).** Definitions 7→10 / 0→0 / 25→49; queue junk rows down
+> 37–65%; Komatsu still 0, as predicted. Caveats: the `source_wording` gain is largely tautological,
+> and **no definition cited a figure caption on any document** — assets are reachable but unused, which
+> points at the caption-to-filter path (built, flag off) as the next free experiment. Open Pit now sits
+> at **71% of the 2M spend ceiling** for one chapter.
+
 > **Updated 2026-08-21.** The banner here said *"Nothing has landed in this repo"* — that is no
 > longer true. **Both halves are now in-tree** (`src/knowledge_parsing/` + `src/knowledge_extraction/`
 > + `eval/knowledge/`, ~5,360 lines across commits `024c30a…a7bdeae`), and three dependencies were
@@ -382,6 +450,19 @@ scores **1.000** on a committed fixture; see the K9 row for how to read that num
 | K10 | Fixture artifact committed | Rifqi | ✅ | **Done 2026-08-27** — `eval/knowledge/fixtures/parsed_STD_2026_006_MNO.json` (BUMA standard, MinerU 3.4.4 `hybrid`, `schema_version 0.3.0`, 31 chunks). The eval and the extraction CLI both run with MinerU absent. The Open Pit textbook fixture is still open (S7 in the checklist) |
 | K11 | Output schema v2 (Harry's contract review) | Rifqi | 🔎 | **Landed 2026-08-26, unevaluated.** Deterministic entity ids, glossary references formulas instead of restating them, rules link to formulas and terms, 16 fields cut (50 → 42). Contract: [docs/knowledge/KNOWLEDGE_OUTPUT_CONTRACT.md](docs/knowledge/KNOWLEDGE_OUTPUT_CONTRACT.md); detail in the checklist as X24. Fixed two live defects — neither `cluster_id` (a frequency ordinal) nor `rule_id` (model-written) survived a re-run, which breaks an expert's approvals, not just links. Suite 497 passed / 7 skipped; **no eval run** — the prompt edits join X20's in the unevaluated queue |
 | K12 | Output schema v3 — `DomainContext` | Rifqi | ✅ | **Done 2026-09-02** (`7f47c61` · `6f5d619` · `e2865ac` · `172d8dc`; checklist **X26**). `BriefContext` → **`DomainContext`**, renamed while nothing is persisted — afterwards an id-prefix change is a migration that loses an expert's approvals. **The branch is no longer the pipeline's one unverifiable output:** it was made guardable by changing the question rather than the model — the prompt asks it to *locate* the document's own statement of purpose (`purpose` → `purpose_verbatim`) instead of composing a summary, so `GUARDED_FIELDS["summary"]` goes `()` → `("title", "purpose_verbatim")`. Adds derived `outline`, `subdomains` and entry counts; `key_parameters` gains a corroborating `term_id`, and an unresolved one earns its own review-queue row. **Contract change for Harry:** queue rows now carry `row_kind`, and a `key_parameter` row has no `term_id`, definition or mention count. Measured on the BUMA fixture (nano, 12 calls): 0 fields rejected, fabricated purpose refused, key-parameter linking 0/5 → 4/5 after a bug the smoke caught. Suite 541 passed / 7 skipped |
+| K13 | **Clustering: retire the fuzzy pass** | Rifqi | ✅ | **Done 2026-09-10.** `token_set_ratio` is blind to the subset case (returns 100), so the threshold never applied and head nouns ate their compounds. Measured across three documents: 131 of 140 merges were subset matches. Now `exact (plural-folded) → abbreviation ↔ expansion` only; `depluralise()` handles the 6 legitimate plural merges. Subset merges **15/18/97 → 0**; E1c_strict **0.8293 → 0.8537** (`EWH` recovered). New metric **E1c** added to `score.py`/`run_eval.py`; **E2's kill line removed** (it rises when the defect fires). Suite 676 passed / 7 skipped |
+| K14 | **Parsing 0.4.0 — asset pointers** | Rifqi (Sofhia's half) | ✅ | **Done 2026-09-10.** Figures fold into the prose chunk as `![](asset://<id>)` at reading-order position; `Asset` carries a content-addressed id, persisted bytes, a **verbatim** caption and a **model-written** description that never enters `text`. Image chunks **2/3/8 → 0/0/0**, 0 dangling refs, 0 description leaks, captions 17/17 preserved, **recall unchanged**. Tables keep their own chunk + two-way pointers. Seam: `adapter.py` carries `assets`/`referenced_by`/**`table_html`**; `SUPPORTED_SCHEMA` → (0,4), so parsing and extraction ship together. Found and fixed a Mistral bug: `image_id` mislabelled on one block, collapsing two figures onto one hash. ⚠️ **Sofhia needs the diff before the team export goes out** |
+| K15 | Paid v1-vs-v2 comparison | Rifqi | ✅ | **Done 2026-09-11**, 3 runs, ~9¢. Definitions **7→10 / 0→0 / 25→49** — Open Pit nearly doubled; junk review-queue rows −37% (Komatsu) / −65% (Open Pit); cache rate +7% on all three; 0 retries. **Komatsu stays at 0 definitions**, as predicted: a troubleshooting manual states symptoms, not definitions, and no plumbing change reaches that. Two caveats recorded rather than buried: `source_wording`'s headline +1608% is mostly **tautological** (informative 12→101, not 13→222), and **`definitions_citing_a_caption` = 0 on every document** — the assets are reachable but nothing consumed them, most likely because captions do not feed the term filter (flag off by decision). ⚠️ Cost: Open Pit is now **71% of the 2M pre-flight ceiling** (was 56%) for one chapter. Three variables moved at once, so no single number is attributable. Write-up: `knowledge_pipeline_runs/V1_VS_V2_PAID_COMPARISON.md` |
+| K16 | Caption hypothesis for the 0 caption-citations | Rifqi | ✅ | **Tested 2026-09-11, HYPOTHESIS FALSIFIED.** Both candidate explanations for `definitions_citing_a_caption = 0` are false: captions ALREADY reach the term filter (they are part of `Chunk.text`; 2/2 BUMA and 3/8 Open Pit captions already produce mentions), and figure-bearing chunks are ALREADY selected (a captioned chunk is in the top-5 for 22/83 BUMA and 64/427 Open Pit clusters). Built the missing ranker signal `asset_caption_names_term` to test it properly: at weight 2.5 it changed the top-5 of **1 cluster out of 603** and moved none from no-figure to has-figure. **The real reason: not one caption in the corpus is a definition** — they are figure titles ("Figure 1.3. Relative ability to influence costs"). A model declining to quote one is CORRECT. Signal ships at **weight 0.0** with the measurement beside it. ⚠️ Also corrects a false claim in the earlier docs: the caption-to-filter flag was described as "built, off by decision" — it was never built. Redirects the remaining figure value to Phase 4 (vision call on the bytes), which is now the only untested path |
+
+| K17 | **Step 2 — review queue ordering + key-parameter heads** | Rifqi | ✅ | **Done 2026-09-11.** Unresolved key parameters held their own sort tier, so every one outranked every routine term; the branch emits them in bulk on the documents it understands least, so the promotion fired hardest where least deserved. Now spliced at most **1 per 5** term rows. Replayed on the three persisted v2 runs: Komatsu `Engine` (108 mentions) **rank 25 → 1**, Open Pit `Resource` **8 → 1**, first junk row **1 → 6**, **row counts identical** (nothing dropped), BUMA unchanged (0 unresolved). `term` now carries a head (first `.` `:` or newline, only past 60 chars — "No. of units" survives) with the full text in new **`surface_full`**. Contract: OUTPUT_CONTRACT §9 |
+| K18 | **Step 0 — measurement hardening** | Rifqi | 🔄 | **Harness done 2026-09-11, the paid band is NOT run.** `eval/knowledge/invariants.py`: asserts on **fields and invariants, never exact text** — span locatable, entity ids present and unique, a rejected field still null (rejected-never-repaired, checkable only from outside), page in range, queue ranks contiguous, no leaked sort keys. `--runs N` repeats **only the paid stage** (free stages are deterministic) and reports min/median/max with the **median** as the value, plus whether all runs minted identical entity ids — the K11 defect class, unwatched since K11. `EXPECTED_MODEL_SNAPSHOT` turns X25's recording into an **assertion** (exit 6). New exit codes: 5 invariant, 6 snapshot. **Needs Rifqi's go-ahead to produce a real band** |
+| K19 | **Step 6 — spend ceiling in cluster units** | Rifqi | ✅ | **Done 2026-09-11.** Found first: `estimate_cost` summed **only the glossary branch** under a field named `estimated_prompt_tokens`, so the ceiling read a partial number as a total — undercount **BUMA 18.4% · Komatsu 8.8% · Open Pit 15.2%**, moving Open Pit 67.7% → **78.0%** of the 2M ceiling. Now all four branches (glossary exact, the rest approximated from prefix + chunk text, both labelled). Second ceiling **`MAX_CLUSTERS = 750`** — exact, free, and what drives spend; at ~3,200 tokens/cluster the token ceiling still binds first for these documents, so this guards the *other* shape (many cheap clusters, 494 calls ≈ 15 min). Refusals name the ceiling and the declined call count; `IngestJob.estimated_cost` (declared since the job model, **never populated**) now carries the per-document breakdown; a wholly-refused batch settles **`blocked`**, which the contract documented and the code could not produce |
+| K20 | **Step 5 — genre classification** | Rifqi | 🔄 | **Classifier done and correct on all three documents; routing deliberately OFF.** `src/knowledge_extraction/genre.py` → `standard` · `diagnostic` · `conceptual` · `unknown` from four free signals (heading coverage · tabular ratio · abbrev pairs · rule density), which separate in *different directions* — measured BUMA 46%/29%/5/0.57, Komatsu 1%/74%/0/0.01, Open Pit 86%/7%/0/1.48. **`ROUTE_LABELS` and `ROUTE_BRANCHES` are both `False`**: the genre label sets are unscoreable without a second gold set (more mentions ≠ better recall) and branch-skipping rests on one document. Today it is observability — `well_served=False` warns *before* spend that a shop manual yields almost nothing, and heading coverage raises a per-document warning (**P5**, partly: the warning exists, a parsing-side fallback does not). **Flipping either lever is Rifqi's call** |
+| K21 | **Finding 6 — escalation retired** | Rifqi | ✅ | **Done 2026-09-11, with the number Finding 6 asked for.** Across all six persisted paid runs (v1 K=3 and v2 K=5 × three documents): **48 escalation calls, 0 definitions recovered** — `extraction_status == "escalated"` appears zero times in every file. Why it cannot work as built: escalation hands the model the *next* k evidence chunks, ranked **below** ones it already declined, and 16 of 18 gold definitions sit at rank ≤3. `MAX_ESCALATION_ROUNDS = 0`; loop and call site untouched. Reviving it means changing the **question** (different prompt or tier), not the evidence |
+| K22 | **The over-cap window (pre-existing)** | Rifqi | ✅ | **Done 2026-09-11.** `WINDOW_WORDS` caps words; GLiNER truncates on **subwords** and warns rather than failing. Measured with GLiNER's own tokeniser: **1 window of 368** exceeds the 384-token cap — BUMA `STD_2026_006_MNO::0024`, 130 words / 738 chars → **409 tokens** — losing its tail at *every* window size including 130, exactly as the 2026-09-10 sweep suspected and left. Windows are now checked against the real tokeniser and split only when they do not fit (a character budget would have to assume the tightest ratio seen, 1.57, and chop up the 367 that fit). Halves overlap by `SPAN_TOKEN_CAP` words so no term straddles a cut. Fixture: 3 splits, mentions **186 → 208**, clusters 74 → 80 (**+8% spend**), **E1 0.8293 and E1c unchanged** |
+| K23 | Step 8 + Step 7 design proposals | Rifqi | 💡 | **Written 2026-09-11, nothing built, both need sign-off.** [TABLE_BRANCH_PROPOSAL.md](docs/knowledge/TABLE_BRANCH_PROPOSAL.md) — deterministic (row header, column header, cell mark) → assertion, one call per **table** (59 on Komatsu, fewer than the 93 the glossary branch spends to abstain 93 times), provenance at the cell; five decisions listed, **T4 (what `table_html` shapes the parser emits) is free and comes first**. [TERM_HIERARCHY_PROPOSAL.md](docs/knowledge/TERM_HIERARCHY_PROPOSAL.md) — parent/child **with** cross-document identity because SciCo's point is they are one problem; the dominating constraint is that **hierarchy must not enter `entity_id`**, or every correction invalidates expert approvals. Sequenced behind **R4** (still not asked) and Step 0 |
+| K24 | **The 2026-09-08 discussion list — structure, incremental ingest, review** | Rifqi | ✅ | **Done 2026-09-14 (pr/21, 7 commits).** *Struktur data 7/8:* `label` carries a readable `TYPE_SUBJECT` key (`keys.py`) — prefixes from the `kind` vocabulary, rules borrow the term they govern plus a digest since a rule has no canonical surface, and the key is a **label, never an identity** (`entity_id` stays the hash approvals hang on). `content`/`provenance`/`doc_ids` promoted out of `payload`; `get_by_key()` + `glossary_index()` make lookup O(1) with no model call. **m7 parked** — the `kind` CHECK would reject `'summary'`. *Ingest bertahap 7/7:* same `doc_id` + new `content_hash` = a new VERSION (reuse check moved to any-version, or idempotency would become a paid no-op); an approved term skips the paid call; staleness compares **only the ruled-on fields** (`mutation.py`) because comparing the payload re-opens everything on every ingest and comparing document `content_hash` flips untouched entries; `impacted_by_document()`; withdrawal marks and keeps rather than deletes. *Sistem review 4/4:* `review_mode` at ingest, `decided_by` + `authorised_by` so an LLM-mode approval can say whose decision it was, and `review_diff()` naming which fields moved. **7 columns added to dedorch** (§6/§7 of `knowledge_tables.sql`, run by hand **after** the handoff went to Harry, verified read-only against live — Harry still needs both in the migration set). Suite **816 passed / 7 skipped** (was 742), ruff clean on touched paths, `import main` 0. **Inert until the review queue has rows:** reuse and staleness both key on approved entries, and `knowledge_reviews` is at 0 |
 
 **Not decided, do not assume:** whether the admin ingest surface is HTTP at all (the offline CLI is
 the honest milestone and both halves are there now); and GLiNER + LangExtract ensembling stays ⏸️

REPO_STATUS.md

@@ -5,7 +5,7 @@
 
 **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
 **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only — no roadmap or to-dos.
-**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§12/§13 — `ReportStore.save` never wrote `reports.user_id`, and on the new **Neon** dedorch instance that column is `text NOT NULL`, so **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates every deployment shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. **Root cause (verified against the Go source after pulling `dd37b38`):** the column is *not* new — `0001_create_core_schema.sql` has declared it `NOT NULL` all along, but `0002`/`0004` retrofit it onto pre-existing DBs as a **nullable** `ADD COLUMN IF NOT EXISTS`. The old DB therefore accepted the column-less INSERT and the bug stayed latent; a fresh DB enforces it. Generalized as a §13 gotcha — **Go's migration set is not convergent, so fresh and migrated dedorch DBs differ in NOT NULL constraints** and the migrations alone can't tell you which shape you have. A full ORM-vs-live drift scan the same day found all five Python-written tables clean; the remaining drift is on read-only or unwired paths. **Knowledge pipeline 2026-08-21:** new **§14** — both halves of the knowledge pipeline are now in-tree (`src/knowledge_parsing/` · `src/knowledge_extraction/` · `eval/knowledge/`), as **offline CLIs with no HTTP surface, no table and nothing mounted in `main.py`**. They meet at one versioned `ParsedDocument` artifact; extraction never takes a file path. Term-filter recall **0.8537**, matching the prototype baseline. The existing `src/knowledge/` OCR→pgvector RAG path is untouched. Deps: `rapidfuzz`/`gliner`/`torch` main, `mineru` an optional extra. **Superseded 2026-09-07:** `mineru` dropped (commented out) and `torch` pinned to the Linux CPU wheel — 36 `nvidia-*` packages and `triton` gone from `uv.lock`; §14. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
+**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§12/§13 — `ReportStore.save` never wrote `reports.user_id`, and on the new **Neon** dedorch instance that column is `text NOT NULL`, so **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates every deployment shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. **Root cause (verified against the Go source after pulling `dd37b38`):** the column is *not* new — `0001_create_core_schema.sql` has declared it `NOT NULL` all along, but `0002`/`0004` retrofit it onto pre-existing DBs as a **nullable** `ADD COLUMN IF NOT EXISTS`. The old DB therefore accepted the column-less INSERT and the bug stayed latent; a fresh DB enforces it. Generalized as a §13 gotcha — **Go's migration set is not convergent, so fresh and migrated dedorch DBs differ in NOT NULL constraints** and the migrations alone can't tell you which shape you have. A full ORM-vs-live drift scan the same day found all five Python-written tables clean; the remaining drift is on read-only or unwired paths. **Knowledge pipeline 2026-08-21:** new **§14** — both halves of the knowledge pipeline are now in-tree (`src/knowledge_parsing/` · `src/knowledge_extraction/` · `eval/knowledge/`), as **offline CLIs with no HTTP surface, no table and nothing mounted in `main.py`**. They meet at one versioned `ParsedDocument` artifact; extraction never takes a file path. Term-filter recall **0.8537**, matching the prototype baseline. The existing `src/knowledge/` OCR→pgvector RAG path is untouched. Deps: `rapidfuzz`/`gliner`/`torch` main, `mineru` an optional extra. **Superseded 2026-09-07:** `mineru` dropped (commented out) and `torch` pinned to the Linux CPU wheel — 36 `nvidia-*` packages and `triton` gone from `uv.lock`; §14. **Knowledge 8-Sep task list 2026-09-14 (pr/21):** §8/§14 — the discussion's structure-data, incremental-ingest and review items are built. `knowledge_entries.label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`) so a lookup is a dictionary hit rather than a scan, with `entity_id` untouched as the thing approvals hang on; a re-uploaded document becomes a **new version** rather than a new document, so its approvals survive; an already-approved term is no longer re-extracted (one LLM call per cluster is the whole cost of that branch); an approval goes stale only when the **content the expert ruled on** changed, never when a count moved; a withdrawn document stops feeding the baseline without anything being deleted; and ingest takes a `review_mode` — `llm` approves on arrival with no reviewer step, recorded against the person who chose it. Seven columns added to dedorch (run by hand 2026-09-14 **after** the handoff went to Harry, and verified read-only against live). **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
 the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
 own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
 **`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet — those skills
@@ -195,6 +195,7 @@ unless `SKIP_INIT_DB=true`.
 | `analyses_messages` *(dedorch)* | the analysis chat room (`role ∈ user\|ai`); replaces deprecated `rooms`/`chat_messages` | Go `/analyses/{id}/messages` | Python chat path **not yet migrated here** (§12) |
 | `message_traceability` *(Python-owned)* | one jsonb `TraceabilityPayload` per assistant turn (PK `message_id`); flushed before `done` | chat pipeline (KM-691) | `GET /api/v1/traceability` |
 | `message_charts` *(Python-owned, added 2026-07-13)* | one row per `render_chart` chart — `spec` jsonb holds the full `dataeyond.chart.v1` envelope; keyed (`analysis_id`, `message_id`), multiple rows per turn allowed; written before `done`, never-throw | slow-path chart persist (`chat_handler._run_slow_path`) | `GET /api/v1/charts` |
+| `knowledge_documents` · `knowledge_extraction_runs` · `knowledge_entries` · `knowledge_reviews` · `knowledge_jobs` *(Python-written, Go-owned schema)* | The knowledge pipeline's five tables (§14). **2026-09-14: 7 columns added** — `knowledge_entries.doc_ids`/`.content`/`.provenance` (promoted out of `payload`), `knowledge_documents.withdrawn_at`, `knowledge_jobs.review_mode`, `knowledge_reviews.decided_by`/`.authorised_by`. `label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`), not the term. DDL + adoption status: `docs/knowledge/knowledge_tables.sql` | knowledge ingest pipeline | `/api/v1/knowledge/*`, `KnowledgeService` |
 
 > ✅ **Python ORM ↔ dedorch drift — reconciled 2026-07-01.** `AnalysisStateRow` (`analyses`) dropped
 > `problem_statement`/`problem_validated` and added `objective`/`business_questions` (Harry's #3);
@@ -522,6 +523,12 @@ records-based report; floor: ≥1 `analyze_*` success). Wiring Go → Python is
 
 ## 14. Knowledge pipeline — offline subsystems (added 2026-08-21)
 
+**Updated 2026-09-14 — the 8 Sep discussion landed.** Two claims below were stale and are
+corrected here: the surface is **ten** routes, not eight (`/knowledge/domain/propose` and
+`/knowledge/domain` were added 2026-09-08), and **the analytics path does read what this
+produces** — the consumption path was armed 2026-09-08 (§0.8 K5/K9/K11), approved entries only.
+The 2026-09-08 task list (m*, i*, r*) is built: see **§0.8 K24**.
+
 **Updated 2026-09-07 — this is now on the live surface.** The line that stood here
 ("no endpoint, no database table, nothing mounted") is out of date. Eight routes under
 `/api/v1/knowledge` are mounted in `main.py` (`src/api/v1/knowledge.py`), and five tables exist in
@@ -532,8 +539,8 @@ RAG.**
 
 | | |
 |---|---|
-| Surface | `POST /knowledge/ingest` · `GET /knowledge/jobs/{id}` · `GET /knowledge/jobs` · `GET /knowledge/documents` · `POST /knowledge/documents/{id}/extract` · `GET /knowledge/entries` · `GET /knowledge/queue` · `POST /knowledge/entries/{entity_id}/reviews` |
-| Tables | `knowledge_documents` · `knowledge_extraction_runs` · `knowledge_entries` · `knowledge_reviews` · `knowledge_jobs` |
+| Surface | **10 routes** (2026-09-14 count): `POST /knowledge/ingest` · `GET /knowledge/jobs/{id}` · `GET /knowledge/jobs` · `GET /knowledge/documents` · `POST /knowledge/documents/{id}/extract` · `GET /knowledge/entries` · `GET /knowledge/queue` · `POST /knowledge/entries/{entity_id}/reviews` · `POST /knowledge/domain/propose` · `GET /knowledge/domain` |
+| Tables | `knowledge_documents` · `knowledge_extraction_runs` · `knowledge_entries` · `knowledge_reviews` · `knowledge_jobs`. **7 columns added 2026-09-14** — see §8 |
 | Orchestration | `src/knowledge_ingest/` — job lifecycle, tenant scope, the pipeline port. Owns no parsing and no extraction logic |
 | Contract | `API_CONTRACT_BE_PYTHON.md` §Knowledge |
 
@@ -569,8 +576,102 @@ behind them in [KNOWLEDGE_PIPELINE_CALIBRATION.md](docs/knowledge/KNOWLEDGE_PIPE
 | Extraction | `src/knowledge_extraction/` | Rifqi | `python -m src.knowledge_extraction.cli <artifact.json> --extract` |
 | Eval harness | `eval/knowledge/` | Rifqi | gold set + scorer + frozen baseline |
 
+> ## Updated 2026-09-11 — the open-queue pass (queue ordering · measurement · ceilings · genre)
+>
+> Six items from `knowledge_pipeline_runs/SESSION_HANDOFF.md` §4, **all measured at zero API spend**
+> (replays of the persisted paid runs, plus free stages on the three v2 artifacts).
+>
+> **The review queue no longer buries its own product (Step 2).** Unresolved key parameters held a
+> sort tier beneath conflicts, so *every* one of them outranked *every* routine term. Replaying the
+> three v2 runs: Komatsu's `Engine` (108 mentions) moves **rank 25 → 1**, Open Pit's `Resource`
+> **8 → 1**, the first junk row moves **1 → 6** on both, and row counts are identical — nothing is
+> dropped. They are now spliced in at most 1 per 5 term rows. A `key_parameter`'s `term` also carries
+> a **head** (split at the first `.` `:` or newline, only past 60 chars) with the untruncated text in
+> the new **`surface_full`** — the branch picks the right concepts and returns them at paragraph
+> length (185 chars in one case).
+>
+> **Measurement hardening (Step 0).** `eval/knowledge/invariants.py` asserts on **fields and
+> invariants, not exact text** — a span that cannot be located, a duplicate entity id, a rejected
+> field that still carries a value, a queue whose ranks are not contiguous. `--runs N` repeats only
+> the paid stage and reports **min/median/max** (the value becomes the *median*) plus whether every
+> run minted identical entity ids, which is the K11 defect class and has been unwatched since. The
+> pinned model snapshot is now an **assertion**, not a log line: a run served by a different snapshot
+> exits 6 and must not be compared to the committed results.
+>
+> **Two spend ceilings, and a corrected estimate (Step 6).** `estimate_cost` summed **only the
+> glossary branch** while reporting a field named `estimated_prompt_tokens` — so the ceiling gating
+> on it read a partial number as a whole one. Measured undercount: **8.8%–18.4%**; Open Pit moves
+> from 67.7% to **78.0%** of the token ceiling. A second ceiling now gates on **cluster count**
+> (exact, free, and what actually drives spend — one call per cluster), `MAX_CLUSTERS = 750`. A
+> refusal names which ceiling bound and how many calls were declined; `IngestJob.estimated_cost`,
+> declared since the job model was written and **never populated**, now carries the per-document
+> breakdown; and a wholly-refused batch settles **`blocked`** rather than `failed`, which the
+> contract had documented and the code could not produce.
+>
+> **Genre classification (Step 5).** `src/knowledge_extraction/genre.py` names a document
+> `standard` · `diagnostic` · `conceptual` · `unknown` from four free signals, and gets all three
+> reference documents right. **Routing is OFF** (`ROUTE_LABELS` / `ROUTE_BRANCHES`) — the label sets
+> are unscoreable without a second gold set, and branch-skipping is confirmed on one document — so
+> today this is observability: `well_served=False` says *before* spending that a shop manual will
+> yield almost nothing. Heading coverage now raises a per-document warning (Komatsu: 1 of 80 chunks).
+>
+> **Escalation is retired (Finding 6), with the number that was asked for.** Counted off all six
+> persisted paid runs: **48 escalation calls, 0 definitions recovered**. `extraction_status` is set
+> to `escalated` only when a later round succeeds where round 0 failed, and it appears zero times in
+> every file. `MAX_ESCALATION_ROUNDS = 0`; the loop and its call site are untouched.
+>
+> **The over-cap window is fixed.** `WINDOW_WORDS` caps *words*; GLiNER truncates on *subwords* and
+> warns rather than failing. Measured with GLiNER's own tokeniser: **1 window in 368** is over the
+> 384-token cap — BUMA `::0024`, 130 words / 738 chars → **409 tokens** — and it has been losing its
+> tail at every window size, including 130. Windows are now checked against the model's real
+> tokeniser and split only if they do not fit. On the committed fixture: 3 windows split, mentions
+> **186 → 208**, clusters 74 → 80 (**+8% spend**), **E1 and E1c unchanged**.
+>
+> Proposals written, nothing built: [TABLE_BRANCH_PROPOSAL.md](docs/knowledge/TABLE_BRANCH_PROPOSAL.md)
+> (Step 8) and [TERM_HIERARCHY_PROPOSAL.md](docs/knowledge/TERM_HIERARCHY_PROPOSAL.md) (Step 7 + R7).
+
+> **Updated 2026-09-10 — parsing `schema_version` 0.4.0 (asset pointers) + clustering fix.** Two
+> changes, both measured on three documents (BUMA standard · Komatsu shop manual · Open Pit textbook
+> Ch.1), at **zero API spend** — the parse cache fingerprint is unchanged so re-parses reuse the
+> cached Mistral responses.
+>
+> **Clustering: the fuzzy pass is RETIRED.** `token_set_ratio` returns exactly 100 when one token set
+> is a subset of the other, so a head noun always absorbed its own compounds and the threshold of 92
+> was inert. 131 of its 140 merges were subset matches (all wrong); 6 were plurals, now folded
+> deterministically by `normalize(lemma=True)`; 3 were distinct metrics wrongly merged. Matching is now
+> `exact (plural-folded) → abbreviation ↔ expansion`, nothing else. Token-subset merges **15/18/97 → 0
+> on all three documents**, and post-cluster gold recall 0.8293 → **0.8537** (the gold term `EWH`, which
+> `Working Hours` had absorbed, is recovered). `_fuzzy_match` is parked, not deleted.
+>
+> **Parsing 0.4.0: a figure is no longer its own chunk.** It used to be, with an EMPTY `text`, which
+> made it unreachable by construction — no text → no mentions → no cluster → never evidence. Measured:
+> **13 figures, each described by a vision model we pay for, and ZERO reaching any prompt.** Now the
+> `![](asset://<id>)` reference is folded into the prose chunk at its reading-order position and the
+> figure travels as an `Asset` (content-addressed `sha256(bytes)[:16]`, persisted bytes, verbatim
+> caption, unwrapped model description). **Image chunks 2/3/8 → 0/0/0**; 0 dangling references; 0
+> model-written descriptions leaked into `text` (the span-check haystack). **Tables deliberately stay
+> their own chunk** — their linearised text earns its evidence slot (74.6% of Komatsu's top-3 evidence
+> is tabular) — and gain an `Asset` plus two-way pointers. `adapter.py` now carries `assets`,
+> `referenced_by` and, finally, **`table_html`**, which parsing had emitted for months while the
+> adapter dropped it. `SUPPORTED_SCHEMA` → (0,4); parsing and extraction ship together by design.
+>
+> **New metric `E1c` (post-cluster recall)** in `eval/knowledge/score.py`: the gold set scored AFTER
+> clustering, which is the window E1 (measured before) and E3 (only over entries that produced a
+> definition) both miss. **E2's compression kill line was REMOVED** — over-merging *raises* compression,
+> so the metric was rewarding the defect.
+>
+> **Two settings changes were measured and REJECTED:** `WINDOW_WORDS` 256 (truncates 46 of 99 Open Pit
+> windows — the "flat recall" that justified it was BUMA-only, the one document whose truncation count
+> does not vary with window size) and `SPAN_SCORE_THRESHOLD` 0.15 (+50% clusters against a +20% bar,
+> despite E1 0.878 → 0.976). `EVIDENCE_K` was raised 3 → 5. Detail:
+> `knowledge_pipeline_runs/CHANGELOG_v2.md`.
+>
+> ⚠️ **Nothing about definition QUALITY is claimed yet** — the paid comparison has not run. Everything
+> above is recall and structural integrity.
+
 > **Current (2026-09-07):** the parser is **Mistral OCR** (`--backend mistral`, no GPU; normalizer
-> `normalize_mistral.py`, section-level). `schema_version` is **`0.3.3`** (adds `source_title`,
+> `normalize_mistral.py`, section-level). `schema_version` is **`0.4.0`** as of 2026-09-10 (asset
+> pointers, K14); 0.3.3 added `source_title`,
 > `normaliser_version`, and a computed 1-based `page_no`).
 >
 > **MinerU was dropped 2026-09-07** — commented out of the `knowledge-parsing` extra and installed
@@ -586,7 +687,7 @@ behind them in [KNOWLEDGE_PIPELINE_CALIBRATION.md](docs/knowledge/KNOWLEDGE_PIPE
 ### The seam
 
 The two halves meet at **one persisted, versioned artifact** — `ParsedDocument` in
-`src/knowledge_parsing/contracts.py` (`schema_version` **`0.3.3`** as of 2026-09-02) — and nowhere else. **Extraction
+`src/knowledge_parsing/contracts.py` (`schema_version` **`0.4.0`** as of 2026-09-10; `SUPPORTED_SCHEMA` is `(0, 4)`, so parsing and extraction ship together) — and nowhere else. **Extraction
 never takes a file path and never touches the parser**, which is what keeps MinerU swappable: a
 Tesseract or Azure Document Intelligence path emits the same artifact and extraction never learns
 which one ran. `knowledge_extraction/adapter.py` is the only module that knows the shape.
@@ -666,9 +767,15 @@ silent content drop worth **+0.1220** on the way: `normalize.py` was discarding
 Indonesian definitions, 0 span rejections, 7,168 of 8,029 prompt tokens cached. Full-document cost
 on the prototype was **$0.0069**.
 
-**Not yet done:** persistence (stage output is JSON on disk; the DDL handoff to Harry is open), any
+~~**Not yet done:** persistence (stage output is JSON on disk; the DDL handoff to Harry is open), any
 HTTP surface, a full-document run on v2, and **schema-fill precision has never been scored on v2** —
-the one prototype experiment that failed.
+the one prototype experiment that failed.~~
+
+**RESOLVED — all four, and the paragraph above is kept only for the record.** Persistence landed
+2026-09-07 (five tables, `knowledge_store.py`) and the HTTP surface with it (8 routes); the
+full-document run is routine (K9, 2026-08-27) and E3 scores **1.000** on the committed fixture —
+read as *precision on a scoreable base of 6*, since 57 of 63 entries abstain. **What is genuinely
+open is in DEV_PLAN §0.8** and, ahead of it, `knowledge_pipeline_runs/SESSION_HANDOFF.md` §4.
 
 ### Dependencies
 

docs/README.md

@@ -39,6 +39,8 @@ Client documents → candidate knowledge entries an expert reviews rather than a
 | [knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md](knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md) | 🔄 | You're working either side of the seam |
 | [knowledge/KNOWLEDGE_SURFACE_PLAN.md](knowledge/KNOWLEDGE_SURFACE_PLAN.md) | 🔄 | Persistence + endpoints: who owns what, phased tasks, blockers. Approved 2026-09-01 |
 | [knowledge/KNOWLEDGE_CONSUMPTION_PLAN.md](knowledge/KNOWLEDGE_CONSUMPTION_PLAN.md) | 💡 | How agents READ the entries (N14). Which agent gets what, and why. **Proposal, nothing built** |
+| [knowledge/TABLE_BRANCH_PROPOSAL.md](knowledge/TABLE_BRANCH_PROPOSAL.md) | 💡 | Step 8 — the symptom x cause matrix a shop manual actually holds. Deterministic triples, LLM only to phrase. **Proposal, nothing built** |
+| [knowledge/TERM_HIERARCHY_PROPOSAL.md](knowledge/TERM_HIERARCHY_PROPOSAL.md) | 💡 | Step 7 + R7 — parent/child terms and cross-document identity, designed together. **Touches `entity_id`; read before building** |
 | [knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V4.md](knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V4.md) | 💡 | What a *real* domain context is. v3's object is a document brief wearing the name. **Proposal, nothing built** |
 | [knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md](knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md) | 💡 | The four-table DDL sent to Harry 2026-09-01. **Pending — no ORM written against it yet** |
 | [knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V3.md](knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V3.md) | ✅ | `DomainContext` design reasoning. **Shipped in full 2026-09-02** — the live shape is the output contract §8 |

docs/knowledge/KNOWLEDGE_OUTPUT_CONTRACT.md

@@ -24,7 +24,7 @@ nothing consumes.
 
 | Change | Detail |
 |---|---|
-| **Ids on every entity** | `term_id`, `formula_id`, `domain_id`; `rule_id` re-specified. All deterministic and content-derived |
+| **Ids on every entity** | `term_id`, `formula_id`, `brief_id`; `rule_id` re-specified. All deterministic and content-derived |
 | **Glossary no longer restates formulas** | `formula_latex` removed → `defining_formula_id` reference |
 | **Rules link to formulas and terms** | `formula_ids[]`, `term_ids[]` |
 | **`rule_type` added** | `interpretation` \| `calculation` — see §2 for why |
@@ -52,7 +52,8 @@ None is model-generated; none is positional.
 term_id     = "t_" + sha256(doc_id + "|" + canonical)[:10]
 formula_id  = "f_" + sha256(doc_id + "|" + name_or_latex_or_chunk_id)[:10]
 rule_id     = "r_" + sha256(doc_id + "|" + chunk_id + "|" + char_start)[:10]
-domain_id   = "d_" + sha256(doc_id)[:10]
+brief_id    = "b_" + sha256(doc_id)[:10]    # per-document card; `d_` belongs to the
+                                            # scope-level DomainContext in src/knowledge_domain/
 ```
 
 Implemented in `src/knowledge_extraction/ids.py`. `term_id` is keyed on the **cluster canonical**,
@@ -80,10 +81,42 @@ registry, which belongs with persistence (DEV_PLAN §0.8 D2).
 | `glossary.json` | `GlossaryEntry[]` | Terms and their definitions |
 | `rules.json` | `RuleEntry[]` | Rules of thumb — how to interpret data or behaviour. **Renamed from `interpretation_pack.json` 2026-09-08** — the file, the entity and the DB `kind` now share one name. *The Interpretation Pack* remains the deliverable's name in prose |
 | `formulas.json` | `FormulaEntry[]` | Formulas, transcribed from the document |
-| `domain_context.json` | `DomainContext` | One domain-context entry per document (renamed from `brief_context.json` / `BriefContext`, 2026-09-01) |
+| `document_brief.json` | `DocumentBrief` | One per-document card. Prefix history, not churn: `b_` (BriefContext) -> `d_` (DomainContext, 2026-09-01) -> **`b_` again** (DocumentBrief, 2026-09-08), once v4 gave `d_` to the real scope-level `DomainContext` in `src/knowledge_domain/` |
 
 Plus three derived/audit payloads: `review_queue.json`, `rejected.json`, `usage.json`.
 
+### Rule vs domain knowledge — where the line is
+
+*Added 2026-09-14 (o2). The two blurred on screen during the 2026-09-08 review,
+and a boundary nobody has written down is one the prompts cannot hold.*
+
+| | **`rule`** | **`domain`** |
+|---|---|---|
+| Grain | One statement, from one document | One per company, composed across all of them |
+| Shape | A condition and a consequence | Identity, boundary, conventions, measures |
+| Evidence | Span-checked against its source | Derived from entries, or expert-declared |
+| Answers | *"…and how do I compute this correctly?"* | *"…what does this company measure, and what does it call it?"* |
+
+**The test to apply.** If removing it would change **how a number is computed
+or sourced**, it is a rule — *"if joint-survey data is unavailable, use truck
+count"* changes the figure. If removing it would only leave an agent **less
+oriented** — not knowing that this company works in overburden and BCM — it is
+domain.
+
+Two consequences worth stating, because they are what the distinction is *for*:
+
+- **A rule is actionable alone; domain knowledge is not.** A planner can apply
+  a rule to a specific calculation. It cannot apply "this is a mining company"
+  to anything — that shapes which questions make sense, not which column to sum.
+- **Domain is high-level and rules are specific, but "high-level" is not the
+  test.** A long, general rule is still a rule. The grain question — *does this
+  come from one document's sentence, or from the corpus as a whole?* — settles
+  it faster and does not drift.
+
+Where it is genuinely both, prefer the **rule**: it carries provenance and a
+span, so an expert can check it. Domain fields are the ones the pipeline can
+least easily prove.
+
 ### Why `RuleEntry` carries a `rule_type`
 
 The Interpretation Pack exists to **improve analytics insight from expert interpretation rules** —
@@ -104,6 +137,36 @@ A consumer wanting only interpretation logic filters on `rule_type`.
 
 ---
 
+## 2c · Evidence now carries figures and tables *(parsing 0.4.0, 2026-09-10)*
+
+The artifact this half consumes changed shape: a figure is no longer its own chunk. It is folded into
+the prose chunk at its reading-order position as `![](asset://<id>)`, and travels as an `Asset` on that
+chunk. `adapter.py` carries `assets`, `referenced_by` and `table_html` across; `evidence_block` renders
+them into the prompt under `ASSETS REFERENCED BY THIS CHUNK`.
+
+**What that means for an extraction branch**, and the distinction is enforced in the prompts:
+
+| | may be quoted? | may be a provenance span? |
+|---|---|---|
+| `Asset.caption` — printed in the document | **yes** | **yes** |
+| `Asset.description` — written by a vision model | no | **no** — it is not in the document, so the span check rejects it and the field is discarded |
+
+Why it mattered: before 0.4.0 a figure chunk had an EMPTY `text`, so it produced no mentions, joined no
+cluster, and was never selected as evidence. 13 figures across three documents, each already described
+by a paid vision call, reached **zero** prompts. Full detail:
+`knowledge_pipeline_runs/CHANGELOG_v2.md` §2.
+
+> ✅ **RESOLVED 2026-09-11 — §1, §2 and §8 now match the code.** The note is kept for the record.
+>
+> ⚠️ **Correction to §8 of this document (noted 2026-09-10).** The code writes
+> `document_brief.json` with **`brief_id`** and a `b_` prefix; §8 below documents
+> `domain_context.json` with `domain_id` and `d_`. The v3 rename was partly reverted for the
+> per-document object once the scope-level `DomainContext` became a separate thing in
+> `src/knowledge_domain/`. **§8 does not describe what the code emits.** Worth one edit, and it matters
+> because `entity_id` prefixes are the key an expert's approvals hang on.
+
+---
+
 ## 3 · Transport reality
 
 Every payload is written by the CLI as a file into `--out-dir` (default `out/knowledge/`), as a
@@ -343,7 +406,7 @@ consequence can.
 
 ---
 
-## 8 · `domain_context.json` → `DomainContext` — 6 fields *(single object)*
+## 8 · `document_brief.json` → `DocumentBrief` — 6 fields *(single object)*
 
 > ✅ **The v3 reshape is complete, corrected 2026-09-08.** This banner previously said the
 > LLM-facing parts — `purpose` as a span-guarded verbatim quote, and the `subdomains` aggregate —
@@ -358,7 +421,7 @@ consequence can.
 
 ```json
 {
-  "domain_id": "d_2ef60a8c19",
+  "brief_id": "d_2ef60a8c19",
   "title": "Standard Parameter Produksi & ECA",
   "purpose_verbatim": "Standard Parameter ini merupakan pedoman dalam perhitungan parameter produksi yang meliputi Quantity (Qty) Physical Availability (PA), Utilization of Availability (UA), Productivity (PTY), serta Equipment Capacity Analysis (ECA).",
   "outline": [
@@ -387,7 +450,7 @@ consequence can.
 
 | Field | Type | Req |
 |---|---|---|
-| `domain_id` | string | ✅ **New in v2** |
+| `brief_id` | string | ✅ **New in v2.** `b_` prefix — reconciled with the code 2026-09-11 |
 | `title` | string \| null | — |
 | `purpose_verbatim` | string \| null | — **Renamed + guarded 2026-09-02** (was `purpose`) |
 | `outline` | string[] | ✅ Default `[]`. **Added 2026-09-02** |
@@ -436,7 +499,7 @@ document with no headings.
 
 ---
 
-## 9 · `review_queue.json` → `ReviewQueueRow[]` — 11 fields
+## 9 · `review_queue.json` → `ReviewQueueRow[]` — 12 fields
 
 **This is the consumption surface** — the payload a review UI binds to. Deliberately denormalised so
 a row renders with no joins.
@@ -465,7 +528,7 @@ a row renders with no joins.
     "rank": 2,
     "row_kind": "key_parameter",
     "term_id": null,
-    "domain_id": "d_2ef60a8c19",
+    "brief_id": "d_2ef60a8c19",
     "term": "Grouping (Composite)",
     "definition": null,
     "source_wording": null,
@@ -483,7 +546,8 @@ a row renders with no joins.
 | `rank` | int | ✅ | 1-based display order — see sorting below |
 | `row_kind` | `term` \| `key_parameter` | ✅ | **New 2026-09-02.** Switch on this before reading anything else |
 | `term_id` | string \| null | ✅ | **New in v2.** What an approve / edit / reject decision attaches to, so it survives a re-run. **Null on a `key_parameter` row** — that it resolves to no term is the reason the row exists |
-| `domain_id` | string | — | `key_parameter` rows only — what the decision attaches to instead |
+| `brief_id` | string | — | `key_parameter` rows only — what the decision attaches to instead. **Named `brief_id` on the wire**, not `domain_id`; see the correction under Sort order |
+| `surface_full` | string \| null | — | `key_parameter` rows only. **New 2026-09-11.** The untruncated surface when `term` had to be shortened to a head; `null` when it did not |
 | `term` · `definition` · `source_wording` · `mention_count` | | | Copied from the entry |
 | `page` · `page_no` · `section_no` · `span` | | | **Lifted out of `provenance`** so a row is self-contained. Render **`page_no`** (1-based), not `page` |
 | `review_reason` | string | ✅ | Human-readable, one of five |
@@ -492,11 +556,42 @@ a row renders with no joins.
 `conflict_variants` — all four are already expressed by `review_reason`, and remain available on the
 entry itself.
 
-**Sort order** — `(conflicts, then unresolved key parameters, then mention_count descending)`.
-Conflicts outrank everything, because a contradiction is a decision only the expert can make.
-Unresolved key parameters sit second: the branch that produced them has **no span check at all**, so
-an unresolvable surface is the one hallucination signal it can raise, and it must not sink beneath
-sixty routine rows where nobody reaches it.
+**Sort order** — **changed 2026-09-11.** `(conflicts, then mention_count descending, with unresolved
+key parameters spliced in at most one per five term rows)`.
+
+Conflicts still outrank everything, because a contradiction is a decision only the expert can make.
+What changed is the tier beneath them. Unresolved key parameters used to occupy one, which meant
+**every** unresolved parameter outranked **every** routine term regardless of how many there were —
+and the branch produces them in bulk on exactly the documents it understands least. On the Komatsu
+shop manual that was **24 junk rows standing in front of `Engine`, a term with 108 mentions**, which
+sat at rank 25.
+
+The original reason for promoting them is preserved and is still sound: that branch has **no span
+check at all**, so an unresolvable surface is the one hallucination signal it can raise, and it must
+not sink beneath sixty routine rows where nobody reaches it. A **cap** expresses that without letting
+the tail take the queue over. Measured by replaying the three persisted v2 runs:
+
+| Document | unresolved key parameters | first one at rank | top term, rank |
+|---|---|---|---|
+| BUMA | 0 | — | `Qty` 1 → 1 (unchanged) |
+| Komatsu | 24 | 1 → **6** | `Engine` **25 → 1** |
+| Open Pit | 7 | 1 → **6** | `Resource` **8 → 1** |
+
+Row counts are identical before and after on all three — **nothing is dropped**; parameters the
+splice cannot place are appended rather than discarded.
+
+**Also new on `key_parameter` rows: `surface_full`.** The branch is supposed to return a parameter
+*name* and on a textbook it returns the right concept wrapped in its whole defining sentence (185
+characters, in one measured case). `term` now carries a head — split at the first `.`, `:` or newline
+— and `surface_full` carries the untruncated text, or `null` when no shortening was needed. The head
+is only taken past 60 characters, which is what keeps a legitimate short name containing a period
+("No. of units") intact.
+
+> ⚠️ **Field-name correction, 2026-09-11.** This document described the `key_parameter` row's owning
+> id as **`domain_id`**. The code emits **`brief_id`** and always has: the K12 rename changed the
+> entity from `BriefContext` to `DomainContext` but left the id field's name alone, deliberately —
+> renaming it is a migration that invalidates every expert approval recorded against the old prefix.
+> **The wire field is `brief_id`.** Corrected here rather than in code, per the trust order.
 
 **`review_reason` values:**
 

docs/knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md

@@ -11,6 +11,40 @@
 Companion to `KNOWLEDGE_OUTPUT_CONTRACT.md` (extraction). This is the half **upstream** of it:
 parsing emits `ParsedDocument`, extraction consumes it and never touches the parser.
 
+> ## ⚠️ schema_version **0.4.0** — asset pointers (2026-09-10)
+>
+> **A figure is no longer its own chunk.** Its markdown reference is folded into the prose chunk at
+> its reading-order position, and the figure travels as an `Asset` on that chunk. This is a **MINOR**
+> bump, not a patch: chunk boundaries change, so a 0.3.x consumer would silently read a different
+> document shape. `adapter.check_schema_version` rejects a newer minor by design — parsing and
+> extraction ship together.
+>
+> **Why.** Before 0.4.0 an image became its own chunk with an **empty `text`**, which made it
+> unreachable by construction: no text → no term mentions → joins no cluster → never selected as
+> evidence. Measured across three documents: **13 figures, every one described by a vision model we
+> pay for, and ZERO appearing in any cluster's evidence.** Meanwhile the prose that said
+> "Figure 1.1" had its `![](…)` reference stripped, so nothing connected the sentence to the picture.
+>
+> **What changed**
+>
+> | | 0.3.3 | 0.4.0 |
+> |---|---|---|
+> | A figure | its own chunk, `text=""`, unreachable | folded into the prose chunk as `![](asset://<id>)`, carried in `Chunk.assets` |
+> | A table | its own chunk | **unchanged** — its linearised text earns its evidence slot (74.6% of the Komatsu manual's top-3 evidence is tabular). Gains an `Asset` + two-way pointers |
+> | Asset bytes | returned by the API, **written nowhere**; `images[]` pointed at files that did not exist | content-addressed and persisted; `Asset.storage_key` |
+> | Asset identity | the parser's page-local filename | **`sha256(bytes)[:16]`** |
+> | A caption | landed in a *neighbouring* prose chunk, unlinked | attached to the `Asset` **and** left in the prose text (it is verbatim document content) |
+> | A description | stored raw, JSON braces included | unwrapped; `description_source` records what produced it |
+>
+> **Measured effect** (same three documents, no re-parse — the cache fingerprint is unchanged):
+> chunks 32→28, 85→80, 59→44 · **image chunks 2/3/8 → 0/0/0** · empty-text chunks → 0 ·
+> 13 assets persisted · 0 dangling references · **0 descriptions leaked into `text`** ·
+> captions still 17/17 in text · recall unmoved.
+>
+> **The invariant that must not break:** a model-written `description` never enters `Chunk.text`.
+> `text` is the haystack the span check searches, so generated prose in there would let a
+> fabrication pass the control built to catch it.
+
 > **Current state (2026-09-02) — additive changes since the reference run below.** The default
 > parser is now **Mistral OCR** (`parser_name: "mistral-ocr"`, `parser_backend: "mistral-ocr"`),
 > API-based and needing **no GPU**. The artifact shape is **identical whichever parser produced it** —
@@ -90,7 +124,7 @@ Three conventions hold:
 | `source_title` | string \| null | — | **(0.3.2)** The document's human-readable name. Equals `doc_id` today; a separate field so the two can diverge once `doc_id` becomes a content hash |
 | `normaliser_version` | string \| null | — | **(0.3.1)** Fingerprint of the normalisation code that produced these chunks. `parser_config` covers the parser's settings; this covers our half |
 | `version` | int | ✅ | Version of **this artifact for this document**; bumped on re-parse |
-| `schema_version` | string | ✅ | Versions **the contract**. `0.3.0` on this reference run; **current `0.3.3`** (see the Current-state note above) |
+| `schema_version` | string | ✅ | Versions **the contract**. `0.3.0` on this reference run; **current `0.4.0`** (see the banner above) |
 | `created_at` | string \| null | — | ISO 8601, seconds precision |
 | `raw_output_dir` | string \| null | — | Untouched MinerU output, kept content-addressed and never copied |
 
@@ -141,11 +175,64 @@ Three conventions hold:
 | `is_tabular` | bool | ✅ | |
 | `source_items` | int[] | ✅ | Indices into MinerU's `content_list.json` — the trace back |
 | `bbox` | int[] \| null | — | `[x0, y0, x1, y1]` of the first source item. Carried for a curation UI; **nothing in the pipeline reasons about it** — ordering uses item sequence, never coordinates |
-| `images` | string[] | ✅ | Paths relative to `raw_output_dir`. Formula images, table/chart crops |
+| `images` | string[] | ✅ | ⚠️ **DEPRECATED 0.4.0** — superseded by `assets`. Still populated for one version so a 0.3.x reader degrades rather than breaking. Holds the parser's own filenames, which are **not reliable**: Mistral reported `image_id: img-2.jpeg` on a block whose markdown said `img-1.jpeg`, so two figures collided and one image's bytes were lost. Read `assets` |
+| `assets` | Asset[] | ✅ | **New 0.4.0.** Figures and tables this chunk contains or points at. Default `[]` |
+| `referenced_by` | string[] | ✅ | **New 0.4.0.** `chunk_id`s pointing AT this chunk. Populated on table chunks. Default `[]` |
 | `latex` | string[] | ✅ | Source markup for formulas, kept beside the rendered prose. ⚠️ **LOAD-BEARING** — extraction's formula guard validates against this field. Reflowing it or leaving it empty weakens the check with no error. See below |
 | `table_html` | string \| null | — | Source markup for tables |
 | `generated_description` | string \| null | — | ⚠️ **MODEL-WRITTEN, NOT FROM THE DOCUMENT.** MinerU's description of an image or chart. **Never span-checkable** — treat as a model claim about a picture, never as evidence. See below |
 
+## 2b · `Asset` *(new at 0.4.0)*
+
+```json
+{
+  "asset_id": "d2e7b4ffe31006ec",
+  "kind": "figure",
+  "storage_key": "assets/d2e7b4ffe31006ec.jpeg",
+  "page_idx": 2,
+  "page_no": 3,
+  "bbox": [93, 562, 434, 907],
+  "caption": "Figure 1.1. The relationship between exploration information, resources and reserves (SME, 1991).",
+  "description": "The diagram illustrates the classification of mineral resources and reserves based on increasing levels of geological knowledge…",
+  "description_source": "mistral-ocr/bbox_annotation"
+}
+```
+
+| Field | Type | Req | Notes |
+|---|---|---|---|
+| `asset_id` | string | ✅ | **`sha256(bytes)[:16]`.** Not the parser's id — see the `images` row. Content addressing makes a re-parse idempotent and de-duplicates a figure that appears twice (Open Pit has one such pair) |
+| `kind` | enum | ✅ | `figure` · `chart` · `diagram` · `table` — one vocabulary for both cases |
+| `storage_key` | string \| null | — | Where the bytes live. `null` for a table (no bytes of its own) and whenever persistence was skipped or failed — writing assets is a **never-throw** seam |
+| `page_idx` | int | ✅ | 0-based, as the parser reports |
+| `page_no` | int | ✅ | **Computed** `page_idx + 1`. The number a human reads; derived so it cannot drift |
+| `bbox` | int[] \| null | — | `[x0,y0,x1,y1]` of the **asset**, so a UI can highlight the figure. Distinct from the host chunk's `bbox`, which stays where the text is |
+| `caption` | string \| null | — | **VERBATIM** from the document's own caption block. Quotable as evidence, span-checkable. Also left in the prose `text`, where it already landed — the asset gets a copy, not a move |
+| `description` | string \| null | — | ⚠️ **MODEL-WRITTEN.** Never span-checkable, never evidence, and **never in `text`** |
+| `description_source` | string \| null | — | e.g. `mistral-ocr/bbox_annotation`, so a later model swap is traceable rather than silently changing what the field means |
+
+**A figure's reference appears inline in its host chunk's `text`**, at the reading-order position the
+document put it:
+
+```text
+…so closely and the geological character is so well defined that size, shape, depth
+and mineral content of the resource are well established.
+
+![](asset://d2e7b4ffe31006ec)
+
+Figure 1.1. The relationship between exploration information, resources and reserves (SME, 1991).
+
+– Indicated. Quantity and grade…
+```
+
+Two consumer obligations:
+
+- **Resolve `asset://<id>` against that chunk's own `assets`.** Every reference is guaranteed to
+  resolve locally (0 dangling across the three test documents).
+- **Switch on the caption/description distinction.** A caption may be quoted and used as a provenance
+  span; a description may not, and a span taken from one is rejected by the validator.
+
+---
+
 ### Why `text` is rendered but markup is kept
 
 The term filter downstream is an NER model reading prose. MinerU writes formulas character-spaced

docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md

@@ -26,7 +26,7 @@ Companions: [KNOWLEDGE_PIPELINE_TODO.md](KNOWLEDGE_PIPELINE_TODO.md) (plan) ·
 | Model | `urchade/gliner_multi-v2.1` | CPU, ~400 MB. Cheap enough to run unconditionally on every chunk, which is what lets us reject a standalone relevance gate |
 | Score threshold | **0.25** | Not 0.35. The sweep measured recall **0.854 @ 0.25** vs **0.658 @ 0.35** on the `broad` label set. Precision falls (0.41 vs 0.50) and that is accepted: the filter is meant to be over-inclusive, clustering and evidence ranking absorb the noise, and **a term the filter never proposes can never be recovered downstream** |
 | Label variant | **`broad`** | Two earlier variants both missed the same class — mining activities and materials (coal mining, land clearing, waste removal, joint survey). `broad` adds `mining activity`, `material or commodity`, `data source or system`, `work category`, `analysis method` to cover it |
-| Window size | **130 words**, overlap **30** | Not 220. GLiNER truncates past ~384 of its own tokens and **warns rather than failing** — silently dropping the tail of every long chunk. Indonesian technical prose subword-tokenises at roughly **2.5×**, so a 220-word window still tripped the cap. Chunks are fed as overlapping word windows with offsets remapped |
+| Window size | **130 words**, overlap **30** (re-confirmed 2026-09-10: 256 truncates 46 of 99 Open Pit windows, 180 truncates 4; 130 is the only size measured safe on both English documents. One BUMA window exceeds the cap at EVERY size — pre-existing) | Not 220. GLiNER truncates past ~384 of its own tokens and **warns rather than failing** — silently dropping the tail of every long chunk. Indonesian technical prose subword-tokenises at roughly **2.5×**, so a 220-word window still tripped the cap. Chunks are fed as overlapping word windows with offsets remapped |
 | Span token cap | **12** (GLiNER's own) | Compound terms ("UA plan grouping (composite) actual") exceed it. Candidates that hit the cap are **flagged, not silently truncated**, so a recall shortfall can be attributed to the cap |
 
 **Label sets** (`labels.yaml`) — three variants were tried and A/B'd without touching code. Keep
@@ -40,7 +40,7 @@ Indonesian-phrasing variant (`istilah produksi tambang`, …) did **not** win; `
 | Constant | Value | Why this value |
 |---|---|---|
 | Match order | exact (normalised) → abbreviation↔expansion → fuzzy | Cheapest and most certain first |
-| Fuzzy threshold | **92** (`rapidfuzz.token_set_ratio`) | Conservative on purpose |
+| ~~Fuzzy threshold~~ | ~~**92**~~ | ⛔ **RETIRED 2026-09-10.** `token_set_ratio` returns exactly 100 when one token set is a SUBSET of the other, so this threshold was **inert** and a head noun always absorbed its compounds. 131 of 140 merges were subset matches; the 6 correct ones were plurals, now handled by `normalize(lemma=True)`. The pass is commented out at its call site; `_fuzzy_match` is parked. Subset merges went to 0 on all three test documents |
 | **Fuzzy disabled below 5 characters** | `min(len(a), len(b)) < 5` → skip | **The one to not lose.** `PA` vs `UA` scores high on `token_set_ratio`. Below 5 chars only exact matching is allowed |
 | Canonical surface | shortest variant ≥ 2 chars | That is how a reader looks a term up — `PA`, not `Physical Availability (PA) untuk …` |
 
@@ -77,17 +77,54 @@ needs the tail.
 
 | Constant | Value | Why |
 |---|---|---|
-| `evidence_k` | **3** | Chunks fed per extraction round |
+| `evidence_k` | **5** (raised from 3, 2026-09-10) | Chunks fed per extraction round |
 | Cue proximity | **100 chars** | Distance from mention to a definitional cue |
+| `evidence_link_partners` | **1** (added 2026-09-11) | Linked partner chunks pulled into the window on top of the ranked K |
+
+### Linked partners (2026-09-11)
+
+Schema 0.4.0 pairs a table chunk with the prose that introduces it, via `referenced_by` in both
+directions. `top_k` now pulls **at most one** partner into the evidence window.
+
+**Why it is not redundant — the measurement that decided it.** Ranking only, on the three v2
+artifacts and their recorded filter payloads (free; no GLiNER, no spend):
+
+| | clusters | top-5 touches a linked pair | **both sides already there** | only one side |
+|---|---|---|---|---|
+| BUMA | 83 | 31 | **0** | 31 |
+| Komatsu | 93 | 55 | **0** | 55 |
+| Open Pit | 427 | 70 | **0** | 70 |
+
+**Zero clusters out of 603 saw both sides.** The ranker never co-selects a table and the prose that
+names it — and that prose is frequently the table's only title (*"Troubleshooting of engine
+(S-mode)"* above a bare `S-1 | 6` grid). Cost at the cap: evidence read **+14.5% / +24.5% / +9.9%**,
+which is **+1.6% / +4.1% / +4.2%** of total prompt spend; Open Pit moves 71% → ~74% of the 2M ceiling.
+
+**Two placement rules, both load-bearing.** The expansion lives in `top_k`, *not* `rank_evidence`:
+
+1. The **ranked list is never mutated**, so everything reading `evidence_chunk_ids` sees the number
+   it saw before.
+2. `evidence_block` (the prompt) and `evidence_text` (the span check) both take their ids from the
+   same `top_k` call, so *what the model reads is exactly what the span check searches* stays true.
+   Expanding in the prompt renderer instead would null every quote from a partner as unlocatable.
+   Verified across all 603 clusters: **0 parity failures**.
+
+**Scope, stated honestly.** The document this helps most (Komatsu, 55 of 93 clusters) is the one that
+returns 0 definitions for *genre* reasons. This is groundwork for the table branch (Step 8) rather
+than a glossary win, and it has **not** been measured against a paid run — with this corpus's noise a
+single run could not have shown anything either way. Set `EVIDENCE_LINK_PARTNERS = 0` to disable.
 
 **Heading matching is word-boundary, never substring.** `PA` is a substring of `Parameter`, `pada`
 and `composite` — substring matching handed the +4.0 heading bonus to almost every Indonesian
 heading and pushed real definition sections *below* formula tables.
 
 **Why this filter is legitimate when the relevance gate was rejected:** no term is dropped (only its
-*evidence* is narrowed), every mention stays in provenance so a bad top-3 is visible, and the
-escalation loop self-corrects. The rejected gate had none of those three properties. Preserve all
-three if ranking is ever revisited.
+*evidence* is narrowed), every mention stays in provenance so a bad top-3 is visible, and the full
+ranked list is retained so the tail stays reachable. The rejected gate had none of those three
+properties. Preserve all three if ranking is ever revisited. (The third property used to be phrased
+as "the escalation loop self-corrects"; escalation was retired on 2026-09-11 with
+`MAX_ESCALATION_ROUNDS = 0`, but retaining the tail is what made the property true and it still
+holds.)
 
 ---
 

docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md

@@ -10,6 +10,13 @@
 
 **Status legend:** ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 built but unverified · ⏸️ deferred
 
+> **Updated 2026-09-11 — this file had drifted, and the drift is worth naming.** Rows **X17**, **D2**
+> and **D4** read ⬜ *"not started"* for four days after persistence and the HTTP surface shipped
+> (2026-09-07), while DEV_PLAN §0.8 recorded both ✅. Anyone reading this file first would have
+> concluded the pipeline was still offline. Corrected below. This file also records **nothing of v2**
+> (K13–K16, 2026-09-10/11) or of the 2026-09-11 open-queue pass (K17–K23) — for those, **DEV_PLAN
+> §0.8 is canonical** and this checklist covers the original build-out only.
+
 > **Updated 2026-08-26.** Both halves are in-tree and run end to end as offline CLIs; the seam is
 > settled and consumed from both sides. Tracked in DEV_PLAN §0.8, described in REPO_STATUS §14.
 > The full-document run landed (X18) and scored E3 = 0.90. **Schema v2 landed 2026-08-26 (X24)** —
@@ -203,7 +210,7 @@ envelope shape proposed in §3 — so S1 can settle either way without touching
 | **X14** | Diff vs. active glossary version | 🔎 | ✅ | `diff/glossary_diff.py`. The baseline is passed in explicitly (`--active-glossary`) rather than read from wherever the last run wrote, so the duplicate and conflicting paths are reachable — all three verified by test |
 | **X15** | Frequency-sorted review queue | ✅ | ✅ | `queue/review_queue.py`. Each row carries page, section and the verbatim span so review is quote-vs-page. Gained a review reason for **wording discrepancies**, which is how the literal-wording decision reaches the expert |
 | **X16** | Bulk vs. incremental ingest | ⬜ | ⬜ | Bulk setup ingest (corpus-frequency statistics as a candidate booster) and incremental single-file add (no corpus context; diff against the active version). Neither exists |
-| **X17** | Persistence | ⬜ | 🔄 | Stage artifacts are written as JSON by the CLI, which is enough for the offline path. Tables still needed → D2 |
+| **X17** | Persistence | ✅ | ✅ | **Landed 2026-09-07** — five tables + `knowledge_store.py`; 88 entry rows persisted from the BUMA standard. This row said ⬜ until 2026-09-11 while DEV_PLAN §0.8 recorded it done. See D2 for the part that is still open (Harry's migration) |
 | **X24** | Output schema v2 (Harry's contract review) | ✅ | 🔎 | **Landed 2026-08-26, UNEVALUATED.** Four asks, all adopted: deterministic ids (`ids.py` — `term_id`/`formula_id`/`rule_id`/`brief_id`, sha256 over content, never positional and never model-written); the glossary references a formula (`defining_formula_id`) instead of restating it; rules link to formulas and terms (`formula_ids[]`, `term_ids[]`); `brief_id` added. New deterministic `link.py` resolves every reference after the branches run — no LLM, no spend, dangle counts reported not swallowed. **16 fields cut, 8 added (50 → 42).** Two defects fixed on the way: `cluster_id` was a frequency-rank ordinal and `rule_id` was written by the model, so **neither survived a re-run** — which silently breaks approval continuity, not just linking. `rule_type` (`interpretation`\|`calculation`) keeps the Interpretation Pack's name while retaining the calculation conventions the corpus actually yields — **0 of 15 gold rules are interpretation rules**, because a *Standard Parameter* document defines how to compute a parameter, not how to read one. Verified: ruff clean on touched paths, `import main` OK, suite **497 passed / 7 skipped**, and a synthetic end-to-end link check (PA resolves to its own defining formula, not the `Production` formula it merely appears in; `Total Hours` dangles to null as designed). **Not verified: any eval** — see the banner; the prompt edits are in the same unevaluated position as X20 |
 | **X26** | Output schema v3 — `DomainContext` | Rifqi | ✅ | ✅ | **Done 2026-09-02** (`7f47c61` · `6f5d619` · `e2865ac` · `172d8dc`). `BriefContext` → **`DomainContext`** (`brief_id` → `domain_id`, `b_` → `d_`) — renamed before persistence exists, because after it a prefix change is an id migration that loses approvals. **The branch stopped being the pipeline's one unverifiable output:** `GUARDED_FIELDS["summary"]` went `()` → `("title", "purpose_verbatim")`, achieved by changing the QUESTION rather than the model — the prompt now asks the model to *locate* the document's own statement of purpose instead of composing a summary, and a quote is checkable where a summary never was. `_run_summary` also never called `validate_entry`, so the guard table alone would have been inert. New derived fields: **`outline`** (ordered unique `heading_path`; needed `heading_path` to cross the adapter, which was dropping it exactly as it once dropped `latex`), **`subdomains`** (roll-up of per-term `subdomain_tags`, frequency-ordered with an alphabetical tiebreak — load-bearing, since `production`/`equipment` tie on the reference doc), and **`n_terms`/`n_formulas`/`n_rules`**. `key_parameters` became `KeyParameter{surface, term_id}`, corroborated in `link.py`. **Smoke found a real bug:** exact lookup resolved 0/5 because the document names parameters `Full Name (ABBR)` while the glossary is keyed on the abbreviation — five FALSE hallucination signals sorted above every real term. `_match_parameter` now tries whole → expansion → abbreviation: **0/5 → 4/5**, ECA correctly still dangling. Measured (BUMA, nano, 12 calls): **0 fields rejected**, fabricated purpose refused, 11-entry outline. Contract §8/§9 quote that run literally. Suite 541 passed / 7 skipped |
 
@@ -214,9 +221,9 @@ envelope shape proposed in §3 — so S1 can settle either way without touching
 | # | Decision | Owner | Status | Description |
 |---|---|---|---|---|
 | **D1** | New dependencies | Rifqi | ✅ | **All approved and added 2026-08-19:** `rapidfuzz>=3.14.5`, `gliner>=0.2.13` (resolved 0.2.28), `torch>=2.6` (resolved 2.11.0+cpu). Install verified clean — `import torchgen` OK, so not the corrupt-wheel failure the calibration doc warns about. `PyYAML` was already present transitively |
-| **D2** | New tables (DDL handoff) | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns the dedorch schema — Python never executes DDL. One consolidated Harry-ready handoff beats three |
+| **D2** | New tables (DDL handoff) | Rifqi → Harry | ✅ (Python) / ⬜ (Go) | **Python side done 2026-09-07**: five tables exist in dedorch exactly as `docs/knowledge/knowledge_tables.sql`, with ORM models and `PostgresJobStore`. ⚠️ **They were created by a ONE-TIME MANUAL RUN.** Go owns the migration and Harry must still fold the SQL in, or a fresh environment comes up without them — the same gap that left `message_traceability` and `message_charts` out of Go's set (Q5) |
 | **D3** | Model tier per branch | Rifqi | ✅ | **Decided 2026-08-19: stay on nano for all four branches.** The `.env` carries the quad `azureai__{api_key,endpoint__url,deployment__name,api__version}__54n`, mirroring the existing `__54m` scheme. Accepted knowingly: nano measured 0.75 against a 0.80 line, and `rule`/`summary` run on the tier whose failure mode is least detectable. Revisit if extraction quality blocks the demo. **`src/config/settings.py` does not expose the `__54n` quad yet** — four `Field(alias=…)` entries, first build step |
-| **D4** | Admin ingest surface | Rifqi | ⬜ | Whether it is HTTP at all, and if so its auth posture. A write surface triggered by an admin is a different risk class from the current unauthenticated read-only chat surface — raise it, don't inherit the posture by default. Only after the offline path works |
+| **D4** | Admin ingest surface | Rifqi | ✅ built / ⛔ auth | **Built 2026-09-07** — 8 routes under `/api/v1/knowledge`, async batch (202 + poll), company-scoped, rate-limited, spend-gated, single-flight. **The auth question it was raised to answer is still open and was re-opened 2026-09-07**: the posture was inherited by default, `POST …/reviews` accepts content that reaches a later prompt, and the caller is the browser SPA (#37/#43). Recorded, not solved |
 | **D5** | v2 module name | Rifqi | ✅ | **`src/knowledge_extraction/`** (2026-08-19). Offline runner in `scripts/`, per the repo's existing precedent |
 | **D6** | Expert review of the gold set | Rifqi → Mas Beta | ⏸️ | **Deferred 2026-08-19: provisional gold is good enough for now** so the rebuild is not blocked on expert time. Standing caveat: E1 and E3 both move when it is reviewed, and E3's scoreable base is 8 entries, so each adjudication shifts it ~12 points. Every E3 failure is listed verbatim in the frozen baseline for when he does review it |
 | **D7** | GLiNER + LangExtract ensembling | — | ⏸️ | Proposed, not decided; flagged as scope creep. Cheap in API terms but adds a reconciliation step between two candidate sets that may disagree. Not in scope for the test integration |

docs/knowledge/TABLE_BRANCH_PROPOSAL.md

@@ -0,0 +1,196 @@
+# Table branch — design proposal (Step 8)
+
+<!-- doc-status -->
+> 💡 **PROPOSAL — nothing is built from this.** Written 2026-09-11 against the open queue in
+> `knowledge_pipeline_runs/SESSION_HANDOFF.md` §4.6. Needs Rifqi's sign-off before any code.
+> Roll-up: [DEV_PLAN.md](../../DEV_PLAN.md) §0.8 · built state: [REPO_STATUS.md](../../REPO_STATUS.md) §14.
+
+**Owner:** Rifqi · **Size:** weeks · **Depends on:** parsing 0.4.0 (shipped), genre classification
+(shipped 2026-09-11, advisory) · **Blocks:** any useful result on diagnostic documents.
+
+---
+
+## 1 · The problem, stated as a number
+
+The Komatsu HD785-7 troubleshooting manual is **74% tabular** — 59 tables across 80 chunks — and the
+pipeline extracts **zero** knowledge entries from it. Two paid runs, v1 and v2, both returned **0
+definitions** from 75 and 93 clusters respectively. That is not a tuning failure and no prompt change
+reaches it: a shop manual does not define its terms, it **asserts relationships between them**, and
+it does so in a matrix.
+
+The genre classifier (`src/knowledge_extraction/genre.py`) now names this document `diagnostic` and
+reports `well_served = False` before anything is spent. This proposal is what would make that answer
+change.
+
+## 2 · What the document actually contains
+
+A troubleshooting table is a **symptom × cause matrix**. Each marked cell is one assertion:
+
+| Symptom \ Cause | Fuel filter clogged | Injector worn | Air cleaner blocked |
+|---|---|---|---|
+| Engine does not start | ● | ● | |
+| Black exhaust smoke | | ● | ● |
+| Output drops | ● | | ● |
+
+Three assertions come out of the first row alone, and **none of them is a definition**:
+
+> *"Engine does not start" may be caused by "Fuel filter clogged".*
+
+The unit of knowledge here is the **(row header, column header, cell mark) triple**, and it is
+recoverable **deterministically** — the mark is in the parsed HTML, the headers are in the parsed
+HTML, nothing needs to be inferred.
+
+## 3 · The shape of the branch
+
+**Deterministic extraction, LLM only to phrase.** This is the inversion of every existing branch, and
+it is the whole point: the assertion's *existence* is read off the table, so the model cannot
+fabricate a relationship that is not in the document — it can only phrase one that is.
+
+```
+table chunk ──► parse table_html ──► (row header, col header, cell mark) triples
+                                              │
+                                              ├── deterministic: the assertion EXISTS
+                                              │
+                                              ▼
+                                    one LLM call per TABLE (not per cell)
+                                              │
+                                              ▼
+                                    natural-language phrasing + typing
+                                              │
+                                              ▼
+                                    span check against the cell's own headers
+```
+
+**Provenance points at the cell**, which is stronger than anything the pipeline produces today: the
+span is the row header plus the column header, both verbatim document text, both locatable by the
+existing `span_present`.
+
+### 3.1 · New entity
+
+```python
+class RelationEntry(BaseModel):
+    relation_id: str          # sha256 over (doc_id, table asset_id, row, col) — positional
+                              # within a table, but the table's asset_id is content-addressed,
+                              # so a re-parse of identical bytes re-mints identical ids.
+    kind: Literal["symptom_cause", "component_action", "condition_effect"]
+    subject: str              # verbatim row header
+    object: str               # verbatim column header
+    statement: str | None     # model-phrased; span-checked against subject + object + caption
+    table_asset_id: str
+    provenance: Provenance    # span = the two headers; page from the chunk
+    term_ids: list[str]       # links into the glossary where headers match known terms
+```
+
+### 3.2 · Why one call per table, not per cell
+
+Komatsu's 59 tables carry on the order of 10³ marked cells. One call per cell is a different cost
+class entirely and would blow the pre-flight ceiling by itself. One call per table hands the model
+the linearised table plus the extracted triples and asks it to phrase each one — 59 calls, which is
+**less than the 93 the glossary branch currently spends to abstain 93 times.**
+
+## 4 · What has to be decided before coding
+
+| # | Question | Why it blocks |
+|---|---|---|
+| T1 | Is a relation a first-class **entry kind** (a fifth `kind` in `knowledge_entries`) or a new table? | Persistence contract + Harry's migration. A fifth kind is additive; a table is a DDL handoff |
+| T2 | What does a reviewer **do** with 1,000 relations? | The review queue is frequency-sorted and finishable by construction. 1,000 relations is not finishable. Likely answer: review per TABLE, not per relation — one approve covers the matrix |
+| T3 | Does a mark type carry meaning (`●` vs `○` vs `▲`)? | Komatsu uses more than one glyph. If they mean "primary/secondary cause" the triple is really a quad, and getting this wrong silently inverts assertions |
+| T4 | Which `table_html` shapes does the parser actually emit — merged cells, multi-row headers? | Determines whether header resolution is 20 lines or a small parser. **Measure before designing** |
+| T5 | Does this ship behind the genre router (`diagnostic` only) or for every document? | BUMA is 25% tabular and its tables are *data*, not relation matrices. Running this on them produces noise |
+
+**T4 is the one to do first** and it is free: inspect the 59 Komatsu tables and count the shapes.
+
+## 5 · Kill lines, agreed before the first paid run
+
+Copying the discipline that made the clustering defect findable — a metric that moves the *wrong* way
+is worse than no metric (E2), so these are stated as recall against a **human-read sample**:
+
+| Metric | What it measures | Kill line |
+|---|---|---|
+| **T-E1** | Triples extracted ÷ triples a human reads off the same 5 tables | ≥ 0.90 — it is deterministic; anything less is a parser bug |
+| **T-E2** | Phrasings whose span check passes | ≥ 0.95 |
+| **T-E3** | Phrasings a human judges to *invert or distort* the relation | 0 — this is the one that matters, and it is the reason the mark glyph question (T3) is not cosmetic |
+| **T-cost** | Calls per document | ≤ 1 per table |
+
+**T-E3 has no automated proxy.** It needs Mas Beta or an equivalent reader on a sample, which makes
+the R4 conversation (does "found but undefined" help an expert?) worth having in the same sitting.
+
+## 6 · What this does NOT solve
+
+A relation matrix is not a glossary. After this branch ships, the Komatsu manual still yields **zero
+definitions** — correctly, because it contains none. If the product's promise is "a glossary from
+your documents", a diagnostic document will always disappoint it, and the honest place to say so is
+the ingest response, which the genre classifier now makes possible.
+
+---
+
+## 7 · Groundwork already landed (2026-09-11)
+
+Two pieces of this branch's plumbing exist before the branch does, and both are measured:
+
+**The pointer is populated and durable.** 0.4.0 gives every table chunk an `Asset` whose id comes
+from `synthetic_asset_id(doc_id, chunk_id, item_index)` — stable document coordinates, so it survives
+a re-parse — plus `referenced_by` naming the prose chunk that introduces it, and the reverse pointer
+on that prose chunk's `assets`.
+
+**One linked partner now joins the evidence window** (`EVIDENCE_LINK_PARTNERS = 1`, in `rank/evidence.py`).
+The measurement that motivated it: across 603 clusters on the three test documents, **zero** had both
+sides of a linked pair in their top-5, so a table reached the model without the prose naming it, and
+vice versa. 31/83, 55/93 and 70/427 clusters now gain a partner, at +1.6%/+4.1%/+4.2% of prompt spend.
+
+**What this does *not* do, and the branch must:** `table_html` crosses the seam into
+`Chunk.table_html` and **nothing reads it** — the prompt shows the linearised `render_table` text
+from `Chunk.text` instead. That is deliberate: the span check validates against `heading + text`
+only, so putting HTML in front of the model would let it quote markup that is then nulled as
+unlocatable. A branch that wants the real cell structure has to extend the span-checked text too, or
+carry provenance that points at a cell rather than a character span (§3.1's `cell_ref` does the
+latter, which is the cleaner of the two).
+
+
+---
+
+## 8 · Correction and re-diagnosis of the Komatsu tables (2026-09-11)
+
+> **Correction.** An earlier reading of this corpus (same day, recorded in the session
+> notes) said the troubleshooting charts had **lost their column headers**, on the
+> evidence that only 4 of 59 tabular chunks contain a `<th>`. That count is accurate
+> and the conclusion drawn from it was **wrong**. The headers are present; they are
+> emitted as ordinary `<td>` cells in a **separate chunk**.
+
+### What Mistral actually produces
+
+Each chart is split into three or four sibling tables, and the first cell of each names
+the fragment. Across the 59 tabular chunks:
+
+| first cell | count | what it holds |
+|---|---|---|
+| `Causes` | 17 | **the column headers** — the cause list |
+| `Troubleshooting` | 16 | check rows |
+| `Questions` | 15 | interview/symptom rows |
+| `Remedy` | 10 | the action row |
+
+So the (row header, column header, cell mark) triple is recoverable, but its three parts
+live in **different chunks**. The branch's first job is re-assembly, not OCR recovery —
+the fragments are adjacent, same-page, and regularly labelled.
+
+### The defect that is real
+
+**Marks survive in only 20 of 59 tabular chunks.** Doc-wide there are 278 marks
+(18 ◎, 137 ○, 25 △, 98 ●), and they cluster on the pages where the matrix fragment
+kept them; on pages 9-10 the `Questions` and `Troubleshooting` fragments come through
+with row labels and **zero marks**. Column structure and `rowspan`/`colspan` are
+preserved where the marks are.
+
+A chart whose marks were dropped carries no symptom-to-cause link at all, so roughly
+two thirds of the document's assertions are currently unrecoverable from the artifact.
+**That, not the headers, is what gates this branch**, and it has to be fixed at the
+parsing layer.
+
+### Free evidence that the content is reachable
+
+Adding a `fault code` label to the span filter (see `config/labels.yaml`) surfaced
+**66 cause phrases** as clusters on this document — *"Clogged fuel filter and/or
+strainer"*, *"Improper injection timing"*, *"Defective alternator (regulator section)"* —
+plus 17 service codes. Those are the column headers, read out of the `Causes`
+fragments. The raw material for the branch is already in the artifact and already
+findable; what is missing is the mark matrix that connects it to the symptoms.

docs/knowledge/TERM_HIERARCHY_PROPOSAL.md

@@ -0,0 +1,122 @@
+# Parent/child terms and cross-document identity — design proposal (Step 7 + R7)
+
+<!-- doc-status -->
+> 💡 **PROPOSAL — nothing is built from this.** Written 2026-09-11 against `SESSION_HANDOFF.md` §4.7.
+> Needs Rifqi's sign-off, and **must not be built before the reading in §6 is done.**
+> Roll-up: [DEV_PLAN.md](../../DEV_PLAN.md) §0.8 · built state: [REPO_STATUS.md](../../REPO_STATUS.md) §14.
+
+**Owner:** Rifqi · **Size:** a quarter or more · **Touches:** `entity_id` derivation, which is what
+every expert approval hangs on. **This is the one change on the queue that cannot be done twice.**
+
+---
+
+## 1 · Why v2 sharpened this rather than solving it
+
+Retiring the fuzzy clustering pass (K13) was correct and is not in question: `token_set_ratio`
+returned 100 on token-subset pairs, so a head noun always absorbed its own compounds — 131 of 140
+merges were wrong, and post-cluster recall rose 0.8293 → 0.8537 when the pass was removed.
+
+But look at what the corpus now says about `engine` and `engine oil`:
+
+| | before K13 | after K13 | the truth |
+|---|---|---|---|
+| `engine` / `engine oil` | **the same cluster** (wrong) | **unrelated** (also wrong) | `engine oil` is a *child concept* of `engine` |
+
+The pipeline has exactly two ways to express a relationship between two terms — same cluster, or no
+relationship — and neither is right. **The defect was never the threshold; it was the vocabulary.**
+Fixing the merge removed the loud wrong answer and left the quiet one.
+
+## 2 · The two problems are one problem
+
+Step 7 (parent/child within a document) and R7 (the same concept across documents) look separable and
+are not. Both ask *"are these two surfaces the same entity, and if not, how are they related?"* — one
+within a document, one across. SciCo's contribution is precisely that **coreference and hierarchy are
+solved jointly**; solving one and bolting on the other produces a model where `engine oil` in
+document A relates to `engine` in document A but has no defined relationship to `engine oil` in
+document B.
+
+They also share the thing that makes this irreversible: **both change `entity_id`.**
+
+## 3 · The constraint that dominates the design
+
+`entity_id` is `sha256` over content and is what an expert's approval is recorded against
+(`knowledge_reviews.entity_id`). Three consequences, all of them already learned here:
+
+1. **K11** — `cluster_id` was a frequency ordinal and `rule_id` was model-written; neither survived a
+   re-run, which broke approvals rather than links. Both are now deterministic.
+2. **K12** — `BriefContext` → `DomainContext` was renamed *while nothing was persisted*, explicitly
+   because afterwards an id-prefix change is a migration that loses approvals.
+3. **`scope_id` is deliberately excluded from the derivation** (2026-09-07), which is what keeps
+   re-pointing a tenant a value migration rather than a re-key.
+
+**So: a hierarchy must not be encoded in the id.** `t_<hash>` must stay a hash of the term's own
+content. Parenthood is an edge between two entities, stored beside them — never a path, never a
+compound key. Getting this wrong means every subsequent hierarchy correction invalidates approvals.
+
+## 4 · Proposed model
+
+```python
+class TermRelation(BaseModel):
+    parent_id: str
+    child_id: str
+    kind: Literal["narrower", "variant", "abbreviation", "related"]
+    evidence: Literal["lexical_subset", "abbrev_pair", "shared_definition", "expert"]
+    confidence: float          # never a hard merge; see §5
+```
+
+Four relationship kinds, because collapsing them is what the fuzzy pass did:
+
+- **`narrower`** — `engine oil` under `engine`. Lexical subset **plus** type agreement.
+- **`variant`** — `Working Hours` / `working hour`. Already handled deterministically by
+  `depluralise()`; this records it rather than changing it.
+- **`abbreviation`** — `PA` / `Physical Availability`. Already a first-class cluster rule.
+- **`related`** — co-defined, no hierarchy. The escape hatch that stops the other three being abused.
+
+Clustering is **unchanged**. Relations are computed *after* clustering, between clusters. That is the
+property that makes this safe to ship incrementally: with the relation table empty, behaviour is
+exactly today's.
+
+## 5 · GLiNER types are a signal, not a veto
+
+Measured 2026-09-10: **48% of token-subset over-merges involved disagreeing GLiNER types.** That is a
+strong enough signal to *type* a proposed relation and far too noisy to *gate* one — the term filter
+runs at threshold 0.25 precisely because it is meant to be over-inclusive, and its labels are
+correspondingly loose. Using a 0.25-threshold label as a hard merge veto would reintroduce the
+clustering defect with the sign flipped: instead of silently merging distinct terms, it would
+silently refuse real relationships.
+
+**Rule: type agreement raises confidence; type disagreement lowers it; neither decides.**
+
+## 6 · Read before building
+
+Non-negotiable, and the reason this is a quarter rather than a sprint:
+
+| Source | What it settles |
+|---|---|
+| **SciCo** (hierarchical cross-document coreference) | Why the two problems are one, and what a jointly-solved model looks like |
+| **ILLOD** (abbreviation–expansion detection) | Whether the abbreviation path should change at all — it is the one relation already working |
+| **Nested-NER survey** | Whether nested decoding helps here after all. Note the local measurement said no (2026-09-10: +7–12% mentions, **zero** recall gain, +40–80% over-merges) — the survey is for understanding *why*, not for re-opening the setting |
+
+## 7 · Sequencing, and what must come first
+
+**This must land before an expert approves anything at volume.** Every approval recorded under the
+current flat model is an approval of a term that may later acquire a parent, and re-keying after the
+fact is the one migration this design exists to avoid.
+
+Concretely, in order:
+
+1. **R4 — ask Mas Beta** whether "term found, no definition in document" is useful or noise. It is one
+   conversation, it is still not had, and it **gates the queue design** this feeds: 48 of 58 BUMA
+   entries and 376 of 425 Open Pit entries are exactly that shape, and whether a hierarchy should
+   surface them differently depends entirely on the answer.
+2. **Step 0** — measurement hardening. A hierarchy change is a *quality* change, and a single run
+   stopped being evidence the moment quality became the subject.
+3. This proposal, reviewed against the reading in §6.
+
+## 8 · What it would buy, honestly
+
+No recall. Recall is a term-filter and clustering property and this changes neither. What it buys is
+**a reviewable structure**: an expert approving `engine` once, with `engine oil`, `engine coolant`
+and `engine speed` beneath it, rather than meeting four unrelated rows at four different ranks. On
+the Open Pit textbook — 425 entries, 49 of them defined — that is the difference between a queue
+someone finishes and a queue someone abandons.

docs/knowledge/knowledge_tables.sql

@@ -1,26 +1,70 @@
 -- Knowledge pipeline tables — dedorch
 --
--- Source of record: docs/knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md §2.
--- This file is the exact SQL that was executed, kept so that what ran is
--- reproducible and reviewable rather than retyped from prose.
+-- HANDOFF TO HARRY. Updated 2026-09-14. Source of record:
+-- docs/knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md §2.
 --
 -- ⚠️ OWNERSHIP. Go owns every dedorch migration (CLAUDE.md §2.2). Python
 -- executes NO DDL, ever: `SKIP_INIT_DB` stays true, there is no `create_all`
--- and no Alembic against this database. These statements were run ONCE by an
--- operator to unblock the build, following the 2026-07-13 `message_charts`
--- precedent — and that precedent's mistake must not be repeated: the schema was
--- run by hand and never handed over, so a fresh environment built from Go's
--- migrations comes up WITHOUT those tables and both write paths fail silently.
+-- and no Alembic against this database. Everything in this file was run ONCE by
+-- an operator to unblock the build, following the 2026-07-13 `message_charts`
+-- precedent — and that precedent's mistake is the one this file exists to avoid:
+-- the schema was run by hand and never handed over, so a fresh environment built
+-- from Go's migrations comes up WITHOUT those tables, and because both write
+-- paths are never-throw seams, nothing errors. They just stop recording.
 --
 --     Run by hand, then hand over. Never run by hand only.
 --
--- Harry: these five tables now exist in dedorch. Nothing here needs designing —
--- the ask is that the migration set match what is already running, so a fresh
--- environment reproduces it. See §5 Q5 for the two older tables in the same
--- position (message_traceability, message_charts).
+-- ─────────────────────────────────────────────────────────────────────────────
+-- WHAT IS LIVE IN DEDORCH RIGHT NOW, AND WHAT YOU ALREADY HAVE
+-- ─────────────────────────────────────────────────────────────────────────────
 --
--- Executed: 2026-09-07 · Idempotent (IF NOT EXISTS throughout) · Additive only:
--- no DROP, no ALTER, nothing touching an existing table.
+--   §1-§5  five tables                      run 2026-09-07   sent ✅
+--   §6     4 columns + 2 indexes            run 2026-09-14   sent ✅
+--   §7     3 columns (review modes)         run 2026-09-14   NOT SENT ⬅ new
+--
+-- **If you adopted the copy sent before 2026-09-14, it is now incomplete in two
+-- ways, and one of them is silent:**
+--
+--   1. §7 is missing entirely. Without it, `POST /knowledge/ingest` with
+--      `review_mode=llm` fails on insert and no entry is ever auto-approved.
+--   2. §1-§5 DRIFTED AFTER the 2026-09-07 run, and `CREATE TABLE IF NOT EXISTS`
+--      will not repair an existing table. Two edits landed:
+--        * `knowledge_entries.kind` CHECK gained 'document'
+--        * `knowledge_entries.run_id` / `.knowledge_document_id` became NULLABLE
+--          for kind='domain' only, guarded by knowledge_entries_parent_check
+--      Both are reflected in §3 below. **Take the current file, not the old one.**
+--
+-- Nothing here needs designing. The ask is only that the migration set
+-- reproduce what is already running, so a fresh environment matches.
+--
+-- ─────────────────────────────────────────────────────────────────────────────
+-- STILL OUTSTANDING FROM AN EARLIER ROUND — not in this file
+-- ─────────────────────────────────────────────────────────────────────────────
+--
+-- `message_traceability` (80 rows) and `message_charts` (6 rows) have been live
+-- since July, created the same way, and are still absent from the migration set.
+-- Their DDL is in KNOWLEDGE_PERSISTENCE_CONTRACT.md §appendix rather than here.
+-- Say the word and it gets folded into this file so one artifact covers every
+-- Python-written table you are missing.
+--
+-- ─────────────────────────────────────────────────────────────────────────────
+-- VERIFIED AGAINST LIVE — 2026-09-14, read-only
+-- ─────────────────────────────────────────────────────────────────────────────
+--
+-- Not 'what we believe ran'. Read back out of dedorch after the fact:
+--
+--   5 tables        knowledge_documents 19 cols · knowledge_entries 18 ·
+--                   knowledge_extraction_runs 20 · knowledge_jobs 16 ·
+--                   knowledge_reviews 13
+--   §6+§7 columns   7 of 7 present (doc_ids, content, provenance,
+--                   withdrawn_at, review_mode, decided_by, authorised_by)
+--   indexes         19 on knowledge_* (12 named here + 7 PK/unique)
+--
+-- The ORM also reads every one of them against live, so the Python models
+-- and this file agree with the database rather than only with each other.
+--
+-- Idempotent throughout (IF NOT EXISTS) · Additive only: no DROP, no type
+-- change, nothing that rewrites an existing column.
 
 -- 1. Parsed document artifact. Chunks live INSIDE `artifact` (jsonb), deliberately
 --    not a child table: documents are small (9 pages = 31 chunks) and the artifact
@@ -168,3 +212,97 @@ CREATE TABLE IF NOT EXISTS knowledge_jobs (
     completed_at   timestamptz
 );
 CREATE INDEX IF NOT EXISTS idx_knowledge_jobs_user ON knowledge_jobs (user_id, created_at DESC);
+
+
+-- ============================================================================
+-- §6. EXECUTED 2026-09-14 — from the 2026-09-08 discussion (m3, m4, m5, m8, i7).
+--
+-- Verified after the run: 4 new columns and 2 new indexes present, and the
+-- ORM reads all of them against live. Harry still needs this in the
+-- migration set so a fresh environment reproduces it.
+--
+-- Additive only: three columns and one index on `knowledge_entries`. Nothing
+-- is dropped and no column changes type, so a pre-migration reader keeps
+-- working and Python's reads stay tolerant of both shapes while this lands.
+-- ============================================================================
+
+-- m3. "doc_id jadi list". One entry fuses meaning from several documents, but
+--     every source stays recorded. `doc_id` REMAINS the primary source — every
+--     existing filter and index uses it, and a single extracted assertion
+--     always has exactly one — while `doc_ids` carries the full list. Additive
+--     rather than a type change: rewriting `doc_id text` into jsonb in place
+--     would break every read at once, for a value that is still needed.
+ALTER TABLE knowledge_entries ADD COLUMN IF NOT EXISTS doc_ids jsonb NOT NULL DEFAULT '[]';
+
+-- m4. "kolom content". The definition or rule body, lifted out of `payload` so
+--     a consumer does not have to open jsonb to use it — the objection raised
+--     on 2026-09-08 was that too much JSON makes the data contract hard and
+--     easy to break. `payload` stays the source of truth; this is a promoted
+--     copy written from the same object in the same insert, exactly the
+--     convention `label`, `mention_count` and `diff_status` already follow.
+ALTER TABLE knowledge_entries ADD COLUMN IF NOT EXISTS content text;
+
+-- m5. "kolom provenance". Which document, which page — a list of objects:
+--       [{"doc_id": "STD_2026_006_MNO", "page": 3, "page_no": 4,
+--         "section_no": "2.1.3", "chunk_id": "STD_2026_006_MNO::0007"}]
+--     Both page forms travel together on purpose: `page` is 0-based as the
+--     parser reports it, `page_no` is the 1-based number a human reads, and
+--     they are derived from one value so they cannot disagree.
+--     `section_no` is included because the review queue already renders it —
+--     whether it earns a place was left open in the meeting, so drop it from
+--     this shape if the answer is no.
+ALTER TABLE knowledge_entries ADD COLUMN IF NOT EXISTS provenance jsonb NOT NULL DEFAULT '[]';
+
+-- m8. Makes the readable-key lookup a hit instead of a scan within the tenant.
+--     `label` now carries the TYPE_SUBJECT key (`GLOSSARY_PA`), and
+--     `KnowledgeStore.get_by_key` filters on exactly this pair. Without the
+--     index the lookup is still correct, just not the O(1) the design intends.
+CREATE INDEX IF NOT EXISTS idx_knowledge_entries_key ON knowledge_entries (scope_id, label);
+
+-- i7. "kasus dokumen dihapus" — ruling 2026-09-14: mark withdrawn, keep the
+--     rows, flag what is left. A withdrawn document stops counting towards the
+--     knowledge without anything being destroyed: an approval is an act someone
+--     took, and the record of what it referred to must survive the source going
+--     away. Null = live. Set = withdrawn.
+--     Entries whose last remaining source is withdrawn are surfaced to the
+--     expert to rule on rather than deleted — whether the knowledge leaves with
+--     the document is their call, not the pipeline's.
+ALTER TABLE knowledge_documents ADD COLUMN IF NOT EXISTS withdrawn_at timestamptz;
+CREATE INDEX IF NOT EXISTS idx_knowledge_documents_live
+    ON knowledge_documents (scope_id, doc_id) WHERE withdrawn_at IS NULL;
+
+
+-- ============================================================================
+-- §7. EXECUTED 2026-09-14 — the review system (r1, r2, r3) from the
+--     2026-09-08 discussion. Verified after the run: the ORM reads
+--     review_mode, decided_by and authorised_by against live.
+--
+-- Additive only: three columns, no drops, no type changes.
+--
+-- The point of these, in Mas's words: an entry approved without a human
+-- looking at it must still say WHO allowed that to happen. *"Kalau ada apa-apa
+-- jangan sampai kita yang disalahin… ya udah itu by mereka aja."* It is a
+-- risk-mitigation requirement, not bookkeeping — which is why it belongs in
+-- columns rather than in a free-text note.
+-- ============================================================================
+
+-- r1. Which review mode a batch was ingested under. Chosen by the caller at
+--     ingest time: 'expert' leaves entries waiting for a human; 'llm' takes
+--     them straight to approved with no reviewer step at all.
+--     Nullable so every job written before this column existed stays legal.
+ALTER TABLE knowledge_jobs ADD COLUMN IF NOT EXISTS review_mode text
+    CHECK (review_mode IS NULL OR review_mode IN ('expert','llm'));
+
+-- r2. WHO decided. 'human' is the existing behaviour and the default, so every
+--     review already recorded keeps its meaning without a backfill.
+ALTER TABLE knowledge_reviews ADD COLUMN IF NOT EXISTS decided_by text
+    NOT NULL DEFAULT 'human'
+    CHECK (decided_by IN ('human','llm'));
+
+-- r3. On WHOSE AUTHORITY. The admin who clicked ingest and the expert who
+--     reviews can be different people, and in 'llm' mode there is no reviewer
+--     at all — so the clicker's id has to reach this row or the decision has
+--     no owner. `reviewer_id` stays "who performed the review";
+--     `authorised_by` is "who permitted it to happen this way".
+--     Together they render: approved by LLM with permission of <person>.
+ALTER TABLE knowledge_reviews ADD COLUMN IF NOT EXISTS authorised_by text;

eval/knowledge/invariants.py

@@ -0,0 +1,203 @@
+"""Step 0 — assert on FIELDS AND INVARIANTS, never on exact text.
+
+Why this file exists. `temperature=0` is argmax over logits that a shared
+inference server does not hand back identically twice, and this corpus has a
+recorded case of identical input scoring 6/7 one day and 0/12 the next. So an
+assertion of the form "the definition equals this sentence" is not a test — it
+is a coin flip that fails on a day the wording moves by a comma, and passes on a
+day the pipeline is quietly broken in a way the wording does not show.
+
+What survives that noise is structure. "A definition, if present, is locatable
+verbatim in the document" is true of every correct run and false of every broken
+one, no matter which sentence the model picked. These are those claims.
+
+Every check here is free: deterministic, local, no API call. They run after the
+paid stage on whatever it produced, so they cost nothing to keep on.
+
+**A violation is a defect in the pipeline, not a bad score.** A metric moving is
+information; an invariant breaking means a control leaked, so the runner exits
+non-zero on any violation regardless of how the scores looked.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from src.knowledge_extraction.validate import span_present
+
+# Entry kinds and the field each one calls its identity. Checked by name rather
+# than by isinstance: entries reach here as `model_dump()` dicts, which is the
+# same shape that gets persisted, so this tests what is stored.
+ID_FIELDS = {
+    "glossary": "term_id",
+    "rule": "rule_id",
+    "formula": "formula_id",
+    # The domain entry kept `brief_id` through the v3 rename. Recorded here
+    # rather than corrected: renaming it is a migration that invalidates every
+    # expert approval recorded against the old prefix (K12's own reasoning).
+    "document": "brief_id",
+}
+
+
+@dataclass
+class Violation:
+    check: str
+    kind: str
+    detail: str
+
+    def as_dict(self) -> dict[str, str]:
+        return {"check": self.check, "kind": self.kind, "detail": self.detail}
+
+
+@dataclass
+class InvariantReport:
+    violations: list[Violation] = field(default_factory=list)
+    checked: dict[str, int] = field(default_factory=dict)
+
+    @property
+    def ok(self) -> bool:
+        return not self.violations
+
+    def as_dict(self) -> dict[str, Any]:
+        return {
+            "ok": self.ok,
+            "n_violations": len(self.violations),
+            "checked": self.checked,
+            "violations": [v.as_dict() for v in self.violations[:50]],
+        }
+
+
+def check_invariants(result, doc) -> InvariantReport:
+    """Structural claims that must hold of ANY correct run of ANY document."""
+    report = InvariantReport()
+    add = report.violations.append
+
+    source = "\n".join(chunk.text or "" for chunk in doc.chunks)
+    n_pages = doc.n_pages or 0
+
+    by_kind = {
+        "glossary": result.glossary,
+        "rule": result.rules,
+        "formula": result.formulas,
+        "document": [result.brief] if result.brief else [],
+    }
+    report.checked = {kind: len(entries) for kind, entries in by_kind.items()}
+
+    for kind, entries in by_kind.items():
+        id_field = ID_FIELDS[kind]
+        seen: set[str] = set()
+        for entry in entries:
+            label = entry.get("term") or entry.get("name") or entry.get(id_field) or "?"
+
+            # 1 — identity. Without it a re-run cannot tell an expert which rows
+            # they already cleared, which is worse than a wrong definition.
+            entity_id = entry.get(id_field)
+            if not entity_id:
+                add(Violation("entity_id_present", kind, f"{label}: {id_field} missing"))
+            elif entity_id in seen:
+                add(Violation("entity_id_unique", kind, f"{entity_id} appears twice"))
+            else:
+                seen.add(entity_id)
+
+            provenance = entry.get("provenance") or {}
+            span = (provenance.get("span") or "").strip()
+
+            # 2 — a claim that survived carries a span, and that span is real.
+            # This is the span check restated from the outside: if it ever
+            # fires, the guard leaked rather than the model misbehaved.
+            if _has_content(entry, kind):
+                if not span:
+                    add(Violation("span_present", kind, f"{label}: content with no span"))
+                elif not span_present(span, source):
+                    add(
+                        Violation(
+                            "span_locatable",
+                            kind,
+                            f"{label}: span not found verbatim in the document "
+                            f"({span[:60]!r})",
+                        )
+                    )
+
+            # 3 — provenance points somewhere that exists.
+            page = provenance.get("page")
+            if page is not None and n_pages and not 0 <= page < n_pages:
+                add(
+                    Violation(
+                        "page_in_range",
+                        kind,
+                        f"{label}: page {page} outside 0..{n_pages - 1}",
+                    )
+                )
+
+    # 4 — a rejected field is NULL on the entry it came from. The contract says
+    # rejected, never repaired; this is the only check that can tell the two
+    # apart after the fact.
+    for rejection in result.rejected:
+        entry = _find_entry(by_kind, rejection.entry_term)
+        if entry is not None and entry.get(rejection.field) not in (None, "", []):
+            add(
+                Violation(
+                    "rejection_is_null",
+                    rejection.branch,
+                    f"{rejection.entry_term}.{rejection.field} was rejected but still "
+                    "carries a value — a repaired span is an unfalsifiable claim",
+                )
+            )
+
+    _check_queue(result.review_queue, report)
+    return report
+
+
+def _check_queue(queue: list[dict], report: InvariantReport) -> None:
+    """The consumption surface. A UI binds these, so their shape is a contract."""
+    add = report.violations.append
+    report.checked["review_queue"] = len(queue)
+
+    ranks = [row.get("rank") for row in queue]
+    if ranks != list(range(1, len(queue) + 1)):
+        add(Violation("queue_ranks_contiguous", "review_queue", "ranks are not 1..N"))
+
+    for row in queue:
+        kind = row.get("row_kind")
+        if kind not in ("term", "key_parameter"):
+            add(Violation("row_kind_known", "review_queue", f"row_kind={kind!r}"))
+        if kind == "key_parameter" and row.get("term_id") is not None:
+            add(
+                Violation(
+                    "key_parameter_has_no_term",
+                    "review_queue",
+                    f"{row.get('term')!r} carries a term_id, which is the one thing "
+                    "a key_parameter row is defined by NOT having",
+                )
+            )
+        if "_conflict" in row or "_sort" in row:
+            add(
+                Violation(
+                    "no_private_keys",
+                    "review_queue",
+                    f"{row.get('term')!r} leaks a sorting key into the payload",
+                )
+            )
+
+
+def _has_content(entry: dict, kind: str) -> bool:
+    """Did this entry actually assert something? An abstention is correct
+    behaviour and carries no span obligation — 48 of 58 entries abstain on the
+    reference document, and treating that as a violation would make the whole
+    report noise."""
+    if kind == "glossary":
+        return bool(entry.get("definition"))
+    if kind == "rule":
+        return bool(entry.get("statement"))
+    if kind == "formula":
+        return bool(entry.get("formula_latex"))
+    return bool(entry.get("purpose_verbatim"))
+
+
+def _find_entry(by_kind: dict[str, list[dict]], label: str) -> dict | None:
+    for entries in by_kind.values():
+        for entry in entries:
+            if label in (entry.get("term"), entry.get("name")):
+                return entry
+    return None

eval/knowledge/results/v2_filters_2026-09-10_151557.json

@@ -0,0 +1,109 @@
+{
+  "run": {
+    "created_at": "2026-09-10_151557",
+    "doc_id": "STD_2026_006_MNO",
+    "artifact": "eval\\knowledge\\fixtures\\parsed_STD_2026_006_MNO.json",
+    "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
+    "parser": "mineru/3.4.4/hybrid",
+    "parser_backend": "hybrid",
+    "extracted": false,
+    "mock": false,
+    "limit": null,
+    "seconds": 68.7,
+    "note": "Free stages only. E3 and rule recall require --extract, which spends."
+  },
+  "funnel": {
+    "pages": 9,
+    "chunks": 31,
+    "mentions_raw": 186,
+    "clusters": 74,
+    "compression_ratio": 2.176,
+    "abbrev_pairs": 5,
+    "rule_candidates": 16
+  },
+  "scores": {
+    "E1": {
+      "label": "term-filter recall",
+      "value": 0.8293,
+      "detail": {
+        "label": "term_filter_recall(E1)",
+        "n_gold": 41,
+        "n_pred": 84,
+        "true_positives": 34,
+        "precision": 0.4048,
+        "recall": 0.8293,
+        "f1": 0.544,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Grouping (Composite)",
+          "Weight average",
+          "Uncontrollable",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E1c_loose": {
+      "label": "post-cluster recall (loose)",
+      "value": 0.8293,
+      "detail": {
+        "label": "post_cluster_recall(E1c_loose)",
+        "n_gold": 41,
+        "n_pred": 74,
+        "true_positives": 34,
+        "precision": 0.4595,
+        "recall": 0.8293,
+        "f1": 0.5913,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Grouping (Composite)",
+          "Weight average",
+          "Uncontrollable",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E1c_strict": {
+      "label": "post-cluster recall (owns its own cluster)",
+      "value": 0.7805,
+      "detail": {
+        "label": "post_cluster_recall(E1c_strict)",
+        "n_gold": 41,
+        "n_pred": 74,
+        "true_positives": 32,
+        "precision": 0.4324,
+        "recall": 0.7805,
+        "f1": 0.5565,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Grouping (Composite)",
+          "Joint survey",
+          "Weight average",
+          "Plan",
+          "Uncontrollable",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E2": {
+      "label": "review-burden compression (descriptive, NOT gated)",
+      "value": 2.176,
+      "detail": {
+        "mentions": 161,
+        "clusters": 74
+      }
+    }
+  },
+  "baseline": {
+    "file": "v2_full_2026-09-02_160326.json",
+    "E1": 0.878,
+    "E2": 2.697,
+    "E3": 1.0,
+    "rule_recall": 0.5
+  }
+}

eval/knowledge/results/v2_filters_2026-09-11_092207.json

@@ -0,0 +1,137 @@
+{
+  "run": {
+    "created_at": "2026-09-11_092207",
+    "doc_id": "STD_2026_006_MNO",
+    "artifact": "eval\\knowledge\\fixtures\\parsed_STD_2026_006_MNO.json",
+    "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
+    "parser": "mineru/3.4.4/hybrid",
+    "parser_backend": "hybrid",
+    "extracted": false,
+    "mock": false,
+    "limit": null,
+    "runs": 1,
+    "expected_model": "gpt-5.4-nano-2026-03-17",
+    "seconds": 87.1,
+    "note": "Free stages only. E3 and rule recall require --extract, which spends."
+  },
+  "funnel": {
+    "pages": 9,
+    "chunks": 31,
+    "mentions_raw": 208,
+    "clusters": 80,
+    "compression_ratio": 2.125,
+    "abbrev_pairs": 5,
+    "rule_candidates": 16
+  },
+  "genre": {
+    "genre": "standard",
+    "labels_variant": "broad",
+    "branches": [
+      "glossary",
+      "rule",
+      "formula",
+      "summary"
+    ],
+    "well_served": true,
+    "routing_applied": {
+      "labels": false,
+      "branches": false
+    },
+    "signals": {
+      "n_chunks": 31,
+      "heading_coverage": 0.355,
+      "tabular_ratio": 0.452,
+      "abbrev_pairs": 5,
+      "rule_density": 0.516
+    },
+    "reasons": [
+      "5 abbreviation definitions in headed prose — a document that names its own terms"
+    ],
+    "warnings": []
+  },
+  "scores": {
+    "E1": {
+      "label": "term-filter recall",
+      "value": 0.8293,
+      "detail": {
+        "label": "term_filter_recall(E1)",
+        "n_gold": 41,
+        "n_pred": 92,
+        "true_positives": 34,
+        "precision": 0.3696,
+        "recall": 0.8293,
+        "f1": 0.5113,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Grouping (Composite)",
+          "Weight average",
+          "Uncontrollable",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E1c_loose": {
+      "label": "post-cluster recall (loose)",
+      "value": 0.8293,
+      "detail": {
+        "label": "post_cluster_recall(E1c_loose)",
+        "n_gold": 41,
+        "n_pred": 80,
+        "true_positives": 34,
+        "precision": 0.425,
+        "recall": 0.8293,
+        "f1": 0.562,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Grouping (Composite)",
+          "Weight average",
+          "Uncontrollable",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E1c_strict": {
+      "label": "post-cluster recall (owns its own cluster)",
+      "value": 0.7805,
+      "detail": {
+        "label": "post_cluster_recall(E1c_strict)",
+        "n_gold": 41,
+        "n_pred": 80,
+        "true_positives": 32,
+        "precision": 0.4,
+        "recall": 0.7805,
+        "f1": 0.5289,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Grouping (Composite)",
+          "Joint survey",
+          "Weight average",
+          "Plan",
+          "Uncontrollable",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E2": {
+      "label": "review-burden compression (descriptive, NOT gated)",
+      "value": 2.125,
+      "detail": {
+        "mentions": 170,
+        "clusters": 80
+      }
+    }
+  },
+  "baseline": {
+    "file": "v2_full_2026-09-02_160326.json",
+    "E1": 0.878,
+    "E2": 2.697,
+    "E3": 1.0,
+    "rule_recall": 0.5
+  }
+}

eval/knowledge/results/v2_filters_2026-09-11_100546.json

@@ -0,0 +1,131 @@
+{
+  "run": {
+    "created_at": "2026-09-11_100546",
+    "doc_id": "STD_2026_006_MNO",
+    "artifact": "eval\\knowledge\\fixtures\\parsed_mistral_STD_2026_006_MNO.json",
+    "content_hash": "0a95466ea17e67bb0a1a83ca4b7022974cf1171b3450cf2ca92cb8903277920d",
+    "parser": "mistral/mistral-ocr-latest/mistral",
+    "parser_backend": "mistral",
+    "extracted": false,
+    "mock": false,
+    "limit": null,
+    "runs": 1,
+    "expected_model": "gpt-5.4-nano-2026-03-17",
+    "seconds": 69.7,
+    "label": "linked_partner",
+    "note": "Free stages only. E3 and rule recall require --extract, which spends."
+  },
+  "funnel": {
+    "pages": 9,
+    "chunks": 32,
+    "mentions_raw": 249,
+    "clusters": 84,
+    "compression_ratio": 2.298,
+    "abbrev_pairs": 5,
+    "rule_candidates": 16
+  },
+  "genre": {
+    "genre": "standard",
+    "labels_variant": "broad",
+    "branches": [
+      "glossary",
+      "rule",
+      "formula",
+      "summary"
+    ],
+    "well_served": true,
+    "routing_applied": {
+      "labels": false,
+      "branches": false
+    },
+    "signals": {
+      "n_chunks": 32,
+      "heading_coverage": 0.406,
+      "tabular_ratio": 0.25,
+      "abbrev_pairs": 5,
+      "rule_density": 0.5
+    },
+    "reasons": [
+      "5 abbreviation definitions in headed prose — a document that names its own terms"
+    ],
+    "warnings": []
+  },
+  "scores": {
+    "E1": {
+      "label": "term-filter recall",
+      "value": 0.878,
+      "detail": {
+        "label": "term_filter_recall(E1)",
+        "n_gold": 41,
+        "n_pred": 98,
+        "true_positives": 36,
+        "precision": 0.3673,
+        "recall": 0.878,
+        "f1": 0.518,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Weight average",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E1c_loose": {
+      "label": "post-cluster recall (loose)",
+      "value": 0.878,
+      "detail": {
+        "label": "post_cluster_recall(E1c_loose)",
+        "n_gold": 41,
+        "n_pred": 84,
+        "true_positives": 36,
+        "precision": 0.4286,
+        "recall": 0.878,
+        "f1": 0.576,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Weight average",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E1c_strict": {
+      "label": "post-cluster recall (owns its own cluster)",
+      "value": 0.8537,
+      "detail": {
+        "label": "post_cluster_recall(E1c_strict)",
+        "n_gold": 41,
+        "n_pred": 84,
+        "true_positives": 35,
+        "precision": 0.4167,
+        "recall": 0.8537,
+        "f1": 0.56,
+        "misses": [
+          "Breakdown",
+          "Overburden removal",
+          "Joint survey",
+          "Weight average",
+          "Fleet management",
+          "Mineplan"
+        ]
+      }
+    },
+    "E2": {
+      "label": "review-burden compression (descriptive, NOT gated)",
+      "value": 2.298,
+      "detail": {
+        "mentions": 193,
+        "clusters": 84
+      }
+    }
+  },
+  "baseline": {
+    "file": "v2_full_2026-09-02_160326.json",
+    "E1": 0.878,
+    "E2": 2.697,
+    "E3": 1.0,
+    "rule_recall": 0.5
+  }
+}

eval/knowledge/run_eval.py

@@ -20,7 +20,14 @@ written by `python -m src.knowledge_parsing.run`.
 Module mode only (`-m`). A file-path invocation breaks `sys.path` and the
 `src.` imports below fail.
 
-Two things this refuses to do, both for the same reason — a number that cannot
+    # a BAND instead of a number — repeats the paid stage only. SPENDS N TIMES.
+    ./.venv/Scripts/python.exe -m eval.knowledge.run_eval <artifact.json> --extract --runs 5
+
+Exit codes: `0` clean · `1` below a kill line · `2` bad arguments or artifact ·
+`3` no extractor · `4` would overwrite a result · `5` an invariant broke ·
+`6` a different model snapshot served the run.
+
+Four things this refuses to do, all for the same reason — a number that cannot
 be compared is worse than no number:
 
 * **It will not run on an artifact that does not declare `parser_backend`.**
@@ -30,12 +37,20 @@ be compared is worse than no number:
   overrides, and stamps the result file so the caveat travels with it.
 * **It never overwrites a result file.** Every run writes a new timestamped
   file beside the others; the frozen baseline stays frozen.
+* **It refuses a run served by a different model snapshot** than the committed
+  results were measured on (`EXPECTED_MODEL_SNAPSHOT`). X25 recorded what served
+  a run; this asserts it, so a silent tier move stops the comparison instead of
+  quietly poisoning the delta column.
+* **It refuses a run that broke a structural invariant** (`invariants.py`) —
+  a span that is not locatable, a duplicate entity id, a repaired rejection.
+  Those are defects rather than scores, so they fail the run outright.
 """
 
 from __future__ import annotations
 
 import argparse
 import json
+import statistics
 import sys
 from collections import Counter
 from datetime import datetime
@@ -44,6 +59,7 @@ from typing import Any
 
 from src.knowledge_extraction.adapter import parsed_doc_from_artifact
 from src.knowledge_extraction.extract import MockExtractor, cacheable, prefix_tokens
+from src.knowledge_extraction.genre import classify as classify_genre
 from src.knowledge_extraction.service import (
     build_clusters,
     estimate_cost,
@@ -52,14 +68,34 @@ from src.knowledge_extraction.service import (
 )
 from src.knowledge_extraction.settings import EVIDENCE_K
 
-from .score import GOLD_PATH, load_gold, score_glossary, score_rules, score_term_filter
+from .invariants import check_invariants
+from .score import (
+    GOLD_PATH,
+    load_gold,
+    score_glossary,
+    score_post_cluster,
+    score_rules,
+    score_term_filter,
+)
 
 _HERE = Path(__file__).resolve().parent
 RESULTS_DIR = _HERE / "results"
 
 # The kill lines. Each is a decision already taken, recorded in
 # KNOWLEDGE_PIPELINE_CALIBRATION.md — not a target invented per run.
-KILL_LINES = {"E1": 0.70, "E2": 2.0, "E3": 0.80}
+# E2's kill line was REMOVED 2026-09-10. Compression is now reported as a
+# descriptive statistic only, never as pass/fail.
+#
+# The reason is not that 2.0 was the wrong number — it is that the metric moves
+# the WRONG WAY. Over-merging distinct terms into one cluster RAISES compression,
+# so the defect that cost the gold term EWH its own entry was pushing this score
+# up and through its own pass line. A metric whose pass condition is partly met
+# by the defect it should expose is worse than no metric. Retiring the fuzzy pass
+# correctly LOWERED compression (BUMA 2.712 -> 2.183), which under a kill line
+# would have read as a regression.
+#
+# E1c (post-cluster recall) is the metric that actually watches this stage.
+KILL_LINES = {"E1": 0.70, "E3": 0.80}
 
 # What the last committed run measured, for the delta column. Rule recall has
 # no kill line: it is tracked, not gated, and 3/15 is the post-X22 scoring of
@@ -80,6 +116,29 @@ BASELINE = {
     "rule_recall": 0.5,
 }
 
+# ── Step 0: measurement hardening ───────────────────────────────────────
+# The snapshot every committed result on this branch was measured against. X25
+# started RECORDING what served a run; this makes it an ASSERTION, which is the
+# difference between a fact discoverable by diffing two files afterwards and one
+# that stops a run at the moment it stops being true.
+#
+# A deployment name is OUR label and never changes; the snapshot behind it can
+# move without notice. When it does, every number in `results/` becomes a
+# measurement of a different system and the delta column starts lying. Comparing
+# against a moved model is the failure this catches — not the move itself, which
+# is Azure's business.
+#
+# `--expect-model` overrides for a deliberate tier change; `--allow-model-drift`
+# downgrades the refusal to a stamped warning when you need the number anyway.
+EXPECTED_MODEL_SNAPSHOT = "gpt-5.4-nano-2026-03-17"
+
+# Below this spread, two runs of one document are telling the same story. Above
+# it, a single run is not evidence and the band must be quoted instead of the
+# point. 0.10 is the handoff's own read of this corpus (identical input scored
+# 6/7 one day and 0/12 the next), not a measured tolerance — raise or lower it
+# once `--runs` has produced a real band on a real document.
+NOISE_BAND = 0.10
+
 
 def _azure_extractor():
     """Built lazily: importing it requires credentials to be configured."""
@@ -117,6 +176,60 @@ def _delta(value: float, previous: float | None) -> str:
     return f"{diff:+.4f}"
 
 
+def _stability(values: list[float]) -> dict[str, Any]:
+    """A band, not a number.
+
+    The reported value becomes the MEDIAN rather than the mean: one run that
+    fails schema validation drags a mean into a region no run actually
+    occupied, and a metric nobody measured is worse than a noisy one.
+    """
+    ordered = sorted(values)
+    return {
+        "n": len(ordered),
+        "min": round(ordered[0], 4),
+        "median": round(statistics.median(ordered), 4),
+        "max": round(ordered[-1], 4),
+        "spread": round(ordered[-1] - ordered[0], 4),
+        "runs": [round(v, 4) for v in values],
+        "noisy": bool(ordered[-1] - ordered[0] > NOISE_BAND),
+    }
+
+
+def _entity_ids(result) -> dict[str, list[str]]:
+    """Every id a run minted, by kind — the thing an expert's approval hangs on.
+
+    K11 fixed two ids that did not survive a re-run. Nothing has watched them
+    since, and a re-run that re-keys its entries silently orphans every approval
+    recorded against the old ones. Cheap to check, catastrophic to miss.
+    """
+    return {
+        "glossary": sorted(e.get("term_id") or "" for e in result.glossary),
+        "rule": sorted(e.get("rule_id") or "" for e in result.rules),
+        "formula": sorted(e.get("formula_id") or "" for e in result.formulas),
+    }
+
+
+def _id_stability(per_run: list[dict[str, list[str]]]) -> dict[str, Any]:
+    """Did every run mint the same ids? Anything but `True` is a defect."""
+    if len(per_run) < 2:
+        return {"comparable": False, "note": "needs --runs 2 or more"}
+    first = per_run[0]
+    drift = {
+        kind: {
+            "only_in_first": sorted(set(first[kind]) - set(other[kind]))[:10],
+            "only_in_later": sorted(set(other[kind]) - set(first[kind]))[:10],
+        }
+        for other in per_run[1:]
+        for kind in first
+        if set(first[kind]) != set(other[kind])
+    }
+    return {"comparable": True, "identical_across_runs": not drift, "drift": drift}
+
+
+def _served_snapshots(result) -> list[str]:
+    return sorted({u.model_version for u in result.usages if u.model_version})
+
+
 def build_report(
     doc,
     filtered,
@@ -126,6 +239,8 @@ def build_report(
     args,
     started: datetime,
     seconds: float,
+    invariants=None,
+    stability: dict[str, Any] | None = None,
 ) -> dict:
     funnel = {
         "pages": doc.n_pages,
@@ -147,12 +262,19 @@ def build_report(
             "extracted": bool(args.extract),
             "mock": bool(args.mock),
             "limit": args.limit,
+            "runs": args.runs,
+            "expected_model": args.expect_model,
             "seconds": round(seconds, 1),
         },
         "funnel": funnel,
+        "genre": classify_genre(doc, filtered).as_dict(),
         "scores": scores,
         "baseline": BASELINE,
     }
+    if invariants is not None:
+        report["invariants"] = invariants.as_dict()
+    if stability:
+        report["stability"] = stability
 
     if args.allow_unlabelled_artifact and not doc.parser_backend:
         report["run"]["comparability"] = (
@@ -170,12 +292,25 @@ def build_report(
         # measurement of an unknown: on 2026-08-27 identical input scored 6/7
         # one day and 0/12 the next under an unchanged deployment name, and
         # nothing recorded said whether the model had moved. See X25.
+        served = _served_snapshots(result)
         report["run"]["served_by"] = {
-            "model_version": sorted({u.model_version for u in result.usages if u.model_version}),
+            "model_version": served,
             "system_fingerprint": sorted(
                 {u.system_fingerprint for u in result.usages if u.system_fingerprint}
             ),
         }
+        # Step 0(a): the recorded snapshot becomes an assertion. A mock run
+        # reports no snapshot and is exempt — it measures plumbing, not a model.
+        if not args.mock and served and served != [args.expect_model]:
+            report["run"]["model_drift"] = {
+                "expected": args.expect_model,
+                "served": served,
+                "forced": bool(args.allow_model_drift),
+                "note": (
+                    "every committed result was measured on the expected snapshot; "
+                    "a delta against this run compares two different systems"
+                ),
+            }
         prompt, cached, completion = result.total_tokens
         report["cost"] = {
             "prompt_tokens": prompt,
@@ -204,10 +339,17 @@ def format_summary(report: dict) -> str:
     for key, score in report["scores"].items():
         value = score["value"]
         kill = KILL_LINES.get(key)
+        band = score.get("stability")
+        suffix = ""
+        if band:
+            suffix = (
+                f"  [{band['min']:.4f}..{band['max']:.4f}] n={band['n']}"
+                + ("  NOISY" if band["noisy"] else "")
+            )
         rows.append(
             f"  {key:<12} {value:>8.4f}   kill {kill if kill else '  —  '!s:>5}   "
             f"{_verdict(value, kill):<5}  vs baseline {_delta(value, BASELINE.get(key))}"
-            f"   {score['label']}"
+            f"   {score['label']}{suffix}"
         )
     funnel = report["funnel"]
     out = [
@@ -229,6 +371,39 @@ def format_summary(report: dict) -> str:
             " by the span check",
             f"  latex_verification: {report['quality']['latex_verification']}",
         ]
+    genre = report.get("genre")
+    if genre:
+        out += [
+            "",
+            f"  genre: {genre['genre']}  (labels {genre['labels_variant']}, "
+            f"well-served {genre['well_served']})   {genre['signals']}",
+        ]
+        for reason in genre["reasons"]:
+            out += [f"    · {reason}"]
+        for warning in genre["warnings"]:
+            out += [f"    !! {warning}"]
+    if "invariants" in report:
+        inv = report["invariants"]
+        out += [
+            "",
+            f"  invariants: {'OK' if inv['ok'] else str(inv['n_violations']) + ' VIOLATED'}"
+            f"   checked {inv['checked']}",
+        ]
+        for violation in inv["violations"][:10]:
+            out += [f"    !! {violation['check']} [{violation['kind']}] {violation['detail']}"]
+    if "stability" in report:
+        ids = report["stability"]["entity_ids"]
+        if ids.get("comparable"):
+            out += [
+                f"  entity ids identical across runs: {ids['identical_across_runs']}"
+            ]
+    if "model_drift" in report["run"]:
+        drift = report["run"]["model_drift"]
+        out += [
+            "",
+            f"  !! MODEL DRIFT — expected {drift['expected']}, served "
+            f"{', '.join(drift['served'])}",
+        ]
     if "comparability" in report["run"]:
         out += ["", f"  !! {report['run']['comparability']}"]
     return "\n".join(out)
@@ -258,8 +433,39 @@ def main(argv: list[str] | None = None) -> int:
         action="store_true",
         help="run on an artifact with no parser_backend (results NOT comparable)",
     )
+    parser.add_argument(
+        "--runs",
+        type=int,
+        default=1,
+        metavar="N",
+        help=(
+            "repeat the PAID stage N times and report min/median/max instead of a "
+            "single number. MULTIPLIES SPEND BY N — the free stages run once"
+        ),
+    )
+    parser.add_argument(
+        "--expect-model",
+        default=EXPECTED_MODEL_SNAPSHOT,
+        help=f"model snapshot this run must be served by (default {EXPECTED_MODEL_SNAPSHOT})",
+    )
+    parser.add_argument(
+        "--allow-model-drift",
+        action="store_true",
+        help="record a snapshot mismatch as a warning instead of refusing the run",
+    )
     args = parser.parse_args(argv)
 
+    if args.runs < 1:
+        print("--runs must be 1 or more", file=sys.stderr)
+        return 2
+    if args.runs > 1 and not args.extract:
+        print(
+            "--runs only varies the PAID stage; the free stages are deterministic.\n"
+            "  Add --extract, or drop --runs.",
+            file=sys.stderr,
+        )
+        return 2
+
     if not args.artifact.exists():
         print(f"artifact not found: {args.artifact}", file=sys.stderr)
         return 2
@@ -297,14 +503,30 @@ def main(argv: list[str] | None = None) -> int:
         return 0
 
     e1 = score_term_filter(gold, [m.surface for m in filtered.mentions])
+    canonicals = [c.canonical for c in clustered.clusters]
+    e1c_loose = score_post_cluster(gold, canonicals)
+    e1c_strict = score_post_cluster(gold, canonicals, cluster_variants=True)
     scores: dict[str, Any] = {
         "E1": {
             "label": "term-filter recall",
             "value": e1.recall,
             "detail": e1.as_dict(),
         },
+        # E1c watches the merge stage, which E1 (measured before clustering) and
+        # E3 (measured only over entries that produced a definition) both miss.
+        # `E1 - E1c_loose` is the recall clustering destroyed; 0 is the target.
+        "E1c_loose": {
+            "label": "post-cluster recall (loose)",
+            "value": e1c_loose.recall,
+            "detail": e1c_loose.as_dict(),
+        },
+        "E1c_strict": {
+            "label": "post-cluster recall (owns its own cluster)",
+            "value": e1c_strict.recall,
+            "detail": e1c_strict.as_dict(),
+        },
         "E2": {
-            "label": "review-burden compression",
+            "label": "review-burden compression (descriptive, NOT gated)",
             "value": clustered.compression_ratio,
             "detail": {
                 "mentions": clustered.n_mentions,
@@ -314,6 +536,8 @@ def main(argv: list[str] | None = None) -> int:
     }
 
     result = None
+    invariants = None
+    stability: dict[str, Any] = {}
     if args.extract:
         extractor = MockExtractor() if args.mock else _azure_extractor()
         if extractor is None:
@@ -323,24 +547,58 @@ def main(argv: list[str] | None = None) -> int:
             if args.active_glossary
             else None
         )
-        result = extract_all(
-            doc, clustered, filtered, extractor, args.limit, active
-        )
-        e3 = score_glossary(gold, result.glossary)
-        rules = score_rules(gold, result.rules)
+
+        # The free stages are deterministic and already done — only the paid
+        # stage is repeated. Re-running GLiNER N times would cost minutes to
+        # re-derive an identical clustering.
+        e3_values: list[float] = []
+        rule_values: list[float] = []
+        ids_per_run: list[dict[str, list[str]]] = []
+        violations_per_run: list[int] = []
+
+        for run_index in range(args.runs):
+            if args.runs > 1:
+                print(f"  run {run_index + 1}/{args.runs} ...", file=sys.stderr)
+            result = extract_all(doc, clustered, filtered, extractor, args.limit, active)
+            e3 = score_glossary(gold, result.glossary)
+            rules = score_rules(gold, result.rules)
+            e3_values.append(e3.precision)
+            rule_values.append(rules.recall)
+            ids_per_run.append(_entity_ids(result))
+            invariants = check_invariants(result, doc)
+            violations_per_run.append(len(invariants.violations))
+
+        # The LAST run's detail is reported beside the band. Detail from one run
+        # of several is an example, not a summary — labelled as such below.
         scores["E3"] = {
             "label": "glossary schema fill (precision)",
-            "value": e3.precision,
+            "value": round(statistics.median(e3_values), 4),
             "detail": e3.as_dict(),
         }
         scores["rule_recall"] = {
             "label": "rule-branch recall",
-            "value": rules.recall,
+            "value": round(statistics.median(rule_values), 4),
             "detail": rules.as_dict(),
         }
+        if args.runs > 1:
+            scores["E3"]["stability"] = _stability(e3_values)
+            scores["rule_recall"]["stability"] = _stability(rule_values)
+            for key in ("E3", "rule_recall"):
+                scores[key]["detail"]["note"] = (
+                    f"detail is from run {args.runs} of {args.runs}; the value is "
+                    "the median across runs"
+                )
+        stability = {
+            "runs": args.runs,
+            "entity_ids": _id_stability(ids_per_run),
+            "violations_per_run": violations_per_run,
+        }
 
     seconds = (datetime.now() - started).total_seconds()
-    report = build_report(doc, filtered, clustered, scores, result, args, started, seconds)
+    report = build_report(
+        doc, filtered, clustered, scores, result, args, started, seconds,
+        invariants=invariants, stability=stability,
+    )
     if args.label:
         report["run"]["label"] = args.label
     if not args.extract:
@@ -361,6 +619,25 @@ def main(argv: list[str] | None = None) -> int:
     )
     print(f"\n-> saved: {out_path.relative_to(_HERE.parent.parent)}")
 
+    # Invariants first, and regardless of the tier: a broken structural claim
+    # means a control leaked, which invalidates the scores rather than joining
+    # them. Mock runs included — the plumbing is exactly what they test.
+    if invariants is not None and not invariants.ok:
+        print(
+            f"\nINVARIANTS VIOLATED: {len(invariants.violations)} — see the result file",
+            file=sys.stderr,
+        )
+        return 5
+
+    if "model_drift" in report["run"] and not args.allow_model_drift:
+        print(
+            "\nMODEL SNAPSHOT MISMATCH — the run is recorded but must not be compared\n"
+            "  to the committed results. Re-pin with --expect-model, or pass\n"
+            "  --allow-model-drift to accept the caveat.",
+            file=sys.stderr,
+        )
+        return 6
+
     if EVIDENCE_K and args.extract and not args.mock:
         failed = [
             k for k, s in scores.items() if k in KILL_LINES and s["value"] < KILL_LINES[k]

eval/knowledge/score.py

@@ -125,6 +125,60 @@ def score_term_filter(gold: dict, surfaces: list[str]) -> Score:
     )
 
 
+def score_post_cluster(gold: dict, canonicals: list[str], cluster_variants=None) -> Score:
+    """**E1c — added 2026-09-10.** The same gold set as E1, scored AFTER clustering.
+
+    E1 is measured on mention surfaces, which is *before* clustering. That left a
+    window nothing observed: a term the filter found correctly, which clustering
+    then absorbed into another term's cluster, and which therefore never gets an
+    entry an expert could approve. On the reference document that window cost the
+    gold term **EWH** — absorbed into `Working Hours`, which then abstained, so
+    two gold terms became one null entry.
+
+    Nothing in the harness could see it. E1 scores it as found; **E2 compression
+    is INFLATED by the merge** (which is why compression is no longer gated); and
+    E3 only scores entries that produced a definition, so a merged-then-null
+    entry is excluded from the scoreable base by construction.
+
+    Two variants, because the difference between them is informative:
+
+    - **loose** (`cluster_variants=None`) — identical matching to E1, just fed
+      cluster canonicals instead of mention surfaces. Directly comparable to E1,
+      so `E1 - E1c_loose` is the recall the merge stage destroyed.
+    - **strict** (`cluster_variants` supplied) — a gold term counts only if it is
+      some cluster's OWN canonical. Surviving as another term's variant does not
+      count, because that is exactly the loss. Always ≤ loose.
+    """
+    score = score_term_filter(gold, canonicals)
+    score.label = "post_cluster_recall(E1c_loose)"
+    if cluster_variants is None:
+        return score
+
+    canon = {norm(c) for c in canonicals if norm(c)}
+    tp, misses = 0, []
+    for entry in gold.get("terms", []):
+        variants = [entry["term"]] + list(entry.get("variants", []))
+        if entry.get("full_name"):
+            variants.append(entry["full_name"])
+        normed = {norm(v) for v in variants if norm(v)}
+        if normed & canon:
+            tp += 1
+        else:
+            misses.append(entry["term"])
+    n_gold = len(gold.get("terms", []))
+    precision, recall, f1 = _prf(tp, len(canon), n_gold)
+    return Score(
+        label="post_cluster_recall(E1c_strict)",
+        n_gold=n_gold,
+        n_pred=len(canon),
+        true_positives=tp,
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        misses=misses,
+    )
+
+
 def score_glossary(gold: dict, entries: list[dict]) -> Score:
     """E3: when nano fills the schema, is it right?
 

src/api/v1/knowledge.py

@@ -102,6 +102,17 @@ class IngestRequest(BaseModel):
             "artifact. Off by default: on, it silently re-spends on every call."
         ),
     )
+    review_mode: Literal["expert", "llm"] = Field(
+        default="expert",
+        description=(
+            "Who rules on what this batch produces. 'expert' leaves every "
+            "entry waiting for a human. 'llm' approves them on arrival with "
+            "NO reviewer step and no second opinion — the entries were "
+            "written by a model and are accepted as-is. Choosing it records "
+            "`authorised_by` against the caller, so an entry approved this "
+            "way can always say who allowed it."
+        ),
+    )
 
     @field_validator("user_id")
     @classmethod
@@ -145,7 +156,9 @@ class IngestAccepted(BaseModel):
 async def ingest(request: Request, payload: IngestRequest) -> IngestAccepted:
     """Admit a batch and return immediately. Poll `GET /jobs/{job_id}`."""
     job, reused = await _service.start(
-        user_id=payload.user_id, document_ids=payload.document_ids
+        user_id=payload.user_id,
+        document_ids=payload.document_ids,
+        review_mode=payload.review_mode,
     )
     if reused:
         message = "A job for this scope is already running; returning it."

src/db/postgres/models.py

@@ -379,6 +379,11 @@ class KnowledgeDocumentRow(Base):
     raw_output_dir = Column(Text)
     artifact = Column(JSONB, nullable=False)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
+    # i7 (2026-09-14): the source document has been removed. Null = live.
+    # A withdrawn document stops counting towards the knowledge WITHOUT
+    # anything being destroyed -- an approval is an act someone took, and the
+    # record of what it referred to has to survive the source going away.
+    withdrawn_at = Column(DateTime(timezone=True), nullable=True)
 
 
 class KnowledgeExtractionRunRow(Base):
@@ -467,6 +472,20 @@ class KnowledgeEntryRow(Base):
     diff_status = Column(Text)
     definition_conflict = Column(Boolean, nullable=False, default=False)
     payload = Column(JSONB, nullable=False)
+    # m3/m4/m5 (2026-09-08 discussion, migrated 2026-09-14). Promoted out of
+    # `payload` so a consumer does not have to open jsonb to use them -- the
+    # objection raised in that discussion was that too much JSON makes the data
+    # contract hard and easy to break. `payload` REMAINS the source of truth;
+    # these are written from the same object in the same insert, exactly as
+    # `label`, `mention_count` and `diff_status` already are.
+    #
+    # `doc_ids` is the list of documents behind the entry. `doc_id` above stays
+    # the primary source -- every existing filter and index uses it -- and the
+    # two agree for an ordinary entry. They differ only when a definition is
+    # carried forward onto a later document that re-mentions the term.
+    doc_ids = Column(JSONB, nullable=False, server_default=text("'[]'::jsonb"))
+    content = Column(Text)
+    provenance = Column(JSONB, nullable=False, server_default=text("'[]'::jsonb"))
     created_at = Column(DateTime(timezone=True), server_default=func.now())
 
 
@@ -501,6 +520,14 @@ class KnowledgeReviewRow(Base):
     edited_payload = Column(JSONB)
     note = Column(Text)
     reviewer_id = Column(Text, nullable=False)
+    # r2/r3. `decided_by` says WHETHER A HUMAN LOOKED; `authorised_by` says
+    # on whose authority it happened. They are separate because the admin who
+    # clicks ingest and the expert who reviews can be different people, and in
+    # "llm" mode there is no reviewer at all -- so without the clicker's id
+    # reaching this row the decision would have no owner. Together they read:
+    # approved by LLM with permission of <person>.
+    decided_by = Column(Text, nullable=False, server_default=text("'human'"))
+    authorised_by = Column(Text)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
 
 
@@ -520,6 +547,11 @@ class KnowledgeJobRow(Base):
     user_id = Column(Text, nullable=False)
     status = Column(Text, nullable=False, default="queued")
     stage = Column(Text)
+    # r1 (2026-09-08 discussion): which review mode this batch was ingested
+    # under. "expert" leaves entries waiting for a human; "llm" takes them
+    # straight to approved with no reviewer step at all. Nullable so jobs
+    # written before the column existed stay legal.
+    review_mode = Column(Text)
     document_ids = Column(JSONB, nullable=False, server_default=text("'[]'::jsonb"))
     n_requested = Column(Integer, nullable=False, default=0)
     n_succeeded = Column(Integer, nullable=False, default=0)

src/knowledge_extraction/adapter.py

@@ -26,7 +26,7 @@ from typing import Any
 
 from src.middlewares.logging import get_logger
 
-from .models import Chunk, ParsedDoc
+from .models import Chunk, ChunkAsset, ParsedDoc
 
 logger = get_logger("knowledge_adapter")
 
@@ -54,7 +54,7 @@ _PAGE_END_KEYS = ("page_idx_end", "page_end")
 # seam. Drift is caught instead by a local seam test that imports
 # `knowledge_parsing.contracts` directly and asserts the two agree, so a bump on
 # either side fails loudly in tests rather than silently at read time.
-SUPPORTED_SCHEMA = (0, 3)
+SUPPORTED_SCHEMA = (0, 4)
 
 
 class IncompatibleArtifactError(ValueError):
@@ -156,6 +156,28 @@ def chunk_from_dict(raw: dict[str, Any], doc_id: str, ordinal: int = 0) -> Chunk
         # the floor here, which left `formula_latex` unprovable — `text` carries
         # rendered prose, so a LaTeX claim had nothing to be checked against.
         latex=list(raw.get("latex") or []),
+        # Same story a third time: the parsing half emits `table_html` and
+        # extraction dropped it here, so a table reached the model ONLY as
+        # pipe-linearised text with its row/column structure destroyed. It is the
+        # haystack the formula guard needs for tables and the input a future
+        # table branch needs; carrying it costs nothing.
+        table_html=raw.get("table_html") or None,
+        # New at 0.4.0 — see ChunkAsset. A pointer the adapter drops is a pointer
+        # that does not exist, which is exactly what happened to `table_html`.
+        assets=[_asset(a) for a in (raw.get("assets") or [])],
+        referenced_by=list(raw.get("referenced_by") or []),
+    )
+
+
+def _asset(raw: dict[str, Any]) -> ChunkAsset:
+    """Map one artifact asset onto the internal shape, tolerating absences."""
+    return ChunkAsset(
+        asset_id=str(raw.get("asset_id") or ""),
+        kind=str(raw.get("kind") or "figure"),
+        caption=raw.get("caption") or None,
+        description=raw.get("description") or None,
+        page_no=raw.get("page_no"),
+        storage_key=raw.get("storage_key") or None,
     )
 
 

src/knowledge_extraction/cluster/cluster.py

@@ -1,9 +1,13 @@
 """Mention[] → TermCluster[].
 
 Matching order, cheapest and most certain first:
-    1. exact match after normalisation
+    1. exact match after normalisation (plural-folded — `normalize(lemma=True)`)
     2. abbreviation ↔ expansion (from legend blocks)
-    3. conservative fuzzy (token_set_ratio >= 92, and only above 5 characters)
+
+**There is deliberately no fuzzy pass** (retired 2026-09-10; `_fuzzy_match` is
+parked below with the measurement that retired it). Every merge it made that was
+correct is now done deterministically by plural folding; 94% of what it made was
+a token-subset match, which is always wrong.
 
 **Over-merging is much worse than under-merging.** An under-merge costs one
 extra LLM call and one extra review-queue row. A wrong merge silently destroys a
@@ -58,10 +62,37 @@ def cluster_mentions(
             _add(buckets[linked], mention, "abbrev")
             continue
 
-        match = _fuzzy_match(key, buckets.keys(), fuzzy_threshold)
-        if match:
-            _add(buckets[match], mention, "fuzzy")
-            continue
+        # --- FUZZY PASS RETIRED 2026-09-10 (measured, not a style call) ---------
+        # `_fuzzy_match` is kept below for the record; this call site is not.
+        #
+        # `token_set_ratio` returns EXACTLY 100 whenever one string's token set is
+        # a subset of the other's, so a head noun always absorbed its own
+        # compounds and the threshold of 92 was inert. Measured across the BUMA
+        # standard, a Komatsu shop manual and an Open Pit textbook chapter:
+        #
+        #   131 of the 140 merges this pass made were token-subset matches — ALL
+        #   wrong (`Radiator` ate `radiator cap`; `level` ate `oil level sensor`;
+        #   `Working Hours` ate `effective working hours`, which cost the gold
+        #   term EWH its own entry entirely).
+        #
+        #   Of the 9 genuine near-misses, 6 were plain singular/plural and 3 were
+        #   `gain/loss pty|qty|ua grouping` — three DISTINCT metrics, also wrong.
+        #
+        # So the pass produced no correct merge that plural folding does not now
+        # do deterministically (`normalize(lemma=True)`). Retiring it drove
+        # subset-merges to 0 on all three documents and lifted post-cluster gold
+        # recall 0.8293 -> 0.8537, while also giving the FEWEST clusters of any
+        # correct configuration (Open Pit 431, against 434 for the best guarded
+        # fuzzy variant and 445 for no-fuzzy-without-lemma).
+        #
+        # Matching is now: exact (normalised, plural-folded) -> abbreviation
+        # expansion. Nothing else. If you are tempted to restore this, the thing
+        # to beat is subset_pairs = 0; see FREE_STAGE_RESULTS.md.
+        #
+        # match = _fuzzy_match(key, buckets.keys(), fuzzy_threshold)
+        # if match:
+        #     _add(buckets[match], mention, "fuzzy")
+        #     continue
 
         buckets[key] = {
             "surfaces": {mention.surface},

src/knowledge_extraction/cluster/normalize.py

@@ -31,7 +31,54 @@ STOP_SURFACES = {
 }
 
 
-def normalize(surface: str) -> str:
+# Plural folding for the CLUSTERING KEY ONLY (see `normalize(lemma=...)` below).
+# Measured 2026-09-10: of the 140 merges the retired fuzzy pass made across three
+# documents, 131 were token-subset matches (all wrong) and of the 9 genuine
+# near-misses SIX were plain singular/plural pairs — `resource`/`resources`,
+# `ore type`/`ore types`, `deposit`/`deposits`, `measurement`/`measurements`,
+# `project manager`/`project managers`, `reserve`/`reserves`. Folding those
+# deterministically is what let the fuzzy pass be retired instead of guarded: on
+# the Open Pit textbook it gives 431 clusters against 434 for the best guarded
+# fuzzy config and 445 for no-fuzzy-without-lemma.
+#
+# Deliberately hand-rolled rather than a real lemmatiser: a proper one means a
+# new dependency (nltk/spacy) for a measured six-pair gain, and this runs on a
+# normalised key where the only job is collapsing an English plural suffix.
+_PLURAL_EXCEPTIONS = ("ss", "us", "is", "as", "os")
+
+
+def depluralise(token: str) -> str:
+    """Collapse an English plural suffix on a single normalised token.
+
+    Conservative by construction — it only ever SHORTENS a token, and only on
+    suffixes that are plural in this corpus. `status`, `analysis`, `gas` and
+    `bias` are protected by `_PLURAL_EXCEPTIONS`; anything under 4 characters is
+    left alone so an abbreviation (`UAS`, `BCM`) is never touched.
+    """
+    for suffix, replacement in (("ies", "y"), ("ses", "s"), ("xes", "x"), ("hes", "h")):
+        if token.endswith(suffix) and len(token) > len(suffix) + 1:
+            return token[: -len(suffix)] + replacement
+    if (
+        token.endswith("s")
+        and not token.endswith(_PLURAL_EXCEPTIONS)
+        and len(token) > 3
+    ):
+        return token[:-1]
+    return token
+
+
+def normalize(surface: str, lemma: bool = False) -> str:
+    """Normalise a surface for comparison.
+
+    `lemma=True` additionally folds English plurals, and is passed ONLY by the
+    clustering key path (`is_noise` and `AbbrevIndex`). It is deliberately NOT
+    the default: `rank/evidence.py` imports this function directly to match a
+    term against a chunk heading, and the 2026-09-10 measurement that justified
+    plural folding patched this module's global — which that direct import
+    bypasses. Making it the default would therefore change evidence ranking in a
+    way nothing has measured. Keep the ranker on the unfolded form until there
+    is a number for it.
+    """
     s = unicodedata.normalize("NFKC", surface).casefold()
     s = s.replace("-", " ").replace("_", " ")
     s = re.sub(r"[.’']", "", s)
@@ -45,11 +92,15 @@ def normalize(surface: str) -> str:
     # and gave one concept two clusters. Checked after the strip, not before.
     if s.count("(") != s.count(")"):
         s = re.sub(r"\s+", " ", s.replace("(", " ").replace(")", " ")).strip()
+    # Applied LAST, per token, so it never interferes with the bracket and
+    # punctuation handling above.
+    if lemma:
+        s = " ".join(depluralise(t) for t in s.split())
     return s
 
 
 def is_noise(surface: str) -> bool:
-    n = normalize(surface)
+    n = normalize(surface, lemma=True)
     if len(n) < 2:
         return True
     if n in STOP_SURFACES:
@@ -122,7 +173,8 @@ class AbbrevIndex:
         self.to_expansion: dict[str, str] = {}
         self.to_abbrev: dict[str, str] = {}
         for pair in pairs:
-            abbrev, expansion = normalize(pair.abbrev), normalize(pair.expansion)
+            abbrev = normalize(pair.abbrev, lemma=True)
+            expansion = normalize(pair.expansion, lemma=True)
             if not abbrev or not expansion:
                 continue
             self.to_expansion[abbrev] = expansion
@@ -152,9 +204,9 @@ class AbbrevIndex:
                 continue
             outer, inner = parts
             if looks_like_abbreviation(inner) and is_initialism(inner, outer):
-                abbrev, expansion = normalize(inner), normalize(outer)
+                abbrev, expansion = normalize(inner, lemma=True), normalize(outer, lemma=True)
             elif looks_like_abbreviation(outer) and is_initialism(outer, inner):
-                abbrev, expansion = normalize(outer), normalize(inner)
+                abbrev, expansion = normalize(outer, lemma=True), normalize(inner, lemma=True)
             else:
                 continue
             if abbrev and expansion and abbrev not in self.to_expansion:
@@ -166,7 +218,7 @@ class AbbrevIndex:
     def canonical_key(self, surface: str) -> str:
         """Map a surface to a shared key so an abbreviation and its expansion
         collide into the same bucket."""
-        n = normalize(surface)
+        n = normalize(surface, lemma=True)
         if n in self.to_abbrev:
             return self.to_abbrev[n]
         if n in self.to_expansion:
@@ -179,7 +231,7 @@ class AbbrevIndex:
         parts = split_parenthetical(surface)
         if parts:
             outer, inner = parts
-            outer_n, inner_n = normalize(outer), normalize(inner)
+            outer_n, inner_n = normalize(outer, lemma=True), normalize(inner, lemma=True)
             if inner_n in self.to_expansion:      # the legend knows the bracket
                 return inner_n
             if outer_n in self.to_abbrev:         # the legend knows the outer
@@ -193,7 +245,7 @@ class AbbrevIndex:
         return n
 
     def linked(self, a: str, b: str) -> bool:
-        na, nb = normalize(a), normalize(b)
+        na, nb = normalize(a, lemma=True), normalize(b, lemma=True)
         if self.to_expansion.get(na) == nb or self.to_expansion.get(nb) == na:
             return True
         # Same reasoning as `canonical_key`: two surfaces are linked when one is

src/knowledge_extraction/config/labels.yaml

@@ -37,8 +37,111 @@ broad:
   - work category
   - analysis method
 
+# ── Genre variants, added 2026-09-10. NOT WIRED: `broad` stays the default. ──────
+#
+# Measured on three documents. `broad` wins where we can score it (BUMA E1 0.8780
+# vs 0.8537 for `conceptual` and 0.4146 for `diagnostic`), which confirms the
+# original calibration. What the other two showed is that label fit is
+# GENRE-dependent, not that either is better:
+#
+#   diagnostic  -> 1108 mentions on the Komatsu shop manual against `broad`'s 449
+#                  (2.5x), and 133 on the textbook. Collapses on the standard.
+#   conceptual  ->  940 mentions on the Open Pit textbook against `broad`'s 792.
+#
+# Those gains are UNSCOREABLE — neither document has a gold set, so "more
+# mentions" is not "better recall", and `diagnostic` on Komatsu also brought 128
+# token-subset merges. They are parked here for the genre router to select once
+# there is a gold set to justify the switch. Selecting one today would be shipping
+# on a mention count.
+
+# Shop manuals / troubleshooting: components, symptoms, remedies. The other sets
+# have no notion of a fault or a corrective action, which is most of a manual.
+#
+# ⚠️ `fault code` added 2026-09-11 after a measured miss. The Komatsu manual prints
+# 27 service codes (`[CA559]`, `[CB2249]`) in plain prose, and BOTH `broad` and the
+# original `diagnostic` set returned **zero** mentions for them — a code is not a
+# component, a symptom, a check or an action, so no label was asking. GLiNER finds
+# them readily once asked: 6 of 6 on the probe chunk at 0.54-0.69, well over the
+# 0.25 floor. Worth recording because it corrects an assumption in the Step 5 plan:
+# routing this document to `diagnostic` would NOT have recovered the codes.
+diagnostic:
+  - machine component
+  - fault or symptom
+  - fault code
+  - inspection or check
+  - corrective action
+  - consumable or fluid
+  - measurement unit
+  - operating condition
+  - warning or indicator
+
+# `broad` plus the one label it was measurably missing. Exists ONLY as the A/B arm;
+# **do not make it the default.** Measured 2026-09-11, `broad` vs `broad_codes`,
+# both arms on current code (so both include the subword window split):
+#
+#                mentions        clusters       code clusters   E1c_strict
+#   BUMA        251 ->  200     86 ->  67          0 ->  0     0.8537 -> 0.7805
+#   Komatsu     449 ->  548     93 -> 161          0 -> 17          —
+#   Open Pit    787 ->  711    427 -> 377          0 ->  0          —
+#
+# **A label is not additive — it perturbs the whole decode.** GLiNER runs flat NER,
+# so a new label competes for spans: BUMA lost 51 mentions, 19 clusters and THREE
+# gold terms (`Coal barging`, `Grouping (Composite)`, `Other Activity`), and
+# `dropped_overlapping` moved 10 -> 6. Open Pit lost 76 mentions. Adding this to
+# `broad` would buy 17 code clusters on one document by damaging the other two.
+#
+# On Komatsu it also costs +73% clusters (93 -> 161) = one LLM call each, against
+# the +20% bar that rejected threshold 0.15. 17 of the 68 new clusters are codes;
+# **66 are cause phrases** ("Clogged fuel filter and/or strainer", "Improper
+# injection timing") — the diagnostic content the other label sets never reached.
+# That is the case for routing, not for widening the default.
+broad_codes:
+  - mining production metric
+  - equipment availability term
+  - operational abbreviation
+  - machine or equipment type
+  - time category
+  - mining activity
+  - material or commodity
+  - measurement unit
+  - data source or system
+  - work category
+  - analysis method
+  - fault code
+
+# Textbooks / standards-of-practice: concepts and categories rather than equipment.
+conceptual:
+  - technical concept
+  - classification category
+  - economic measure
+  - mining activity
+  - material or commodity
+  - analysis method
+  - regulatory or reporting term
+  - measurement unit
+
 # 0.25, not 0.35: the E1 sweep measured recall 0.854 @ 0.25 vs 0.658 @ 0.35 on the
 # `broad` variant. Precision falls (0.41 vs 0.50) but the term filter is meant to be
 # over-inclusive — clustering and evidence ranking absorb the noise, and a term the
 # filter never proposes can never be recovered downstream.
+#
+# ⚠️ RE-SWEPT 2026-09-10. 0.15 had never been tested and measures E1 0.9756
+# (40/41) against 0.8780 at 0.25 — four more gold terms, and post-cluster recall
+# 0.8537 -> 0.9512. It is NOT shipped, and the reason is cost:
+#
+#   clusters vs the 0.25 baseline:  BUMA +50%   Komatsu +29%   Open Pit +10%
+#
+# Clusters drive LLM spend one call each, and a single 46-page chapter already
+# consumes 56% of the 2M pre-flight ceiling. The agreed rule was "ship 0.15 if it
+# lands within +20% of today's cluster count"; the worst case is +50%, so the rule
+# says keep 0.25.
+#
+# The hoped-for escape was pairing it with a larger WINDOW_WORDS to cancel the
+# cost. That is now closed too: 256 truncates 46 of 99 windows on the Open Pit
+# textbook (see settings.WINDOW_WORDS), so the cluster reduction it offered was
+# partly content loss, not efficiency.
+#
+# What would reopen this: a cheaper way to spend a cluster (batching several terms
+# per call), or a gold set for a second document so the recall gain can be
+# confirmed as general rather than BUMA-specific.
 threshold: 0.25

src/knowledge_extraction/extract/base.py

@@ -63,5 +63,41 @@ def evidence_block(chunks: list[Chunk], scores: list[float] | None = None) -> st
         parts.append(
             f"[chunk_id={chunk.chunk_id} section={chunk.section_no or '-'} "
             f"page={chunk.page_start}{score}]\n{head}{chunk.text}"
+            + _asset_block(chunk)
         )
     return "EVIDENCE\n" + "\n\n---\n\n".join(parts)
+
+
+def _asset_block(chunk: Chunk) -> str:
+    """Figures and tables this chunk references, appended to its evidence. 0.4.0.
+
+    The chunk's own text already carries each figure's `![](asset://…)` reference
+    inline, at the position the document put it. This resolves those references so
+    the model can actually use them — before 0.4.0 a figure was an unreachable
+    chunk with empty text, and its vision-model description never reached a prompt
+    at all.
+
+    **The verbatim/generated split is labelled, not merely recorded.** A caption is
+    printed in the document, so it is quotable as evidence and the span check can
+    locate it. A description is a model's claim about a picture: it cannot be
+    span-checked against anything, so quoting it would produce a field that the
+    validator then nulls — wasting the call and looking like a model failure. The
+    label is what stops that.
+    """
+    if not chunk.assets:
+        return ""
+    lines = ["", "ASSETS REFERENCED BY THIS CHUNK"]
+    for asset in chunk.assets:
+        where = f" · page {asset.page_no}" if asset.page_no else ""
+        lines.append(f"  [asset://{asset.asset_id} · {asset.kind}{where}]")
+        if asset.caption:
+            lines.append(f"    caption (VERBATIM, quotable as evidence): {asset.caption}")
+        if asset.description:
+            lines.append(
+                "    description (MODEL-GENERATED, not from the document — "
+                "do NOT quote as evidence):"
+            )
+            lines.append(f"      {asset.description}")
+        if not asset.caption and not asset.description:
+            lines.append("    (no caption or description available)")
+    return "\n".join(lines)

src/knowledge_extraction/extract/branches.py

@@ -49,7 +49,12 @@ def _evidence_for(
     cluster: TermCluster, chunks: list[Chunk], k: int, round_index: int
 ) -> tuple[list[Chunk], list[float]]:
     by_id = {c.chunk_id: c for c in chunks}
-    ids = top_k(cluster, k=k, round_index=round_index)
+    # `chunks` is passed so a linked partner (0.4.0 table pointers) joins the
+    # window. `service` passes it to `top_k` too, so the prompt and the span-check
+    # source stay identical — that equality is load-bearing, not incidental.
+    ids = top_k(cluster, k=k, round_index=round_index, chunks=chunks)
+    # Ranked scores only. A partner was LINKED in, not ranked in, so it has no
+    # score — `evidence_block` shows none for it, which is the honest rendering.
     scores = cluster.evidence_scores[round_index * k : round_index * k + k]
     return [by_id[i] for i in ids if i in by_id], scores
 

src/knowledge_extraction/extract/prompts/glossary.txt

@@ -70,4 +70,15 @@ COMMON MISTAKES TO AVOID:
 - Do NOT put units, table captions, or figure labels in the definition field.
 - If the evidence contains only a formula and no prose, set definition to null.
 
+FIGURES AND TABLES (asset references):
+- Evidence may contain `![](asset://<id>)` markers and an "ASSETS REFERENCED BY THIS
+  CHUNK" section. The marker is where the document placed a figure.
+- A caption marked VERBATIM is printed in the document. You MAY quote it, and you may
+  use it as a provenance span.
+- A description marked MODEL-GENERATED is another model's claim about a picture. You
+  may read it for orientation, but you must NOT quote it and must NOT use it as a
+  provenance span — it does not appear in the document, so any span taken from it is
+  rejected and the field is discarded.
+- Never treat the `asset://` marker itself as part of a definition or a span.
+
 Return a single JSON object. No prose, no markdown fence.

src/knowledge_extraction/extract/prompts/rule.txt

@@ -54,4 +54,15 @@ corpus, so neither an answer nor a NON-answer for a real document leaks into thi
 prompt. Keep it that way — an example lifted from the corpus either hands the
 model a gold answer or forbids one.
 
+FIGURES AND TABLES (asset references):
+- Evidence may contain `![](asset://<id>)` markers and an "ASSETS REFERENCED BY THIS
+  CHUNK" section. The marker is where the document placed a figure.
+- A caption marked VERBATIM is printed in the document. You MAY quote it, and you may
+  use it as a provenance span.
+- A description marked MODEL-GENERATED is another model's claim about a picture. You
+  may read it for orientation, but you must NOT quote it and must NOT use it as a
+  provenance span — it does not appear in the document, so any span taken from it is
+  rejected and the field is discarded.
+- Never treat the `asset://` marker itself as part of a definition or a span.
+
 Return a single JSON object. No prose, no markdown fence.

src/knowledge_extraction/filters/span_filter.py

@@ -28,6 +28,10 @@ from functools import lru_cache
 from ...middlewares.logging import get_logger
 from ..models import Chunk, Mention
 from ..settings import (
+    SPAN_FALLBACK_CHARS_PER_TOKEN,
+    SPAN_MAX_SPLIT_DEPTH,
+    SPAN_SUBWORD_CAP,
+    SPAN_SUBWORD_MARGIN,
     SPAN_TOKEN_CAP,
     WINDOW_OVERLAP,
     WINDOW_WORDS,
@@ -74,10 +78,19 @@ def extract_mentions(
 
     mentions: list[Mention] = []
     capped = 0
+    split = 0
     started = time.time()
+    fits = _token_fitter(model)
 
     for chunk in chunks:
-        for window_text, offset in _windows(chunk.text):
+        windows = _windows(chunk.text)
+        budgeted: list[tuple[str, int]] = []
+        for text, offset in windows:
+            pieces = _fit_to_budget(text, offset, fits)
+            split += len(pieces) - 1
+            budgeted.extend(pieces)
+
+        for window_text, offset in budgeted:
             try:
                 found = model.predict_entities(
                     window_text, labels, threshold=score_floor
@@ -113,11 +126,100 @@ def extract_mentions(
         mentions=len(deduped),
         dropped_overlapping=len(mentions) - len(deduped),
         hit_span_cap=capped,
+        # Windows that exceeded the model's subword budget and were split rather
+        # than silently truncated. Expected to be small and non-zero: 1 of 368
+        # on the three reference documents. A sudden rise means the corpus got
+        # denser, not that something broke.
+        split_over_budget=split,
         seconds=round(time.time() - started, 1),
     )
     return deduped
 
 
+def _token_fitter(model):
+    """A `fits(text) -> bool` using the model's OWN tokeniser where possible.
+
+    The word window is a proxy for the thing that actually matters — the
+    subword count the transformer truncates at — and the two diverge by text
+    density. Asking the tokeniser removes the proxy.
+
+    Falls back to a character budget if the attribute has moved (a GLiNER
+    version change), erring toward over-splitting: a split window costs a little
+    context at one boundary, while a truncated one loses its tail entirely and
+    says nothing about it.
+    """
+    tokenizer = None
+    for path in ("data_processor.transformer_tokenizer", "data_processor.tokenizer"):
+        probe = model
+        try:
+            for part in path.split("."):
+                probe = getattr(probe, part)
+        except AttributeError:
+            continue
+        if hasattr(probe, "tokenize"):
+            tokenizer = probe
+            break
+
+    budget_chars = int(SPAN_SUBWORD_CAP * SPAN_FALLBACK_CHARS_PER_TOKEN)
+    if tokenizer is None:
+        logger.warning(
+            "span tokeniser not found; falling back to a character budget",
+            budget_chars=budget_chars,
+        )
+        return lambda text: len(text) <= budget_chars
+
+    limit = int(SPAN_SUBWORD_CAP * SPAN_SUBWORD_MARGIN)
+
+    def fits(text: str) -> bool:
+        try:
+            return len(tokenizer.tokenize(text)) <= limit
+        except Exception as exc:  # noqa: BLE001 - never break the widest net
+            logger.warning("span tokenise failed", error=repr(exc))
+            return len(text) <= budget_chars
+
+    return fits
+
+
+def _fit_to_budget(
+    text: str, offset: int, fits, depth: int = 0
+) -> list[tuple[str, int]]:
+    """Halve a window until every piece fits the model's subword budget.
+
+    Slices the ORIGINAL text rather than re-joining words, so character offsets
+    stay exact — the same reason `_windows` walks positions. An off-by-one here
+    would fail every span check downstream, and it would look like a bad model.
+
+    The halves overlap by `SPAN_TOKEN_CAP` words — the longest span the filter
+    can emit — which is exactly enough that no term can straddle the cut and be
+    lost, and no more. (`WINDOW_OVERLAP`, the top-level overlap, is 30 words and
+    would make the right half LARGER than the left on a 130-word window, so the
+    split would not converge.) The overlap re-finds terms, and `_dedupe` removes
+    them exactly, keyed on position.
+    """
+    if fits(text) or depth >= SPAN_MAX_SPLIT_DEPTH:
+        return [(text, offset)]
+
+    cut = text.rfind(" ", 0, len(text) // 2 + 1)
+    if cut <= 0:
+        # One enormous unbroken token. Nothing to split on; the model will
+        # truncate it, and there is no representation that avoids that.
+        return [(text, offset)]
+
+    left = text[:cut]
+    # Start the right half SPAN_TOKEN_CAP words back, so the boundary is covered.
+    back = cut
+    for _ in range(SPAN_TOKEN_CAP):
+        previous = text.rfind(" ", 0, back)
+        if previous <= 0:
+            break
+        back = previous
+    right_start = back + 1 if back > 0 else cut + 1
+
+    return _fit_to_budget(left, offset, fits, depth + 1) + _fit_to_budget(
+        text[right_start:], offset + right_start, fits, depth + 1
+    )
+
+
 def _windows(text: str) -> list[tuple[str, int]]:
     """Overlapping word windows plus each window's character offset.
 

src/knowledge_extraction/genre.py

@@ -0,0 +1,213 @@
+"""Step 5 — what KIND of document is this, decided before anything is spent.
+
+The pipeline was calibrated on one genre (an Indonesian mining standard) and is
+now handed three. The failure is not subtle: on the Komatsu shop manual the
+glossary branch burns ~104 calls to abstain 93 times, because a troubleshooting
+manual states symptoms and remedies, not definitions. No amount of prompt work
+reaches that — it is the wrong question asked of the wrong document.
+
+Four signals separate the three genres cleanly, and all four are already
+computed for free before the paid stage:
+
+                        heading coverage   tabular ratio   abbrev pairs   rule density
+    BUMA standard              41%              25%             5             0.50
+    Komatsu shop manual         1%              69%             0             0.01
+    Open Pit textbook          63%               5%             0             1.10
+
+They do not merely differ, they differ in DIFFERENT DIRECTIONS, which is what
+makes a rule-based classifier honest here rather than a model call: a shop
+manual is mostly tables with no headings, a textbook is mostly headed prose, a
+standard sits between them and is the only one that defines its abbreviations.
+
+**What this module does NOT do yet, and why.** Classification is on; acting on
+it is not. Both levers below default to off because neither can be justified by
+a measurement today, and shipping an unmeasured recall change is how the
+clustering defect survived for months:
+
+* `ROUTE_LABELS` — switching the GLiNER label set per genre. The `diagnostic`
+  and `conceptual` sets produce far more mentions on their genres (2.5x on
+  Komatsu), but only BUMA has a gold set, so "more mentions" cannot be shown to
+  be "better recall". Unlocked by: a gold set for a second document.
+* `ROUTE_BRANCHES` — skipping branches that do not fit. Measured on exactly one
+  document (Komatsu: 0 definitions across two paid runs, as predicted), which
+  is a prediction confirmed once, not a rule. Unlocked by: a second diagnostic
+  document behaving the same way, or Rifqi accepting one document as enough.
+
+Until then this is observability: the verdict and its warnings are recorded, an
+operator can see why a document is expensive and unproductive, and turning
+either lever on is a one-line change with the reasoning already written down.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+# Both default OFF. See the module docstring for what unlocks each.
+ROUTE_LABELS = False
+ROUTE_BRANCHES = False
+
+# Thresholds sit between the measured clusters rather than on top of any one of
+# them, so a fourth document of a known genre is likely to land correctly.
+# Widely separated by construction — see the table above.
+HEADING_COVERAGE_LOW = 0.10
+HEADING_COVERAGE_HIGH = 0.50
+TABULAR_RATIO_HIGH = 0.50
+TABULAR_RATIO_LOW = 0.20
+RULE_DENSITY_LOW = 0.10
+
+# Below this, the document's own structure is missing and every heading-derived
+# signal downstream degrades: `source_wording` (read off headings), the
+# `term_in_heading` ranking weight, and `DocumentContext.outline`. Komatsu has
+# exactly ONE headed chunk of 85, and nothing warned about it.
+HEADING_COVERAGE_WARN = 0.10
+
+GENRE_LABELS = {
+    "standard": "broad",
+    "diagnostic": "diagnostic",
+    "conceptual": "conceptual",
+    "unknown": "broad",
+}
+
+# Which branches earn their spend on each genre. `unknown` runs everything —
+# not knowing the genre is not a reason to extract less.
+GENRE_BRANCHES: dict[str, tuple[str, ...]] = {
+    "standard": ("glossary", "rule", "formula", "summary"),
+    # A shop manual defines nothing. What it holds is a symptom x cause matrix
+    # across 59 tables, and the branch that reads one does not exist yet
+    # (Step 8) — so today the honest answer is "almost nothing here fits".
+    "diagnostic": ("summary",),
+    "conceptual": ("glossary", "rule", "summary"),
+    "unknown": ("glossary", "rule", "formula", "summary"),
+}
+
+
+@dataclass
+class GenreSignals:
+    n_chunks: int
+    heading_coverage: float
+    tabular_ratio: float
+    abbrev_pairs: int
+    rule_density: float
+
+    def as_dict(self) -> dict:
+        return {
+            "n_chunks": self.n_chunks,
+            "heading_coverage": round(self.heading_coverage, 3),
+            "tabular_ratio": round(self.tabular_ratio, 3),
+            "abbrev_pairs": self.abbrev_pairs,
+            "rule_density": round(self.rule_density, 3),
+        }
+
+
+@dataclass
+class GenreVerdict:
+    genre: str
+    signals: GenreSignals
+    reasons: list[str] = field(default_factory=list)
+    warnings: list[str] = field(default_factory=list)
+
+    @property
+    def labels_variant(self) -> str:
+        """The label set this genre wants — advisory unless `ROUTE_LABELS`."""
+        return GENRE_LABELS.get(self.genre, "broad")
+
+    @property
+    def branches(self) -> tuple[str, ...]:
+        """The branches that fit — advisory unless `ROUTE_BRANCHES`."""
+        return GENRE_BRANCHES.get(self.genre, GENRE_BRANCHES["unknown"])
+
+    @property
+    def well_served(self) -> bool:
+        """Is this a document the pipeline can actually do something with?
+
+        False is not a failure — it is the honest answer for a shop manual
+        today, and it is worth saying BEFORE spending ~104 calls to find out.
+        """
+        return self.genre != "diagnostic"
+
+    def as_dict(self) -> dict:
+        return {
+            "genre": self.genre,
+            "labels_variant": self.labels_variant,
+            "branches": list(self.branches),
+            "well_served": self.well_served,
+            "routing_applied": {"labels": ROUTE_LABELS, "branches": ROUTE_BRANCHES},
+            "signals": self.signals.as_dict(),
+            "reasons": self.reasons,
+            "warnings": self.warnings,
+        }
+
+
+def measure(doc, filtered) -> GenreSignals:
+    """Four free signals off the parsed artifact and the free filter stage."""
+    chunks = doc.chunks or []
+    n = len(chunks) or 1
+    return GenreSignals(
+        n_chunks=len(chunks),
+        heading_coverage=sum(1 for c in chunks if (c.heading or "").strip()) / n,
+        tabular_ratio=sum(1 for c in chunks if c.is_tabular) / n,
+        abbrev_pairs=len(filtered.abbrev_pairs),
+        rule_density=len(filtered.rule_candidates) / n,
+    )
+
+
+def classify(doc, filtered) -> GenreVerdict:
+    """Name the genre, or say `unknown` rather than guess.
+
+    Deliberately rule-based and deliberately conservative. A misclassification
+    that routes a document to the wrong label set costs recall silently, which
+    is the one failure mode this pipeline has repeatedly proven bad at noticing
+    — so an unclear document gets `unknown`, which routes to exactly today's
+    behaviour and spends nothing extra.
+    """
+    signals = measure(doc, filtered)
+    reasons: list[str] = []
+
+    if (
+        signals.tabular_ratio >= TABULAR_RATIO_HIGH
+        and signals.heading_coverage <= HEADING_COVERAGE_LOW
+        and signals.rule_density <= RULE_DENSITY_LOW
+    ):
+        genre = "diagnostic"
+        reasons.append(
+            f"{signals.tabular_ratio:.0%} tabular with {signals.heading_coverage:.0%} "
+            "headed and almost no rule language — a symptom/remedy matrix, not prose"
+        )
+    elif (
+        signals.heading_coverage >= HEADING_COVERAGE_HIGH
+        and signals.tabular_ratio <= TABULAR_RATIO_LOW
+    ):
+        genre = "conceptual"
+        reasons.append(
+            f"{signals.heading_coverage:.0%} of chunks are headed and only "
+            f"{signals.tabular_ratio:.0%} tabular — structured expository prose"
+        )
+    elif signals.abbrev_pairs >= 3 and signals.heading_coverage > HEADING_COVERAGE_LOW:
+        genre = "standard"
+        reasons.append(
+            f"{signals.abbrev_pairs} abbreviation definitions in headed prose — "
+            "a document that names its own terms"
+        )
+    else:
+        genre = "unknown"
+        reasons.append(
+            "no genre signature matched; routing is unchanged and nothing is skipped"
+        )
+
+    warnings: list[str] = []
+    if signals.heading_coverage < HEADING_COVERAGE_WARN:
+        warnings.append(
+            f"heading coverage is {signals.heading_coverage:.0%} "
+            f"({round(signals.heading_coverage * signals.n_chunks)} of "
+            f"{signals.n_chunks} chunks): `source_wording`, the term_in_heading "
+            "ranking weight and the document outline all degrade together, and "
+            "nothing downstream can tell that from a document with no terms"
+        )
+    if genre == "diagnostic":
+        warnings.append(
+            "diagnostic documents define almost nothing — the glossary branch is "
+            "expected to abstain on nearly every cluster. The content is in the "
+            "tables, and the branch that reads them is not built (Step 8)"
+        )
+
+    return GenreVerdict(genre=genre, signals=signals, reasons=reasons, warnings=warnings)

src/knowledge_extraction/keys.py

@@ -0,0 +1,160 @@
+"""Readable concept keys — `TYPE_SUBJECT`, the label a person or an agent searches by.
+
+Asked for in the 2026-09-08 discussion (m1). The shape is the one agreed there:
+the type first, then what the entry is *about*, in capitals — `GLOSSARY_PA`,
+never the entry's full text. Two properties it exists to provide:
+
+- **One concept, one key.** `PA` drawn from five documents still reads
+  `GLOSSARY_PA`, so an agent that needs a term's definition writes the key and
+  looks it up directly — no scan, no embedding, no model call.
+- **Readable at a glance.** Capitals were a deliberate call (lowercase reads
+  worse in a list), and lookups are capitals too.
+
+**A key is a label, never an identity — this is m2.** `entity_id` in `ids.py`
+stays the thing an expert's approval hangs on, because it is a hash of content
+and survives a re-run untouched. A key is built from a *surface*, so it moves
+the moment clustering canonicalises that surface differently. Both travel
+together and neither substitutes for the other: the hash identifies, the key
+reads. Collapsing the two is exactly how approvals get orphaned, which is the
+failure `ids.py` was written to prevent.
+
+Prefixes use the `kind` column's own vocabulary (English, singular) so the
+system holds one spelling of each branch rather than two to keep in sync
+(decided 2026-09-14).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+import unicodedata
+from collections.abc import Iterable, Mapping
+from typing import Any
+
+# Keyed by the `kind` column's values, so a new kind that forgets to register
+# here returns None rather than silently inventing a prefix.
+_PREFIX: dict[str, str] = {
+    "glossary": "GLOSSARY",
+    "rule": "RULE",
+    "formula": "FORMULA",
+    "document": "DOCUMENT",
+    "domain": "DOMAIN",
+}
+
+# Past this a subject stops being a label and becomes the content it was meant
+# to stand in for. A rule's `statement` can be a whole paragraph.
+MAX_SUBJECT = 40
+
+# Short, and deliberately NOT the full `entity_id`: this digest separates two
+# rules that govern the same term. It does not identify the row — `entity_id`
+# does, and shortening a hash for readability is only safe because of that.
+DIGEST_LEN = 6
+
+
+def slugify(surface: str) -> str:
+    """`Physical of Availability (PA)` -> `PHYSICAL_OF_AVAILABILITY_PA`.
+
+    Accents are folded rather than dropped so an Indonesian or a borrowed term
+    keeps its letters. Everything that is not a letter or a digit becomes a
+    single underscore, because a key that carries brackets or slashes cannot be
+    typed from memory — which is the whole point of having one.
+    """
+    folded = unicodedata.normalize("NFKD", surface or "")
+    folded = "".join(c for c in folded if not unicodedata.combining(c))
+    slug = re.sub(r"[^A-Za-z0-9]+", "_", folded).upper()
+    slug = re.sub(r"_+", "_", slug).strip("_")
+    if len(slug) <= MAX_SUBJECT:
+        return slug
+    # Cut on a word boundary so the tail is not half a word — but a single long
+    # token has no boundary to cut on, and an empty subject is worse than a
+    # blunt one.
+    cut = slug[:MAX_SUBJECT]
+    head, _, _tail = cut.rpartition("_")
+    return (head or cut).strip("_")
+
+
+def _digest(value: str) -> str:
+    return hashlib.sha256(value.encode("utf-8")).hexdigest()[:DIGEST_LEN].upper()
+
+
+def term_surfaces(glossary_payloads: Iterable[Mapping[str, Any]] | None) -> dict[str, str]:
+    """`term_id` -> the term's surface, for keying the rules that govern it.
+
+    Built by the caller from the same run's glossary payloads, because a rule
+    payload carries `term_ids` and no surfaces — the words live on the glossary
+    entry, not on the rule.
+    """
+    out: dict[str, str] = {}
+    for payload in glossary_payloads or []:
+        term_id = payload.get("term_id")
+        term = payload.get("term")
+        if term_id and isinstance(term, str) and term.strip():
+            out[str(term_id)] = term.strip()
+    return out
+
+
+def _rule_subject(
+    payload: Mapping[str, Any], entity_id: str, surfaces: Mapping[str, str]
+) -> str:
+    """A rule has no canonical surface of its own — it is a sentence.
+
+    So it borrows the term it governs and adds a short digest, because one term
+    usually governs several rules: `RULE_PA_C07BE4`. Asking the model to name
+    the rule was the obvious alternative and is the one `ids.py` already rules
+    out — a model cannot produce a stable identifier, and a model-written
+    `rule_id` was a real defect here, returning a different slug for the same
+    rule on consecutive runs at temperature 0. A rule that governs no term
+    keeps the digest alone; that it is not searchable by word was accepted when
+    this was agreed.
+    """
+    for term_id in payload.get("term_ids") or []:
+        surface = slugify(surfaces.get(str(term_id), ""))
+        if surface:
+            return f"{surface}_{_digest(entity_id)}"
+    return _digest(entity_id)
+
+
+def _subject(
+    kind: str,
+    payload: Mapping[str, Any],
+    entity_id: str,
+    doc_id: str,
+    surfaces: Mapping[str, str],
+) -> str:
+    if kind == "glossary":
+        return slugify(payload.get("term") or "")
+    if kind == "formula":
+        # A formula may legitimately have no name; `entity_id` already falls
+        # back name -> latex -> chunk_id, so the digest inherits that work.
+        return slugify(payload.get("name") or "") or _digest(entity_id)
+    if kind == "document":
+        # The document's own id reads better than its title. A title is a
+        # verbatim transcription and routinely carries the PDF's broken
+        # spacing — correct to store, useless to type.
+        return slugify(doc_id) or _digest(entity_id)
+    if kind == "domain":
+        return slugify(payload.get("domain_name") or "")
+    if kind == "rule":
+        return _rule_subject(payload, entity_id, surfaces)
+    return ""
+
+
+def concept_key(
+    kind: str,
+    payload: Mapping[str, Any],
+    *,
+    entity_id: str,
+    doc_id: str = "",
+    surfaces: Mapping[str, str] | None = None,
+) -> str | None:
+    """The `TYPE_SUBJECT` key for one entry; None when the kind is unregistered.
+
+    Never raises: the caller writes this into a row alongside content it has
+    already validated, and a key that cannot be built is a degraded label, not
+    a reason to lose the entry.
+    """
+    prefix = _PREFIX.get(kind)
+    if prefix is None:
+        return None
+    subject = _subject(kind, payload, entity_id, doc_id, surfaces or {})
+    return f"{prefix}_{subject}" if subject else prefix

src/knowledge_extraction/models.py

@@ -67,6 +67,30 @@ class SubdomainEnum(str, Enum):
 # ── Stage 1: the chunk (internal view of the seam artifact) ─────────────
 
 
+class ChunkAsset(BaseModel):
+    """A figure or table referenced by a chunk. Crosses the seam at 0.4.0.
+
+    The subset of the parsing half's `Asset` that extraction actually uses. The
+    split between the two content fields is the whole point and must be preserved
+    into the prompt:
+
+    - `caption` is **verbatim document text**, so it is quotable as evidence and
+      the span check can locate it.
+    - `description` is **model-written** (a vision model's claim about the
+      picture). It is never span-checkable, never evidence, and must never be
+      copied into `Chunk.text` — `text` is the haystack the span check searches,
+      so generated prose in there would let a fabrication pass the control built
+      to catch it.
+    """
+
+    asset_id: str
+    kind: str
+    caption: str | None = None
+    description: str | None = None
+    page_no: int | None = None
+    storage_key: str | None = None
+
+
 class Chunk(BaseModel):
     """One unit of a parsed document, as the extraction stages need it.
 
@@ -98,6 +122,16 @@ class Chunk(BaseModel):
     # it on the floor at the adapter — the same way it once dropped `latex`.
     heading_path: list[str] = Field(default_factory=list)
 
+    # Figures and tables this chunk carries or points at. New at schema 0.4.0.
+    # Before it, a figure was its own chunk with EMPTY text, which made it
+    # unreachable: no text -> no mentions -> no cluster -> never evidence. 13
+    # figures across three documents, each described by a vision model we paid
+    # for, and none ever reached a prompt.
+    assets: list[ChunkAsset] = Field(default_factory=list)
+    # chunk_ids pointing AT this chunk (set on table chunks, which stay
+    # standalone because their linearised text earns its evidence slot).
+    referenced_by: list[str] = Field(default_factory=list)
+
     # Cheap downstream filters / ranking signals
     has_formula: bool = False
     is_tabular: bool = False
@@ -115,6 +149,14 @@ class Chunk(BaseModel):
     # "fails verification" — see `validate/span_check.py`.
     latex: list[str] = Field(default_factory=list)
 
+    # Source table markup, kept beside the pipe-linearised `text`. Crosses the
+    # seam at 0.4.0; before that the parsing half emitted it and the adapter
+    # dropped it, so a symptom x cause matrix reached the model as a wall of
+    # pipe-delimited text with the row/column correspondence gone. `colspan` and
+    # `rowspan` live here, which is what a table branch needs to recover
+    # (row header, column header, cell) triples.
+    table_html: str | None = None
+
 
 class ParsedDoc(BaseModel):
     """A document's chunks plus the identity needed to version and cache them."""

src/knowledge_extraction/mutation.py

@@ -0,0 +1,113 @@
+"""Has an entry changed in a way that invalidates an expert's ruling? (i3)
+
+From the 2026-09-08 discussion: *"kalau dia sudah termutate harusnya dia perlu
+review lagi… kalau tidak berubah, tetap approved."* The whole difficulty is in
+the second half. Flipping everything a new document touches is easy and useless
+— a document that merely mentions `PA` again moves `mention_count`, and if that
+counts as a mutation the expert comes back to forty rows where nothing has
+actually changed, finds that out one row at a time, and stops trusting the queue.
+
+So the comparison is deliberately narrow: **only the fields an expert ruled on**
+(decided 2026-09-14). Everything the pipeline computes about an entry — how
+often it was mentioned, how it was classified, where it was found — can move
+freely without disturbing an approval, because none of it is what the expert was
+looking at when they approved.
+
+Whitespace is collapsed before comparing, so re-extraction that reflows a
+definition does not re-open it. Any other difference counts: this stays
+conservative, because missing a real change means serving knowledge an expert
+never agreed to, and that is the failure mode this exists to prevent.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+# What a reviewer is actually judging, per kind — the fields the review queue
+# puts in front of them. Everything else on the entry is machinery.
+RULED_FIELDS: dict[str, tuple[str, ...]] = {
+    "glossary": ("term", "full_name", "definition", "source_wording"),
+    "rule": ("statement", "condition", "consequence"),
+    "formula": ("name", "formula_latex", "unit"),
+    "document": ("title", "purpose_verbatim"),
+    "domain": ("domain_name", "purpose_verbatim"),
+}
+
+# Pipeline bookkeeping. Named explicitly so the fallback below cannot mistake a
+# recomputed count for a changed claim.
+VOLATILE_FIELDS: frozenset[str] = frozenset(
+    {
+        "mention_count",
+        "diff_status",
+        "extraction_status",
+        "definition_conflict",
+        "conflict_variants",
+        "provenance",
+        "doc_id",
+        "doc_ids",
+    }
+)
+
+
+def _norm(value: Any) -> Any:
+    """Collapse whitespace in strings; recurse into lists and dicts.
+
+    Reflowed prose is not a changed claim. Nothing else is normalised — each
+    further normalisation is a difference this would stop noticing.
+    """
+    if isinstance(value, str):
+        return " ".join(value.split())
+    if isinstance(value, list | tuple):
+        return [_norm(v) for v in value]
+    if isinstance(value, Mapping):
+        return {k: _norm(v) for k, v in sorted(value.items())}
+    return value
+
+
+def ruled_content(kind: str, payload: Mapping[str, Any] | None) -> dict[str, Any]:
+    """The subset of an entry that an approval actually covers.
+
+    An unregistered kind falls back to the whole payload minus the volatile
+    fields — conservative on purpose: a new kind should over-report changes
+    until someone decides what its reviewer is looking at, never under-report.
+    """
+    payload = payload or {}
+    fields = RULED_FIELDS.get(kind)
+    if fields is not None:
+        return {field: _norm(payload.get(field)) for field in fields}
+    return {
+        key: _norm(value)
+        for key, value in sorted(payload.items())
+        if key not in VOLATILE_FIELDS
+    }
+
+
+def changed_fields(
+    kind: str,
+    reviewed: Mapping[str, Any] | None,
+    current: Mapping[str, Any] | None,
+) -> list[str]:
+    """Which ruled-on fields moved — the before/after a reviewer is shown (r4).
+
+    Same comparison as `has_mutated`, reported field by field so the queue can
+    say *what* changed instead of making the expert re-read the whole entry to
+    find out. That was the ask: show before / after, do not send them back to
+    the start.
+    """
+    before = ruled_content(kind, reviewed)
+    after = ruled_content(kind, current)
+    return [field for field in before if before[field] != after.get(field)]
+
+
+def has_mutated(
+    kind: str,
+    reviewed: Mapping[str, Any] | None,
+    current: Mapping[str, Any] | None,
+) -> bool:
+    """True when `current` differs from what was approved, in ways that matter.
+
+    `reviewed` is what the expert had in front of them — for an `edited`
+    decision that is their corrected payload, not the model's original claim.
+    """
+    return ruled_content(kind, reviewed) != ruled_content(kind, current)

src/knowledge_extraction/queue/review_queue.py

@@ -22,6 +22,22 @@ on the entry for a consumer that wants them.
 
 from __future__ import annotations
 
+import re
+import textwrap
+
+# At most one unresolved key parameter per this many term rows. See
+# `_interleave` for why a cap replaced the promotion tier it used to have.
+KEY_PARAMETER_INTERLEAVE = 5
+
+# A key parameter NAME longer than this is not a name. The branch that produces
+# them has no span check and no length guard, and on a textbook it returns the
+# whole defining paragraph as the surface — see `_parameter_name`.
+KEY_PARAMETER_NAME_MAX = 60
+
+# The first sentence/label boundary in a returned surface. Applied only past
+# `KEY_PARAMETER_NAME_MAX`, which is what keeps "No. of units" intact.
+_HEAD_SPLIT = re.compile(r"[.:\n]")
+
 
 def build_queue(entries: list[dict], domain: dict | None = None) -> list[dict]:
     """Rows an expert works through, most-decision-worthy first.
@@ -36,12 +52,12 @@ def build_queue(entries: list[dict], domain: dict | None = None) -> list[dict]:
             entry.get("diff_status") == "conflicting"
             or entry.get("definition_conflict") is True
         )
-        return (0 if conflicting else 2, -int(entry.get("mention_count", 0) or 0))
+        return (0 if conflicting else 1, -int(entry.get("mention_count", 0) or 0))
 
-    queue = []
+    term_rows = []
     for entry in sorted(entries, key=sort_key):
         provenance = entry.get("provenance") or {}
-        queue.append(
+        term_rows.append(
             {
                 "row_kind": "term",
                 # What an approve/edit/reject decision attaches to. Without it a
@@ -57,21 +73,51 @@ def build_queue(entries: list[dict], domain: dict | None = None) -> list[dict]:
                 "section_no": provenance.get("section_no"),
                 "span": provenance.get("span"),
                 "review_reason": _reason(entry),
-                "_sort": sort_key(entry),
+                "_conflict": sort_key(entry)[0] == 0,
             }
         )
 
-    queue.extend(_key_parameter_rows(domain))
-
-    # Ranked once, over both kinds. Unresolved parameters sit between conflicts
-    # and routine confirmations: a contradiction is still the more urgent call,
-    # but "the model named a parameter nothing in the document corroborates" is
-    # a judgement only the expert can make, and it should not sink beneath 60
-    # routine rows where nobody reaches it.
-    queue.sort(key=lambda row: row["_sort"])
+    queue = _interleave(term_rows, _key_parameter_rows(domain))
     for rank, row in enumerate(queue, start=1):
         row["rank"] = rank
-        del row["_sort"]
+        row.pop("_conflict", None)
+    return queue
+
+
+def _interleave(term_rows: list[dict], parameter_rows: list[dict]) -> list[dict]:
+    """Conflicts, then terms by frequency with key parameters spliced in.
+
+    Unresolved key parameters used to occupy their own tier directly beneath
+    conflicts, which meant EVERY one of them outranked EVERY routine term no
+    matter how many there were. On the Komatsu shop manual that is 24 junk rows
+    standing in front of `Engine` (108 mentions) — the branch produces them in
+    bulk on exactly the documents where it understands least, so the promotion
+    fired hardest where it was least deserved and the expert met a wall of
+    noise before reaching anything real.
+
+    The original reason for promoting them is still sound and is preserved: the
+    key-parameter branch has no span check, so an unresolvable surface is the
+    one hallucination signal it can raise, and it must not sink beneath sixty
+    routine rows where nobody reaches it. A CAP expresses that without letting
+    the tail take the queue over — one parameter per five terms, so they are
+    always visible early and never crowd out frequency.
+
+    Conflicts stay strictly on top: a contradiction is a decision only the
+    expert can make, and nothing interleaves above it.
+    """
+    conflicts = [row for row in term_rows if row.get("_conflict")]
+    routine = [row for row in term_rows if not row.get("_conflict")]
+
+    queue = list(conflicts)
+    pending = list(parameter_rows)
+    for index, row in enumerate(routine, start=1):
+        queue.append(row)
+        if pending and index % KEY_PARAMETER_INTERLEAVE == 0:
+            queue.append(pending.pop(0))
+
+    # Whatever the interleave could not place — a document with more unresolved
+    # parameters than terms, which is itself a signal — rather than dropped.
+    queue.extend(pending)
     return queue
 
 
@@ -88,6 +134,7 @@ def _key_parameter_rows(domain: dict | None) -> list[dict]:
     for parameter in (domain or {}).get("key_parameters") or []:
         if parameter.get("term_id"):
             continue
+        name, verbatim = _parameter_name(parameter.get("surface"))
         rows.append(
             {
                 "row_kind": "key_parameter",
@@ -95,7 +142,12 @@ def _key_parameter_rows(domain: dict | None) -> list[dict]:
                 # there is no term_id, which is the entire point of the row.
                 "term_id": None,
                 "brief_id": (domain or {}).get("brief_id"),
-                "term": parameter.get("surface"),
+                "term": name,
+                # Null unless the surface had to be shortened. The full text is
+                # never discarded: it is what the expert judges "is this real?"
+                # against, and the branch picks the right CONCEPTS even when it
+                # returns them at paragraph length.
+                "surface_full": verbatim,
                 "definition": None,
                 "source_wording": None,
                 "mention_count": 0,
@@ -107,12 +159,38 @@ def _key_parameter_rows(domain: dict | None) -> list[dict]:
                     "named as a key parameter but no extracted term corroborates it "
                     "— confirm it is real"
                 ),
-                "_sort": (1, 0),
             }
         )
     return rows
 
 
+def _parameter_name(surface: str | None) -> tuple[str, str | None]:
+    """A key parameter's displayable name, and its full text when shortened.
+
+    The branch returns a `surface` that is supposed to be a parameter name, and
+    on the Open Pit textbook it returns the right concept wrapped in its whole
+    defining sentence. Nothing downstream caps it, so the queue renders a
+    paragraph where a name belongs and the row is unreadable at a glance.
+
+    Shortening is applied ONLY past `KEY_PARAMETER_NAME_MAX`, which is what
+    keeps a legitimate short name containing a period ("No. of units") intact —
+    splitting unconditionally on the first `.` is the obvious implementation and
+    it silently truncates that class of name.
+    """
+    text = (surface or "").strip()
+    if not text:
+        return "", None
+    if len(text) <= KEY_PARAMETER_NAME_MAX and "\n" not in text:
+        return text, None
+
+    head = _HEAD_SPLIT.split(text, 1)[0].strip()
+    if not head or len(head) > KEY_PARAMETER_NAME_MAX:
+        # No usable boundary — a run-on clause. Cut on a word boundary rather
+        # than mid-token so the row still reads as language.
+        head = textwrap.shorten(text, width=KEY_PARAMETER_NAME_MAX, placeholder="…")
+    return head, text
+
+
 def _reason(entry: dict) -> str:
     if entry.get("definition_conflict") or entry.get("diff_status") == "conflicting":
         return "conflicting definitions — expert decision required"

src/knowledge_extraction/rank/evidence.py

@@ -10,6 +10,13 @@ survive any future change here:
 3. **The full ranked list is retained**, so escalation can take the next K and
    self-correct.
 
+One signal ranks on ASSET metadata (`asset_caption_names_term`, added 2026-09-11),
+and it is deliberately limited to the **caption**. A caption is printed in the
+document and is span-checkable; the `description` beside it is model-written and
+cannot be, so ranking on a description would let a hallucinated sentence pull a
+figure into evidence — the retrieval-side version of the failure the span check
+exists to prevent.
+
 The rejected relevance gate had none of the three. If ranking is ever revisited,
 that is the test to apply.
 """
@@ -22,7 +29,12 @@ from ..cluster.normalize import normalize
 from ..filters.cue_filter import definitional_hits
 from ..filters.legend_filter import find_legend_lines
 from ..models import Chunk, TermCluster
-from ..settings import CUE_PROXIMITY_CHARS, EVIDENCE_K, EVIDENCE_WEIGHTS
+from ..settings import (
+    CUE_PROXIMITY_CHARS,
+    EVIDENCE_K,
+    EVIDENCE_LINK_PARTNERS,
+    EVIDENCE_WEIGHTS,
+)
 
 
 def rank_evidence(
@@ -59,11 +71,62 @@ def rank_evidence(
 
 
 def top_k(
-    cluster: TermCluster, k: int = EVIDENCE_K, round_index: int = 0
+    cluster: TermCluster,
+    k: int = EVIDENCE_K,
+    round_index: int = 0,
+    chunks: list[Chunk] | None = None,
 ) -> list[str]:
-    """Evidence for extraction round `round_index` (0 = best K, 1 = next K…)."""
+    """Evidence for extraction round `round_index` (0 = best K, 1 = next K…).
+
+    When `chunks` is supplied, up to `EVIDENCE_LINK_PARTNERS` **linked partner**
+    chunks are appended to the window: 0.4.0 records a table and the prose that
+    introduces it as a pair (`referenced_by`, both directions), and the ranker
+    measured as never co-selecting them — 0 of 603 clusters across the three test
+    documents had both sides in their top-5. The partner is therefore unseen
+    content, and for a bare table it is usually the only thing that names it.
+
+    Both consumers of this function pass `chunks` — `_evidence_for` (the prompt)
+    and `service` (the span-check source) — so the model reads exactly what the
+    span check searches. The full ranked list is NOT modified, so anything reading
+    `evidence_chunk_ids` (`rounds_available` included) is unaffected. See
+    `EVIDENCE_LINK_PARTNERS` for the measurement and why the expansion lives here
+    rather than in `rank_evidence`.
+
+    Omitting `chunks` keeps the pre-0.4.0 behaviour exactly.
+    """
     start = round_index * k
-    return cluster.evidence_chunk_ids[start : start + k]
+    window = cluster.evidence_chunk_ids[start : start + k]
+    if not chunks or EVIDENCE_LINK_PARTNERS <= 0 or not window:
+        return window
+    return window + _linked_partners(window, chunks, EVIDENCE_LINK_PARTNERS)
+
+
+def _linked_partners(
+    window: list[str], chunks: list[Chunk], cap: int
+) -> list[str]:
+    """Partners of chunks in `window`, best-ranked side first, capped.
+
+    Walks the window in rank order so the partner chosen belongs to the
+    highest-ranked linked chunk, and sorts each chunk's own partners by id so a
+    chunk with several links resolves identically on every run.
+    """
+    links: dict[str, set[str]] = {}
+    for chunk in chunks:
+        for other in chunk.referenced_by:
+            links.setdefault(chunk.chunk_id, set()).add(other)
+            links.setdefault(other, set()).add(chunk.chunk_id)
+
+    seen = set(window)
+    picked: list[str] = []
+    for chunk_id in window:
+        for partner in sorted(links.get(chunk_id, ())):
+            if partner in seen:
+                continue
+            picked.append(partner)
+            seen.add(partner)
+            if len(picked) >= cap:
+                return picked
+    return picked
 
 
 def rounds_available(cluster: TermCluster, k: int, max_rounds: int) -> int:
@@ -126,6 +189,21 @@ def _score_chunk(
                 score += w["in_legend_block"]
                 break
 
+    # A figure in this chunk whose CAPTION names the term (0.4.0 assets).
+    #
+    # Word-boundary matched against the caption only — never the model-written
+    # `description`. A description is a claim about a picture that cannot be
+    # span-checked, so letting it influence retrieval would let a hallucinated
+    # sentence pull a figure into evidence. The caption is printed in the
+    # document, so it is safe to rank on.
+    if any(
+        asset.caption
+        and any(_word_match(normalize(v), normalize(asset.caption))
+                for v in cluster.variants)
+        for asset in getattr(chunk, "assets", []) or []
+    ):
+        score += w.get("asset_caption_names_term", 0.0)
+
     if chunk.has_formula:
         score += w["formula_present"]
 

src/knowledge_extraction/service.py

@@ -15,6 +15,7 @@ import time
 
 from ..middlewares.logging import get_logger
 from .cluster import cluster_mentions
+from .cluster.normalize import normalize
 from .diff import diff_glossary
 from .extract import (
     build_glossary_prompt,
@@ -23,8 +24,10 @@ from .extract import (
     extract_glossary,
     extract_rule,
     extract_summary,
+    prefix_tokens,
 )
 from .filters import abbrev_pairs, extract_mentions, rule_candidates
+from .ids import term_id
 from .link import link_all
 from .models import (
     CallUsage,
@@ -56,6 +59,10 @@ class ExtractionResult:
         self.brief: dict | None = None
         self.review_queue: list[dict] = []
         self.rejected: list[RejectedField] = []
+        # i2: clusters answered from an approved entry instead of a paid
+        # call. Reported rather than inferred - a branch that skips work
+        # silently looks identical to one that found nothing.
+        self.reused: int = 0
         self.usages: list[CallUsage] = []
         # Dangle counts from the linking stage. Reported rather than swallowed:
         # a link stage that resolves nothing looks identical to one that works.
@@ -91,19 +98,67 @@ def build_clusters(doc: ParsedDoc, filtered: FilterResult) -> ClusterResult:
 def estimate_cost(
     doc: ParsedDoc, clustered: ClusterResult, filtered: FilterResult, limit: int | None = None
 ) -> dict:
-    """Dry run: exact prompts are built, nothing is sent."""
+    """Dry run: exact prompts are built, nothing is sent.
+
+    **`estimated_prompt_tokens` covers all four branches as of 2026-09-11.** It
+    used to sum the glossary branch alone while carrying a name that reads as a
+    total, so every consumer — including the spend ceiling that gates on it —
+    was reading a partial number as a whole one. The glossary figure stays exact
+    (its prompts are really built); the other three are approximated from the
+    cached prefix plus the chunk text they are handed, which undercounts the
+    evidence block's own framing by a few percent. Both are labelled, because a
+    ceiling that does not know which half of its input is a guess is not a
+    ceiling.
+
+    `clusters` is reported alongside because **cluster count, not page count, is
+    what drives spend** — one call per cluster — and it is known here for free.
+    """
     clusters = clustered.clusters[:limit] if limit else clustered.clusters
-    prompt_tokens = 0
+    glossary_tokens = 0
     for cluster in clusters:
         system, user = build_glossary_prompt(cluster, doc.chunks)
-        prompt_tokens += est_tokens(system) + est_tokens(user)
+        glossary_tokens += est_tokens(system) + est_tokens(user)
+
+    by_id = {c.chunk_id: c for c in doc.chunks}
+    rule_chunks = [
+        by_id[c.chunk_id]
+        for c in (filtered.rule_candidates[:limit] if limit else filtered.rule_candidates)
+        if c.chunk_id in by_id
+    ]
+    formula_chunks = [c for c in doc.chunks if c.has_formula]
+    formula_chunks = formula_chunks[:limit] if limit else formula_chunks
+
+    rule_tokens = sum(prefix_tokens("rule") + est_tokens(c.text or "") for c in rule_chunks)
+    formula_tokens = sum(
+        prefix_tokens("formula") + est_tokens(c.text or "") for c in formula_chunks
+    )
+    # The summary branch is handed the WHOLE document — few calls, but roughly a
+    # quarter of all input tokens, and it cannot be filtered (X7).
+    summary_tokens = prefix_tokens("summary") + sum(
+        est_tokens(c.text or "") for c in doc.chunks
+    )
+
+    by_branch = {
+        "glossary": glossary_tokens,
+        "rule": rule_tokens,
+        "formula": formula_tokens,
+        "summary": summary_tokens,
+    }
     return {
+        "clusters": len(clusters),
         "glossary_calls": len(clusters),
-        "rule_calls": len(filtered.rule_candidates),
-        "formula_calls": sum(1 for c in doc.chunks if c.has_formula),
+        "rule_calls": len(rule_chunks),
+        "formula_calls": len(formula_chunks),
         "summary_calls": 1,
-        "estimated_prompt_tokens": prompt_tokens,
-        "note": "estimate only — real counts come from the API usage object",
+        "total_calls": len(clusters) + len(rule_chunks) + len(formula_chunks) + 1,
+        "estimated_prompt_tokens": sum(by_branch.values()),
+        "estimated_prompt_tokens_by_branch": by_branch,
+        "glossary_tokens_are_exact": True,
+        "note": (
+            "estimate only — real counts come from the API usage object. Glossary "
+            "prompts are built for real; rule/formula/summary are approximated from "
+            "prefix + chunk text and run slightly low."
+        ),
     }
 
 
@@ -121,7 +176,9 @@ def extract_all(
     started = time.time()
 
     if "glossary" in branches:
-        _run_glossary(doc, clustered, extractor, out, limit)
+        _run_glossary(
+            doc, clustered, extractor, out, limit, _approved_index(active_glossary)
+        )
     if "rule" in branches:
         _run_rules(doc, filtered, extractor, out, limit)
     if "formula" in branches:
@@ -162,9 +219,59 @@ def extract_all(
     return out
 
 
-def _run_glossary(doc, clustered, extractor, out, limit) -> None:
+def _approved_index(active_glossary: list[dict] | None) -> dict[str, dict]:
+    """Canonical key -> the approved payload, for recognising a known term (i2).
+
+    Keyed with the SAME normaliser that buckets mentions inside a document, so
+    one rule decides sameness at both scales rather than two that can drift.
+    Both `term` and `full_name` are indexed because a later document may write
+    the expansion where the first wrote the abbreviation.
+    """
+    index: dict[str, dict] = {}
+    for payload in active_glossary or []:
+        for surface in (payload.get("term"), payload.get("full_name")):
+            key = normalize(surface or "")
+            if key:
+                index.setdefault(key, payload)
+    return index
+
+
+def _reuse_approved(approved: dict, doc: ParsedDoc, cluster) -> dict:
+    """Carry an approved definition onto this document without paying for it.
+
+    The entry is this document's — `term_id` is re-minted from *its* `doc_id`,
+    because ids stay document-scoped (decision (a), 2026-09-14) and sharing one
+    id across documents would move every existing `entity_id` and orphan the
+    approvals this whole scheme protects.
+
+    **Provenance is deliberately left pointing at the document the definition
+    actually came from.** It would be easy to rewrite it to this document and
+    wrong: nothing was extracted here, so there is no span in this document to
+    cite, and a fabricated one is exactly the unfalsifiable claim the span check
+    exists to refuse. Which documents re-mention the term is what `doc_ids`
+    (m3) will record once it lands.
+    """
+    entry = dict(approved)
+    entry["term_id"] = term_id(doc.doc_id, cluster.canonical)
+    entry["mention_count"] = getattr(cluster, "mention_count", 0)
+    entry["diff_status"] = "duplicate"
+    return entry
+
+
+def _run_glossary(doc, clustered, extractor, out, limit, approved=None) -> None:
     clusters = clustered.clusters[:limit] if limit else clustered.clusters
+    approved = approved or {}
     for cluster in clusters:
+        # i2: a term this company has already approved is not re-extracted. One
+        # LLM call per cluster is the whole cost of this branch, so every term
+        # recognised here is a call not made — and it is also what stops the
+        # same concept arriving as `term 1.1` and `term 1.2`.
+        reused = approved.get(normalize(cluster.canonical))
+        if reused is not None:
+            out.glossary.append(_reuse_approved(reused, doc, cluster))
+            out.reused += 1
+            continue
+
         entry = None
         max_round = rounds_available(cluster, EVIDENCE_K)
 
@@ -177,7 +284,11 @@ def _run_glossary(doc, clustered, extractor, out, limit) -> None:
                 continue
 
             source = evidence_text(
-                top_k(cluster, EVIDENCE_K, round_index), doc.chunks
+                # `doc.chunks` passed so a linked partner joins the window here
+                # exactly as it does in the prompt (`_evidence_for`). If these two
+                # ever diverge, a quote from a partner chunk is nulled as
+                # unlocatable and the call is wasted.
+                top_k(cluster, EVIDENCE_K, round_index, doc.chunks), doc.chunks
             )
             entry, rejections = validate_entry(entry, "glossary", source, cluster.canonical)
             out.rejected.extend(rejections)

src/knowledge_extraction/settings.py

@@ -33,28 +33,215 @@ SPAN_SCORE_THRESHOLD = 0.25
 # The span NER model truncates past ~384 of its own tokens and *warns rather
 # than failing*, so a long chunk silently loses its tail. Indonesian technical
 # prose subword-tokenises at roughly 2.5x, so 220-word windows still tripped the
-# cap; 130 does not. Chunks are fed as overlapping windows with offsets remapped.
+# cap; 130 did not. Chunks are fed as overlapping windows with offsets remapped.
+#
+# ⚠️ STAYS AT 130. A 2026-09-10 sweep appeared to show 256 was free — recall flat
+# across 130/180/256/320 (E1 0.878 at every size) while clusters fell 6%/23%/34%
+# across the three test documents. It was raised, then REVERTED the same day after
+# counting the truncation directly with GLiNER's own tokeniser:
+#
+#   windows exceeding the 384-token cap, by WINDOW_WORDS
+#                130    180    200    256    320
+#   BUMA (ID)      1      1      1      1      1     <- pre-existing, see below
+#   Komatsu (EN)   0      0      0      1     10
+#   Open Pit (EN)  0      4     11     46     49     <- 46 of 99 windows!
+#
+# At 256 nearly HALF of the Open Pit textbook's windows lose their tail. The
+# "flat recall" that justified the raise was measured on BUMA, the one document
+# whose truncation count does not change with window size — so the metric was
+# structurally blind to the cost, in exactly the way E1 was blind to the
+# clustering over-merge. The cluster reduction was not better grouping; it was
+# content disappearing.
+#
+# This vindicates the original reasoning rather than overturning it: the ~2.5x
+# Indonesian subword ratio was the motivating case, but dense ENGLISH technical
+# prose also trips the cap by 180. 130 is the only size measured safe on both
+# English documents.
+#
+# ⚠️ Separate, PRE-EXISTING finding from the same measurement: BUMA has one window
+# that exceeds the cap at EVERY size including 130 (409 tokens from 130 words — a
+# dense Indonesian chunk). One chunk in that document has always been losing its
+# tail. Not caused by this work and not fixed by it; a smaller window or a
+# per-chunk token budget would be the fix, and neither is measured yet.
 WINDOW_WORDS = 130
 WINDOW_OVERLAP = 30
 SPAN_TOKEN_CAP = 12
 
+# ── The per-window SUBWORD budget (fixes the pre-existing over-cap window) ──
+#
+# `WINDOW_WORDS` counts WORDS; the model truncates on SUBWORDS, and the ratio
+# between them is a property of the text, not of the setting. Measured with
+# GLiNER's own tokeniser across all three documents, 2026-09-11:
+#
+#                windows   over the 384-token cap   chars/subword (min / mean)
+#   BUMA              31                        1            1.57 / 3.43
+#   Komatsu          144                        0            2.53 / 3.58
+#   Open Pit         193                        0            2.79 / 4.10
+#
+# One window in 368 is over, and it is the one the 2026-09-10 sweep found and
+# left unfixed: `STD_2026_006_MNO::0024`, 130 words and 738 characters, which
+# tokenise to 409 subwords. That chunk has ALWAYS been losing its tail, at every
+# window size including 130, and the model warns rather than failing so nothing
+# ever surfaced it.
+#
+# The fix measures instead of guessing: each window is checked against the
+# model's own tokeniser and split only if it does not fit. A character budget
+# would have to assume the worst ratio seen anywhere (1.57) and would therefore
+# split at ~600 characters — chopping up the other 367 windows, which fit
+# comfortably, to fix one.
+SPAN_SUBWORD_CAP = 384
+
+# Split at 90% of the cap. The tokeniser count excludes the special tokens and
+# the label prompt GLiNER prepends, so measuring exactly to 384 still truncates.
+SPAN_SUBWORD_MARGIN = 0.9
+
+# Used only when the tokeniser cannot be located on the model (a GLiNER version
+# change). 1.57 is the tightest ratio measured anywhere in the corpus, so this
+# over-splits rather than silently truncating — the safe direction.
+SPAN_FALLBACK_CHARS_PER_TOKEN = 1.57
+
+# A split halves and recurses. 4 takes a 130-word window down to ~8 words, which
+# is past any plausible cap; the bound exists so a pathological input cannot
+# recurse indefinitely.
+SPAN_MAX_SPLIT_DEPTH = 4
+
+# Flat (non-overlapping) span decoding. MEASURED AND KEPT 2026-09-10: nested
+# decoding (`flat_ner=False`) was tested on all three documents on the hypothesis
+# that flat mode was discarding the parent/child spans compound terms need. It
+# was not. Nested added 7-12% more mentions, delivered EXACTLY ZERO recall gain
+# (E1 0.878 either way, E1c identical too), and increased token-subset
+# over-merges by 40-80% (BUMA 15->24, Komatsu 18->32, Open Pit 97->134).
+# `span_filter` leaves this at the library default rather than passing it; this
+# constant documents the decision. Do not re-open without a new measurement.
+SPAN_FLAT_NER = True
+
 # ── Clustering ──────────────────────────────────────────────────────────
+# ⚠️ BOTH CONSTANTS BELOW ARE RETIRED as of 2026-09-10. The fuzzy matching pass
+# they configured is commented out at its call site in `cluster/cluster.py`; they
+# are still imported there because the `_fuzzy_match` function itself is parked
+# (comment-out-don't-delete), and it reads them. Changing them now affects
+# NOTHING — the pass does not run.
+#
+# Why it was retired, measured across three documents: `token_set_ratio` returns
+# exactly 100 when one token set is a subset of the other, so `FUZZY_THRESHOLD`
+# was inert and head nouns always absorbed their compounds. 131 of 140 merges
+# were subset matches (all wrong); of the 9 remaining, 6 were plain plurals —
+# now handled deterministically by `normalize(lemma=True)` — and 3 were distinct
+# metrics wrongly merged. Retiring the pass took subset-merges to 0 on all three
+# documents and lifted post-cluster gold recall 0.8293 -> 0.8537.
+#
+# `FUZZY_MIN_LEN` did its own job correctly to the end: it stopped "PA" and "UA"
+# colliding. It simply never gated multi-word English compounds, which is the
+# case that did the damage.
 FUZZY_THRESHOLD = 92
-
-# Below this length only exact matching is allowed: "PA" and "UA" score highly
-# against each other on token_set_ratio. Over-merging is far worse than
-# under-merging — an under-merge costs one extra call and one extra review row,
-# a wrong merge destroys a distinct term and the expert never sees it.
 FUZZY_MIN_LEN = 5
 
 # ── Evidence ranking ────────────────────────────────────────────────────
-EVIDENCE_K = 3
+# 3 -> 5 on 2026-09-10 (Rifqi's call). How many top-ranked chunks each extraction
+# call sees. Three things this changes, recorded because none is obvious:
+#
+# 1. **Cost.** Evidence is the variable part of the prompt, so this is the single
+#    biggest lever on spend per call after cluster count. Measured effect on the
+#    three test documents: see CHANGELOG_v2.md. It also lowers the CACHE HIT RATE,
+#    because the fixed cached prefix becomes a smaller share of each prompt.
+#
+# 2. **Escalation gets rarer but deeper.** `rounds_available` needs more than K
+#    evidence chunks to fire at all, so at 5 a cluster needs 6+ (not 4+) to
+#    escalate; when it does, a round covers ranks 6-10 instead of 4-6. On the three
+#    test documents most clusters have 1-2 evidence chunks, so escalation fires
+#    less often than before.
+#
+# 3. **It is NOT supported by the ranker measurement, and that is fine.** The
+#    2026-09-10 ceiling check found 16 of 16 gold definitions already at rank <=3
+#    (14 at rank 1), so on the BUMA standard ranks 4-5 add cost and no recall. The
+#    case for 5 is the two ENGLISH documents, where definitions are dispersed
+#    across pages and there is no gold set to measure with — a judgement that more
+#    context beats a cheaper call on that document class. Revisit with a number if
+#    a second gold set ever exists.
+EVIDENCE_K = 5
 CUE_PROXIMITY_CHARS = 100
 
+# How many LINKED partner chunks may be pulled into an evidence window on top of
+# the ranked K (0.4.0 two-way table pointers; added 2026-09-11). 1, not more.
+#
+# A table keeps its own chunk, and the prose that introduces it is a separate
+# chunk; `referenced_by` records the pair in both directions. Measured on the
+# three v2 artifacts, ranking only (free — no GLiNER, no spend):
+#
+#   clusters whose top-5 touches a linked pair, by side
+#                 clusters   touches a pair   BOTH sides   only ONE side
+#   BUMA              83           31              0             31
+#   Komatsu           93           55              0             55
+#   Open Pit         427           70              0             70
+#
+# **Zero clusters see both sides.** The ranker never co-selects a table and the
+# prose that names it, so the partner is genuinely unseen content rather than a
+# duplicate — and that prose is frequently the table's only title ("Troubleshooting
+# of engine (S-mode)" above a bare `S-1 | 6` grid).
+#
+# Cost AT THE CAP OF 1, measured on the same artifacts: evidence the model reads
+# +14.5% / +24.5% / +9.9%, which is +3.4k / +11.2k / +59.5k tokens, i.e.
+# **+1.6% / +4.1% / +4.2%** of each document's total v2 prompt spend (the fixed
+# cached prefix dominates). Open Pit moves from 71% to ~74% of the 2M ingest
+# ceiling. Clusters that actually gain a partner: 31/83, 55/93, 70/427.
+#
+# **Expansion happens in `top_k`, NOT in `rank_evidence`.** Two invariants depend
+# on that placement:
+#
+# 1. The full ranked list stays untouched, so nothing that reads
+#    `evidence_chunk_ids` — `rounds_available` included — sees a different number.
+#    Appending partners to the RANKED list would inflate the evidence count and
+#    change escalation's arithmetic as a side effect. That is currently moot
+#    (`MAX_ESCALATION_ROUNDS = 0` since 2026-09-11; escalation produced 0
+#    definitions in 48 calls) but the separation is what keeps this change
+#    orthogonal to that one, and to whatever replaces it.
+# 2. `evidence_text` (the span check) and `evidence_block` (the prompt) both take
+#    their ids from `top_k`, so expanding there keeps "what the model reads is
+#    exactly what the span check searches" true for free. Expanding in the prompt
+#    renderer instead would break it, and every quote from a partner chunk would
+#    be nulled as unlocatable. Verified: 0 parity failures across all 603 clusters.
+#
+# Set to 0 to disable. Two honest scope notes:
+#
+# - The document this helps most (Komatsu, 59% of clusters gain a partner) is the
+#   one that returns 0 definitions for GENRE reasons. This is groundwork for the
+#   table branch more than a glossary win.
+# - The counts above were measured on the v2 artifacts and the RECORDED v2 filter
+#   payloads, so they predate the subword window-splitting change. Mention and
+#   cluster counts move with that change; the pairing structure does not.
+EVIDENCE_LINK_PARTNERS = 1
+
 EVIDENCE_WEIGHTS: dict[str, float] = {
     "definitional_cue_near": 5.0,
     "term_in_heading": 4.0,
     "in_legend_block": 3.5,
+    # A figure whose CAPTION names this term sits in this chunk. Built and
+    # measured 2026-09-11, then set to **0.0 — OFF, because it measured as a
+    # no-op.** The code stays because the A/B harness and the reasoning are worth
+    # more than the three lines, and a diagram-heavy document could revive it.
+    #
+    # It was built to explain `definitions_citing_a_caption = 0` on all three
+    # documents in the first paid v2 run. Two hypotheses, both now falsified:
+    #
+    # 1. "Captions don't reach the term filter." FALSE — a caption is part of
+    #    `Chunk.text`, so GLiNER already scans it and already produces mentions
+    #    from it (2 of 2 captions on BUMA, 3 of 8 on Open Pit).
+    # 2. "Figure-bearing chunks don't get selected." FALSE — a captioned chunk is
+    #    already inside the top-5 for 22 of 83 BUMA clusters and 64 of 427 Open
+    #    Pit clusters. Turning this weight up to 2.5 changed the top-5 of **1
+    #    cluster out of 603** across all three documents, and moved zero clusters
+    #    from "no figure" to "has a figure".
+    #
+    # The actual reason is simpler, and visible by reading them: **not one caption
+    # in the corpus is a definition.** They are figure titles — "Figure 1.3.
+    # Relative ability to influence costs (Lee, 1984)", "Gambar 2.1 Analisis
+    # Gain/Loss Berdasarkan Setiap Parameter Produksi". A model declining to quote
+    # one as a definition is CORRECT, not a retrieval failure.
+    #
+    # If revived, weight it BELOW `term_in_heading`: a heading naming a term means
+    # "this section is about X"; a caption naming it means only "a figure about X
+    # is here".
+    "asset_caption_names_term": 0.0,
     "formula_present": 2.0,
     "bold_or_italic": 1.5,
     "first_occurrence": 1.0,
@@ -75,7 +262,37 @@ TEMPERATURE = 0.0
 CACHE_MIN_TOKENS = 1024
 
 # ── Validation ──────────────────────────────────────────────────────────
-MAX_ESCALATION_ROUNDS = 2
+# ⚠️ RETIRED 2026-09-11 by setting it to 0 — the loop and its call site are
+# untouched (comment-out-don't-delete), and restoring it is this one character.
+#
+# Finding 6 asked for a number either way. Here it is, counted off the six
+# persisted paid runs (v1 K=3 and v2 K=5, three documents):
+#
+#                    glossary calls   clusters   extra calls   definitions FROM escalation
+#   BUMA      v1                 69         66             3                             0
+#   BUMA      v2                 83         83             0                             0
+#   Komatsu   v1                 94         75            19                             0
+#   Komatsu   v2                102         93             9                             0
+#   Open Pit  v1                361        349            12                             0
+#   Open Pit  v2                430        425             5                             0
+#
+# 48 escalation calls across three documents and two configurations, and NOT ONE
+# of them produced a definition the first round had missed. `extraction_status`
+# is set to "escalated" only when a later round succeeds where round 0 failed,
+# and it appears zero times in every one of those six files.
+#
+# Why it fails is worth keeping, because it argues against reviving it naively:
+# escalation hands the model the NEXT k evidence chunks, ranked below the ones it
+# already declined. The 2026-09-10 ranker ceiling check found 16 of 18 gold
+# definitions at rank <=3 — so if the definition is anywhere, it was already in
+# round 0, and ranks 6-10 are where it is least likely to be. Raising EVIDENCE_K
+# to 5 made this worse in both directions at once: escalation needs 6+ evidence
+# chunks to fire at all, and the rounds it does reach are even further down.
+#
+# What would justify reviving it: a retry that changes the QUESTION (a different
+# prompt, a larger tier) rather than one that changes the evidence to lower-ranked
+# evidence. That is a different feature, not this constant.
+MAX_ESCALATION_ROUNDS = 0
 CONFLICT_OVERLAP_THRESHOLD = 0.4
 DUPLICATE_OVERLAP_THRESHOLD = 0.8
 

src/knowledge_ingest/knowledge_store.py

@@ -34,7 +34,7 @@ import uuid
 from datetime import UTC, datetime
 from typing import Any
 
-from sqlalchemy import select
+from sqlalchemy import select, update
 
 from src.db.postgres.connection import AsyncSessionLocal
 from src.db.postgres.models import (
@@ -43,6 +43,7 @@ from src.db.postgres.models import (
     KnowledgeExtractionRunRow,
     KnowledgeReviewRow,
 )
+from src.knowledge_extraction import keys, mutation
 from src.middlewares.logging import get_logger
 
 logger = get_logger("knowledge_store")
@@ -62,6 +63,84 @@ _ENTRY_KINDS: dict[str, tuple[str, str]] = {
 }
 
 
+# m4: the body a reviewer or a consumer actually wants, per kind. First
+# non-empty wins, so a kind whose primary field abstained still promotes
+# something rather than silently leaving the column null.
+_CONTENT_FIELDS: dict[str, tuple[str, ...]] = {
+    "glossary": ("definition",),
+    "rule": ("statement", "consequence"),
+    "formula": ("formula_latex", "name"),
+    "document": ("purpose_verbatim", "title"),
+    "domain": ("purpose_verbatim", "domain_name"),
+}
+
+# m5: the provenance keys worth lifting out. `page` is 0-based as the parser
+# reports it and `page_no` is the 1-based number a human reads; both travel
+# because they are derived from one value and cannot disagree.
+_PROVENANCE_KEYS = ("doc_id", "page", "page_no", "section_no", "chunk_id")
+
+
+def _content(kind: str, payload: dict[str, Any]) -> str | None:
+    """m4: the entry's body, promoted out of `payload`."""
+    for field in _CONTENT_FIELDS.get(kind, ()):
+        value = payload.get(field)
+        if isinstance(value, str) and value.strip():
+            return value.strip()
+    return None
+
+
+def _provenance(payload: dict[str, Any]) -> list[dict]:
+    """m5: provenance as a LIST, so a fused entry can carry several sources.
+
+    The extraction contract emits a single object per entry today; it is
+    wrapped rather than reshaped, because the list is what an entry drawn from
+    more than one document needs and normalising the shape here means every
+    consumer reads one thing.
+    """
+    raw = payload.get("provenance")
+    if isinstance(raw, dict):
+        raw = [raw]
+    if not isinstance(raw, list):
+        return []
+    out: list[dict] = []
+    for item in raw:
+        if not isinstance(item, dict):
+            continue
+        kept = {k: item[k] for k in _PROVENANCE_KEYS if item.get(k) is not None}
+        if kept:
+            out.append(kept)
+    return out
+
+
+def _doc_ids(doc_id: str, provenance: list[dict]) -> list[str]:
+    """m3: every document behind this entry, this one first.
+
+    Derived rather than declared, so it cannot drift from the provenance it
+    summarises. For an ordinary entry it is `[doc_id]`; it grows only when a
+    definition was carried forward from an earlier document that a later one
+    re-mentions (i2), which is the fusion case m3 was asked for.
+    """
+    ids = [doc_id]
+    for item in provenance:
+        source = item.get("doc_id")
+        if isinstance(source, str) and source and source not in ids:
+            ids.append(source)
+    return ids
+
+
+def _has_live_source(row: Any, withdrawn: set[str]) -> bool:
+    """Does this entry still rest on a document that has not been withdrawn? (i7)
+
+    Reads `doc_ids` when it is populated and falls back to `doc_id` when it is
+    not, so rows written before m3 landed keep answering correctly instead of
+    all looking orphaned at once.
+    """
+    sources = [d for d in (getattr(row, "doc_ids", None) or []) if isinstance(d, str)]
+    if not sources:
+        sources = [row.doc_id]
+    return any(source not in withdrawn for source in sources)
+
+
 def _label(payload: dict[str, Any], key: str) -> str | None:
     """The listing label, trimmed. A rule's `statement` can be a paragraph, and
     a column meant for scanning a queue should not carry one."""
@@ -95,6 +174,43 @@ class KnowledgeStore:
             )
             return result.scalars().first()
 
+    async def find_document_by_hash(
+        self, doc_id: str, content_hash: str
+    ) -> KnowledgeDocumentRow | None:
+        """The same content-addressing lookup, across every version (i5).
+
+        Version is assigned by us now, not read off the artifact, so the reuse
+        check cannot key on it: computing the next version and *then* looking
+        it up would miss the row that already holds these exact bytes and
+        re-parse a document we already have — turning the idempotency guarantee
+        into a paid no-op.
+        """
+        async with AsyncSessionLocal() as session:
+            result = await session.execute(
+                select(KnowledgeDocumentRow)
+                .where(
+                    KnowledgeDocumentRow.doc_id == doc_id,
+                    KnowledgeDocumentRow.content_hash == content_hash,
+                )
+                .order_by(KnowledgeDocumentRow.version.desc())
+            )
+            return result.scalars().first()
+
+    async def latest_document(self, doc_id: str) -> KnowledgeDocumentRow | None:
+        """The current version of a document — the highest `version` (i5).
+
+        What "current" means for a document that has been amended: the newest
+        version supersedes, and the older rows stay for the audit trail rather
+        than being rewritten.
+        """
+        async with AsyncSessionLocal() as session:
+            result = await session.execute(
+                select(KnowledgeDocumentRow)
+                .where(KnowledgeDocumentRow.doc_id == doc_id)
+                .order_by(KnowledgeDocumentRow.version.desc())
+            )
+            return result.scalars().first()
+
     async def save_document(
         self,
         artifact: Any,
@@ -120,9 +236,8 @@ class KnowledgeStore:
         )
         doc_id = payload["doc_id"]
         content_hash = payload["content_hash"]
-        version = int(payload.get("version", 1))
 
-        existing = await self.find_document(doc_id, content_hash, version)
+        existing = await self.find_document_by_hash(doc_id, content_hash)
         if existing is not None:
             logger.info(
                 "knowledge_document_reused",
@@ -130,6 +245,20 @@ class KnowledgeStore:
             )
             return existing
 
+        # i5 (ruling 2026-09-14): identity is `doc_id`, so the same document
+        # arriving with different bytes is a NEW VERSION of it, never a new
+        # document. That is what keeps an amended file's approvals attached —
+        # `entity_id` derives from `doc_id`, which does not move. The artifact's
+        # own `version` is ignored: the parsing half stamps 1 on every parse and
+        # has no way to know what this tenant already holds.
+        previous = await self.latest_document(doc_id)
+        version = (previous.version + 1) if previous else 1
+        if previous is not None:
+            logger.info(
+                "knowledge_document_versioned",
+                extra={"doc_id": doc_id, "version": version, "supersedes": previous.id},
+            )
+
         row = KnowledgeDocumentRow(
             id=str(uuid.uuid4()),
             doc_id=doc_id,
@@ -262,6 +391,10 @@ class KnowledgeStore:
         # so, hence the warning and the count on the run row.
         seen: set[tuple[str, str]] = set()
         collisions: list[str] = []
+        # m1/m6: `label` carries the readable `TYPE_SUBJECT` key. Rules key off
+        # the term they govern, and a rule payload holds `term_ids` rather than
+        # words, so the surfaces are collected from this run's glossary first.
+        surfaces = keys.term_surfaces(entries_by_kind.get("glossary"))
         for kind, payloads in entries_by_kind.items():
             if kind not in _ENTRY_KINDS:
                 raise ValueError(f"unknown entry kind {kind!r}")
@@ -277,6 +410,7 @@ class KnowledgeStore:
                     collisions.append(f"{kind}:{entity_id}")
                     continue
                 seen.add((kind, entity_id))
+                prov = _provenance(payload)
                 rows.append(
                     KnowledgeEntryRow(
                         id=str(uuid.uuid4()),
@@ -287,7 +421,19 @@ class KnowledgeStore:
                         user_id=user_id,
                         kind=kind,
                         entity_id=entity_id,
-                        label=_label(payload, label_field),
+                        # m6 (2026-09-08): the key, not the term. The readable
+                        # term is still in `payload`, so nothing is lost to a
+                        # reader — but the column an agent searches now holds
+                        # something it can type from memory. `_label` remains
+                        # the fallback for a kind with no registered prefix.
+                        label=keys.concept_key(
+                            kind,
+                            payload,
+                            entity_id=entity_id,
+                            doc_id=doc_id,
+                            surfaces=surfaces,
+                        )
+                        or _label(payload, label_field),
                         mention_count=int(payload.get("mention_count") or 0),
                         extraction_status=payload.get("extraction_status"),
                         diff_status=payload.get("diff_status"),
@@ -295,6 +441,12 @@ class KnowledgeStore:
                             payload.get("definition_conflict", False)
                         ),
                         payload=payload,
+                        # m3/m4/m5: promoted from the SAME object in the SAME
+                        # insert, so the column and the jsonb can never
+                        # disagree about what this entry says.
+                        content=_content(kind, payload),
+                        provenance=prov,
+                        doc_ids=_doc_ids(doc_id, prov),
                     )
                 )
 
@@ -349,6 +501,188 @@ class KnowledgeStore:
             result = await session.execute(stmt)
             return list(result.scalars().all())
 
+    async def get_by_key(
+        self, scope_id: str, key: str
+    ) -> KnowledgeEntryRow | None:
+        """One entry by its readable key (m8) — `GLOSSARY_PA`, not a search.
+
+        This is the lookup the 2026-09-08 discussion asked for: an agent that
+        meets a term it does not know writes the key and hits the row. **No
+        model, no embedding, no scan of payloads** — the whole reason the key
+        exists is that finding a glossary term should not cost a round trip to
+        a language model.
+
+        Keys are uppercased on the way in so a caller that types one from
+        memory in lower case still lands, matching how they are minted.
+
+        Note for whoever adds the index: this filters on `label`, which has no
+        index today, so Postgres scans within the tenant. Correct, and fine at
+        the current corpus size — but it is an index away from being the O(1)
+        the design intends, and that index rides along with the m3/m4/m5 DDL.
+        """
+        stmt = (
+            select(KnowledgeEntryRow)
+            .where(KnowledgeEntryRow.scope_id == scope_id)
+            .where(KnowledgeEntryRow.label == (key or "").strip().upper())
+            .order_by(KnowledgeEntryRow.created_at.desc())
+            .limit(1)
+        )
+        async with AsyncSessionLocal() as session:
+            result = await session.execute(stmt)
+            return result.scalars().first()
+
+    async def glossary_index(
+        self, scope_id: str, *, limit: int = 1000
+    ) -> dict[str, dict]:
+        """The company's glossary as a dictionary keyed by concept key (m8).
+
+        `{"GLOSSARY_PA": {...payload...}, "GLOSSARY_QTY": {...}}` — the shape
+        Mas asked for when he said the glossary should be key-value rather than
+        a list. Searching a list is O(N) and searching this is O(1); it is also
+        the shape the JSON export writes, so the file on disk and the lookup in
+        memory agree with each other rather than drifting.
+
+        A row whose key could not be built is skipped rather than given a
+        made-up one: a glossary you cannot trust the keys of is worse than a
+        smaller glossary.
+        """
+        rows = await self.list_entries(
+            scope_id=scope_id, kind="glossary", limit=limit
+        )
+        index: dict[str, dict] = {}
+        for row in rows:
+            if row.label and isinstance(row.payload, dict):
+                index.setdefault(row.label, row.payload)
+        return index
+
+    async def review_diff(
+        self, scope_id: str, entity_id: str, kind: str = "glossary"
+    ) -> dict | None:
+        """Before / after for one entry, for the review screen (r4).
+
+        Returns None when there is nothing to compare — no decision on record,
+        or the entry is gone. `changed` is empty when the approval still
+        stands, which is the common case and the one worth making cheap: the
+        screen can say "nothing to re-read" instead of showing two identical
+        panes.
+
+        `before` is what the expert had in front of them; for an `edited`
+        decision that is their own corrected text, not the model's claim.
+        """
+        reviews = await self.latest_reviews(scope_id, [entity_id])
+        review = reviews.get(entity_id)
+        if review is None:
+            return None
+
+        rows = await self.list_entries(scope_id=scope_id, kind=kind, limit=1000)
+        current = next(
+            (r for r in sorted(rows, key=lambda r: r.created_at or 0, reverse=True)
+             if r.entity_id == entity_id),
+            None,
+        )
+        if current is None:
+            return None
+
+        if review.decision == "edited" and review.edited_payload:
+            before = review.edited_payload
+        else:
+            async with AsyncSessionLocal() as session:
+                result = await session.execute(
+                    select(KnowledgeEntryRow).where(
+                        KnowledgeEntryRow.id == review.entry_id
+                    )
+                )
+                reviewed_row = result.scalars().first()
+            before = reviewed_row.payload if reviewed_row else None
+
+        changed = mutation.changed_fields(kind, before, current.payload)
+        return {
+            "entity_id": entity_id,
+            "kind": kind,
+            "decision": review.decision,
+            "decided_at": review.created_at,
+            "changed": changed,
+            "status": "waiting_approval" if changed else review.decision,
+            "before": mutation.ruled_content(kind, before),
+            "after": mutation.ruled_content(kind, current.payload),
+        }
+
+    async def withdraw_document(self, scope_id: str, doc_id: str) -> int:
+        """Mark every version of a document withdrawn (i7). Returns how many.
+
+        **Nothing is deleted**, here or downstream. The entries the document
+        produced stay exactly where they are, and so do the reviews attached to
+        them — an approval is an act someone took, and destroying the record of
+        what it referred to would destroy the audit trail that makes the
+        approval mean anything.
+
+        What withdrawal *does* change is what the knowledge base will serve: a
+        withdrawn document stops feeding the approved baseline, so the pipeline
+        cannot keep answering from a standard the client has taken back. That
+        was the concern behind the whole discussion — never serve knowledge the
+        source no longer supports.
+        """
+        async with AsyncSessionLocal() as session:
+            result = await session.execute(
+                update(KnowledgeDocumentRow)
+                .where(
+                    KnowledgeDocumentRow.scope_id == scope_id,
+                    KnowledgeDocumentRow.doc_id == doc_id,
+                    KnowledgeDocumentRow.withdrawn_at.is_(None),
+                )
+                .values(withdrawn_at=datetime.now(UTC))
+            )
+            await session.commit()
+        count = result.rowcount or 0
+        logger.info(
+            "knowledge_document_withdrawn",
+            extra={"scope_id": scope_id, "doc_id": doc_id, "versions": count},
+        )
+        return count
+
+    async def withdrawn_doc_ids(self, scope_id: str) -> set[str]:
+        """Documents this company has taken back."""
+        async with AsyncSessionLocal() as session:
+            result = await session.execute(
+                select(KnowledgeDocumentRow.doc_id).where(
+                    KnowledgeDocumentRow.scope_id == scope_id,
+                    KnowledgeDocumentRow.withdrawn_at.is_not(None),
+                )
+            )
+            return {r for (r,) in result.all() if r}
+
+    async def orphaned_entities(self, scope_id: str, kind: str = "glossary") -> set[str]:
+        """Entries whose every source document has been withdrawn (i7).
+
+        Surfaced for the expert to rule on rather than deleted: whether the
+        knowledge leaves with the document is their judgement, not the
+        pipeline's. A term defined in a withdrawn standard may well still be
+        the company's term.
+        """
+        withdrawn = await self.withdrawn_doc_ids(scope_id)
+        if not withdrawn:
+            return set()
+        rows = await self.list_entries(scope_id=scope_id, kind=kind, limit=1000)
+        return {r.entity_id for r in rows if not _has_live_source(r, withdrawn)}
+
+    async def impacted_by_document(
+        self, scope_id: str, doc_id: str, limit: int = 1000
+    ) -> list[KnowledgeEntryRow]:
+        """Which entries a document contributed to — the impact map (i4).
+
+        This is what makes a re-ingest proportionate: a document is amended,
+        these are the entries that can possibly have moved, and **only these**
+        go back to the reviewer. Everything else the company has approved is
+        untouched and stays untouched.
+
+        No separate lineage structure is needed for it — the contribution is
+        already recorded on the entry. Once `doc_ids` lands (m3) this also has
+        to match membership in that list, for entries fused from several
+        documents; today every entry has exactly one source, so the single
+        column is the whole answer.
+        """
+        return await self.list_entries(scope_id=scope_id, doc_id=doc_id, limit=limit)
+
     async def latest_run_for(
         self, scope_id: str, doc_id: str
     ) -> KnowledgeExtractionRunRow | None:
@@ -435,6 +769,8 @@ class KnowledgeStore:
         reviewer_id: str,
         edited_payload: dict | None = None,
         note: str | None = None,
+        decided_by: str = "human",
+        authorised_by: str | None = None,
     ) -> KnowledgeReviewRow:
         """Record one expert decision. APPEND-ONLY.
 
@@ -468,6 +804,8 @@ class KnowledgeStore:
                 edited_payload=edited_payload,
                 note=note,
                 reviewer_id=reviewer_id,
+                decided_by=decided_by,
+                authorised_by=authorised_by,
             )
             session.add(row)
             await session.commit()
@@ -478,11 +816,65 @@ class KnowledgeStore:
             extra={
                 "entity_id": entity_id,
                 "decision": decision,
+                "decided_by": decided_by,
                 "content_hash": content_hash[:12],
             },
         )
         return row
 
+    async def auto_approve_run(
+        self, *, run_id: str, scope_id: str, authorised_by: str
+    ) -> int:
+        """Approve every entry a run produced, on a named person's authority (r1).
+
+        This is what `review_mode="llm"` does, and it is worth being plain about
+        what it is: **no model re-examines anything.** The entries were written
+        by a model already; choosing this mode means accepting them without a
+        human looking. There is no second opinion in here.
+
+        What makes it offerable rather than reckless is the record. Every row
+        carries `decided_by="llm"` and `authorised_by=<the person who chose
+        it>`, so an entry approved this way can always answer "who allowed
+        this?" — which was the entire point when it was agreed: if something
+        goes wrong it was their choice, made knowingly, and the system can show
+        whose.
+
+        Written through the ordinary append-only review path, so an expert can
+        later disagree with any of it exactly as they would with each other.
+        """
+        rows = await self.list_entries(scope_id=scope_id, run_id=run_id, limit=1000)
+        approved = 0
+        for row in rows:
+            try:
+                await self.save_review(
+                    entity_id=row.entity_id,
+                    scope_id=scope_id,
+                    decision="approved",
+                    reviewer_id="llm",
+                    decided_by="llm",
+                    authorised_by=authorised_by,
+                    note=f"auto-approved (review_mode=llm) on behalf of {authorised_by}",
+                )
+                approved += 1
+            except LookupError:
+                # The entry vanished between listing and reviewing. Not fatal:
+                # an un-approved entry is a reviewable one, which is the safe
+                # direction to fail in.
+                logger.warning(
+                    "knowledge_auto_approve_skipped",
+                    extra={"entity_id": row.entity_id, "run_id": run_id},
+                )
+        logger.info(
+            "knowledge_run_auto_approved",
+            extra={
+                "run_id": run_id,
+                "scope_id": scope_id,
+                "approved": approved,
+                "authorised_by": authorised_by,
+            },
+        )
+        return approved
+
     async def latest_reviews(
         self, scope_id: str, entity_ids: list[str] | None = None
     ) -> dict[str, KnowledgeReviewRow]:
@@ -513,6 +905,33 @@ class KnowledgeStore:
         is worked. An `edited` decision contributes the expert's corrected
         payload, not the model's original claim.
         """
+        fresh, _stale = await self._approval_state(scope_id, "glossary")
+        return fresh
+
+    async def stale_approvals(
+        self, scope_id: str, kind: str = "glossary"
+    ) -> set[str]:
+        """Entity ids whose approval no longer stands — 'termutasi' (i3).
+
+        A later document changed the content the expert ruled on, so the entry
+        is back to waiting approval and must not feed the baseline in the
+        meantime. What counts as changed is `mutation.RULED_FIELDS`, and it is
+        deliberately narrow — see that module for why a re-mention must not
+        re-open anything.
+        """
+        _fresh, stale = await self._approval_state(scope_id, kind)
+        return stale
+
+    async def _approval_state(
+        self, scope_id: str, kind: str
+    ) -> tuple[list[dict], set[str]]:
+        """Split approved entries into those that still stand and those that do not.
+
+        One pass, because the baseline and the status display must never
+        disagree about whether an approval is current — the same defect as
+        having two notions of readiness, which is why `readiness.py` shares one
+        predicate between the report API and Help.
+        """
         reviews = await self.latest_reviews(scope_id)
         approved = {
             entity_id: review
@@ -520,31 +939,69 @@ class KnowledgeStore:
             if review.decision in ("approved", "edited")
         }
         if not approved:
-            return []
+            return [], set()
 
+        reviewed_ids = [r.entry_id for r in approved.values() if r.entry_id]
         async with AsyncSessionLocal() as session:
             result = await session.execute(
                 select(KnowledgeEntryRow).where(
                     KnowledgeEntryRow.scope_id == scope_id,
-                    KnowledgeEntryRow.kind == "glossary",
+                    KnowledgeEntryRow.kind == kind,
                     KnowledgeEntryRow.entity_id.in_(list(approved)),
                 )
             )
             rows = list(result.scalars().all())
+            # The exact rows the decisions were made against. Without these the
+            # only available comparison is document-level `content_hash`, which
+            # flips every entry in an edited document including the untouched
+            # ones — the outcome the i3 ruling rejected.
+            reviewed_rows: dict[str, KnowledgeEntryRow] = {}
+            if reviewed_ids:
+                reviewed = await session.execute(
+                    select(KnowledgeEntryRow).where(
+                        KnowledgeEntryRow.id.in_(reviewed_ids)
+                    )
+                )
+                reviewed_rows = {r.id: r for r in reviewed.scalars().all()}
+
+        withdrawn = await self.withdrawn_doc_ids(scope_id)
 
         out: list[dict] = []
+        stale: set[str] = set()
         seen: set[str] = set()
         for row in sorted(rows, key=lambda r: r.created_at or 0, reverse=True):
             if row.entity_id in seen:
                 continue
             seen.add(row.entity_id)
+            # i7: a withdrawn source stops feeding the baseline, approval or
+            # not. The approval itself is untouched and the rows stay — this
+            # only stops the pipeline answering from a document the client has
+            # taken back.
+            if withdrawn and not _has_live_source(row, withdrawn):
+                continue
             review = approved[row.entity_id]
-            out.append(
-                review.edited_payload
-                if review.decision == "edited" and review.edited_payload
-                else row.payload
+
+            if review.decision == "edited" and review.edited_payload:
+                # An edit is a stronger statement than an approval: the expert
+                # wrote the text themselves, so a later run producing different
+                # model output does not unsettle it. Their payload IS the entry.
+                out.append(review.edited_payload)
+                continue
+
+            ruled = reviewed_rows.get(review.entry_id)
+            if ruled is not None and mutation.has_mutated(
+                kind, ruled.payload, row.payload
+            ):
+                stale.add(row.entity_id)
+                continue
+            out.append(row.payload)
+
+        if stale:
+            logger.info(
+                "knowledge_approvals_stale",
+                extra={"scope_id": scope_id, "kind": kind, "count": len(stale)},
             )
-        return out
+        return out, stale
 
     async def save_declaration(
         self, *, scope_id: str, user_id: str, payload: dict
@@ -588,7 +1045,11 @@ class KnowledgeStore:
                     entity_id=entity_id,
                 )
                 session.add(row)
-            row.label = _label(payload, "domain_name")
+            # m6: the declaration is keyed like every other entry, so one
+            # lookup path covers all five kinds.
+            row.label = keys.concept_key(
+                "domain", payload, entity_id=entity_id
+            ) or _label(payload, "domain_name")
             row.payload = payload
             row.extraction_status = "ok"
             await session.commit()

src/knowledge_ingest/models.py

@@ -35,6 +35,15 @@ JobStage = Literal["parsing", "filtering", "estimating", "extracting", "persisti
 
 DocumentStatus = Literal["pending", "succeeded", "failed", "skipped"]
 
+# r1 (2026-09-08 discussion). Chosen by the caller at ingest time.
+#   "expert" - entries wait for a human to rule on them.
+#   "llm"    - entries go straight to approved, with NO reviewer step at all.
+# The second is not a second model call: the entries were produced by a model
+# already, so "review by LLM" means accepting them without human review. What
+# makes it safe to offer is that the decision is RECORDED against the person
+# who chose it (r2/r3), not that the pipeline checks itself.
+ReviewMode = Literal["expert", "llm"]
+
 
 def _now() -> datetime:
     return datetime.now(UTC)
@@ -55,6 +64,18 @@ class DocumentResult(BaseModel):
     n_entries: int = 0
     error: str | None = None
 
+    # The pre-flight estimate, recorded whether or not the document was
+    # refused. An operator cannot plan a batch from a number they only ever see
+    # when it is already too late, and "why was this one refused when that one
+    # ran" is not answerable without the shape of a run that succeeded.
+    estimate: dict | None = None
+
+    # Which ceiling refused this document (`clusters` / `prompt_tokens`), or
+    # None if it was not refused. Distinguishes "declined before spending" from
+    # every other failure, which is what lets a wholly-refused batch settle
+    # `blocked` rather than `failed`.
+    blocked_by: str | None = None
+
 
 class SkippedDocument(BaseModel):
     """A document the batch declined to take, and why.
@@ -77,6 +98,7 @@ class IngestJob(BaseModel):
     user_id: str
     status: JobStatus = "queued"
     stage: JobStage | None = None
+    review_mode: ReviewMode = "expert"
     document_ids: list[str] = Field(default_factory=list)
     n_requested: int = 0
     n_succeeded: int = 0
@@ -102,10 +124,17 @@ class IngestJob(BaseModel):
         """
         self.n_succeeded = sum(1 for r in self.results if r.status == "succeeded")
         self.n_failed = sum(1 for r in self.results if r.status == "failed")
+        failures = [r for r in self.results if r.status == "failed"]
         if self.n_succeeded and self.n_failed:
             self.status = "partial"
         elif self.n_succeeded:
             self.status = "succeeded"
+        elif failures and all(r.blocked_by for r in failures):
+            # `blocked` was documented from the start and was unreachable until
+            # 2026-09-11: a spend refusal settled as `failed`, which is the one
+            # outcome an operator can act on (raise the ceiling, or split the
+            # document) hidden inside the one they cannot.
+            self.status = "blocked"
         else:
             self.status = "failed"
         self.stage = None

src/knowledge_ingest/pipeline.py

@@ -36,6 +36,7 @@ from functools import partial
 from pathlib import Path
 
 from src.db.postgres.models import Document
+from src.knowledge_extraction.genre import classify as classify_genre
 from src.knowledge_extraction.service import (
     build_clusters,
     estimate_cost,
@@ -54,6 +55,45 @@ logger = get_logger("knowledge_pipeline")
 # accident without getting in the way of anything real.
 MAX_PROMPT_TOKENS = 2_000_000
 
+# The SECOND ceiling, and the one that tracks what actually scales.
+#
+# Spend is driven by CLUSTER COUNT, not pages: the pipeline makes one call per
+# term cluster, so a dense chapter costs more than a long simple one. Two things
+# the token ceiling alone cannot see:
+#
+#   1. **Per-call cost is not linear in prompt tokens.** Each call carries its
+#      own completion tokens and its own latency, and the fixed cached prefix
+#      becomes a smaller share of each prompt as evidence grows - so the CACHE
+#      HIT RATE degrades as a run gets bigger, and the real bill rises faster
+#      than the token estimate does.
+#   2. **The estimate is partly an approximation** (see `estimate_cost`), while
+#      cluster count is exact and known for free before anything is spent.
+#
+# This became urgent rather than theoretical on 2026-09-10: retiring the fuzzy
+# clustering pass was correct, and it produces MORE, SMALLER clusters - the Open
+# Pit textbook's single 46-page chapter went from 56% to 71% of the token
+# ceiling on that change alone. A ceiling that only counts tokens would have let
+# the next such improvement through unnoticed.
+#
+# Measured free on the three v2 artifacts, 2026-09-11:
+#
+#                clusters   calls   est. prompt tokens   % of the token ceiling
+#   BUMA               83     106              201,751                    10.1%
+#   Komatsu            93      95              227,329                    11.4%
+#   Open Pit          427     494            1,560,548                    78.0%
+#
+# Read those honestly: at this corpus's density (~3,200 prompt tokens per
+# cluster) the TOKEN ceiling binds first, at roughly 630 clusters. So this
+# ceiling is not the tighter of the two for documents that look like these - it
+# is the guard for the other shape, a document with many cheap clusters, where
+# the cost is call COUNT rather than prompt size: 494 calls already runs about a
+# quarter of an hour, and a batch is twenty documents.
+#
+# 750 is ~1.75x the largest real document measured. Generous enough not to
+# obstruct anything observed, tight enough that one document cannot occupy the
+# single-flight worker for an hour. Raise it with a measurement, not a hunch.
+MAX_CLUSTERS = 750
+
 # Extraction is synchronous and blocks for ~170 s on nine pages, so it cannot run
 # on the event loop. Its own pool, following the #49 `dbexec` precedent: sharing
 # the default `to_thread` pool means a long parse competes with every other
@@ -75,16 +115,42 @@ BRANCHES = ("glossary", "rule", "formula", "summary")
 
 
 class SpendCeilingExceededError(RuntimeError):
-    """The pre-flight estimate exceeded the ceiling; nothing was spent."""
+    """The pre-flight estimate exceeded a ceiling; nothing was spent.
 
-    def __init__(self, estimate: dict) -> None:
+    Names WHICH ceiling bound and by how much. An operator's next move differs
+    completely between the two - a cluster refusal means the document is dense
+    and wants splitting, a token refusal means it is simply long - and "over
+    budget" tells them neither.
+    """
+
+    def __init__(self, estimate: dict, ceiling: str) -> None:
         self.estimate = estimate
+        self.ceiling = ceiling
+        if ceiling == "clusters":
+            measured, limit, unit = estimate.get("clusters", 0), MAX_CLUSTERS, "term clusters"
+        else:
+            measured, limit, unit = (
+                estimate.get("estimated_prompt_tokens", 0),
+                MAX_PROMPT_TOKENS,
+                "prompt tokens",
+            )
+        self.measured, self.limit = measured, limit
         super().__init__(
-            f"estimated {estimate.get('estimated_prompt_tokens'):,} prompt tokens "
-            f"exceeds the ceiling of {MAX_PROMPT_TOKENS:,}"
+            f"estimated {measured:,} {unit} exceeds the ceiling of {limit:,} "
+            f"({measured / limit:.0%} of it) — nothing was spent; "
+            f"{estimate.get('total_calls', 0):,} LLM calls were declined"
         )
 
 
+def check_ceilings(estimate: dict) -> None:
+    """Raise if either ceiling binds. Clusters are checked FIRST because they
+    are exact where the token figure is partly approximate."""
+    if (estimate.get("clusters") or 0) > MAX_CLUSTERS:
+        raise SpendCeilingExceededError(estimate, "clusters")
+    if (estimate.get("estimated_prompt_tokens") or 0) > MAX_PROMPT_TOKENS:
+        raise SpendCeilingExceededError(estimate, "prompt_tokens")
+
+
 def _extractor():
     """The Azure client the extraction branches call.
 
@@ -121,6 +187,12 @@ class KnowledgePipeline:
 
     def __init__(self, store: KnowledgeStore | None = None) -> None:
         self._store = store or KnowledgeStore()
+        # The most recent pre-flight estimate, so `run_document` can report it
+        # on the SUCCESS path too (a refusal carries its own on the exception).
+        # Safe as instance state only because documents are run one at a time —
+        # `IngestService` holds a single-flight lock and `_EXECUTOR` has one
+        # worker. If either ever changes, pass it back instead of stashing it.
+        self.last_estimate: dict | None = None
 
     async def run_document(
         self, document: Document, scope_id: str, user_id: str
@@ -151,10 +223,18 @@ class KnowledgePipeline:
         except SpendCeilingExceededError as e:
             logger.warning(
                 "knowledge_spend_ceiling",
-                extra={"document_id": document.id, "estimate": e.estimate},
+                extra={
+                    "document_id": document.id,
+                    "ceiling": e.ceiling,
+                    "measured": e.measured,
+                    "limit": e.limit,
+                    "estimate": e.estimate,
+                },
             )
             result.status = "failed"
             result.error = str(e)
+            result.estimate = e.estimate
+            result.blocked_by = e.ceiling
             return result
         except Exception as e:
             logger.error(
@@ -167,6 +247,7 @@ class KnowledgePipeline:
 
         result.run_id = run_id
         result.n_entries = n_entries
+        result.estimate = self.last_estimate
         result.status = "succeeded"
         return result
 
@@ -229,8 +310,56 @@ class KnowledgePipeline:
         clustered = await loop.run_in_executor(_EXECUTOR, build_clusters, doc, filtered)
 
         estimate = estimate_cost(doc, clustered, filtered)
-        if (estimate.get("estimated_prompt_tokens") or 0) > MAX_PROMPT_TOKENS:
-            raise SpendCeilingExceededError(estimate)
+
+        # What KIND of document this is, decided before anything is spent.
+        # Advisory today — see `genre.ROUTE_LABELS` / `ROUTE_BRANCHES`, both
+        # off — but recorded, because "104 calls, 93 abstentions" is a fact an
+        # operator should be able to see the reason for.
+        #
+        # A never-throw seam, like every other observability path here: this
+        # describes the run, it does not decide it, and a classifier defect must
+        # not take down an extraction that was about to work. `repr(e)` because
+        # an empty `str(e)` is how the Fernet mismatch hid for a day.
+        try:
+            verdict = classify_genre(doc, filtered)
+            estimate["genre"] = verdict.as_dict()
+            for warning in verdict.warnings:
+                logger.warning(
+                    "knowledge_genre_warning",
+                    extra={
+                        "doc_id": doc.doc_id,
+                        "genre": verdict.genre,
+                        "warning": warning,
+                    },
+                )
+        except Exception as e:  # noqa: BLE001 - described above
+            logger.warning(
+                "knowledge_genre_failed",
+                extra={"doc_id": doc.doc_id, "error": repr(e)},
+            )
+
+        self.last_estimate = estimate
+        # Logged on EVERY run, not only on a refusal: an operator who can only
+        # see the number when it is already too late cannot plan a batch, and
+        # "why was this refused" is answerable only against what a run that
+        # succeeded looked like.
+        logger.info(
+            "knowledge_pre_flight_estimate",
+            extra={
+                "doc_id": doc.doc_id,
+                "clusters": estimate.get("clusters"),
+                "total_calls": estimate.get("total_calls"),
+                "estimated_prompt_tokens": estimate.get("estimated_prompt_tokens"),
+                "cluster_ceiling_pct": round(
+                    100 * (estimate.get("clusters") or 0) / MAX_CLUSTERS, 1
+                ),
+                "token_ceiling_pct": round(
+                    100 * (estimate.get("estimated_prompt_tokens") or 0) / MAX_PROMPT_TOKENS,
+                    1,
+                ),
+            },
+        )
+        check_ceilings(estimate)
 
         # Built before the run row so a missing Azure deployment fails without
         # leaving an orphan `running` row behind.

src/knowledge_ingest/service.py

@@ -38,6 +38,39 @@ from src.middlewares.logging import get_logger
 logger = get_logger("knowledge_ingest")
 
 
+def _roll_up_estimates(results: list[DocumentResult]) -> dict | None:
+    """The batch's pre-flight shape, per document and in total.
+
+    `IngestJob.estimated_cost` has existed since the job model was written and
+    was never populated, so the field a caller would poll to answer "what is
+    this going to cost, and why was that document refused" always came back
+    null. Spend is driven by CLUSTER COUNT — one LLM call per term cluster — so
+    the per-document breakdown is what an operator needs to see; a total alone
+    cannot tell them which document to split.
+    """
+    estimates = [(r.document_id, r.estimate) for r in results if r.estimate]
+    if not estimates:
+        return None
+    return {
+        "clusters": sum(e.get("clusters") or 0 for _, e in estimates),
+        "total_calls": sum(e.get("total_calls") or 0 for _, e in estimates),
+        "estimated_prompt_tokens": sum(
+            e.get("estimated_prompt_tokens") or 0 for _, e in estimates
+        ),
+        "per_document": {
+            doc_id: {
+                "clusters": e.get("clusters"),
+                "total_calls": e.get("total_calls"),
+                "estimated_prompt_tokens": e.get("estimated_prompt_tokens"),
+            }
+            for doc_id, e in estimates
+        },
+        "blocked": {
+            r.document_id: r.blocked_by for r in results if r.blocked_by
+        },
+    }
+
+
 @runtime_checkable
 class PipelinePort(Protocol):
     """Run the whole pipeline over ONE document and report its outcome.
@@ -102,7 +135,10 @@ class IngestService:
         return await self._store.list_for_user(user_id, limit)
 
     async def start(
-        self, user_id: str, document_ids: list[str] | None = None
+        self,
+        user_id: str,
+        document_ids: list[str] | None = None,
+        review_mode: str = "expert",
     ) -> tuple[IngestJob, bool]:
         """Admit a batch. Returns (job, reused).
 
@@ -127,6 +163,7 @@ class IngestService:
             job = IngestJob(
                 scope_id=scope_id,
                 user_id=user_id,
+                review_mode=review_mode,
                 document_ids=[d.id for d in selected],
                 n_requested=len(selected),
                 results=[
@@ -195,8 +232,26 @@ class IngestService:
                     status="failed",
                     error=repr(e),
                 )
+            # r1: in `llm` mode there is no reviewer step - the run's
+            # entries are approved as soon as they land, recorded against
+            # the person who chose the mode (r2/r3). Never-throw: an
+            # auto-approval that fails leaves the entries REVIEWABLE, which
+            # is the safe direction, and must not fail the document.
+            if job.review_mode == "llm" and result.status == "succeeded" and result.run_id:
+                try:
+                    await self._knowledge.auto_approve_run(
+                        run_id=result.run_id,
+                        scope_id=job.scope_id,
+                        authorised_by=job.user_id,
+                    )
+                except Exception as e:  # noqa: BLE001
+                    logger.error(
+                        "ingest_auto_approve_failed",
+                        extra={"job_id": job.id, "run_id": result.run_id, "error": repr(e)},
+                    )
             by_id[result.document_id] = result
             job.results = list(by_id.values())
+            job.estimated_cost = _roll_up_estimates(job.results)
             await self._store.save(job)
 
         job.settle()

src/knowledge_ingest/store.py

@@ -123,6 +123,7 @@ class PostgresJobStore:
             user_id=row.user_id,
             status=row.status,
             stage=row.stage,
+            review_mode=row.review_mode or "expert",
             document_ids=list(row.document_ids or []),
             n_requested=row.n_requested or 0,
             n_succeeded=row.n_succeeded or 0,
@@ -161,6 +162,7 @@ class PostgresJobStore:
             user_id=job.user_id,
             status=job.status,
             stage=job.stage,
+            review_mode=job.review_mode,
             document_ids=list(job.document_ids),
             n_requested=job.n_requested,
             n_succeeded=job.n_succeeded,

src/knowledge_parsing/assets.py

@@ -0,0 +1,156 @@
+"""Asset identity and persistence for figures pulled out of a parse. Added 0.4.0.
+
+Two jobs, deliberately small:
+
+1. **Identity.** `asset_id()` content-addresses the bytes. The parser's own id is
+   not usable: Mistral names images per PAGE (`img-0.jpeg`), so the same name
+   recurs across pages — the Open Pit textbook produces **7 distinct names for 8
+   images**, and a flat store keyed on the parser's name loses one. Content
+   addressing also makes a re-parse idempotent and de-duplicates a figure that
+   appears in two documents.
+
+2. **Persistence.** An `AssetSink` protocol plus a local-filesystem default.
+
+**Why a protocol and not a blob client.** `src/knowledge_parsing/` imports nothing
+from the rest of `src` and that is load-bearing — the package is handed to the
+team as a standalone unit, so reaching into `src.storage` (which pulls app
+settings, which needs `.env`) would break the export and make the parser
+un-runnable outside this repo. The app injects a blob-backed sink instead; see
+`knowledge_ingest.asset_sink`.
+
+Persistence is a **never-throw seam**. A figure whose bytes cannot be stored
+yields `storage_key=None` and a logged warning, never a failed parse: the parse is
+the expensive, cached, hard-to-repeat half, and losing a whole document because
+one PNG could not be written is the wrong trade.
+"""
+
+from __future__ import annotations
+
+import base64
+import binascii
+import hashlib
+import logging
+from pathlib import Path
+from typing import Protocol, runtime_checkable
+
+logger = logging.getLogger("knowledge_parsing.assets")
+
+ASSET_ID_CHARS = 16
+
+# Mistral reports ids like "img-0.jpeg"; the suffix is the only extension signal
+# available. Anything unrecognised falls back to .bin rather than guessing, so a
+# stored object is never mislabelled.
+_KNOWN_EXT = {"jpg", "jpeg", "png", "gif", "webp", "bmp", "tiff"}
+
+
+def asset_id(data: bytes) -> str:
+    """Content address for asset bytes: sha256, first 16 hex chars.
+
+    16 chars is ~64 bits. At corpus scale (thousands of figures) the collision
+    probability is negligible, and it keeps the id short enough to read in a
+    `![](asset://…)` reference inside a prose chunk.
+    """
+    return hashlib.sha256(data).hexdigest()[:ASSET_ID_CHARS]
+
+
+def extension_for(parser_id: str | None) -> str:
+    """File extension from the parser's own id, defaulting to `bin`."""
+    if not parser_id or "." not in parser_id:
+        return "bin"
+    ext = parser_id.rsplit(".", 1)[-1].strip().lower()
+    return ext if ext in _KNOWN_EXT else "bin"
+
+
+def decode_base64(payload: str | None) -> bytes | None:
+    """Decode a base64 image payload, tolerating a `data:` URI prefix.
+
+    Returns None rather than raising on anything malformed — the caller treats a
+    missing payload as "no bytes to store", which is the same degraded path as a
+    failed write.
+    """
+    if not payload:
+        return None
+    text = payload.strip()
+    if text.startswith("data:"):
+        _, _, text = text.partition(",")
+    try:
+        return base64.b64decode(text, validate=False)
+    except (binascii.Error, ValueError) as exc:
+        logger.warning("asset base64 decode failed: %r", exc)
+        return None
+
+
+@runtime_checkable
+class AssetSink(Protocol):
+    """Somewhere to put asset bytes. Returns a storage key, or None on failure."""
+
+    def put(self, identifier: str, extension: str, data: bytes) -> str | None:
+        ...
+
+
+class LocalAssetSink:
+    """Write assets to a local directory, named by content address.
+
+    The default, and what the CLI uses. Idempotent by construction: an asset whose
+    id already exists on disk is not rewritten, because the id IS the content.
+    """
+
+    def __init__(self, root: Path, prefix: str = "assets") -> None:
+        self.root = Path(root)
+        self.prefix = prefix
+
+    def put(self, identifier: str, extension: str, data: bytes) -> str | None:
+        try:
+            target_dir = self.root / self.prefix
+            target_dir.mkdir(parents=True, exist_ok=True)
+            target = target_dir / f"{identifier}.{extension}"
+            if not target.exists():
+                target.write_bytes(data)
+            return f"{self.prefix}/{identifier}.{extension}"
+        except OSError as exc:
+            logger.warning("asset write failed for %s: %r", identifier, exc)
+            return None
+
+
+class NullAssetSink:
+    """Discard asset bytes, keeping the asset metadata. For `--no-assets`.
+
+    Not the same as having no assets: the figure is still folded into its prose
+    chunk with its caption and description, and still carries an `asset_id`. Only
+    the bytes are not retained, so `storage_key` is None.
+    """
+
+    def put(self, identifier: str, extension: str, data: bytes) -> str | None:
+        return None
+
+
+def store_asset(
+    sink: AssetSink | None,
+    parser_id: str | None,
+    base64_payload: str | None,
+) -> tuple[str | None, str | None]:
+    """Content-address and persist one asset. Returns `(asset_id, storage_key)`.
+
+    `asset_id` is None only when there are no bytes to address — a table, or a
+    figure whose payload the parser did not return. `storage_key` is None whenever
+    the bytes were not retained, which is a normal outcome (`NullAssetSink`) as
+    well as a failure one.
+    """
+    data = decode_base64(base64_payload)
+    if data is None:
+        return None, None
+    identifier = asset_id(data)
+    if sink is None:
+        return identifier, None
+    return identifier, sink.put(identifier, extension_for(parser_id), data)
+
+
+def synthetic_asset_id(*parts: str) -> str:
+    """Id for an asset with no bytes of its own — a table.
+
+    Derived from the document and chunk coordinates rather than content, because
+    there is no content to hash. Deterministic for the same document, which is
+    what matters: a re-parse produces the same id, so a pointer to a table
+    survives re-extraction the way a figure's does.
+    """
+    return hashlib.sha256("|".join(parts).encode("utf-8")).hexdigest()[:ASSET_ID_CHARS]

src/knowledge_parsing/config.py

@@ -88,6 +88,11 @@ class PipelineConfig:
 
     # --- behaviour ---
     resume: bool = True                  # skip documents already finished
+    # Keep asset METADATA but discard the bytes. The figure is still folded into
+    # its prose chunk with its caption, description and content-addressed id —
+    # only `storage_key` comes back null. For a run where the images are not
+    # wanted on disk, or where storage is unavailable.
+    no_assets: bool = False
     write_debug_pdf: bool = True         # _layout.pdf & _span.pdf: fastest quality check there is
 
     # --- quality-check thresholds (see checks.py) ---

src/knowledge_parsing/contracts.py

@@ -42,13 +42,85 @@ from typing import Literal
 from pydantic import BaseModel, Field, computed_field
 
 ChunkKind = Literal["text", "table", "chart", "equation", "image"]
-
+AssetKind = Literal["figure", "chart", "diagram", "table"]
+
+# 0.4.0 — ASSET POINTERS. A figure is no longer its own chunk: its markdown
+#         reference is folded into the prose chunk at its reading-order position
+#         and the figure travels as an `Asset` on that chunk. Tables stay their
+#         own chunk (their linearised text earns its place as evidence) but gain
+#         an `Asset` and a two-way pointer. Adds `Chunk.assets` and
+#         `Chunk.referenced_by`; `Chunk.images` is kept as a DEPRECATED alias.
+#
+#         A MINOR bump, not a patch: chunk boundaries change, so a 0.3.x consumer
+#         reading a 0.4.0 artifact would silently see a different document shape.
+#         `adapter.check_schema_version` rejects a newer minor, which is the
+#         designed behaviour — parsing and extraction ship together here.
 # 0.3.3 — adds the computed 1-based `page_no` (S6b). Derived from `page_idx`.
 # 0.3.2 — adds the optional `source_title`.
 # 0.3.1 — adds the optional `normaliser_version`.
-# All patch bumps: purely additive, so every 0.3.0 consumer reads them unchanged
-# and the new fields are null on anything written earlier.
-SCHEMA_VERSION = "0.3.3"
+# 0.3.x were all patch bumps: purely additive, so every 0.3.0 consumer reads them
+# unchanged and the new fields are null on anything written earlier.
+SCHEMA_VERSION = "0.4.0"
+
+
+class Asset(BaseModel):
+    """A figure, chart, diagram or table referenced by a chunk. Added 0.4.0.
+
+    **Why this exists.** Before 0.4.0 an image became its own chunk with an EMPTY
+    `text`, which made it unreachable by construction: no text means no term
+    mentions, no mentions means the chunk joins no cluster, and a chunk in no
+    cluster is never selected as evidence. Measured across three documents: 13
+    figures, every one described by a vision model we paid for, and **zero of
+    them appearing in any cluster's evidence.** Meanwhile the prose that said
+    "Figure 1.1" had its `![](…)` reference stripped, so nothing connected the
+    sentence to the picture.
+
+    The asset carries both halves of the provenance split, and the distinction is
+    load-bearing:
+
+    - `caption` is **verbatim document text** — printed on the page, therefore
+      quotable as evidence and span-checkable like any other transcription.
+    - `description` is **model-written** — a claim *about* the picture. It is
+      never span-checkable and must never be quoted as evidence, which is why it
+      lives here and not in `Chunk.text`.
+    """
+
+    # Content-addressed: sha256 of the asset bytes, first 16 hex chars.
+    #
+    # NOT the parser's own id. Mistral names images per PAGE (`img-0.jpeg`), which
+    # already collides in practice — the Open Pit textbook yields 7 distinct names
+    # for 8 images, so a flat store keyed on the parser's name silently loses one.
+    # Content addressing also makes a re-parse idempotent and de-duplicates a
+    # figure that appears in two documents.
+    asset_id: str
+    kind: AssetKind
+
+    # Where the bytes live. `null` for a table (it has no bytes of its own) and
+    # for a figure whose persistence failed — writing assets is a never-throw
+    # seam, so a storage outage degrades to "no key" rather than failing a parse.
+    storage_key: str | None = None
+
+    page_idx: int = 0
+
+    @computed_field  # type: ignore[prop-decorator]
+    @property
+    def page_no(self) -> int:
+        """1-based page number, for a human and for a review UI. Derived from
+        `page_idx` so the two cannot drift."""
+        return self.page_idx + 1
+
+    # `[x0, y0, x1, y1]` of the asset itself, so a UI can highlight the figure.
+    # Distinct from the host chunk's own `bbox`, which stays where the TEXT is.
+    bbox: list[int] | None = None
+
+    # Verbatim, from the document's own caption block. Quotable as evidence.
+    caption: str | None = None
+
+    # ⚠️ MODEL-WRITTEN. See the class docstring. Never evidence.
+    description: str | None = None
+    # Which model/feature produced `description`, so a future swap is traceable
+    # rather than silently changing what the field means.
+    description_source: str | None = None
 
 
 class Chunk(BaseModel):
@@ -124,7 +196,24 @@ class Chunk(BaseModel):
     # uses item sequence, never coordinates.
     bbox: list[int] | None = None
 
-    # Non-text attachments (formula images, table/chart crops)
+    # Figures, charts, diagrams and tables this chunk contains or points at.
+    # Added 0.4.0 — see the `Asset` docstring for why images stopped being chunks.
+    #
+    # A prose chunk that holds a figure carries it here AND carries the figure's
+    # `![](asset://<asset_id>)` reference inline in `text` at its reading-order
+    # position, so the model reads the reference exactly where the document put it.
+    assets: list[Asset] = Field(default_factory=list)
+
+    # chunk_ids of OTHER chunks that point at this one. Populated on table chunks,
+    # which stay standalone (their linearised text earns its place as evidence —
+    # 74.6% of the Komatsu manual's top-3 evidence is tabular) while still being
+    # reachable from the prose that introduces them.
+    referenced_by: list[str] = Field(default_factory=list)
+
+    # ⚠️ DEPRECATED 0.4.0 — superseded by `assets`. Still populated for one
+    # version so a 0.3.x reader degrades rather than breaking; it holds the bare
+    # parser-local filenames, which are NOT unique across a document (see
+    # `Asset.asset_id`). Read `assets` instead.
     images: list[str] = Field(default_factory=list)
 
     # Source markup, kept verbatim beside the rendered prose in `text`.

src/knowledge_parsing/normalize_mistral.py

@@ -30,13 +30,15 @@ whitespace normalisation. See contracts.py for why.
 from __future__ import annotations
 
 import hashlib
+import json
 import re
 from collections.abc import Iterator
 from functools import lru_cache
 from pathlib import Path
 from typing import Any
 
-from .contracts import Chunk
+from .assets import AssetSink, store_asset, synthetic_asset_id
+from .contracts import Asset, Chunk
 from .normalize import MAX_CHUNK_CHARS, _heading  # identical section-boundary logic
 from .render import render_latex, render_table
 
@@ -62,8 +64,15 @@ def normaliser_version() -> str:
 # the source anyway.
 _DISCARDED = {"page_number", "footer"}
 
+# Recorded on every Asset so a later swap of the description model is traceable
+# rather than silently changing what the field means.
+_DESCRIPTION_SOURCE = "mistral-ocr/bbox_annotation"
+
 _LEADING_HASHES = re.compile(r"^\s*(#{1,6})\s*")
 _MD_IMAGE = re.compile(r"!\[[^\]]*\]\([^)]*\)")
+# Same shape, but capturing the target so a block's own markdown can be read
+# for the image id it refers to (see _block_image_id).
+_MD_IMAGE_TARGET = re.compile(r"!\[[^\]]*\]\(([^)]*)\)")
 _MD_LINK = re.compile(r"\[([^\]]*)\]\([^)]*\)")
 _MD_BOLD = re.compile(r"\*\*(.+?)\*\*", re.S)
 _MD_ITALIC = re.compile(r"\*(.+?)\*", re.S)
@@ -127,7 +136,75 @@ def _image_descriptions(response: dict[str, Any]) -> dict[str, str]:
             ann = im.get("image_annotation")
             iid = im.get("id")
             if iid and ann:
-                out[iid] = ann if isinstance(ann, str) else str(ann)
+                out[iid] = _unwrap_description(ann)
+    return out
+
+
+def _unwrap_description(annotation: Any) -> str:
+    """Pull the prose out of an annotation payload.
+
+    Mistral returns bbox annotations JSON-WRAPPED — literally
+    `{"description": "The diagram illustrates…"}` — and 0.3.x stored that string
+    raw, braces and all, so every consumer got JSON where it expected a sentence.
+    Unwrapped here, at the one place that reads the field.
+
+    Never raises: anything unparseable falls back to the string form, because a
+    slightly ugly description is worth more than a failed parse.
+    """
+    if isinstance(annotation, dict):
+        value = annotation.get("description") or annotation.get("text")
+        return str(value) if value else json.dumps(annotation, ensure_ascii=False)
+    if isinstance(annotation, str):
+        text = annotation.strip()
+        if text.startswith("{"):
+            try:
+                parsed = json.loads(text)
+            except (ValueError, TypeError):
+                return text
+            if isinstance(parsed, dict):
+                value = parsed.get("description") or parsed.get("text")
+                if value:
+                    return str(value)
+            return text
+        return text
+    return str(annotation)
+
+
+def _block_image_id(block: dict[str, Any]) -> str | None:
+    """The image this block refers to, preferring the id in its own markdown.
+
+    ⚠️ **`block["image_id"]` is not trustworthy.** Measured on the Open Pit
+    textbook: page 6 carries two image blocks, and the FIRST one's `content` reads
+    `![img-1.jpeg](img-1.jpeg)` while its `image_id` field says **`img-2.jpeg`** —
+    so both blocks claim the same image. Keying on `image_id` therefore resolved
+    both to img-2's bytes, gave two different figures one content hash, and lost
+    img-1 entirely (7 files written for 8 figures).
+
+    The `content` markdown is the document's own reference and was correct in
+    every case, so it wins; `image_id` is the fallback for a block whose content
+    carries no reference. This also corrects an earlier diagnosis: the ids are
+    globally unique across the response (`img-0` … `img-7`), so the duplication
+    was never page-local naming — it was this mislabelled field.
+    """
+    match = _MD_IMAGE_TARGET.search(block.get("content") or "")
+    if match and match.group(1).strip():
+        return match.group(1).strip()
+    return block.get("image_id")
+
+
+def _image_payloads(response: dict[str, Any]) -> dict[str, str]:
+    """image_id -> base64 payload. Empty when `include_image_base64` was off.
+
+    0.3.x never read these: the bytes were returned by the API, recorded nowhere,
+    and `Chunk.images` pointed at filenames that did not exist on disk. 0.4.0
+    content-addresses and persists them.
+    """
+    out: dict[str, str] = {}
+    for page in response.get("pages", []):
+        for im in page.get("images") or []:
+            iid, payload = im.get("id"), im.get("image_base64")
+            if iid and payload:
+                out[iid] = payload
     return out
 
 
@@ -221,7 +298,10 @@ def _chunk_by_page(
 
 
 def normalise_mistral(
-    response: dict[str, Any], doc_id: str, granularity: str = "section"
+    response: dict[str, Any],
+    doc_id: str,
+    granularity: str = "section",
+    asset_sink: AssetSink | None = None,
 ) -> list[Chunk]:
     """Mistral OCR response (include_blocks=True) → chunks.
 
@@ -268,13 +348,38 @@ def normalise_mistral(
 
     # --- page mode: recall-only; a page chunk has no single heading ---
     if granularity == "page":
-        return _chunk_by_page(blocks, descriptions, chapter_per_page, doc_id)
+        # ⚠️ FENCED OFF AT 0.4.0, deliberately, rather than given the asset
+        # treatment. Page mode emits 0.3.x-shaped chunks — images standalone with
+        # empty text, no `assets` — so letting it run would quietly produce an
+        # artifact that CLAIMS schema_version 0.4.0 while having the old shape.
+        # Two normalisers silently diverging is exactly how the `type: "list"`
+        # content drop survived unnoticed and cost five gold terms.
+        #
+        # Page mode exists only for a recall-only measurement (it nulls
+        # `heading_path`, which the outline and `source_wording` both need), so it
+        # is not worth duplicating the asset logic into. Restore by porting the
+        # image/table branches from the section path below.
+        raise NotImplementedError(
+            "granularity='page' is not supported at schema 0.4.0: it would emit "
+            "0.3.x-shaped chunks (standalone image chunks, no assets) under a "
+            "0.4.0 version stamp. Use granularity='section', or port the asset "
+            "handling into _chunk_by_page first."
+        )
 
     # --- section mode (default): a numbered heading opens a new chunk (mirrors normalize.py) ---
     chunks: list[Chunk] = []
     running: Chunk | None = None
     pieces: list[str] = []
     stack: list[tuple[int, str]] = []
+    payloads = _image_payloads(response)
+    # A table's asset has to be added to the PROSE chunk that precedes it, but
+    # that chunk is already closed and appended by then — so the back-reference is
+    # queued and resolved once, after the loop.
+    _pending_back_refs: list[tuple[str, Asset]] = []
+    # A printed caption arrives as its own `caption` block AFTER its image block.
+    # The figure waiting for one is held here; the caption is attached to the asset
+    # AND left in the prose text, because it is verbatim document content.
+    _pending_caption_for: list[Asset] = []
 
     def push_heading(level: int, text: str) -> None:
         while stack and stack[-1][0] >= level:
@@ -285,7 +390,7 @@ def normalise_mistral(
         nonlocal running, pieces
         if running is not None:
             running.text = "\n\n".join(pieces).strip()
-            if running.text or running.images:
+            if running.text or running.assets or running.images:
                 running.page_idxs = sorted(set(running.page_idxs))
                 chunks.append(running)
         running, pieces = None, []
@@ -308,6 +413,12 @@ def normalise_mistral(
             continue
 
         if kind == "table":
+            # Tables stay their OWN chunk (0.4.0 kept this deliberately): unlike a
+            # figure, a table's linearised text carries real terms the filter
+            # finds — 74.6% of the Komatsu manual's top-3 evidence is tabular. What
+            # 0.4.0 adds is reachability: an `Asset` plus a two-way pointer, so the
+            # prose that introduces a table can be followed to it and back.
+            prose_before = running.chunk_id if running is not None else None
             close()
             c = open_chunk("table", page)
             c.text = render_table(block.get("content") or "")
@@ -315,22 +426,67 @@ def normalise_mistral(
             c.table_html = block.get("content") or None
             c.source_items = [i]
             c.bbox = _bbox(block)
-            if c.text or c.images:
+            table_asset = Asset(
+                # No bytes to hash, so the id is derived from stable document
+                # coordinates instead — deterministic across re-parses, which is
+                # what a durable pointer needs.
+                asset_id=synthetic_asset_id(doc_id, c.chunk_id, str(i)),
+                kind="table",
+                page_idx=page,
+                bbox=_bbox(block),
+            )
+            c.assets = [table_asset]
+            if prose_before is not None:
+                c.referenced_by = [prose_before]
+                _pending_back_refs.append((prose_before, table_asset))
+            if c.text or c.assets:
                 chunks.append(c)
             continue
 
         if kind == "image":
-            close()
-            c = open_chunk("image", page)
-            iid = block.get("image_id")
-            c.generated_description = descriptions.get(iid) if iid else None
-            c.source_items = [i]
-            c.bbox = _bbox(block)
+            # ── 0.4.0: a figure is NO LONGER ITS OWN CHUNK. ────────────────────
+            # It used to be, with an EMPTY `text`, which made it unreachable by
+            # construction: no text -> no term mentions -> joins no cluster ->
+            # never selected as evidence. Measured across three documents: 13
+            # figures, every one described by a vision model we pay for, and ZERO
+            # appearing in any cluster's evidence. Meanwhile the prose saying
+            # "Figure 1.1" had its `![](…)` reference stripped, so nothing
+            # connected the sentence to the picture.
+            #
+            # Now the reference is folded into the RUNNING prose chunk at its
+            # reading-order position — exactly where the document put it — and the
+            # figure travels as an `Asset` on that chunk.
+            iid = _block_image_id(block)
+            identifier, storage_key = store_asset(
+                asset_sink, iid, payloads.get(iid) if iid else None
+            )
+            # Fall back to document coordinates when the parser returned no bytes,
+            # so a figure is still referenced rather than silently vanishing.
+            if identifier is None:
+                identifier = synthetic_asset_id(doc_id, str(page), str(i))
+            asset = Asset(
+                asset_id=identifier,
+                kind="figure",
+                storage_key=storage_key,
+                page_idx=page,
+                bbox=_bbox(block),
+                description=descriptions.get(iid) if iid else None,
+                description_source=_DESCRIPTION_SOURCE if iid else None,
+            )
+            # P5 edge case: a figure can precede any prose on its page (Open Pit
+            # p.3 is close to this shape). Open a chunk rather than drop the asset.
+            if running is None:
+                running = open_chunk("text", page)
+            running.source_items.append(i)
+            running.page_idxs.append(page)
+            running.assets.append(asset)
+            # Deprecated alias, one version only.
             if iid:
-                c.images = [iid]
-            # text stays "" — a printed caption arrives as its own `caption`
-            # block; the model description is kept OUT of text on purpose.
-            chunks.append(c)
+                running.images.append(iid)
+            # The reference itself, inline in the prose. `asset://` rather than the
+            # parser's filename because the filename is page-local and collides.
+            pieces.append(f"![](asset://{identifier})")
+            _pending_caption_for.append(asset)
             continue
 
         if kind == "equation":
@@ -362,6 +518,14 @@ def normalise_mistral(
                 running.source_items.append(i)
                 running.page_idxs.append(page)
                 pieces.append(text)
+                # A caption directly after a figure names that figure. Attach it to
+                # the asset so a consumer can render "Figure 1.1 …" beside the
+                # image — and LEAVE it in `pieces` too, because it is printed in
+                # the document: verbatim, quotable as evidence, span-checkable.
+                # Copying it is the point; moving it would silently delete
+                # evidence that survived 17/17 times on the Open Pit textbook.
+                if kind == "caption" and _pending_caption_for:
+                    _pending_caption_for.pop(0).caption = text.strip()
             continue
 
         # `text` / `title` / `signature`: a NUMBERED line may open a section.
@@ -413,4 +577,19 @@ def normalise_mistral(
                                  heading=continued_from.heading)
 
     close()
+
+    # Two-way table pointers, resolved once the loop is done. The table chunk
+    # already names the prose chunk in `referenced_by`; this adds the table's asset
+    # to that prose chunk so the link is navigable from the prose side too — the
+    # direction a reader, and a model reading evidence, actually travels. Deferred
+    # rather than done inline because the prose chunk is already closed and
+    # appended by the time its table is reached.
+    by_id = {c.chunk_id: c for c in chunks}
+    for prose_id, table_asset in _pending_back_refs:
+        host = by_id.get(prose_id)
+        if host is not None and not any(
+            a.asset_id == table_asset.asset_id for a in host.assets
+        ):
+            host.assets.append(table_asset)
+
     return chunks

src/knowledge_parsing/run.py

@@ -21,6 +21,7 @@ from datetime import datetime
 from pathlib import Path
 from typing import Any
 
+from .assets import AssetSink, LocalAssetSink, NullAssetSink
 from .checks import check_items, vocabulary_per_page
 from .config import PipelineConfig
 from .contracts import ParsedDocument
@@ -114,7 +115,12 @@ class _ParseOutcome:
     normalise_seconds: float
 
 
-def _parse_one(source: Path, cfg: PipelineConfig, doc_id: str) -> _ParseOutcome:
+def _parse_one(
+    source: Path,
+    cfg: PipelineConfig,
+    doc_id: str,
+    asset_sink: AssetSink | None = None,
+) -> _ParseOutcome:
     """The per-document core: parse, normalise, assemble the `ParsedDocument`.
 
     The single place a `ParsedDocument` is built. Both `parse_one` and `run()` go
@@ -130,7 +136,7 @@ def _parse_one(source: Path, cfg: PipelineConfig, doc_id: str) -> _ParseOutcome:
         # character-spaced formulas, so neither applies.
         checks: dict[str, Any] = {"backend": "mistral-ocr"}
         t0 = time.perf_counter()
-        chunks = normalise_mistral(response, doc_id)
+        chunks = normalise_mistral(response, doc_id, asset_sink=asset_sink)
         normalise_seconds = time.perf_counter() - t0
     else:
         items = json.loads(result.content_list.read_text(encoding="utf-8"))
@@ -190,6 +196,7 @@ def parse_one(
     source: Path,
     cfg: PipelineConfig,
     doc_id: str | None = None,
+    asset_sink: AssetSink | None = None,
 ) -> ParsedDocument:
     """Parse one document and return the artifact in memory. Raises on failure.
 
@@ -206,7 +213,7 @@ def parse_one(
     """
     if doc_id is None:
         doc_id = source.stem
-    return _parse_one(source, cfg, doc_id).artifact
+    return _parse_one(source, cfg, doc_id, asset_sink).artifact
 
 
 def run(cfg: PipelineConfig) -> Manifest:
@@ -245,7 +252,13 @@ def run(cfg: PipelineConfig) -> Manifest:
             # Same core the ingestion service calls — the artifact is built once,
             # here and there. run() adds only the folder-batch concerns around it:
             # the manifest record and writing the result to disk.
-            outcome = _parse_one(source, cfg, doc_id)
+            # Assets land beside the artifact they belong to, so a run folder is
+            # self-describing and movable. `--no-assets` keeps the metadata and
+            # drops only the bytes (see assets.NullAssetSink).
+            sink: AssetSink = (
+                NullAssetSink() if cfg.no_assets else LocalAssetSink(target)
+            )
+            outcome = _parse_one(source, cfg, doc_id, asset_sink=sink)
             artifact = outcome.artifact
             result = outcome.parse_result
 
@@ -323,6 +336,8 @@ def main() -> None:
     # on `vlm`.
     p.add_argument("--effort", choices=["medium", "high"], default=None)
     p.add_argument("--no-resume", action="store_true", help="reprocess everything")
+    p.add_argument("--no-assets", action="store_true",
+                   help="keep asset metadata but do not write image bytes to disk")
     p.add_argument("--start-page", type=int, default=None)
     p.add_argument("--end-page", type=int, default=None)
     a = p.parse_args()
@@ -338,6 +353,8 @@ def main() -> None:
         cfg.effort = a.effort
     if a.no_resume:
         cfg.resume = False
+    if a.no_assets:
+        cfg.no_assets = True
     if a.start_page is not None:
         cfg.start_page = a.start_page
     if a.end_page is not None: