[NOTICKET] docs: knowledge pipeline — reframe kex as prototype, not port target

kex is a prototype that answered the design questions; extraction v2 gets rebuilt
in this repo. What carries forward is the evidence and the tuned constants, not
the code. Restructures the checklist accordingly:

- New §2 prototype disposition (P1 freeze / P2 harvest constants / P3 migrate the
gold sets + scorer into eval/knowledge/ / P4 record negative findings). P3 lands
before any v2 pipeline code — without a standing baseline "improved" cannot be
shown, and the nano schema-fill experiment already failed.
- §5 splits every stage into DESIGN (validated, with a number) vs BUILD (exists in
this repo). The previous table read 11 done, which was honest about the design
and misleading about a rebuild — none of it exists here.
- Adds the module-name collision: src/knowledge/ is the existing OCR->pgvector
service, so v2 needs its own name.
- Seam section reflects Sofhia's contracts.py draft as the agreed base, with the
envelope / section-structure / source_wording items still open.

DEV_PLAN §0.8 and the §6 open decisions follow the same reframing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DEV_PLAN.md

@@ -289,9 +289,11 @@ two independently runnable halves — **parsing (Sofhia, MinerU)** and **knowled
 (Rifqi)** — meeting only at a **persisted, versioned parsed-document artifact**. Extraction never
 takes a file path; that constraint is the whole point of the split.
 
-**Where it stands (audited 2026-08-19).** The extraction half is **not greenfield**: a working POC
-(`kex`, ~2,763 LOC, 21 passing tests, 8-command CLI, four experiments against real Azure calls)
-already exists outside version control at `Documents/Dataeyond_catalog/knowledge-pipeline/`.
+**Where it stands (audited 2026-08-19).** A working **prototype** (`kex`, ~2,763 LOC, 21 passing
+tests, 8-command CLI, four experiments against real Azure calls) exists outside version control at
+`Documents/Dataeyond_catalog/knowledge-pipeline/`. It is **not the product and will not be ported** —
+it answered the design questions; extraction v2 gets rebuilt in this repo. What carries forward is
+the *evidence* (gold sets, experiment results) and the *tuned constants*, not the code.
 Measured on the BUMA standard (9 pages): GLiNER Indonesian recall **0.854** (kill line 0.70, PASS —
 this closes the context doc's headline open risk), clustering compression **2.56×** (kill 2.0, PASS),
 nano schema-fill precision **0.75** (kill 0.80, **FAIL**), span-validation rejection **1.00** with
@@ -299,18 +301,22 @@ nano schema-fill precision **0.75** (kill 0.80, **FAIL**), span-validation rejec
 
 | # | Task | Owner | Status | Note |
 |---|---|---|---|---|
-| K1 | Put `kex` under version control | Rifqi | ⬜ | No `.git`. ~2,763 LOC + both gold sets + all four experiment results exist on one machine. Highest-priority item; repo home undecided |
-| K2 | Agree the parsed-artifact schema (A1) | Sofhia + Rifqi | 🔄 | Sofhia circulated a draft `contracts.py` 2026-08-19, derived from real MinerU output. Adopted as the base; open items are the document-level envelope (no home for `version`), the section-structure claim, and `source_wording` |
-| K3 | Move the seam off the file path | Sofhia + Rifqi | ⬜ | The POC's boundary is `Parser.parse(path) -> list[Page]` — the one shape the plan forbids. Must land **before** the MinerU parser is built against it |
-| K4 | Artifact + candidate-entry persistence | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns dedorch DDL → one consolidated Harry-ready handoff, never DDL from Python (§2.2) |
-| K5 | Fix the literal-source-wording violation | Rifqi | ⬜ | A locked decision is currently violated: the BUMA heading reads "Physical **of** Availability (PA)"; the extracted entry carries "Physical Availability". The discrepancy is meant to reach the expert, not be silently corrected |
-| K6 | Provision a mini deployment | Rifqi | ⛔ | Only nano is provisioned, so all four branches route there — including `rule` and `summary`, whose failure mode is least detectable (a plausible summary cannot be span-checked). Blocks the nano-vs-mini A/B |
-| K7 | Expert review of the gold set | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is self-bootstrapped and provisional. Both the recall and precision figures move when it is reviewed |
-| K8 | Port `kex` into this repo | Rifqi | ⬜ | Python 3.13 → 3.12, new deps (GLiNER + torch), gating, and an admin ingest surface whose auth question is a different risk class from the read-only chat surface. **Gated on §6 sign-off — deps (§6.4) and any new endpoint (§6.3).** |
-
-**Not decided, do not assume:** whether `kex` becomes a subdirectory here, a separate repo, or a
-branch; whether the admin ingest surface is HTTP at all (the offline script path is the honest
-first milestone); and GLiNER + LangExtract ensembling stays ⏸️ flagged as scope creep.
+| P1 | Freeze the `kex` prototype (read-only snapshot) | Rifqi | ⬜ | No `.git`. ~2,763 LOC + both gold sets + all four experiment results on one machine. It is an archive, **not** a codebase to develop — v2 is a rebuild here, not a port |
+| P2 | Harvest the prototype's tuned constants | Rifqi | ⬜ | Every number that took runs to find, with its reason, **before** v2 code: rapidfuzz 92, fuzzy off below 5 chars (`PA`/`UA` collide), K=3, the evidence weight table, the 1024-token prefix pad (below it, caching does not engage — worth the measured 54% hit rate), label + cue sets, the 4 prompts |
+| P3 | Migrate the evidence into `eval/knowledge/` | Rifqi | ⬜ | Gold sets (41 terms + 15 rules) + scorer + experiment runners, following the existing `eval/{intent,help,readiness,planner}/` convention. **Before any v2 pipeline code** — without a standing baseline, "improved" is unfalsifiable |
+| S1 | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | Sofhia's draft `contracts.py` (2026-08-19) adopted as the base. Open: document-level envelope (no home for `version`), the section-structure claim, `source_wording` |
+| S5 | Keep the seam off the file path | Sofhia + Rifqi | ⬜ | The prototype's boundary was `Parser.parse(path) -> list[Page]` — the shape the plan forbids. Settle **before** the MinerU parser is built against it |
+| X* | Build extraction v2 in this repo | Rifqi | ⬜ | 17 stages, `src/query/`-shaped. Design is validated by the prototype; **none of it exists here yet** — see the two-column table in the checklist |
+| D2 | Artifact + candidate-entry persistence | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns dedorch DDL → one consolidated Harry-ready handoff, never DDL from Python (§2.2) |
+| R1 | Fix the literal-source-wording violation | Rifqi | ⬜ | A locked decision is violated: the BUMA heading reads "Physical **of** Availability (PA)"; the extracted entry carries "Physical Availability". The discrepancy is meant to reach the expert |
+| D3 | Provision a mini deployment | Rifqi | ⛔ | Only nano is provisioned, so all four branches route there — including `rule` and `summary`, whose failure mode is least detectable. Blocks the nano-vs-mini A/B |
+| D6 | Expert review of the gold set | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is self-bootstrapped and provisional. Now the baseline v2 is measured against |
+| D5 | v2 module name | Rifqi | ⬜ | `src/knowledge/` is taken by the existing OCR→pgvector service. Pick a non-colliding name before the first commit |
+
+**Not decided, do not assume:** the v2 module name (D5); whether the admin ingest surface is HTTP
+at all (the offline script path is the honest first milestone); and GLiNER + LangExtract ensembling
+stays ⏸️ flagged as scope creep. **`kex` is not being ported** — it is a prototype that answered the
+design questions; v2 is built here against this repo's patterns.
 
 ## 5. Critical path & sequencing
 
@@ -336,8 +342,8 @@ first milestone); and GLiNER + LangExtract ensembling stays ⏸️ flagged as sc
 - **W3 re-open timing** (deferred `analyze_*` activation) — Rifqi (deferred at the 2026-07-13 approval).
 - **INV-6 relaxation for S1b targeted repair** — team, only after S1a `repair_candidate` telemetry
   shows a meaningful hit-rate (SPINE_V2_PLAN §6).
-- **Knowledge pipeline — where `kex` lives** (§0.8 K1/K8): subdirectory of this repo, separate repo,
-  or its own branch. Blocks version control, which blocks everything else in §0.8.
-- **Knowledge pipeline — admin ingest surface** (§0.8 K8): whether it is HTTP at all, and if so its
+- **Knowledge pipeline — v2 module name** (§0.8 D5): `src/knowledge/` is already the OCR→pgvector
+  service, so extraction v2 needs its own module name before the first commit.
+- **Knowledge pipeline — admin ingest surface** (§0.8 D4): whether it is HTTP at all, and if so its
   auth posture. A write surface triggered by an admin is a different risk class from the current
   unauthenticated read-only chat surface (§0.7) — raise it, do not inherit the posture by default.

KNOWLEDGE_PIPELINE_TODO.md

@@ -1,274 +1,201 @@
-# Knowledge Pipeline — Task Checklist (draft for review)
+# Knowledge Pipeline — Task Checklist
 
-**Date:** 2026-08-19 · **From:** Rifqi · **For:** Sofhia (review before we agree on next steps)
-**Sources:** `knowledge_pipeline_context.md` (intent) + an audit of the existing `kex` POC
-(`Documents/Dataeyond_catalog/knowledge-pipeline/`, audited 2026-08-19)
+**Date:** 2026-08-19 · **Owner of this doc:** Rifqi · **Shared with:** Sofhia
+**Intent + rationale:** [knowledge_pipeline_context.md](knowledge_pipeline_context.md)
+**Tracked in:** [DEV_PLAN.md](DEV_PLAN.md) §0.8
 **Target repo:** `Agentic-Service-Data-Eyond-Catalog` (Python agentic service)
 
 **Status legend:** ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 built but unverified · ⏸️ deferred
 
-> **Headline: the extraction half is not a greenfield build.** A working POC already exists —
-> 2,763 LOC across all nine stages, 21 passing tests, four experiments run against real Azure
-> calls on the BUMA standard, and a metrics pack. Most of section C below is ✅ or 🔎, not ⬜.
-> The remaining work is **porting, hardening and the seam**, not building the pipeline.
->
-> Nothing here is agreed yet. Please push back on section B (yours) and section A (both of ours).
-
 ---
 
-## 0. Audit summary — what already exists
+## How to read this
+
+The pipeline splits into two independently runnable halves that meet at **one persisted, versioned
+parsed-document artifact**: **parsing** (Sofhia, MinerU) and **knowledge extraction** (Rifqi).
+Extraction never takes a file path — that constraint is the point of the split, because it keeps the
+parser swappable and lets extraction be re-run cheaply on an already-parsed corpus.
+
+**The extraction half has a working prototype (`kex`) that is not the product.** It lives outside
+this repo, is not under version control, and will **not** be ported. It answered the design
+questions; v2 gets rebuilt here against this repo's patterns. That distinction drives the whole
+structure below — §2 disposes of the prototype, §5 builds v2, and §5 deliberately separates *design
+validated* from *implemented here*, because the two are very different amounts of work.
+
+---
 
-`kex` implements the full six-stage flow: `parse → chunk → filters → cluster → rank → extract →
-validate → diff → queue`, driven by an 8-command CLI where every stage reads and writes JSON, so
-extraction can be re-run without re-running GLiNER.
+## 1. What the prototype already proved
 
-**Measured on `STD_2026_006_MNO` (the BUMA standard, 9 pages, run `20260813-145132`):**
+`kex` — ~2,763 LOC, 21 passing tests, 8-command CLI, four experiments against real Azure calls.
+Measured on the BUMA standard `STD_2026_006_MNO` (9 pages, majority Bahasa Indonesia), run
+`20260813-145132`.
 
-| Funnel stage | Count |
-|---|---|
-| Pages | 9 |
-| Chunks (numbered-heading split) | 13 |
-| Mentions (after noise filter; 195 raw) | 169 |
-| Clusters | 66 |
-| Glossary entries | 66 |
-| Review-queue items | 66 |
+**Funnel:** 9 pages → 13 chunks → 169 mentions (195 raw) → 66 clusters → 66 entries → 66 queue items.
 
 | Experiment | Question | Result | Kill line | Verdict |
 |---|---|---|---|---|
 | **E1** | GLiNER recall on Bahasa Indonesia technical prose | **0.854** | 0.70 | **PASS** |
-| **E2** | Does clustering cut review burden? | **2.56×** (169 → 66) | 2.0× | **PASS** |
+| **E2** | Does clustering cut expert review burden? | **2.56×** (169 → 66) | 2.0× | **PASS** |
 | **E3** | Is nano sufficient for schema fill? | **0.75** | 0.80 | **FAIL** |
 | **E4** | Does verbatim-span validation catch fabrication? | **1.00** (false-rejection 0.00) | 0.90 | **PASS** |
 
-Other measured figures: cost **$0.0069** for the 9-page document (155,313 prompt tokens, of which
-**125,184 cached**), 242 s wall clock, 3 fields rejected by span check, 0 definition conflicts,
-7 rule entries, 7 formula entries, 1 brief-context summary.
+Also measured: **$0.0069** for the document · 155,313 prompt tokens of which **125,184 cached**
+(54% hit rate) · 242 s wall clock · 3 fields rejected by span check · 56 of 66 entries returned no
+definition · 0 escalations · 0 definition conflicts.
 
-**The two things that most change our plan:**
+**Two results that settle open questions:**
 
-1. **E1 answers the standing Indonesian risk.** GLiNER multi-v2.1 scores **0.854 recall** on the
-   majority-Indonesian BUMA standard, comfortably above the 0.70 kill line. The context doc lists
-   this as the headline open risk — it is now measured, on the exact document named as the test case.
-2. **E3 settles half of the model-tier decision.** nano scores **0.75 precision against a 0.80 kill
-   line — a FAIL.** Caveats matter: the scoreable base is 8 entries, and nano is not deterministic at
-   `temperature=0` (consecutive identical runs scored 0.75 and 0.625). No mini deployment is
-   provisioned, so the spec's nano-vs-mini A/B has never run and **all four branches currently route
-   to nano**, including `rule` and `summary`, which the spec explicitly wanted on the larger tier.
+- **E1 closes the Indonesian risk.** The context doc names GLiNER's unverified Indonesian
+  performance as the headline unknown. It is now measured, on the exact document named as the test
+  case, comfortably above the kill line.
+- **E3 half-settles the model tier.** nano **fails** at 0.75 against a 0.80 line. Caveats: the
+  scoreable base is 8 entries, and nano is not deterministic at `temperature=0` (consecutive
+  identical runs gave 0.75 and 0.625). No mini deployment exists, so the intended nano-vs-mini A/B
+  has never run and all four branches currently route to nano.
 
-**Caveat on E2 vs. the context doc.** The context doc claims clustering cuts review burden ~6.2×
-(3,125 → ~500 on a 1,000-page corpus). The **measured** figure on 9 pages is **2.56×**. These are
-not in conflict — compression grows with corpus size as terms repeat — but 6.2× is a projection and
-2.56× is the measurement. Quote the right one to the right audience.
+**Caveat when quoting E2.** The context doc's **6.2×** is a 1,000-page projection; **2.56×** is the
+9-page measurement. Compression grows as terms repeat across a corpus. Both are true — quote the
+right one to the right audience.
 
 ---
 
-## A. The seam — shared, and still the first thing to agree
-
-The audit changes this section substantially: we now have a concrete starting point instead of a
-blank page, **and** a concrete conflict to resolve.
-
-`kex/models.py` already defines `ParseResult` / `Chunk`, which is close to the artifact we need —
-persistable, content-hashed, carrying section numbers, page ranges and formatting signals:
-
-```python
-class Chunk(BaseModel):
-    chunk_id: str            # f"{doc_id}#{section_no}#{ordinal}"
-    doc_id: str
-    section_no: str | None   # "2.1.3"
-    heading: str | None
-    text: str
-    page_start: int; page_end: int
-    ordinal: int
-    has_formula: bool; is_tabular: bool
-    bold_spans: list[str]    # feeds the formatting evidence signal
-
-class ParseResult(BaseModel):
-    doc_id: str; source_path: str; content_hash: str
-    n_pages: int; chunks: list[Chunk]
-    used_heading_split: bool
-```
-
-**But the POC's actual seam is the wrong one.** `kex/parse/base.py` defines the boundary as an
-in-process protocol, `Parser.parse(path: Path) -> list[Page]`, where `Page` is `{page_no, text,
-bold_spans}`. That takes a **file path** — the one shape the plan forbids — it is an in-process
-interface rather than a persisted versioned artifact, and its flat `text` field discards exactly
-what MinerU exists to produce.
+## 2. Prototype disposition — do this before building v2
+
+`kex` is three different assets with three different fates. Treating it as one thing is how you
+either port scaffolding you didn't want or discard measurements you can't recreate.
 
 | # | Task | Owner | Status | Description |
 |---|---|---|---|---|
-| A1 | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | **Sofhia circulated a draft `contracts.py` on 2026-08-19** — derived from real MinerU output, backend-agnostic, with a `kind` discriminator (text/table/chart/equation), `source_items` back-references, `images` as paths, and an explicit verbatim-text warning. **Adopt it as the base.** Open items: merge kex's document-level envelope (§A1a), the `text_level` correction (§A1b), the formatting signal, and 0- vs 1-based pages. **Highest-priority item.** |
-| A1a | Add a document-level envelope | Sofhia + Rifqi | ⬜ | The draft is `list[Chunk]` with no wrapper, so there is nowhere to put `content_hash`, `n_pages`, `source_path`, `parser_name`/`parser_version` — or **`version`**, which A3 requires. kex's `ParseResult` wraps exactly this. Parser identity matters specifically because the seam exists to make parsers swappable: when extraction output changes we must know which parser produced the artifact |
-| A1b | Resolve the section-structure question | Rifqi → Sofhia | ⬜ | The draft states MinerU does not supply section structure (`text_level` on 1 of 84 items). **On the BUMA standard it does:** the full 9-page parse carries `text_level` on **29 of 118 items**, cleanly numbered `1.` → `2.2.5`, including `2.1.3. Physical of Availability (PA)`. The draft's sample appears to be the handbook, which genuinely has no numbered headings. Keeping both fields Optional is right; the stated *reason* is not, and acting on it would starve section-aware chunking on exactly the documents that carry numbering |
-| A2 | Move the seam off the file path | Sofhia + Rifqi | ⬜ | Replace `Parser.parse(path) -> list[Page]` as *the contract* with a persisted, versioned `ParseResult`. `list[Page]` may survive as an internal detail of the PyMuPDF fallback; it must stop being the boundary. |
-| A3 | Artifact versioning + storage | Sofhia + Rifqi | ⬜ | Version number, what triggers a re-parse, where it persists. Today `kex` writes `data/interim/*.chunks.json` with a content-hash cache — real but file-based and unversioned. Needs a DB table → Harry-ready DDL handoff (Go owns the schema; Python never runs DDL). |
-| A4 | MinerU → `ParseResult` adapter | Rifqi (provisional) → Sofhia (owns) | ⬜ | Reads a MinerU output folder and emits `ParseResult` **directly, skipping `list[Page]`**. Routing MinerU through `Page.text` flattens typed blocks, LaTeX and table HTML back to a string, then asks the chunker to re-derive by regex what MinerU had already structured. See §B for which files to read. |
-| A5 | Commit fixture artifacts | Rifqi | 🔄 | Both source PDFs are already in `kex/data/raw/` (BUMA standard + Open Pit textbook) and parsed chunk JSON exists in `data/interim/`. Needs promoting to committed fixtures in the target repo. |
-| A6 | Write the seam contract into the repo docs | Rifqi | ⬜ | Extraction never takes a file path; neither half imports the other's internals; parsing stays swappable behind the artifact. So the rule survives us forgetting it — the POC drifted from it already. |
-
-### A1 review notes — Sofhia's `contracts.py` draft (2026-08-19)
-
-**Adopt as the A1 base.** It is derived from real MinerU output rather than designed on paper, and
-three of its choices are better than what kex has: the `kind` discriminator, `source_items`
-(index back-references into `content_list.json`, which kex has no equivalent of), and the explicit
-**verbatim-text warning** — which independently identifies the same failure mode as our span-check
-control, including that it fails *silently* to null and looks like a bad LLM.
-
-| # | Item | Severity | Note |
-|---|---|---|---|
-| R1 | Document-level envelope missing | **High** | See A1a. No home for `version`, and A3 needs one |
-| R2 | Section-structure claim | **High** | See A1b. True for the handbook, not for the BUMA standard |
-| R3 | Formatting signal dropped | Medium | kex's evidence ranking uses six signals — cue · heading · legend · formula · **formatting** · first-occurrence — and reads formatting from `Chunk.bold_spans`. The draft excludes it (rightly, as `text_format` is `vlm`-only). Either derive it from markdown emphasis in `text`, or drop the signal deliberately. It should not disappear by silence |
-| R4 | Page indexing | Medium | Draft is 0-based "as MinerU reports"; kex is 1-based, and page numbers are shown to Mas Beta in the review queue. Off-by-one straight into the expert-facing UI. Pick one and name the field accordingly (`page_idx` if 0-based) |
-| R5 | Chunk granularity unspecified | Medium | The sample chunk `handbook::0000` spans 32 source items across 2 pages. kex caps ~1,500 tokens with paragraph-boundary splitting. Coarse chunks make evidence ranking blunt (top-3 chunks = many tokens per call) and inflate the summary branch's token share, which §5 already names as the quiet cost risk |
-| R6 | `chapter` is lossy across boundaries | Low | `handbook::0000` covers pages [0,1] and is labelled *"THE HORIZONS OF MAINTENANCE MANAGEMENT"*, while `handbook::0001` on page 1 is *"MAINTENANCE STORES AND INVENTORY CONTROL"*. A chunk that crosses a chapter boundary gets one label. Fine if known |
-| R7 | `TermRecord` is thinner than kex's entry | Low | Marked "declared, not yet used", so no conflict — but point it at `kex/models.py` rather than re-deriving. kex additionally carries `interpretation`, `domain`, `company`, `language`, `definition_conflict`, `conflict_variants`, `diff_status`, and a structured `provenance{doc_id, section_no, page, span}` |
-| R8 | dataclass vs pydantic | Low | kex and the target repo are both pydantic v2. A seam without validation is weakly a seam |
-| R9 | **Add `source_wording` now** | Medium | The contract is open, so this is the cheap moment to fix **G1**. With `heading` carried verbatim, a `source_wording` field span-checked against it makes *"Physical of Availability"* survive instead of being silently normalised to *"Physical Availability"* |
+| **P1** | **Freeze the prototype** | Rifqi | ⬜ | It has no `.git`. ~2,763 LOC, both gold sets and all four experiment results exist on exactly one machine. Snapshot it read-only — zip, orphan branch or private repo, whichever is least ceremony. It is an archive, not a codebase to develop |
+| **P2** | **Harvest the tuned constants** | Rifqi | ⬜ | Every number that took runs to find, written down with its reason, **before** v2 code. Known set: rapidfuzz threshold **92**; fuzzy matching **off below 5 characters** (because `PA` and `UA` score highly against each other); evidence **K=3**; the six-signal evidence weight table; the **1024-token prefix pad** (below that threshold OpenAI-family caching does not engage at all — worth the measured 54% hit rate); `labels.yaml` GLiNER label set; `cues.yaml` discourse cues; the 4 prompt files. None of this is in the spec; it lives only as constants and comments, and a clean rewrite will rediscover them as production defects |
+| **P3** | **Migrate the evidence into `eval/knowledge/`** | Rifqi | ⬜ | Gold sets (41 terms + 15 rules) + scorer + the four experiment runners. Follows the existing convention exactly — `eval/{intent,help,readiness,planner}/` each carry `README.md`, `<area>_dataset.json`, `run_eval.py`, and `results/<name>_result_YYYY-MM-DD_HHMMSS.json` files that are never overwritten. **Do this before any v2 pipeline code**: without a standing baseline, "new and improved" is unfalsifiable, and given E3 already failed, quality is exactly what this work will be judged on |
+| **P4** | **Record the negative findings** | Rifqi | ⬜ | Things v2 must not re-derive: nano fails schema fill; escalation is unreachable on small documents (54 of 66 clusters had exactly 1 evidence chunk against K=3, so the loop correctly never fired); 85% of entries returned no definition; conflict detection had nothing to find in a single internally-consistent standard |
 
 ---
 
-## B. Parsing pipeline — **Sofhia**
+## 3. The seam — shared with Sofhia
 
-Unchanged in scope. One addition from the audit: the MinerU output shape is now known, because a
-real MinerU run of the BUMA standard was inspected (`Downloads/mineru_output_results/…/hybrid_auto/`).
+**Status: under active discussion.** Sofhia circulated a draft `contracts.py` on 2026-08-19,
+derived from real MinerU output rather than designed on paper, and it is adopted as the base. Three
+of its choices improve on the prototype: a `kind` discriminator (text/table/chart/equation),
+`source_items` index back-references into MinerU's `content_list.json`, and an explicit
+verbatim-text warning that independently identifies the same failure mode as the span-check control
+— including that it fails *silently to null* and looks like a bad LLM.
 
-**Which files the adapter should read:**
+| # | Task | Owner | Status | Description |
+|---|---|---|---|---|
+| **S1** | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | Under discussion. Everything else in this section is an open item against her draft |
+| **S2** | Add a document-level envelope | Sofhia + Rifqi | 🔄 | The draft is `list[Chunk]` with no wrapper, so there is nowhere for `content_hash`, `n_pages`, `source_path`, or **`version`** — and a versioned artifact is a plan requirement. Proposed: `ParsedDocument{doc_id, source_path, content_hash, version, n_pages, parser_name, parser_version, chunks}`. Parser identity matters *because* the seam exists to make parsers swappable: when extraction output shifts, we must be able to tell a parser change from a prompt change |
+| **S3** | Correct the section-structure claim | Rifqi → Sofhia | 🔄 | The draft states MinerU does not supply section structure (`text_level` on 1 of 84 items). **On the BUMA standard it does:** the full 9-page parse carries `text_level` on **29 of 118 items**, cleanly numbered `1.` → `2.2.5`, including `2.1.3. Physical of Availability (PA)`. Her sample appears to be the handbook (its `doc_id` is `handbook`, and it has item types `page_number`/`header` absent from the BUMA parse), which genuinely has no numbered headings. Keeping both fields Optional is correct; the stated *reason* is not, and acting on it would starve section-aware chunking on exactly the documents that carry numbering — i.e. every client standard |
+| **S4** | Add `source_wording` while the contract is open | Rifqi → Sofhia | 🔄 | Cheap moment to fix the defect in §7. With `heading` carried verbatim, a `source_wording` field span-checked against it makes *"Physical **of** Availability"* survive instead of being silently normalised |
+| **S5** | Keep the seam off the file path | Sofhia + Rifqi | ⬜ | The prototype's boundary was `Parser.parse(path) -> list[Page]` — the one shape the plan forbids. v2 must not reproduce it. Must be settled **before** the MinerU parser is built against it, or it becomes a rewrite of Sofhia's work |
+| **S6** | Minor contract items (deferred to review) | Rifqi → Sofhia | ⏸️ | Raised but not blocking: **(a)** the formatting evidence signal — ranking uses six signals including formatting, read from `bold_spans`; the draft rightly excludes it as backend-specific, so either derive it from markdown emphasis or drop it deliberately, not by silence. **(b)** page indexing — draft is 0-based "as MinerU reports", prototype is 1-based, and page numbers are shown to the expert; name it `page_idx` if 0-based. **(c)** chunk-size policy — the sample chunk spans 32 source items across 2 pages; the prototype capped ~1,500 tokens. **(d)** `chapter` is lossy when a chunk crosses a chapter boundary. **(e)** pydantic over dataclass, matching this repo. **(f)** point `TermRecord` at the prototype's richer entry rather than re-deriving it |
+| **S7** | Commit fixture artifacts | Rifqi | ⬜ | Both source PDFs exist (BUMA standard + Open Pit textbook). Promote a parsed artifact of each into the target repo as committed fixtures, so the extraction half is testable **with MinerU not installed at all** — the seam doing its job |
+| **S8** | Write the seam rule into the repo docs | Rifqi | ⬜ | Extraction never takes a file path; neither half imports the other's internals; parsing stays swappable behind the artifact. The prototype drifted from this, so the rule needs to outlive our memory of it |
+
+---
+
+## 4. Parsing pipeline — Sofhia
 
-| File | Use | Why |
-|---|---|---|
-| `*_content_list_v2.json` | **Primary** | Page-grouped; richest taxonomy: `paragraph` · `title` (with `level`) · `equation_interline` (LaTeX + image ref) · `table` (HTML + image ref) |
-| `*_content_list.json` (v1) | **Secondary** | Flat, but carries explicit `page_idx` and `text_level`. v2 encodes page only as outer list position — cross-check the two |
-| `*_middle.json` | Later, if needed | Line/span-level bbox. Only if the review UI wants to highlight a quote on the page image |
-| `images/` | **Reference only** | Hash-named crops. The LLM stage is text-only — the artifact carries paths, never bytes |
-| `*.md` | **Do not consume** | A rendering. Loses `page_idx` and block-type boundaries; its image paths couple us to the folder layout |
-| `*_model.json`, `*_layout.pdf`, `*_span.pdf`, `*_origin.pdf` | Ignore | Raw model output and debug visualisations |
+MinerU replaces the current text+table-only extraction: it adds formula, image, chart and diagram
+detection with type labels. Presented to the execs 13 Aug as the prerequisite for Scenario 2.
 
 | # | Task | Owner | Status | Description |
 |---|---|---|---|---|
-| B1 | MinerU cost justification (written) | Sofhia | ⬜ | Owed for the budget proposal, from the 13 Aug exec review. |
-| B2 | MinerU install + run on both sample docs | Sofhia | ✅ | **Complete 9-page parse delivered 2026-08-19**: 118 items (90 text, 14 table, 11 equation, 3 chart) across pages 0–8. The rule-of-thumb grounding case is present (Other Activity, Controllable/Uncontrollable, Standby), so the fixture now exercises all four branches. Supersedes the earlier 5-page run. |
-| B3 | Evaluate MinerU quality vs. the current path | Sofhia | ⬜ | Compare specifically on tables, the p.2 legend block, formulas and figures — against the PyMuPDF path the POC uses today, which is a real baseline with real numbers, not a guess. |
-| B4 | Indonesian check on **parsing** | Sofhia | ⬜ | Distinct from E1, which measured GLiNER, not the parser. Confirm MinerU's text/layout quality holds on Bahasa Indonesia. |
-| B5 | Section-aware chunking | Sofhia | 🔎 | **Already built and working** for the PyMuPDF path (`chunk/section_chunker.py`, 244 LOC, tested): numbered-heading regex, breadcrumb-header dedupe, ~1,500-token cap, page-boundary fallback. On the BUMA standard it produced 13 chunks with `used_heading_split=True`. The open question is whether MinerU's typed blocks should feed it or replace it. |
-| B6 | Emit the agreed artifact (A1) | Sofhia | ⬜ | The parser writes our schema, not MinerU's native shape. This is what keeps MinerU swappable. Currently a `NotImplementedError` stub. |
-| B7 | Fallback path behind the same artifact | Sofhia | 🔎 | PyMuPDF already works end-to-end and is the POC default. It needs re-pointing at `ParseResult` as the contract rather than `list[Page]`. |
-| B8 | Parsing subsystem is additive + gated | Sofhia | ⬜ | The existing unstructured path (OCR → chunk → pgvector) in the target repo stays untouched. MinerU is a new flag-gated path beside it. |
+| **B1** | MinerU cost justification (written) | Sofhia | ⬜ | Owed for the budget proposal, from the 13 Aug exec review |
+| **B2** | MinerU install + run on the sample docs | Sofhia | ✅ | Complete 9-page parse of the BUMA standard delivered 2026-08-19: **118 items** (90 text, 14 table, 11 equation, 3 chart), pages 0–8. Includes the rule-of-thumb grounding content (Other Activity, Controllable/Uncontrollable, Standby), so the fixture exercises all four extraction branches. Supersedes an earlier 5-page run |
+| **B3** | Evaluate MinerU quality vs. the current path | Sofhia | ⬜ | Compare on tables, the p.2 legend block, formulas and figures — against the PyMuPDF baseline the prototype used, which has real numbers rather than a guess |
+| **B4** | Indonesian check on **parsing** | Sofhia | ⬜ | Distinct from E1, which measured GLiNER, not the parser. Confirm MinerU's text and layout quality hold on Bahasa Indonesia |
+| **B5** | Section-aware chunking | Sofhia | ⬜ | Split on numbered headings rather than fixed token windows. Acceptance test: the NSR definition in the Open Pit textbook (intro p.92, formula p.93, remark p.94) stays inside one semantic unit. The prototype's chunker solved this for the PyMuPDF path and is worth reading before rebuilding — it already handles the breadcrumb problem below |
+| **B6** | Emit the agreed artifact | Sofhia | ⬜ | The parser writes our schema, not MinerU's native shape. This is what keeps MinerU swappable. Gated on §3 |
+| **B7** | Fallback path behind the same artifact | Sofhia | ⬜ | Tesseract / Azure Document Intelligence emits the *same* artifact, so extraction never knows which parser ran |
+| **B8** | Keep the parsing subsystem additive + gated | Sofhia | ⬜ | The existing unstructured path (`src/knowledge/`, Tesseract OCR → chunk → pgvector) stays untouched. MinerU is a new flag-gated path beside it |
 
 **Findings from the real MinerU output that affect B5/B6:**
 
 - **Heading depth is flattened** — all 29 `title` blocks carry `"level": 2`, including `1. TUJUAN
   PARAMETER` and `2.1.3. Physical of Availability (PA)`. Hierarchy must come from the numbering
-  text, never from `level`. But the heading *text itself* is supplied — see A1b.
+  text, never from `level`. The heading *text* is supplied (see S3).
 - **Equation LaTeX is character-spaced** — `P r o d u c t i o n = M O H H \times Q t y \times P A
   \times U A \times P t y`. Naive matching will never find `MOHH`. Needs normalising before the
-  formula/legend branch and before span validation can locate anything inside a formula.
+  formula/legend branch, and before span validation can locate anything inside a formula.
 - **Breadcrumb headings repeat per page** — `2. PENJELASAN PARAMETER` / `2.1. Production Parameter`
-  re-appear as `title` blocks on pages 2–8. kex's chunker already dedupes these
-  (`_boilerplate_keys`); whatever emits the artifact must not treat them as new sections.
+  re-appear as `title` blocks on pages 2–8. They must not be treated as new sections.
 - **Charts carry captions only** — all 3 chart items have `content: ""`, with the caption in
-  `chart_caption` (e.g. *"Gambar 2.1 Analisis Gain/Loss…"*). A chart chunk therefore contributes
-  a caption plus an image reference and nothing else to extraction. Expected, but it means chart
-  detection buys Scenario 2, not glossary coverage.
+  `chart_caption`. A chart chunk contributes a caption plus an image reference and nothing else.
+  Chart detection buys Scenario 2, not glossary coverage.
 
 ---
 
-## C. Extraction pipeline — **Rifqi**
+## 5. Extraction pipeline v2 — Rifqi, built in this repo
 
-Statuses below are **verified against the code and the run artifacts**, not estimated.
+**Two separate columns, deliberately.** *Design* = validated by the prototype, with a number behind
+it. *Build* = exists in this repo. The gap between them is the remaining engineering effort, and
+collapsing the two is how a rebuild gets under-estimated.
 
-### C-1. Built and measured
-
-| # | Task | Status | Evidence |
-|---|---|---|---|
-| C1 | Candidate-entry schema | ✅ | `models.py` (222 LOC) encodes all three invariants: content fields Optional, `SubdomainEnum`, `Provenance.span` mandatory |
-| C2 | Offline runner over a fixture | ✅ | 8-command CLI; every stage reads/writes JSON; content-hash cache; `--dry-run` prints exact prompts + token estimate with zero network calls |
-| C3 | GLiNER span filter | ✅ | `filters/gliner_filter.py`, `urchade/gliner_multi-v2.1`, CPU, every chunk, labels in YAML |
-| C4 | **Indonesian performance check** | ✅ | **E1 = 0.854 recall** vs 0.70 kill line, on the BUMA standard, 41-term gold set. The context doc's headline open risk, now measured |
-| C5 | Discourse-cue regex → rule-of-thumb | ✅ | `filters/cue_filter.py`; cues in `config/cues.yaml`; 7 rule entries produced |
-| C6 | Legend-block regex → formula vars | ✅ | `filters/legend_filter.py` + a ≥0.95-precision test; 7 formula entries produced |
-| C7 | Section pass → summary units | ✅ | `extract/summary.py`; produced `brief_context.json` + `.md` |
-| C8 | Normalise + cluster mentions | ✅ | `cluster/` — exact → abbrev → rapidfuzz ≥92; fuzzy off below 5 chars because `PA`/`UA` score highly against each other. **E2 = 2.56×** |
-| C9 | Evidence ranking → top-K | ✅ | `rank/evidence.py`, weight table, K=3, full ranked list retained for escalation |
-| C10 | LLM extraction call | ✅ | `extract/azure_client.py` + 4 branch modules; structured outputs via `json_schema`; prompts as files; prefix padded past 1024 tokens for caching (**54% hit rate measured**) |
-| C11 | Verbatim-span validation | ✅ | `validate/span_check.py`; never repairs a failed span. **E4 = 1.00 rejection, 0.00 false-rejection** |
-| C16 | Frequency-sorted review queue | ✅ | `queue/review_queue.py`; conflicts promoted above frequency. Queue top: Qty(20) · PA(16) · PTY(12) · UA(11) — exactly the terms that carry the document |
-
-### C-2. Built but not exercised — needs verification before we trust it
-
-| # | Task | Status | What the audit found |
-|---|---|---|---|
-| C12 | Null-definition escalation | 🔎 | Implemented and wired, but **`escalated = 0`** across the whole run. Not a bug: **54 of 66 clusters have exactly 1 evidence chunk** and K=3, so `rounds_available()` correctly returns 0 — the document is too small to reach the path. Unexercised until a larger corpus runs |
-| C13 | Conflict detection | 🔎 | `validate/conflict.py` wired, but **0 conflicts** observed — a single internally-consistent standard gives it nothing to find. Untested against real disagreement |
-| C15 | Diff vs. active glossary version | 🔎 | `diff/glossary_diff.py` is wired, but `write_artifacts` reads `existing` from `out/glossary.json` — **the same file it then overwrites**. All 66 entries came back `new`. Only the empty-baseline path has ever run; there is no version history, no active-version pointer, and no approval state |
+**Structural target:** follow `src/query/` (typed models → validator → deterministic stages →
+service facade → thin entry points), with the offline runner in `scripts/`. **Naming collision:**
+`src/knowledge/` is already taken by the existing OCR→pgvector service — v2 needs its own module
+name (`src/kex/`, `src/knowledge_extraction/`, TBD in §6).
 
-### C-3. Not built
-
-| # | Task | Status | Description |
-|---|---|---|---|
-| C14 | **Literal-source-wording behaviour** | ⛔ | **Currently violated — see §G1.** |
-| C17 | Bulk vs. incremental ingest modes | ⬜ | Single-document CLI only. Neither corpus-frequency boosting (bulk) nor active-version diffing (incremental) exists |
-| C18 | Candidate-entry persistence | ⬜ | Everything is JSON files in `out/`. Needs tables → Harry-ready DDL handoff, same rule as A3 |
-| C19 | Port `kex` into the target repo | ⬜ | 2,763 LOC, Python 3.13 + its own venv vs. the target repo's 3.12; new deps (GLiNER + torch). Sequencing and gating to be decided |
-
----
-
-## D. Decisions needed
-
-| # | Decision | Owner | Status | Description |
+| # | Stage | Design | Build | Note |
 |---|---|---|---|---|
-| D1 | New dependencies | Rifqi (asks) | ⬜ | GLiNER + torch (~heavy) on extraction, MinerU on parsing. `pyproject.toml` changes in the target repo need sign-off. Note the POC needs **torch ≥ 2.6** and hit a corrupt-wheel failure worth not re-debugging |
-| D2 | New tables (DDL handoff) | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. One consolidated handoff beats three |
-| D3 | Model tier per branch | Rifqi | ⛔ | **Half-answered: nano FAILS at 0.75 vs the 0.80 line.** Blocked on a **mini deployment being provisioned** — until then all four branches run on nano, including `rule` and `summary`, which the spec never intended for it, and the A/B cannot run |
-| D4 | Admin ingest authentication | Rifqi | ⬜ | A different risk class from the read-only chat surface. Raise it rather than assume — and only after the offline path is ported |
-| D5 | GLiNER + LangExtract ensembling | — | ⏸️ | Proposed, not decided; flagged as scope creep. Not in scope for the test integration |
-| D6 | **Expert review of the gold set** | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is **self-bootstrapped and provisional**. E1 and E3 both move when it is reviewed. E3's scoreable base is 8 entries — each adjudication moves the number ~12 points. Every failure is listed verbatim in `eval/out/report.md` for him |
+| **X1** | Candidate-entry schema | ✅ | ⬜ | Three invariants to re-encode: all content fields Optional (a required field is a fabrication invitation), `subdomain_tags` as an enum, `provenance.span` mandatory. Pydantic v2 |
+| **X2** | Offline runner over a fixture | ✅ | ⬜ | Script-driven run is the honest first milestone; HTTP is a convenience layer over it, never the reverse. Every stage reads/writes its own artifact so extraction re-runs without re-running GLiNER |
+| **X3** | GLiNER span filter | ✅ | ⬜ | Runs on **all** chunks unconditionally. No standalone relevance gate — "zero candidate spans" is the relevance signal, produced as a byproduct of looking rather than a judgment made before looking |
+| **X4** | Indonesian performance | ✅ **0.854** | n/a | Answered by E1. Re-measure once on v2 to confirm no regression |
+| **X5** | Discourse-cue filter → rule-of-thumb | ✅ | ⬜ | Free CPU pass. Grounding case: Other Activity + the controllability hierarchy |
+| **X6** | Legend-block filter → formula vars | ✅ | ⬜ | Grounding case: the MOHH/Qty/PA/UA/Pty legend block. Needs the LaTeX normalisation noted in §4 |
+| **X7** | Section pass → summary units | ✅ | ⬜ | The quiet cost risk: few calls but ~¼ of all input tokens, because summarisation cannot be filtered — it needs whole documents |
+| **X8** | Normalise + cluster mentions | ✅ **2.56×** | ⬜ | The LLM call unit is the **term cluster**. Carry the tuned constants from P2 |
+| **X9** | Evidence ranking → top-K | ✅ | ⬜ | Six signals: cue · heading · legend · formula · formatting · first-occurrence. No term is ever dropped — ranking narrows *evidence*, not terms, and the full ranked list is retained for escalation |
+| **X10** | LLM extraction call | ✅ | ⬜ | One call per term cluster, plus one per cue chunk, per unique formula, per document. Keep prompts as files with a byte-identical cached prefix |
+| **X11** | Verbatim-span validation | ✅ **1.00** | ⬜ | A field whose span cannot be located is rejected, **never repaired** — a repaired span is an unfalsifiable claim, which is what the control exists to prevent |
+| **X12** | Null-definition escalation | 🔎 | ⬜ | Built in the prototype but **never fired**: 54 of 66 clusters had exactly 1 evidence chunk against K=3, so the loop correctly had nowhere to escalate. Unverified until a larger corpus runs |
+| **X13** | Conflict detection | 🔎 | ⬜ | Built but **0 conflicts found** — one internally-consistent standard gives it nothing. Untested against real disagreement. Note it only works *because* all evidence for a term arrives in one call |
+| **X14** | Diff vs. active glossary version | 🔎 | ⬜ | The prototype diffed against the file it then overwrote, so only the empty-baseline path ever ran and every entry came back `new`. v2 needs a real active-version pointer |
+| **X15** | Frequency-sorted review queue | ✅ | ⬜ | Sorted by mention count, conflicts promoted above frequency. Prototype queue topped out Qty(20) · PA(16) · PTY(12) · UA(11) — exactly the terms carrying the document |
+| **X16** | Bulk vs. incremental ingest | ⬜ | ⬜ | Bulk setup ingest (corpus-frequency statistics as a candidate booster) and incremental single-file add (no corpus context; diff against the active version). Neither exists |
+| **X17** | Persistence | ⬜ | ⬜ | Prototype wrote JSON files. v2 needs tables → see D2 |
 
 ---
 
-## E. Outside our two halves
+## 6. Decisions needed
 
-| # | Item | Owner | Status | Description |
+| # | Decision | Owner | Status | Description |
 |---|---|---|---|---|
-| E1 | Curation UI + expert review flow | Mas Har / Mas Beta | ⬜ | Our pipeline's job ends at the queue. `out/review_queue.json` (66 items, ranked, each with page, section, span and a `review_reason`) is a ready-made input contract |
-| E2 | Versioning + approval audit trail (product side) | Mas Har | ⬜ | The expert chooses which version is active. Pairs with C15/A3 |
-| E3 | MCP server — the consumption surface | — | ⬜ | The POC already emits three of the four target artifacts: `glossary.json`, `interpretation_pack.json`, `formulas.json`, `brief_context.json`. Only the data dictionary (DB profiling) is out of scope for extraction |
+| **D1** | New dependencies | Rifqi (asks) | ⬜ | GLiNER + torch (heavy) for extraction; MinerU for parsing. `pyproject.toml` changes need sign-off. Prototype needed **torch ≥ 2.6** and hit a corrupt-wheel failure worth not re-debugging |
+| **D2** | New tables (DDL handoff) | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns the dedorch schema — Python never executes DDL. One consolidated Harry-ready handoff beats three |
+| **D3** | Model tier per branch | Rifqi | ⛔ | **Half-answered: nano FAILS at 0.75 vs 0.80.** Blocked on a **mini deployment being provisioned**. Until then all four branches run on nano — including rule and summary, whose failure mode is least detectable, since a plausible summary cannot be span-checked |
+| **D4** | Admin ingest surface | Rifqi | ⬜ | Whether it is HTTP at all, and if so its auth posture. A write surface triggered by an admin is a different risk class from the current unauthenticated read-only chat surface — raise it, don't inherit the posture by default. Only after the offline path works |
+| **D5** | v2 module name | Rifqi | ⬜ | `src/knowledge/` is taken by the existing OCR→pgvector service. Pick a non-colliding name before the first commit |
+| **D6** | Expert review of the gold set | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is self-bootstrapped and provisional. E1 and E3 both move when it is reviewed, and E3's scoreable base is 8 entries — each adjudication shifts it ~12 points. Now *more* important than before: it is the baseline v2 gets measured against |
+| **D7** | GLiNER + LangExtract ensembling | — | ⏸️ | Proposed, not decided; flagged as scope creep. Cheap in API terms but adds a reconciliation step between two candidate sets that may disagree. Not in scope for the test integration |
 
 ---
 
-## F. Already settled — please do not reopen
-
-1. The LLM call unit is the **term cluster**. Justification is review-burden reduction, **not** tokens.
-2. **No standalone relevance gate.** "Zero candidate spans" is the relevance signal.
-3. **`provenance.span` is mandatory and verbatim.** Never repaired — a repaired span is an unfalsifiable claim.
-4. **All content fields Optional.** Abstention is correct behaviour, not an error.
-5. **API cost is negligible; latency irrelevant.** Measured: $0.0069 for 9 pages. Optimise the review experience, not tokens.
-6. **This is a test integration, not a productionisation.** Both halves are new, gated subsystems.
-7. **Nothing is ever dropped at the term level.** Evidence ranking narrows *evidence*, not terms.
-
----
-
-## G. Defects and risks found by the audit
+## 7. Defects and risks to carry into v2
 
 | # | Finding | Severity | Detail |
 |---|---|---|---|
-| **G1** | **Literal source wording is being normalised away** | **High** | Decision §4.10 says the pipeline must record what the document actually says and surface the discrepancy. It currently does not. The BUMA standard's heading reads **"2.1.3. Physical of Availability (PA)"**; the extracted entry carries `full_name: "Physical Availability"` — silently corrected. Mechanism: `Chunk.heading` is a separate field from `Chunk.text`, and the chunk text does not contain the phrase at all, so the literal wording never reaches the model. Fix shape: feed the heading verbatim, and/or add a `source_wording` field span-checked against the heading |
-| **G2** | **The POC is not under version control** | **High** | No `.git`. 2,763 LOC, four experiment results, a metrics pack and both gold sets exist only on this machine |
-| **G3** | **85% abstention rate** | Medium | 56 of 66 entries have **no definition**; only 10 carry one. Correct behaviour per the abstention rule, but it means the review queue is mostly "term found, no definition in document". Whether that is useful to Mas Beta or noise is a **review-experience question to put to him** — and per §5 of the context doc, review experience is where engineering time is worth spending |
-| **G4** | **The seam drifted from the plan** | Medium | `Parser.parse(path) -> list[Page]` is the exact shape the context doc forbids. Cheap to fix now, expensive once MinerU is built against it — hence A2 before B6 |
-| **G5** | **Everything generalises from one 9-page document** | Medium | Single document, single language, single company. E2's compression, E3's scoreable base and the escalation path are all constrained by it |
-| **G6** | **`rule` and `summary` run on the wrong tier** | Medium | Both route to nano because it is the only provisioned deployment. These are the two branches whose failure mode is least detectable — a plausible summary cannot be span-checked. Same blocker as D3 |
+| **R1** | **Literal source wording is normalised away** | **High** | A locked decision says the pipeline records what the document actually says and surfaces the discrepancy to the expert. It does not. The BUMA heading reads **"2.1.3. Physical of Availability (PA)"**; the extracted entry carries `full_name: "Physical Availability"` — silently corrected. Mechanism: the heading is a separate field from the chunk text, and the chunk text does not contain the phrase, so the literal wording never reaches the model. Fix shape: feed the heading verbatim + add the span-checked `source_wording` field (S4) |
+| **R2** | **Prototype is not under version control** | **High** | Addressed by P1 |
+| **R3** | **85% abstention** | Medium | 56 of 66 entries carry no definition. Correct behaviour — for a term the document never defines, null *is* the right answer — but it means the review queue is mostly "term found, no definition in document". Whether that is useful to Mas Beta or noise is a **review-experience question to put to him**, and review experience is explicitly where engineering time is worth spending |
+| **R4** | **Everything generalises from one 9-page document** | Medium | Single document, single language, single company. E2's compression, E3's scoreable base and the escalation path are all constrained by it |
+| **R5** | **Rule and summary run on the wrong tier** | Medium | Both route to nano because it is the only provisioned deployment. Same blocker as D3 |
+| **R6** | **Rewrites drop accidental knowledge** | Medium | Addressed by P2. The `PA`/`UA` fuzzy-collision guard and the 1024-token caching threshold are the known examples; there will be others |
 
 ---
 
-## Proposed sequence
-
-1. **G2 first, today** — put `kex` under version control. Everything below risks a machine failure until this is done.
-2. **Close A1 on Sofhia's draft** — reply with R1–R9, settle A1a/A1b, and let her push the agreed version to the catalog repo as the module of record. **A2** (move the seam off the file path) rides along with it: doing A2 *before* she builds B6 is the whole point; afterwards it is a rewrite of her work.
-3. **Then split.** Sofhia takes B1–B4 (including a complete 9-page re-parse); Rifqi takes G1 (the literal-wording defect), then A4 (provisional MinerU adapter) and C15/A3 (real versioning behind the diff).
-4. **Reconverge at B6.** Sofhia's parser emits the agreed artifact and replaces the provisional adapter. If A1 was right, nothing in the extraction half changes — that is the test of whether we did A1 properly.
-5. **In parallel, unblock the two ⛔s:** D3 (provision a mini deployment, then run the A/B) and D6 (get the gold set in front of Mas Beta). Both gate quality claims, neither gates engineering.
-6. **Then** C17, C18, C19 and D1/D2/D4 — the port into the target repo.
-
-Rationale: this ordering front-loads the two things that are expensive to get wrong later — the seam
-and the version-control gap — and it keeps the four open decisions in section D off both of our
-critical paths.
+## 8. Settled — do not reopen
+
+1. The LLM call unit is the **term cluster**. Justification is review-burden reduction, **not** tokens (~$0.45 saved is trivial).
+2. **Conflict detection depends on the clustering** — two contradictory definitions can only be spotted because all evidence for a term arrives in one call.
+3. **No standalone relevance gate.** A pre-filter's false negatives drop content before extraction sees it, and dropped content never reaches review to be caught.
+4. **Evidence ranking is allowed** even though it is also a filter — no term is dropped, all mentions stay in provenance, and the escalation loop self-corrects. Preserve those three properties if it is ever revisited.
+5. **`provenance.span` is mandatory and verbatim**, never repaired. The primary anti-hallucination control, and what makes expert review finishable: the reviewer checks a quote against a page, not a claim against their memory.
+6. **All content fields Optional.** Abstention is correct behaviour, not an error.
+7. **Frequency-sorted review queue** — validate the ~40 terms carrying the document first, let the long tail wait.
+8. **API cost is negligible, latency irrelevant** (measured $0.0069 for 9 pages). This runs a few times a year, admin-triggered, not continuous and not per-user. Optimise the review experience, never tokens. Prefer boring, inspectable stages.
+9. **The pipeline records the literal source wording** and surfaces discrepancies rather than normalising them (currently violated — R1).
+10. **This is a test integration, not a productionisation.** Both halves are new, gated subsystems alongside the existing document/catalog pipelines — not a rewrite of them.