[NOTICKET] docs: knowledge pipeline test integration — context, audited task checklist, DEV_PLAN §0.8

Planning-only. No code, dependency, table or endpoint added to this repo.

- knowledge_pipeline_context.md: intent, ownership split, the six stages, and the
settled decisions (term-cluster call unit, no relevance gate, mandatory verbatim
provenance span, all content fields Optional, frequency-sorted review queue).
- KNOWLEDGE_PIPELINE_TODO.md: task checklist with owners, for review with Sofhia.
Statuses verified against the existing kex POC and its run artifacts, not estimated.
- DEV_PLAN.md: new §0.8 tracking the work in the usual place, plus two §6 open
decisions (where kex lives; the admin ingest surface's auth posture).

Audit finding that shapes the plan: the extraction half is not greenfield. A working
POC (~2,763 LOC, 21 passing tests, four experiments against real Azure calls) exists
outside version control. Measured on the BUMA standard: GLiNER Indonesian recall 0.854
(PASS, closing the context doc's headline open risk), clustering 2.56x (PASS), nano
schema-fill 0.75 (FAIL against a 0.80 line), span validation 1.00 (PASS).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DEV_PLAN.md

@@ -273,6 +273,45 @@ by design** until #43 (Go-forwarded identity). Not a gap to re-raise — it is a
 CORS was left at `["*"]` on purpose (tightening it needs the FE origin as config, which we chose
 not to set for now).
 
+## 0.8. Knowledge pipeline — test integration (planning, 2026-08-19)
+
+**Nothing has landed in this repo.** This section exists so the in-flight work is tracked in the
+usual place; no code, dependency, table or endpoint has been added here yet. Do not read any of it
+into REPO_STATUS.
+
+Context and intent: [knowledge_pipeline_context.md](knowledge_pipeline_context.md). Task breakdown,
+owners and verified statuses: [KNOWLEDGE_PIPELINE_TODO.md](KNOWLEDGE_PIPELINE_TODO.md).
+
+**Shape.** A six-stage pipeline (parse → free CPU/regex filter → one LLM call per *term cluster* →
+deterministic span validation → diff vs. the active glossary → frequency-sorted expert review) that
+turns client documents into reviewable knowledge entries, ultimately served over MCP. It splits into
+two independently runnable halves — **parsing (Sofhia, MinerU)** and **knowledge extraction
+(Rifqi)** — meeting only at a **persisted, versioned parsed-document artifact**. Extraction never
+takes a file path; that constraint is the whole point of the split.
+
+**Where it stands (audited 2026-08-19).** The extraction half is **not greenfield**: a working POC
+(`kex`, ~2,763 LOC, 21 passing tests, 8-command CLI, four experiments against real Azure calls)
+already exists outside version control at `Documents/Dataeyond_catalog/knowledge-pipeline/`.
+Measured on the BUMA standard (9 pages): GLiNER Indonesian recall **0.854** (kill line 0.70, PASS —
+this closes the context doc's headline open risk), clustering compression **2.56×** (kill 2.0, PASS),
+nano schema-fill precision **0.75** (kill 0.80, **FAIL**), span-validation rejection **1.00** with
+0.00 false rejections (kill 0.90, PASS). Cost **$0.0069** for the document.
+
+| # | Task | Owner | Status | Note |
+|---|---|---|---|---|
+| K1 | Put `kex` under version control | Rifqi | ⬜ | No `.git`. ~2,763 LOC + both gold sets + all four experiment results exist on one machine. Highest-priority item; repo home undecided |
+| K2 | Agree the parsed-artifact schema (A1) | Sofhia + Rifqi | 🔄 | Sofhia circulated a draft `contracts.py` 2026-08-19, derived from real MinerU output. Adopted as the base; open items are the document-level envelope (no home for `version`), the section-structure claim, and `source_wording` |
+| K3 | Move the seam off the file path | Sofhia + Rifqi | ⬜ | The POC's boundary is `Parser.parse(path) -> list[Page]` — the one shape the plan forbids. Must land **before** the MinerU parser is built against it |
+| K4 | Artifact + candidate-entry persistence | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns dedorch DDL → one consolidated Harry-ready handoff, never DDL from Python (§2.2) |
+| K5 | Fix the literal-source-wording violation | Rifqi | ⬜ | A locked decision is currently violated: the BUMA heading reads "Physical **of** Availability (PA)"; the extracted entry carries "Physical Availability". The discrepancy is meant to reach the expert, not be silently corrected |
+| K6 | Provision a mini deployment | Rifqi | ⛔ | Only nano is provisioned, so all four branches route there — including `rule` and `summary`, whose failure mode is least detectable (a plausible summary cannot be span-checked). Blocks the nano-vs-mini A/B |
+| K7 | Expert review of the gold set | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is self-bootstrapped and provisional. Both the recall and precision figures move when it is reviewed |
+| K8 | Port `kex` into this repo | Rifqi | ⬜ | Python 3.13 → 3.12, new deps (GLiNER + torch), gating, and an admin ingest surface whose auth question is a different risk class from the read-only chat surface. **Gated on §6 sign-off — deps (§6.4) and any new endpoint (§6.3).** |
+
+**Not decided, do not assume:** whether `kex` becomes a subdirectory here, a separate repo, or a
+branch; whether the admin ingest surface is HTTP at all (the offline script path is the honest
+first milestone); and GLiNER + LangExtract ensembling stays ⏸️ flagged as scope creep.
+
 ## 5. Critical path & sequencing
 
 - **Critical path:** ~~#22 (send Harry the `report_inputs` schema)~~ **✅ resolved 2026-07-22** — now **#32** (`message_traceability` + `message_charts` DDL to Harry) and **#31** (non-convergent migration set). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
@@ -297,3 +336,8 @@ not to set for now).
 - **W3 re-open timing** (deferred `analyze_*` activation) — Rifqi (deferred at the 2026-07-13 approval).
 - **INV-6 relaxation for S1b targeted repair** — team, only after S1a `repair_candidate` telemetry
   shows a meaningful hit-rate (SPINE_V2_PLAN §6).
+- **Knowledge pipeline — where `kex` lives** (§0.8 K1/K8): subdirectory of this repo, separate repo,
+  or its own branch. Blocks version control, which blocks everything else in §0.8.
+- **Knowledge pipeline — admin ingest surface** (§0.8 K8): whether it is HTTP at all, and if so its
+  auth posture. A write surface triggered by an admin is a different risk class from the current
+  unauthenticated read-only chat surface (§0.7) — raise it, do not inherit the posture by default.

KNOWLEDGE_PIPELINE_TODO.md

@@ -0,0 +1,274 @@
+# Knowledge Pipeline — Task Checklist (draft for review)
+
+**Date:** 2026-08-19 · **From:** Rifqi · **For:** Sofhia (review before we agree on next steps)
+**Sources:** `knowledge_pipeline_context.md` (intent) + an audit of the existing `kex` POC
+(`Documents/Dataeyond_catalog/knowledge-pipeline/`, audited 2026-08-19)
+**Target repo:** `Agentic-Service-Data-Eyond-Catalog` (Python agentic service)
+
+**Status legend:** ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 built but unverified · ⏸️ deferred
+
+> **Headline: the extraction half is not a greenfield build.** A working POC already exists —
+> 2,763 LOC across all nine stages, 21 passing tests, four experiments run against real Azure
+> calls on the BUMA standard, and a metrics pack. Most of section C below is ✅ or 🔎, not ⬜.
+> The remaining work is **porting, hardening and the seam**, not building the pipeline.
+>
+> Nothing here is agreed yet. Please push back on section B (yours) and section A (both of ours).
+
+---
+
+## 0. Audit summary — what already exists
+
+`kex` implements the full six-stage flow: `parse → chunk → filters → cluster → rank → extract →
+validate → diff → queue`, driven by an 8-command CLI where every stage reads and writes JSON, so
+extraction can be re-run without re-running GLiNER.
+
+**Measured on `STD_2026_006_MNO` (the BUMA standard, 9 pages, run `20260813-145132`):**
+
+| Funnel stage | Count |
+|---|---|
+| Pages | 9 |
+| Chunks (numbered-heading split) | 13 |
+| Mentions (after noise filter; 195 raw) | 169 |
+| Clusters | 66 |
+| Glossary entries | 66 |
+| Review-queue items | 66 |
+
+| Experiment | Question | Result | Kill line | Verdict |
+|---|---|---|---|---|
+| **E1** | GLiNER recall on Bahasa Indonesia technical prose | **0.854** | 0.70 | **PASS** |
+| **E2** | Does clustering cut review burden? | **2.56×** (169 → 66) | 2.0× | **PASS** |
+| **E3** | Is nano sufficient for schema fill? | **0.75** | 0.80 | **FAIL** |
+| **E4** | Does verbatim-span validation catch fabrication? | **1.00** (false-rejection 0.00) | 0.90 | **PASS** |
+
+Other measured figures: cost **$0.0069** for the 9-page document (155,313 prompt tokens, of which
+**125,184 cached**), 242 s wall clock, 3 fields rejected by span check, 0 definition conflicts,
+7 rule entries, 7 formula entries, 1 brief-context summary.
+
+**The two things that most change our plan:**
+
+1. **E1 answers the standing Indonesian risk.** GLiNER multi-v2.1 scores **0.854 recall** on the
+   majority-Indonesian BUMA standard, comfortably above the 0.70 kill line. The context doc lists
+   this as the headline open risk — it is now measured, on the exact document named as the test case.
+2. **E3 settles half of the model-tier decision.** nano scores **0.75 precision against a 0.80 kill
+   line — a FAIL.** Caveats matter: the scoreable base is 8 entries, and nano is not deterministic at
+   `temperature=0` (consecutive identical runs scored 0.75 and 0.625). No mini deployment is
+   provisioned, so the spec's nano-vs-mini A/B has never run and **all four branches currently route
+   to nano**, including `rule` and `summary`, which the spec explicitly wanted on the larger tier.
+
+**Caveat on E2 vs. the context doc.** The context doc claims clustering cuts review burden ~6.2×
+(3,125 → ~500 on a 1,000-page corpus). The **measured** figure on 9 pages is **2.56×**. These are
+not in conflict — compression grows with corpus size as terms repeat — but 6.2× is a projection and
+2.56× is the measurement. Quote the right one to the right audience.
+
+---
+
+## A. The seam — shared, and still the first thing to agree
+
+The audit changes this section substantially: we now have a concrete starting point instead of a
+blank page, **and** a concrete conflict to resolve.
+
+`kex/models.py` already defines `ParseResult` / `Chunk`, which is close to the artifact we need —
+persistable, content-hashed, carrying section numbers, page ranges and formatting signals:
+
+```python
+class Chunk(BaseModel):
+    chunk_id: str            # f"{doc_id}#{section_no}#{ordinal}"
+    doc_id: str
+    section_no: str | None   # "2.1.3"
+    heading: str | None
+    text: str
+    page_start: int; page_end: int
+    ordinal: int
+    has_formula: bool; is_tabular: bool
+    bold_spans: list[str]    # feeds the formatting evidence signal
+
+class ParseResult(BaseModel):
+    doc_id: str; source_path: str; content_hash: str
+    n_pages: int; chunks: list[Chunk]
+    used_heading_split: bool
+```
+
+**But the POC's actual seam is the wrong one.** `kex/parse/base.py` defines the boundary as an
+in-process protocol, `Parser.parse(path: Path) -> list[Page]`, where `Page` is `{page_no, text,
+bold_spans}`. That takes a **file path** — the one shape the plan forbids — it is an in-process
+interface rather than a persisted versioned artifact, and its flat `text` field discards exactly
+what MinerU exists to produce.
+
+| # | Task | Owner | Status | Description |
+|---|---|---|---|---|
+| A1 | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | **Sofhia circulated a draft `contracts.py` on 2026-08-19** — derived from real MinerU output, backend-agnostic, with a `kind` discriminator (text/table/chart/equation), `source_items` back-references, `images` as paths, and an explicit verbatim-text warning. **Adopt it as the base.** Open items: merge kex's document-level envelope (§A1a), the `text_level` correction (§A1b), the formatting signal, and 0- vs 1-based pages. **Highest-priority item.** |
+| A1a | Add a document-level envelope | Sofhia + Rifqi | ⬜ | The draft is `list[Chunk]` with no wrapper, so there is nowhere to put `content_hash`, `n_pages`, `source_path`, `parser_name`/`parser_version` — or **`version`**, which A3 requires. kex's `ParseResult` wraps exactly this. Parser identity matters specifically because the seam exists to make parsers swappable: when extraction output changes we must know which parser produced the artifact |
+| A1b | Resolve the section-structure question | Rifqi → Sofhia | ⬜ | The draft states MinerU does not supply section structure (`text_level` on 1 of 84 items). **On the BUMA standard it does:** the full 9-page parse carries `text_level` on **29 of 118 items**, cleanly numbered `1.` → `2.2.5`, including `2.1.3. Physical of Availability (PA)`. The draft's sample appears to be the handbook, which genuinely has no numbered headings. Keeping both fields Optional is right; the stated *reason* is not, and acting on it would starve section-aware chunking on exactly the documents that carry numbering |
+| A2 | Move the seam off the file path | Sofhia + Rifqi | ⬜ | Replace `Parser.parse(path) -> list[Page]` as *the contract* with a persisted, versioned `ParseResult`. `list[Page]` may survive as an internal detail of the PyMuPDF fallback; it must stop being the boundary. |
+| A3 | Artifact versioning + storage | Sofhia + Rifqi | ⬜ | Version number, what triggers a re-parse, where it persists. Today `kex` writes `data/interim/*.chunks.json` with a content-hash cache — real but file-based and unversioned. Needs a DB table → Harry-ready DDL handoff (Go owns the schema; Python never runs DDL). |
+| A4 | MinerU → `ParseResult` adapter | Rifqi (provisional) → Sofhia (owns) | ⬜ | Reads a MinerU output folder and emits `ParseResult` **directly, skipping `list[Page]`**. Routing MinerU through `Page.text` flattens typed blocks, LaTeX and table HTML back to a string, then asks the chunker to re-derive by regex what MinerU had already structured. See §B for which files to read. |
+| A5 | Commit fixture artifacts | Rifqi | 🔄 | Both source PDFs are already in `kex/data/raw/` (BUMA standard + Open Pit textbook) and parsed chunk JSON exists in `data/interim/`. Needs promoting to committed fixtures in the target repo. |
+| A6 | Write the seam contract into the repo docs | Rifqi | ⬜ | Extraction never takes a file path; neither half imports the other's internals; parsing stays swappable behind the artifact. So the rule survives us forgetting it — the POC drifted from it already. |
+
+### A1 review notes — Sofhia's `contracts.py` draft (2026-08-19)
+
+**Adopt as the A1 base.** It is derived from real MinerU output rather than designed on paper, and
+three of its choices are better than what kex has: the `kind` discriminator, `source_items`
+(index back-references into `content_list.json`, which kex has no equivalent of), and the explicit
+**verbatim-text warning** — which independently identifies the same failure mode as our span-check
+control, including that it fails *silently* to null and looks like a bad LLM.
+
+| # | Item | Severity | Note |
+|---|---|---|---|
+| R1 | Document-level envelope missing | **High** | See A1a. No home for `version`, and A3 needs one |
+| R2 | Section-structure claim | **High** | See A1b. True for the handbook, not for the BUMA standard |
+| R3 | Formatting signal dropped | Medium | kex's evidence ranking uses six signals — cue · heading · legend · formula · **formatting** · first-occurrence — and reads formatting from `Chunk.bold_spans`. The draft excludes it (rightly, as `text_format` is `vlm`-only). Either derive it from markdown emphasis in `text`, or drop the signal deliberately. It should not disappear by silence |
+| R4 | Page indexing | Medium | Draft is 0-based "as MinerU reports"; kex is 1-based, and page numbers are shown to Mas Beta in the review queue. Off-by-one straight into the expert-facing UI. Pick one and name the field accordingly (`page_idx` if 0-based) |
+| R5 | Chunk granularity unspecified | Medium | The sample chunk `handbook::0000` spans 32 source items across 2 pages. kex caps ~1,500 tokens with paragraph-boundary splitting. Coarse chunks make evidence ranking blunt (top-3 chunks = many tokens per call) and inflate the summary branch's token share, which §5 already names as the quiet cost risk |
+| R6 | `chapter` is lossy across boundaries | Low | `handbook::0000` covers pages [0,1] and is labelled *"THE HORIZONS OF MAINTENANCE MANAGEMENT"*, while `handbook::0001` on page 1 is *"MAINTENANCE STORES AND INVENTORY CONTROL"*. A chunk that crosses a chapter boundary gets one label. Fine if known |
+| R7 | `TermRecord` is thinner than kex's entry | Low | Marked "declared, not yet used", so no conflict — but point it at `kex/models.py` rather than re-deriving. kex additionally carries `interpretation`, `domain`, `company`, `language`, `definition_conflict`, `conflict_variants`, `diff_status`, and a structured `provenance{doc_id, section_no, page, span}` |
+| R8 | dataclass vs pydantic | Low | kex and the target repo are both pydantic v2. A seam without validation is weakly a seam |
+| R9 | **Add `source_wording` now** | Medium | The contract is open, so this is the cheap moment to fix **G1**. With `heading` carried verbatim, a `source_wording` field span-checked against it makes *"Physical of Availability"* survive instead of being silently normalised to *"Physical Availability"* |
+
+---
+
+## B. Parsing pipeline — **Sofhia**
+
+Unchanged in scope. One addition from the audit: the MinerU output shape is now known, because a
+real MinerU run of the BUMA standard was inspected (`Downloads/mineru_output_results/…/hybrid_auto/`).
+
+**Which files the adapter should read:**
+
+| File | Use | Why |
+|---|---|---|
+| `*_content_list_v2.json` | **Primary** | Page-grouped; richest taxonomy: `paragraph` · `title` (with `level`) · `equation_interline` (LaTeX + image ref) · `table` (HTML + image ref) |
+| `*_content_list.json` (v1) | **Secondary** | Flat, but carries explicit `page_idx` and `text_level`. v2 encodes page only as outer list position — cross-check the two |
+| `*_middle.json` | Later, if needed | Line/span-level bbox. Only if the review UI wants to highlight a quote on the page image |
+| `images/` | **Reference only** | Hash-named crops. The LLM stage is text-only — the artifact carries paths, never bytes |
+| `*.md` | **Do not consume** | A rendering. Loses `page_idx` and block-type boundaries; its image paths couple us to the folder layout |
+| `*_model.json`, `*_layout.pdf`, `*_span.pdf`, `*_origin.pdf` | Ignore | Raw model output and debug visualisations |
+
+| # | Task | Owner | Status | Description |
+|---|---|---|---|---|
+| B1 | MinerU cost justification (written) | Sofhia | ⬜ | Owed for the budget proposal, from the 13 Aug exec review. |
+| B2 | MinerU install + run on both sample docs | Sofhia | ✅ | **Complete 9-page parse delivered 2026-08-19**: 118 items (90 text, 14 table, 11 equation, 3 chart) across pages 0–8. The rule-of-thumb grounding case is present (Other Activity, Controllable/Uncontrollable, Standby), so the fixture now exercises all four branches. Supersedes the earlier 5-page run. |
+| B3 | Evaluate MinerU quality vs. the current path | Sofhia | ⬜ | Compare specifically on tables, the p.2 legend block, formulas and figures — against the PyMuPDF path the POC uses today, which is a real baseline with real numbers, not a guess. |
+| B4 | Indonesian check on **parsing** | Sofhia | ⬜ | Distinct from E1, which measured GLiNER, not the parser. Confirm MinerU's text/layout quality holds on Bahasa Indonesia. |
+| B5 | Section-aware chunking | Sofhia | 🔎 | **Already built and working** for the PyMuPDF path (`chunk/section_chunker.py`, 244 LOC, tested): numbered-heading regex, breadcrumb-header dedupe, ~1,500-token cap, page-boundary fallback. On the BUMA standard it produced 13 chunks with `used_heading_split=True`. The open question is whether MinerU's typed blocks should feed it or replace it. |
+| B6 | Emit the agreed artifact (A1) | Sofhia | ⬜ | The parser writes our schema, not MinerU's native shape. This is what keeps MinerU swappable. Currently a `NotImplementedError` stub. |
+| B7 | Fallback path behind the same artifact | Sofhia | 🔎 | PyMuPDF already works end-to-end and is the POC default. It needs re-pointing at `ParseResult` as the contract rather than `list[Page]`. |
+| B8 | Parsing subsystem is additive + gated | Sofhia | ⬜ | The existing unstructured path (OCR → chunk → pgvector) in the target repo stays untouched. MinerU is a new flag-gated path beside it. |
+
+**Findings from the real MinerU output that affect B5/B6:**
+
+- **Heading depth is flattened** — all 29 `title` blocks carry `"level": 2`, including `1. TUJUAN
+  PARAMETER` and `2.1.3. Physical of Availability (PA)`. Hierarchy must come from the numbering
+  text, never from `level`. But the heading *text itself* is supplied — see A1b.
+- **Equation LaTeX is character-spaced** — `P r o d u c t i o n = M O H H \times Q t y \times P A
+  \times U A \times P t y`. Naive matching will never find `MOHH`. Needs normalising before the
+  formula/legend branch and before span validation can locate anything inside a formula.
+- **Breadcrumb headings repeat per page** — `2. PENJELASAN PARAMETER` / `2.1. Production Parameter`
+  re-appear as `title` blocks on pages 2–8. kex's chunker already dedupes these
+  (`_boilerplate_keys`); whatever emits the artifact must not treat them as new sections.
+- **Charts carry captions only** — all 3 chart items have `content: ""`, with the caption in
+  `chart_caption` (e.g. *"Gambar 2.1 Analisis Gain/Loss…"*). A chart chunk therefore contributes
+  a caption plus an image reference and nothing else to extraction. Expected, but it means chart
+  detection buys Scenario 2, not glossary coverage.
+
+---
+
+## C. Extraction pipeline — **Rifqi**
+
+Statuses below are **verified against the code and the run artifacts**, not estimated.
+
+### C-1. Built and measured
+
+| # | Task | Status | Evidence |
+|---|---|---|---|
+| C1 | Candidate-entry schema | ✅ | `models.py` (222 LOC) encodes all three invariants: content fields Optional, `SubdomainEnum`, `Provenance.span` mandatory |
+| C2 | Offline runner over a fixture | ✅ | 8-command CLI; every stage reads/writes JSON; content-hash cache; `--dry-run` prints exact prompts + token estimate with zero network calls |
+| C3 | GLiNER span filter | ✅ | `filters/gliner_filter.py`, `urchade/gliner_multi-v2.1`, CPU, every chunk, labels in YAML |
+| C4 | **Indonesian performance check** | ✅ | **E1 = 0.854 recall** vs 0.70 kill line, on the BUMA standard, 41-term gold set. The context doc's headline open risk, now measured |
+| C5 | Discourse-cue regex → rule-of-thumb | ✅ | `filters/cue_filter.py`; cues in `config/cues.yaml`; 7 rule entries produced |
+| C6 | Legend-block regex → formula vars | ✅ | `filters/legend_filter.py` + a ≥0.95-precision test; 7 formula entries produced |
+| C7 | Section pass → summary units | ✅ | `extract/summary.py`; produced `brief_context.json` + `.md` |
+| C8 | Normalise + cluster mentions | ✅ | `cluster/` — exact → abbrev → rapidfuzz ≥92; fuzzy off below 5 chars because `PA`/`UA` score highly against each other. **E2 = 2.56×** |
+| C9 | Evidence ranking → top-K | ✅ | `rank/evidence.py`, weight table, K=3, full ranked list retained for escalation |
+| C10 | LLM extraction call | ✅ | `extract/azure_client.py` + 4 branch modules; structured outputs via `json_schema`; prompts as files; prefix padded past 1024 tokens for caching (**54% hit rate measured**) |
+| C11 | Verbatim-span validation | ✅ | `validate/span_check.py`; never repairs a failed span. **E4 = 1.00 rejection, 0.00 false-rejection** |
+| C16 | Frequency-sorted review queue | ✅ | `queue/review_queue.py`; conflicts promoted above frequency. Queue top: Qty(20) · PA(16) · PTY(12) · UA(11) — exactly the terms that carry the document |
+
+### C-2. Built but not exercised — needs verification before we trust it
+
+| # | Task | Status | What the audit found |
+|---|---|---|---|
+| C12 | Null-definition escalation | 🔎 | Implemented and wired, but **`escalated = 0`** across the whole run. Not a bug: **54 of 66 clusters have exactly 1 evidence chunk** and K=3, so `rounds_available()` correctly returns 0 — the document is too small to reach the path. Unexercised until a larger corpus runs |
+| C13 | Conflict detection | 🔎 | `validate/conflict.py` wired, but **0 conflicts** observed — a single internally-consistent standard gives it nothing to find. Untested against real disagreement |
+| C15 | Diff vs. active glossary version | 🔎 | `diff/glossary_diff.py` is wired, but `write_artifacts` reads `existing` from `out/glossary.json` — **the same file it then overwrites**. All 66 entries came back `new`. Only the empty-baseline path has ever run; there is no version history, no active-version pointer, and no approval state |
+
+### C-3. Not built
+
+| # | Task | Status | Description |
+|---|---|---|---|
+| C14 | **Literal-source-wording behaviour** | ⛔ | **Currently violated — see §G1.** |
+| C17 | Bulk vs. incremental ingest modes | ⬜ | Single-document CLI only. Neither corpus-frequency boosting (bulk) nor active-version diffing (incremental) exists |
+| C18 | Candidate-entry persistence | ⬜ | Everything is JSON files in `out/`. Needs tables → Harry-ready DDL handoff, same rule as A3 |
+| C19 | Port `kex` into the target repo | ⬜ | 2,763 LOC, Python 3.13 + its own venv vs. the target repo's 3.12; new deps (GLiNER + torch). Sequencing and gating to be decided |
+
+---
+
+## D. Decisions needed
+
+| # | Decision | Owner | Status | Description |
+|---|---|---|---|---|
+| D1 | New dependencies | Rifqi (asks) | ⬜ | GLiNER + torch (~heavy) on extraction, MinerU on parsing. `pyproject.toml` changes in the target repo need sign-off. Note the POC needs **torch ≥ 2.6** and hit a corrupt-wheel failure worth not re-debugging |
+| D2 | New tables (DDL handoff) | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. One consolidated handoff beats three |
+| D3 | Model tier per branch | Rifqi | ⛔ | **Half-answered: nano FAILS at 0.75 vs the 0.80 line.** Blocked on a **mini deployment being provisioned** — until then all four branches run on nano, including `rule` and `summary`, which the spec never intended for it, and the A/B cannot run |
+| D4 | Admin ingest authentication | Rifqi | ⬜ | A different risk class from the read-only chat surface. Raise it rather than assume — and only after the offline path is ported |
+| D5 | GLiNER + LangExtract ensembling | — | ⏸️ | Proposed, not decided; flagged as scope creep. Not in scope for the test integration |
+| D6 | **Expert review of the gold set** | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is **self-bootstrapped and provisional**. E1 and E3 both move when it is reviewed. E3's scoreable base is 8 entries — each adjudication moves the number ~12 points. Every failure is listed verbatim in `eval/out/report.md` for him |
+
+---
+
+## E. Outside our two halves
+
+| # | Item | Owner | Status | Description |
+|---|---|---|---|---|
+| E1 | Curation UI + expert review flow | Mas Har / Mas Beta | ⬜ | Our pipeline's job ends at the queue. `out/review_queue.json` (66 items, ranked, each with page, section, span and a `review_reason`) is a ready-made input contract |
+| E2 | Versioning + approval audit trail (product side) | Mas Har | ⬜ | The expert chooses which version is active. Pairs with C15/A3 |
+| E3 | MCP server — the consumption surface | — | ⬜ | The POC already emits three of the four target artifacts: `glossary.json`, `interpretation_pack.json`, `formulas.json`, `brief_context.json`. Only the data dictionary (DB profiling) is out of scope for extraction |
+
+---
+
+## F. Already settled — please do not reopen
+
+1. The LLM call unit is the **term cluster**. Justification is review-burden reduction, **not** tokens.
+2. **No standalone relevance gate.** "Zero candidate spans" is the relevance signal.
+3. **`provenance.span` is mandatory and verbatim.** Never repaired — a repaired span is an unfalsifiable claim.
+4. **All content fields Optional.** Abstention is correct behaviour, not an error.
+5. **API cost is negligible; latency irrelevant.** Measured: $0.0069 for 9 pages. Optimise the review experience, not tokens.
+6. **This is a test integration, not a productionisation.** Both halves are new, gated subsystems.
+7. **Nothing is ever dropped at the term level.** Evidence ranking narrows *evidence*, not terms.
+
+---
+
+## G. Defects and risks found by the audit
+
+| # | Finding | Severity | Detail |
+|---|---|---|---|
+| **G1** | **Literal source wording is being normalised away** | **High** | Decision §4.10 says the pipeline must record what the document actually says and surface the discrepancy. It currently does not. The BUMA standard's heading reads **"2.1.3. Physical of Availability (PA)"**; the extracted entry carries `full_name: "Physical Availability"` — silently corrected. Mechanism: `Chunk.heading` is a separate field from `Chunk.text`, and the chunk text does not contain the phrase at all, so the literal wording never reaches the model. Fix shape: feed the heading verbatim, and/or add a `source_wording` field span-checked against the heading |
+| **G2** | **The POC is not under version control** | **High** | No `.git`. 2,763 LOC, four experiment results, a metrics pack and both gold sets exist only on this machine |
+| **G3** | **85% abstention rate** | Medium | 56 of 66 entries have **no definition**; only 10 carry one. Correct behaviour per the abstention rule, but it means the review queue is mostly "term found, no definition in document". Whether that is useful to Mas Beta or noise is a **review-experience question to put to him** — and per §5 of the context doc, review experience is where engineering time is worth spending |
+| **G4** | **The seam drifted from the plan** | Medium | `Parser.parse(path) -> list[Page]` is the exact shape the context doc forbids. Cheap to fix now, expensive once MinerU is built against it — hence A2 before B6 |
+| **G5** | **Everything generalises from one 9-page document** | Medium | Single document, single language, single company. E2's compression, E3's scoreable base and the escalation path are all constrained by it |
+| **G6** | **`rule` and `summary` run on the wrong tier** | Medium | Both route to nano because it is the only provisioned deployment. These are the two branches whose failure mode is least detectable — a plausible summary cannot be span-checked. Same blocker as D3 |
+
+---
+
+## Proposed sequence
+
+1. **G2 first, today** — put `kex` under version control. Everything below risks a machine failure until this is done.
+2. **Close A1 on Sofhia's draft** — reply with R1–R9, settle A1a/A1b, and let her push the agreed version to the catalog repo as the module of record. **A2** (move the seam off the file path) rides along with it: doing A2 *before* she builds B6 is the whole point; afterwards it is a rewrite of her work.
+3. **Then split.** Sofhia takes B1–B4 (including a complete 9-page re-parse); Rifqi takes G1 (the literal-wording defect), then A4 (provisional MinerU adapter) and C15/A3 (real versioning behind the diff).
+4. **Reconverge at B6.** Sofhia's parser emits the agreed artifact and replaces the provisional adapter. If A1 was right, nothing in the extraction half changes — that is the test of whether we did A1 properly.
+5. **In parallel, unblock the two ⛔s:** D3 (provision a mini deployment, then run the A/B) and D6 (get the gold set in front of Mas Beta). Both gate quality claims, neither gates engineering.
+6. **Then** C17, C18, C19 and D1/D2/D4 — the port into the target repo.
+
+Rationale: this ordering front-loads the two things that are expensive to get wrong later — the seam
+and the version-control gap — and it keeps the four open decisions in section D off both of our
+critical paths.

knowledge_pipeline_context.md

@@ -0,0 +1,299 @@
+# Knowledge Pipeline — Context for the Data Eyond Python Repo (19 Aug 2026)
+
+> **Who this is for:** a Claude Code session working in `Agentic-Service-Data-Eyond-Catalog`
+> (the Python agentic service). This doc carries *context and intent* only — the why, the
+> ownership split, the shape of the pipeline, and the decisions already settled. It does not
+> prescribe folder layout, module names, or endpoint signatures; work those out against
+> `CLAUDE.md`, `REPO_STATUS.md` and the existing subsystem patterns in the repo.
+>
+---
+
+## 1. What we are doing and why now
+
+Data Eyond is an AI data-analyst platform. The strategic thesis from the 30 July exec review
+is that **domain knowledge is the differentiator, not architecture** — orchestrators and
+connectors are replicable in a quarter, model capability is rented, only encoded domain
+knowledge compounds. The end goal is an **MCP product** exposing modular domain knowledge
+that EMA and later clients connect to.
+
+That knowledge currently comes from experts typing it in. **Mas Beta** (plant & maintenance)
+is the single validation bottleneck — nothing becomes an artifact without him. The knowledge
+pipeline exists to change the expert's job from *authoring* to *reviewing*: the pipeline reads
+the client's own documents (standards, SOPs, handbooks) and proposes candidate knowledge
+entries; the expert approves, edits or rejects them.
+
+**Why it lands in this repo now.** We want to integrate the pipeline into Data Eyond to test
+it end-to-end on real documents, in the place where the resulting knowledge will actually be
+consumed. This is a **test integration**, not a productionisation — treat it as a new, gated
+subsystem alongside the existing document/catalog pipelines, not as a rewrite of them.
+
+---
+
+## 2. Ownership — two items, two people, one seam
+
+The pipeline splits cleanly in two, and the split is deliberate: **they are separate,
+independently runnable, independently swappable pipelines.**
+
+| Half | Owner | Scope |
+|---|---|---|
+| **① Parsing (MinerU)** | **Sofhia Az-Zahra** | PDF/DOCX in → structured, typed extracted content out. MinerU replaces the current text+table-only extraction: it adds formula, image, chart and diagram detection, and emits Markdown or JSON with type labels. Presented to the execs 13 Aug as the **prerequisite for Scenario 2**; a written cost justification for MinerU is owed for the budget proposal. |
+| **② Knowledge extraction** | **Rifqi (me)** | Parsed content in → validated candidate knowledge entries out. Term filtering, clustering, evidence ranking, schema-guided LLM attribute fill, span validation, diff against the active glossary version, and the frequency-sorted review queue. |
+
+**The seam.** The two halves must meet only at a **persisted, versioned parsed-document
+artifact** — extraction consumes that artifact, never a raw file, never the parser's API.
+This is the single most important structural constraint in the whole plan, because:
+
+- MinerU can be swapped (or fall back to the existing Tesseract / Azure Document Intelligence
+  paths) without extraction knowing;
+- extraction can be re-run cheaply on an already-parsed corpus when a prompt or schema changes
+  — and it will be, often, because prompt iteration is the main development loop here;
+- Sofhia and I can work and test independently, against a fixture artifact, without blocking
+  each other;
+- parsing is the slow/expensive-to-rerun half; extraction is the fast-to-iterate half. Keeping
+  the boundary hard means we never re-parse to fix an extraction bug.
+
+Anything that makes the two halves import each other's internals, or that makes extraction
+take a file path, breaks the point of the split.
+
+---
+
+## 3. The pipeline
+
+Six stages. The cost story matters as much as the flow: almost everything is free CPU/regex
+work, and the LLM appears in exactly one stage.
+
+```mermaid
+flowchart TD
+    A["Admin / expert triggers ingest<br/><i>not continuous, not per-user</i>"] --> B
+
+    subgraph P["① PARSING — Sofhia"]
+        B["MinerU<br/>text · tables · formulas · figures · charts"] --> C["Section-aware chunking<br/>split on numbered headings 2.1.1, 2.1.2 …"]
+    end
+
+    C --> SEAM[("Parsed-document artifact<br/>versioned · persisted<br/><b>THE SEAM</b>")]
+    SEAM --> D
+
+    subgraph F["② TERM FILTER — CPU / regex, free"]
+        D["GLiNER span filter on ALL chunks → term mentions"] --> H
+        E["Discourse-cue regex → rule-of-thumb candidates"] --> H
+        G["Legend-block regex → formula vars + abbreviation dictionary"] --> H
+        G2["Section pass → summary units"] --> H
+        H["Normalise + cluster mentions<br/>PA · P.A. · Physical Availability → 1 cluster"] --> I["Evidence ranking<br/>cue · heading · legend · formula · formatting · first-occurrence"]
+        I --> J["Top-3 evidence chunks per cluster"]
+    end
+
+    SEAM --> E
+    SEAM --> G
+    SEAM --> G2
+
+    J --> K
+
+    subgraph X["③ KNOWLEDGE EXTRACTION — the only paid stage"]
+        K["1 LLM call per TERM CLUSTER<br/>(not per mention, not per chunk)"]
+        L["1 call per cue chunk · 1 per unique formula · 1 per document (summary)"]
+    end
+
+    K --> M
+    L --> M
+
+    subgraph V["④ VALIDATION — deterministic"]
+        M["Verbatim-span check → reject field if span not locatable"] --> N["Null definition → escalate to next 3 evidence chunks<br/>max 2 rounds → else flag 'term found, no definition located'"]
+        N --> O["Conflicting definitions → definition_conflict + variants[]"]
+    end
+
+    O --> Q["⑤ DIFF vs active glossary version<br/>duplicate (skip) · new · conflicting"]
+    Q --> R
+
+    subgraph RV["⑥ REVIEW — human"]
+        R["Queue sorted by mention frequency<br/>PA 250 → UA 180 → … → long tail"] --> S["Expert (Mas Beta) approves / edits / rejects"]
+        S --> T["Glossary · Interpretation Pack · Brief Context · Formula"]
+        T --> U["Versioning + approval audit trail"]
+    end
+
+    U --> W["MCP server — the consumption surface"]
+
+    style P fill:#e8f4ff,stroke:#4a90d9
+    style F fill:#e6f7e6,stroke:#4caf50
+    style X fill:#dbe9ff,stroke:#2f6fd0
+    style V fill:#f0e6ff,stroke:#8e5fd0
+    style RV fill:#fff0e0,stroke:#e08b3c
+    style SEAM fill:#fffbe0,stroke:#c9a227,stroke-width:3px
+```
+
+Colour convention, carried from the diagrams shown to the team:
+**green = free** (CPU/regex — most of the pipeline) · **blue = LLM** (the only paid part) ·
+**purple = deterministic code** · **orange = human**.
+
+### The five-box version (how this is explained to execs)
+
+```mermaid
+flowchart LR
+    A["PARSE<br/>MinerU"] --> B["FILTER<br/>free, CPU"] --> C["EXTRACT<br/>LLM, per term cluster"] --> D["VALIDATE + DIFF<br/>deterministic"] --> E["EXPERT REVIEW<br/>frequency-sorted queue"]
+    style A fill:#e8f4ff,stroke:#4a90d9
+    style B fill:#e6f7e6,stroke:#4caf50
+    style C fill:#dbe9ff,stroke:#2f6fd0
+    style D fill:#f0e6ff,stroke:#8e5fd0
+    style E fill:#fff0e0,stroke:#e08b3c
+```
+
+### Ownership view
+
+```mermaid
+flowchart LR
+    subgraph S["Sofhia"]
+        P["Parsing pipeline<br/>MinerU + backends"]
+    end
+    subgraph R["Rifqi"]
+        K["Extraction pipeline<br/>filter → LLM → validate → diff → queue"]
+    end
+    subgraph H["Mas Har / Mas Beta"]
+        C["Curation UI + expert review<br/>versioning, approval"]
+    end
+    P -->|"versioned parsed artifact<br/>(the only contract)"| K
+    K -->|"candidate entries + provenance"| C
+    C -->|"active versions"| M["MCP server"]
+```
+
+---
+
+## 4. Decisions already settled — do not reopen these
+
+1. **The LLM call unit is the term cluster** — not the chunk, not the mention. Per-chunk works
+   for a 9-page document and breaks at 1,000+ pages: "PA" mentioned 250× would produce 250
+   near-identical candidates. Clustering first cuts expert review burden ~6.2× (3,125 raw
+   extractions → ~500 term entries on a 1,000-page document). The token saving is trivial
+   (~$0.45); **the review-burden reduction is the justification.**
+2. **Conflict detection depends on the clustering.** Two contradictory definitions of the same
+   term can only be spotted because all evidence for that term arrives in one call. Two
+   separate calls would never meet, and the conflict would surface as two silently-accepted
+   entries.
+3. **No standalone relevance gate.** A binary relevance classifier before the term filter is a
+   second imperfect classifier whose false negatives drop content *before* extraction sees it —
+   and dropped content never reaches expert review to be caught. GLiNER is cheap enough (CPU,
+   ~400MB) to run unconditionally on every chunk; "zero candidate spans" becomes the relevance
+   signal as a byproduct of looking, rather than a judgment made before looking.
+4. **Evidence ranking is allowed even though it is also a filter** — because no term is
+   dropped (only which passages feed the call is narrowed), all mentions stay in provenance so
+   a bad top-3 is visible, and the escalation loop self-corrects. The rejected relevance gate
+   had none of those three properties. This distinction is the reasoning to preserve if the
+   ranking is ever revisited.
+5. **`provenance.span` is mandatory and verbatim.** A field whose span cannot be located in the
+   source is rejected. This is the primary anti-hallucination control and the thing that makes
+   expert review *finishable* — the reviewer checks a quote against a page, not a claim against
+   their memory.
+6. **All content fields are Optional in the schema.** The model must be able to answer null; a
+   required field is a fabrication invitation for smaller models. `subdomain_tags` is an enum —
+   classification, not generation.
+7. **Frequency-sorted review queue.** Directly targets the Mas Beta bottleneck: validate the
+   ~40 terms carrying the document's meaning first, let the long tail wait.
+8. **Model tier is per branch.** Glossary and Formula are extraction tasks well-guarded by span
+   verification → a small/cheap tier is fine. Rule-of-thumb and Summary are reasoning/generation
+   tasks where a small model's failure mode is least detectable (a plausible summary cannot be
+   span-checked) → larger tier. Test before committing the small tier on the rule-of-thumb
+   branch.
+9. **GLiNER-hybrid stays the glossary-branch default.** Pure NER can only fill 2 of 6 schema
+   fields (definition, formula and interpretation are not spans — they are assembled from text
+   that may be pages apart). Pure small-LLM shows recall 0.93–1.00 at precision 0.15–0.43;
+   perfect recall is the failure signature of a model fabricating a value for every field.
+10. **The pipeline records the literal source wording.** It recorded "Physical of Availability"
+    from the BUMA standard rather than the textbook's "Physical Availability" — surfacing the
+    discrepancy to the expert instead of silently normalising it. Keep that behaviour.
+
+---
+
+## 5. Deployment reality — this shapes every engineering trade-off
+
+The pipeline runs **once at setup (bulk corpus ingest)** and **occasionally when an admin or
+expert adds a file**. It is **not continuous** and **not per-user** — normal engineers cannot
+add data. Consequences, in the order they matter:
+
+- **Extraction quality and expert-review efficiency dominate.** A missed term has no
+  self-correcting mechanism, because engineers can't add data to fill a gap later.
+- **Maintainability matters more than it looks.** A glue chain invoked every few months is
+  exactly the thing nobody remembers how to debug. Prefer boring, inspectable stages over
+  clever ones.
+- **API cost is negligible and latency is irrelevant.** ~$8–12 for a 6,000-page corpus;
+  an incremental single 9-page file is ~1.5 US cents. Do not spend engineering time optimising
+  tokens — engineering time on the review experience is worth more than any token saving
+  available here.
+- **No GPU required.** GLiNER is a CPU encoder model (~1.6 seq/s Python, ~6.7 seq/s via the
+  Rust port on an i9 8-core). ~15,000 chunks ≈ 2.6h Python / ~40min Rust for a job that runs a
+  few times a year. The LLM half is an API call. (Runtime as reported to the team on 13 Aug:
+  ~11 s/page GLiNER, ~26 s/page LLM — with the caveat that **cost scales on term count, not
+  page count.**)
+- **Two ingest modes to build:** *bulk setup ingest* (corpus-scale — the only place
+  corpus-frequency statistics are usable as a candidate-term booster) and *incremental
+  single-file add* (no corpus context; diff against the active glossary version).
+- **The summary branch is the quiet cost risk** — only ~300 calls but a quarter of all input
+  tokens, because summarisation can't be filtered; it needs whole documents.
+
+---
+
+## 6. Where the output goes
+
+Approved candidates become the project's **four artifacts** (see the `context` doc for the
+full architecture):
+
+1. **Domain knowledge** — system-prompt material, capped ~1500 tokens: operating and equipment
+   hierarchy, time convention (WITA), seasonality, most-important glossary terms, high-level
+   business process, and agent limitations.
+2. **Data dictionary** — from DB column profiling; used for query generation. Pak Ricky
+   connected the extraction work directly to **automating this**.
+3. **Interpretation pack** — expert-curated interpretation logic, action rules, benchmark
+   history, tied to a use case ("PA missed" reads differently from "PA missed but MTBS
+   achieved"). Revived on 13 Aug after earlier removal for being too normative — reintroduce
+   carefully.
+4. **Skill registry** — query rules and expert-approved formulas. Approval is mandatory because
+   the same term computes differently per company (MTTR at BUMA = breakdown duration ÷
+   breakdown frequency; MTTR in IT = mean time to resolve).
+
+All four carry **versioning plus an approval audit trail**, and the expert chooses which
+version is active. Delivery format is **MCP** — skill registry → MCP skills; data dictionary →
+MCP on demand; interpretation pack → attached to the skill; domain knowledge → system prompt
+but preferably served via MCP so owners can change it without redeploying the engine.
+
+---
+
+## 7. Grounding material
+
+Two real sample documents anchor every design claim above, and should anchor test fixtures too:
+
+- **BUMA STD/2026/006/MNO Rev.0.0** — Production Parameter & ECA. 9 pages, majority Bahasa
+  Indonesia. Carries: PA (glossary, p.4), Other Activity + the controllability hierarchy
+  (rule-of-thumb, pp.8–9), the MOHH/Qty/PA/UA/Pty legend block (formula branch, p.2).
+- **Open Pit Mine Planning & Design** (textbook excerpt) — carries NSR as a **cross-page
+  definition**: intro p.92, formula p.93, interpretive remark p.94 (§2.3.5). This is the case
+  that justifies section-aware chunking over fixed token windows — the whole definition must
+  stay inside one semantic unit.
+
+**Known open risk:** GLiNER multi-v2.1's Indonesian performance is unverified, and the BUMA
+standard is majority Bahasa Indonesia. This is the standing "Indonesian degradation" question,
+now with a concrete test case. Verify before trusting recall numbers.
+
+**Flagged as scope creep, proposed not decided:** ensembling GLiNER + LangExtract on every
+ingest. Cheap in API terms, but it adds a reconciliation/dedup step between two candidate sets
+that may disagree — new engineering surface for uncertain gain.
+
+---
+
+## 8. Integration guidance for this repo
+
+Deliberately shape-agnostic — decide the specifics against `CLAUDE.md` and the existing
+subsystem patterns, not against this doc.
+
+- **Both halves are new subsystems, not modifications** to the existing document/catalog
+  pipelines. The current unstructured path (Tesseract OCR → chunk → pgvector) stays as it is;
+  the MinerU path is additive and, at least initially, gated.
+- **Follow the repo's existing multi-stage precedent.** The query subsystem (IR → validator →
+  compiler → executor, orchestrated from a service facade, triggered by thin entry points) is
+  the closest structural analogue to what both halves need.
+- **Respect the hard boundaries.** Go owns the dedorch schema — any new table needs a
+  Harry-ready DDL handoff, never DDL executed from Python. Any new endpoint on the live surface
+  needs a contract-doc entry, and an admin ingest surface is a different risk class than the
+  read-only chat surface — raise the authentication question rather than assuming.
+- **Build the offline path first.** Because this runs a few times a year and is triggered by an
+  admin, a script-driven run over a fixture document is the honest first milestone; HTTP
+  endpoints are a convenience layer over it, not the other way round.
+- **Fixtures over live parsing in tests.** A committed parsed-artifact fixture from one of the
+  two sample documents lets the extraction half be tested without MinerU installed at all —
+  which is the seam doing its job.