knowledge_pipeline_context.md

# Knowledge Pipeline — Context for the Data Eyond Python Repo (19 Aug 2026)

> **Who this is for:** a Claude Code session working in `Agentic-Service-Data-Eyond-Catalog`
> (the Python agentic service). This doc carries *context and intent* only — the why, the
> ownership split, the shape of the pipeline, and the decisions already settled. It does not
> prescribe folder layout, module names, or endpoint signatures; work those out against
> `CLAUDE.md`, `REPO_STATUS.md` and the existing subsystem patterns in the repo.
>
---

## 1. What we are doing and why now

Data Eyond is an AI data-analyst platform. The strategic thesis from the 30 July exec review
is that **domain knowledge is the differentiator, not architecture** — orchestrators and
connectors are replicable in a quarter, model capability is rented, only encoded domain
knowledge compounds. The end goal is an **MCP product** exposing modular domain knowledge
that EMA and later clients connect to.

That knowledge currently comes from experts typing it in. **Mas Beta** (plant & maintenance)
is the single validation bottleneck — nothing becomes an artifact without him. The knowledge
pipeline exists to change the expert's job from *authoring* to *reviewing*: the pipeline reads
the client's own documents (standards, SOPs, handbooks) and proposes candidate knowledge
entries; the expert approves, edits or rejects them.

**Why it lands in this repo now.** We want to integrate the pipeline into Data Eyond to test
it end-to-end on real documents, in the place where the resulting knowledge will actually be
consumed. This is a **test integration**, not a productionisation — treat it as a new, gated
subsystem alongside the existing document/catalog pipelines, not as a rewrite of them.

---

## 2. Ownership — two items, two people, one seam

The pipeline splits cleanly in two, and the split is deliberate: **they are separate,
independently runnable, independently swappable pipelines.**

| Half | Owner | Scope |
|---|---|---|
| **① Parsing (MinerU)** | **Sofhia Az-Zahra** | PDF/DOCX in → structured, typed extracted content out. MinerU replaces the current text+table-only extraction: it adds formula, image, chart and diagram detection, and emits Markdown or JSON with type labels. Presented to the execs 13 Aug as the **prerequisite for Scenario 2**; a written cost justification for MinerU is owed for the budget proposal. |
| **② Knowledge extraction** | **Rifqi (me)** | Parsed content in → validated candidate knowledge entries out. Term filtering, clustering, evidence ranking, schema-guided LLM attribute fill, span validation, diff against the active glossary version, and the frequency-sorted review queue. |

**The seam.** The two halves must meet only at a **persisted, versioned parsed-document
artifact** — extraction consumes that artifact, never a raw file, never the parser's API.
This is the single most important structural constraint in the whole plan, because:

- MinerU can be swapped (or fall back to the existing Tesseract / Azure Document Intelligence
  paths) without extraction knowing;
- extraction can be re-run cheaply on an already-parsed corpus when a prompt or schema changes
  — and it will be, often, because prompt iteration is the main development loop here;
- Sofhia and I can work and test independently, against a fixture artifact, without blocking
  each other;
- parsing is the slow/expensive-to-rerun half; extraction is the fast-to-iterate half. Keeping
  the boundary hard means we never re-parse to fix an extraction bug.

Anything that makes the two halves import each other's internals, or that makes extraction
take a file path, breaks the point of the split.

---

## 3. The pipeline

Six stages. The cost story matters as much as the flow: almost everything is free CPU/regex
work, and the LLM appears in exactly one stage.

```mermaid
flowchart TD
    A["Admin / expert triggers ingest<br/><i>not continuous, not per-user</i>"] --> B

    subgraph P["① PARSING — Sofhia"]
        B["MinerU<br/>text · tables · formulas · figures · charts"] --> C["Section-aware chunking<br/>split on numbered headings 2.1.1, 2.1.2 …"]
    end

    C --> SEAM[("Parsed-document artifact<br/>versioned · persisted<br/><b>THE SEAM</b>")]
    SEAM --> D

    subgraph F["② TERM FILTER — CPU / regex, free"]
        D["GLiNER span filter on ALL chunks → term mentions"] --> H
        E["Discourse-cue regex → rule-of-thumb candidates"] --> H
        G["Legend-block regex → formula vars + abbreviation dictionary"] --> H
        G2["Section pass → summary units"] --> H
        H["Normalise + cluster mentions<br/>PA · P.A. · Physical Availability → 1 cluster"] --> I["Evidence ranking<br/>cue · heading · legend · formula · formatting · first-occurrence"]
        I --> J["Top-3 evidence chunks per cluster"]
    end

    SEAM --> E
    SEAM --> G
    SEAM --> G2

    J --> K

    subgraph X["③ KNOWLEDGE EXTRACTION — the only paid stage"]
        K["1 LLM call per TERM CLUSTER<br/>(not per mention, not per chunk)"]
        L["1 call per cue chunk · 1 per unique formula · 1 per document (summary)"]
    end

    K --> M
    L --> M

    subgraph V["④ VALIDATION — deterministic"]
        M["Verbatim-span check → reject field if span not locatable"] --> N["Null definition → escalate to next 3 evidence chunks<br/>max 2 rounds → else flag 'term found, no definition located'"]
        N --> O["Conflicting definitions → definition_conflict + variants[]"]
    end

    O --> Q["⑤ DIFF vs active glossary version<br/>duplicate (skip) · new · conflicting"]
    Q --> R

    subgraph RV["⑥ REVIEW — human"]
        R["Queue sorted by mention frequency<br/>PA 250 → UA 180 → … → long tail"] --> S["Expert (Mas Beta) approves / edits / rejects"]
        S --> T["Glossary · Interpretation Pack · Brief Context · Formula"]
        T --> U["Versioning + approval audit trail"]
    end

    U --> W["MCP server — the consumption surface"]

    style P fill:#e8f4ff,stroke:#4a90d9
    style F fill:#e6f7e6,stroke:#4caf50
    style X fill:#dbe9ff,stroke:#2f6fd0
    style V fill:#f0e6ff,stroke:#8e5fd0
    style RV fill:#fff0e0,stroke:#e08b3c
    style SEAM fill:#fffbe0,stroke:#c9a227,stroke-width:3px
```

Colour convention, carried from the diagrams shown to the team:
**green = free** (CPU/regex — most of the pipeline) · **blue = LLM** (the only paid part) ·
**purple = deterministic code** · **orange = human**.

### The five-box version (how this is explained to execs)

```mermaid
flowchart LR
    A["PARSE<br/>MinerU"] --> B["FILTER<br/>free, CPU"] --> C["EXTRACT<br/>LLM, per term cluster"] --> D["VALIDATE + DIFF<br/>deterministic"] --> E["EXPERT REVIEW<br/>frequency-sorted queue"]
    style A fill:#e8f4ff,stroke:#4a90d9
    style B fill:#e6f7e6,stroke:#4caf50
    style C fill:#dbe9ff,stroke:#2f6fd0
    style D fill:#f0e6ff,stroke:#8e5fd0
    style E fill:#fff0e0,stroke:#e08b3c
```

### Ownership view

```mermaid
flowchart LR
    subgraph S["Sofhia"]
        P["Parsing pipeline<br/>MinerU + backends"]
    end
    subgraph R["Rifqi"]
        K["Extraction pipeline<br/>filter → LLM → validate → diff → queue"]
    end
    subgraph H["Mas Har / Mas Beta"]
        C["Curation UI + expert review<br/>versioning, approval"]
    end
    P -->|"versioned parsed artifact<br/>(the only contract)"| K
    K -->|"candidate entries + provenance"| C
    C -->|"active versions"| M["MCP server"]
```

---

## 4. Decisions already settled — do not reopen these

1. **The LLM call unit is the term cluster** — not the chunk, not the mention. Per-chunk works
   for a 9-page document and breaks at 1,000+ pages: "PA" mentioned 250× would produce 250
   near-identical candidates. Clustering first cuts expert review burden ~6.2× (3,125 raw
   extractions → ~500 term entries on a 1,000-page document). The token saving is trivial
   (~$0.45); **the review-burden reduction is the justification.**
2. **Conflict detection depends on the clustering.** Two contradictory definitions of the same
   term can only be spotted because all evidence for that term arrives in one call. Two
   separate calls would never meet, and the conflict would surface as two silently-accepted
   entries.
3. **No standalone relevance gate.** A binary relevance classifier before the term filter is a
   second imperfect classifier whose false negatives drop content *before* extraction sees it —
   and dropped content never reaches expert review to be caught. GLiNER is cheap enough (CPU,
   ~400MB) to run unconditionally on every chunk; "zero candidate spans" becomes the relevance
   signal as a byproduct of looking, rather than a judgment made before looking.
4. **Evidence ranking is allowed even though it is also a filter** — because no term is
   dropped (only which passages feed the call is narrowed), all mentions stay in provenance so
   a bad top-3 is visible, and the escalation loop self-corrects. The rejected relevance gate
   had none of those three properties. This distinction is the reasoning to preserve if the
   ranking is ever revisited.
5. **`provenance.span` is mandatory and verbatim.** A field whose span cannot be located in the
   source is rejected. This is the primary anti-hallucination control and the thing that makes
   expert review *finishable* — the reviewer checks a quote against a page, not a claim against
   their memory.
6. **All content fields are Optional in the schema.** The model must be able to answer null; a
   required field is a fabrication invitation for smaller models. `subdomain_tags` is an enum —
   classification, not generation.
7. **Frequency-sorted review queue.** Directly targets the Mas Beta bottleneck: validate the
   ~40 terms carrying the document's meaning first, let the long tail wait.
8. **Model tier is per branch.** Glossary and Formula are extraction tasks well-guarded by span
   verification → a small/cheap tier is fine. Rule-of-thumb and Summary are reasoning/generation
   tasks where a small model's failure mode is least detectable (a plausible summary cannot be
   span-checked) → larger tier. Test before committing the small tier on the rule-of-thumb
   branch.
9. **GLiNER-hybrid stays the glossary-branch default.** Pure NER can only fill 2 of 6 schema
   fields (definition, formula and interpretation are not spans — they are assembled from text
   that may be pages apart). Pure small-LLM shows recall 0.93–1.00 at precision 0.15–0.43;
   perfect recall is the failure signature of a model fabricating a value for every field.
10. **The pipeline records the literal source wording.** It recorded "Physical of Availability"
    from the BUMA standard rather than the textbook's "Physical Availability" — surfacing the
    discrepancy to the expert instead of silently normalising it. Keep that behaviour.

---

## 5. Deployment reality — this shapes every engineering trade-off

The pipeline runs **once at setup (bulk corpus ingest)** and **occasionally when an admin or
expert adds a file**. It is **not continuous** and **not per-user** — normal engineers cannot
add data. Consequences, in the order they matter:

- **Extraction quality and expert-review efficiency dominate.** A missed term has no
  self-correcting mechanism, because engineers can't add data to fill a gap later.
- **Maintainability matters more than it looks.** A glue chain invoked every few months is
  exactly the thing nobody remembers how to debug. Prefer boring, inspectable stages over
  clever ones.
- **API cost is negligible and latency is irrelevant.** ~$8–12 for a 6,000-page corpus;
  an incremental single 9-page file is ~1.5 US cents. Do not spend engineering time optimising
  tokens — engineering time on the review experience is worth more than any token saving
  available here.
- **No GPU required.** GLiNER is a CPU encoder model (~1.6 seq/s Python, ~6.7 seq/s via the
  Rust port on an i9 8-core). ~15,000 chunks ≈ 2.6h Python / ~40min Rust for a job that runs a
  few times a year. The LLM half is an API call. (Runtime as reported to the team on 13 Aug:
  ~11 s/page GLiNER, ~26 s/page LLM — with the caveat that **cost scales on term count, not
  page count.**)
- **Two ingest modes to build:** *bulk setup ingest* (corpus-scale — the only place
  corpus-frequency statistics are usable as a candidate-term booster) and *incremental
  single-file add* (no corpus context; diff against the active glossary version).
- **The summary branch is the quiet cost risk** — only ~300 calls but a quarter of all input
  tokens, because summarisation can't be filtered; it needs whole documents.

---

## 6. Where the output goes

Approved candidates become the project's **four artifacts** (see the `context` doc for the
full architecture):

1. **Domain knowledge** — system-prompt material, capped ~1500 tokens: operating and equipment
   hierarchy, time convention (WITA), seasonality, most-important glossary terms, high-level
   business process, and agent limitations.
2. **Data dictionary** — from DB column profiling; used for query generation. Pak Ricky
   connected the extraction work directly to **automating this**.
3. **Interpretation pack** — expert-curated interpretation logic, action rules, benchmark
   history, tied to a use case ("PA missed" reads differently from "PA missed but MTBS
   achieved"). Revived on 13 Aug after earlier removal for being too normative — reintroduce
   carefully.
4. **Skill registry** — query rules and expert-approved formulas. Approval is mandatory because
   the same term computes differently per company (MTTR at BUMA = breakdown duration ÷
   breakdown frequency; MTTR in IT = mean time to resolve).

All four carry **versioning plus an approval audit trail**, and the expert chooses which
version is active. Delivery format is **MCP** — skill registry → MCP skills; data dictionary →
MCP on demand; interpretation pack → attached to the skill; domain knowledge → system prompt
but preferably served via MCP so owners can change it without redeploying the engine.

---

## 7. Grounding material

Two real sample documents anchor every design claim above, and should anchor test fixtures too:

- **BUMA STD/2026/006/MNO Rev.0.0** — Production Parameter & ECA. 9 pages, majority Bahasa
  Indonesia. Carries: PA (glossary, p.4), Other Activity + the controllability hierarchy
  (rule-of-thumb, pp.8–9), the MOHH/Qty/PA/UA/Pty legend block (formula branch, p.2).
- **Open Pit Mine Planning & Design** (textbook excerpt) — carries NSR as a **cross-page
  definition**: intro p.92, formula p.93, interpretive remark p.94 (§2.3.5). This is the case
  that justifies section-aware chunking over fixed token windows — the whole definition must
  stay inside one semantic unit.

**Known open risk:** GLiNER multi-v2.1's Indonesian performance is unverified, and the BUMA
standard is majority Bahasa Indonesia. This is the standing "Indonesian degradation" question,
now with a concrete test case. Verify before trusting recall numbers.

**Flagged as scope creep, proposed not decided:** ensembling GLiNER + LangExtract on every
ingest. Cheap in API terms, but it adds a reconciliation/dedup step between two candidate sets
that may disagree — new engineering surface for uncertain gain.

---

## 8. Integration guidance for this repo

Deliberately shape-agnostic — decide the specifics against `CLAUDE.md` and the existing
subsystem patterns, not against this doc.

- **Both halves are new subsystems, not modifications** to the existing document/catalog
  pipelines. The current unstructured path (Tesseract OCR → chunk → pgvector) stays as it is;
  the MinerU path is additive and, at least initially, gated.
- **Follow the repo's existing multi-stage precedent.** The query subsystem (IR → validator →
  compiler → executor, orchestrated from a service facade, triggered by thin entry points) is
  the closest structural analogue to what both halves need.
- **Respect the hard boundaries.** Go owns the dedorch schema — any new table needs a
  Harry-ready DDL handoff, never DDL executed from Python. Any new endpoint on the live surface
  needs a contract-doc entry, and an admin ingest surface is a different risk class than the
  read-only chat surface — raise the authentication question rather than assuming.
- **Build the offline path first.** Because this runs a few times a year and is triggered by an
  admin, a script-driven run over a fixture document is the honest first milestone; HTTP
  endpoints are a convenience layer over it, not the other way round.
- **Fixtures over live parsing in tests.** A committed parsed-artifact fixture from one of the
  two sample documents lets the extraction half be tested without MinerU installed at all —
  which is the seam doing its job.