knowledge_pipeline_context.md

Knowledge Pipeline — Context for the Data Eyond Python Repo (19 Aug 2026)

Who this is for: a Claude Code session working in Agentic-Service-Data-Eyond-Catalog (the Python agentic service). This doc carries context and intent only — the why, the ownership split, the shape of the pipeline, and the decisions already settled. It does not prescribe folder layout, module names, or endpoint signatures; work those out against CLAUDE.md, REPO_STATUS.md and the existing subsystem patterns in the repo.

---

1. What we are doing and why now

Data Eyond is an AI data-analyst platform. The strategic thesis from the 30 July exec review is that domain knowledge is the differentiator, not architecture — orchestrators and connectors are replicable in a quarter, model capability is rented, only encoded domain knowledge compounds. The end goal is an MCP product exposing modular domain knowledge that EMA and later clients connect to.

That knowledge currently comes from experts typing it in. Mas Beta (plant & maintenance) is the single validation bottleneck — nothing becomes an artifact without him. The knowledge pipeline exists to change the expert's job from authoring to reviewing: the pipeline reads the client's own documents (standards, SOPs, handbooks) and proposes candidate knowledge entries; the expert approves, edits or rejects them.

Why it lands in this repo now. We want to integrate the pipeline into Data Eyond to test it end-to-end on real documents, in the place where the resulting knowledge will actually be consumed. This is a test integration, not a productionisation — treat it as a new, gated subsystem alongside the existing document/catalog pipelines, not as a rewrite of them.

---

2. Ownership — two items, two people, one seam

The pipeline splits cleanly in two, and the split is deliberate: they are separate, independently runnable, independently swappable pipelines.

Half	Owner	Scope
① Parsing (MinerU)	Sofhia Az-Zahra	PDF/DOCX in → structured, typed extracted content out. MinerU replaces the current text+table-only extraction: it adds formula, image, chart and diagram detection, and emits Markdown or JSON with type labels. Presented to the execs 13 Aug as the prerequisite for Scenario 2; a written cost justification for MinerU is owed for the budget proposal.
② Knowledge extraction	Rifqi (me)	Parsed content in → validated candidate knowledge entries out. Term filtering, clustering, evidence ranking, schema-guided LLM attribute fill, span validation, diff against the active glossary version, and the frequency-sorted review queue.

The seam. The two halves must meet only at a persisted, versioned parsed-document artifact — extraction consumes that artifact, never a raw file, never the parser's API. This is the single most important structural constraint in the whole plan, because:

• MinerU can be swapped (or fall back to the existing Tesseract / Azure Document Intelligence paths) without extraction knowing;
• extraction can be re-run cheaply on an already-parsed corpus when a prompt or schema changes — and it will be, often, because prompt iteration is the main development loop here;
• Sofhia and I can work and test independently, against a fixture artifact, without blocking each other;
• parsing is the slow/expensive-to-rerun half; extraction is the fast-to-iterate half. Keeping the boundary hard means we never re-parse to fix an extraction bug.

Anything that makes the two halves import each other's internals, or that makes extraction take a file path, breaks the point of the split.

---

3. The pipeline

Six stages. The cost story matters as much as the flow: almost everything is free CPU/regex work, and the LLM appears in exactly one stage.

flowchart TD
    A["Admin / expert triggers ingest<br/><i>not continuous, not per-user</i>"] --> B

    subgraph P["① PARSING — Sofhia"]
        B["MinerU<br/>text · tables · formulas · figures · charts"] --> C["Section-aware chunking<br/>split on numbered headings 2.1.1, 2.1.2 …"]
    end

    C --> SEAM[("Parsed-document artifact<br/>versioned · persisted<br/><b>THE SEAM</b>")]
    SEAM --> D

    subgraph F["② TERM FILTER — CPU / regex, free"]
        D["GLiNER span filter on ALL chunks → term mentions"] --> H
        E["Discourse-cue regex → rule-of-thumb candidates"] --> H
        G["Legend-block regex → formula vars + abbreviation dictionary"] --> H
        G2["Section pass → summary units"] --> H
        H["Normalise + cluster mentions<br/>PA · P.A. · Physical Availability → 1 cluster"] --> I["Evidence ranking<br/>cue · heading · legend · formula · formatting · first-occurrence"]
        I --> J["Top-3 evidence chunks per cluster"]
    end

    SEAM --> E
    SEAM --> G
    SEAM --> G2

    J --> K

    subgraph X["③ KNOWLEDGE EXTRACTION — the only paid stage"]
        K["1 LLM call per TERM CLUSTER<br/>(not per mention, not per chunk)"]
        L["1 call per cue chunk · 1 per unique formula · 1 per document (summary)"]
    end

    K --> M
    L --> M

    subgraph V["④ VALIDATION — deterministic"]
        M["Verbatim-span check → reject field if span not locatable"] --> N["Null definition → escalate to next 3 evidence chunks<br/>max 2 rounds → else flag 'term found, no definition located'"]
        N --> O["Conflicting definitions → definition_conflict + variants[]"]
    end

    O --> Q["⑤ DIFF vs active glossary version<br/>duplicate (skip) · new · conflicting"]
    Q --> R

    subgraph RV["⑥ REVIEW — human"]
        R["Queue sorted by mention frequency<br/>PA 250 → UA 180 → … → long tail"] --> S["Expert (Mas Beta) approves / edits / rejects"]
        S --> T["Glossary · Interpretation Pack · Brief Context · Formula"]
        T --> U["Versioning + approval audit trail"]
    end

    U --> W["MCP server — the consumption surface"]

    style P fill:#e8f4ff,stroke:#4a90d9
    style F fill:#e6f7e6,stroke:#4caf50
    style X fill:#dbe9ff,stroke:#2f6fd0
    style V fill:#f0e6ff,stroke:#8e5fd0
    style RV fill:#fff0e0,stroke:#e08b3c
    style SEAM fill:#fffbe0,stroke:#c9a227,stroke-width:3px

Colour convention, carried from the diagrams shown to the team: green = free (CPU/regex — most of the pipeline) · blue = LLM (the only paid part) · purple = deterministic code · orange = human.

The five-box version (how this is explained to execs)

flowchart LR
    A["PARSE<br/>MinerU"] --> B["FILTER<br/>free, CPU"] --> C["EXTRACT<br/>LLM, per term cluster"] --> D["VALIDATE + DIFF<br/>deterministic"] --> E["EXPERT REVIEW<br/>frequency-sorted queue"]
    style A fill:#e8f4ff,stroke:#4a90d9
    style B fill:#e6f7e6,stroke:#4caf50
    style C fill:#dbe9ff,stroke:#2f6fd0
    style D fill:#f0e6ff,stroke:#8e5fd0
    style E fill:#fff0e0,stroke:#e08b3c

Ownership view

flowchart LR
    subgraph S["Sofhia"]
        P["Parsing pipeline<br/>MinerU + backends"]
    end
    subgraph R["Rifqi"]
        K["Extraction pipeline<br/>filter → LLM → validate → diff → queue"]
    end
    subgraph H["Mas Har / Mas Beta"]
        C["Curation UI + expert review<br/>versioning, approval"]
    end
    P -->|"versioned parsed artifact<br/>(the only contract)"| K
    K -->|"candidate entries + provenance"| C
    C -->|"active versions"| M["MCP server"]

---

4. Decisions already settled — do not reopen these

1. The LLM call unit is the term cluster — not the chunk, not the mention. Per-chunk works for a 9-page document and breaks at 1,000+ pages: "PA" mentioned 250× would produce 250 near-identical candidates. Clustering first cuts expert review burden 6.2× (3,125 raw extractions → ~500 term entries on a 1,000-page document). The token saving is trivial ($0.45); the review-burden reduction is the justification.
2. Conflict detection depends on the clustering. Two contradictory definitions of the same term can only be spotted because all evidence for that term arrives in one call. Two separate calls would never meet, and the conflict would surface as two silently-accepted entries.
3. No standalone relevance gate. A binary relevance classifier before the term filter is a second imperfect classifier whose false negatives drop content before extraction sees it — and dropped content never reaches expert review to be caught. GLiNER is cheap enough (CPU, ~400MB) to run unconditionally on every chunk; "zero candidate spans" becomes the relevance signal as a byproduct of looking, rather than a judgment made before looking.
4. Evidence ranking is allowed even though it is also a filter — because no term is dropped (only which passages feed the call is narrowed), all mentions stay in provenance so a bad top-3 is visible, and the escalation loop self-corrects. The rejected relevance gate had none of those three properties. This distinction is the reasoning to preserve if the ranking is ever revisited.
5. provenance.span is mandatory and verbatim. A field whose span cannot be located in the source is rejected. This is the primary anti-hallucination control and the thing that makes expert review finishable — the reviewer checks a quote against a page, not a claim against their memory.
6. All content fields are Optional in the schema. The model must be able to answer null; a required field is a fabrication invitation for smaller models. subdomain_tags is an enum — classification, not generation.
7. Frequency-sorted review queue. Directly targets the Mas Beta bottleneck: validate the ~40 terms carrying the document's meaning first, let the long tail wait.
8. Model tier is per branch. Glossary and Formula are extraction tasks well-guarded by span verification → a small/cheap tier is fine. Rule-of-thumb and Summary are reasoning/generation tasks where a small model's failure mode is least detectable (a plausible summary cannot be span-checked) → larger tier. Test before committing the small tier on the rule-of-thumb branch.
9. GLiNER-hybrid stays the glossary-branch default. Pure NER can only fill 2 of 6 schema fields (definition, formula and interpretation are not spans — they are assembled from text that may be pages apart). Pure small-LLM shows recall 0.93–1.00 at precision 0.15–0.43; perfect recall is the failure signature of a model fabricating a value for every field.
10. The pipeline records the literal source wording. It recorded "Physical of Availability" from the BUMA standard rather than the textbook's "Physical Availability" — surfacing the discrepancy to the expert instead of silently normalising it. Keep that behaviour.

---

5. Deployment reality — this shapes every engineering trade-off

The pipeline runs once at setup (bulk corpus ingest) and occasionally when an admin or expert adds a file. It is not continuous and not per-user — normal engineers cannot add data. Consequences, in the order they matter:

• Extraction quality and expert-review efficiency dominate. A missed term has no self-correcting mechanism, because engineers can't add data to fill a gap later.
• Maintainability matters more than it looks. A glue chain invoked every few months is exactly the thing nobody remembers how to debug. Prefer boring, inspectable stages over clever ones.
• API cost is negligible and latency is irrelevant. ~$8–12 for a 6,000-page corpus; an incremental single 9-page file is ~1.5 US cents. Do not spend engineering time optimising tokens — engineering time on the review experience is worth more than any token saving available here.
• No GPU required. GLiNER is a CPU encoder model (~1.6 seq/s Python, ~6.7 seq/s via the Rust port on an i9 8-core). ~15,000 chunks ≈ 2.6h Python / ~40min Rust for a job that runs a few times a year. The LLM half is an API call. (Runtime as reported to the team on 13 Aug: ~11 s/page GLiNER, ~26 s/page LLM — with the caveat that cost scales on term count, not page count.)
• Two ingest modes to build: bulk setup ingest (corpus-scale — the only place corpus-frequency statistics are usable as a candidate-term booster) and incremental single-file add (no corpus context; diff against the active glossary version).
• The summary branch is the quiet cost risk — only ~300 calls but a quarter of all input tokens, because summarisation can't be filtered; it needs whole documents.

---

6. Where the output goes

Approved candidates become the project's four artifacts (see the context doc for the full architecture):

1. Domain knowledge — system-prompt material, capped ~1500 tokens: operating and equipment hierarchy, time convention (WITA), seasonality, most-important glossary terms, high-level business process, and agent limitations.
2. Data dictionary — from DB column profiling; used for query generation. Pak Ricky connected the extraction work directly to automating this.
3. Interpretation pack — expert-curated interpretation logic, action rules, benchmark history, tied to a use case ("PA missed" reads differently from "PA missed but MTBS achieved"). Revived on 13 Aug after earlier removal for being too normative — reintroduce carefully.
4. Skill registry — query rules and expert-approved formulas. Approval is mandatory because the same term computes differently per company (MTTR at BUMA = breakdown duration ÷ breakdown frequency; MTTR in IT = mean time to resolve).

All four carry versioning plus an approval audit trail, and the expert chooses which version is active. Delivery format is MCP — skill registry → MCP skills; data dictionary → MCP on demand; interpretation pack → attached to the skill; domain knowledge → system prompt but preferably served via MCP so owners can change it without redeploying the engine.

---

7. Grounding material

Two real sample documents anchor every design claim above, and should anchor test fixtures too:

• BUMA STD/2026/006/MNO Rev.0.0 — Production Parameter & ECA. 9 pages, majority Bahasa Indonesia. Carries: PA (glossary, p.4), Other Activity + the controllability hierarchy (rule-of-thumb, pp.8–9), the MOHH/Qty/PA/UA/Pty legend block (formula branch, p.2).
• Open Pit Mine Planning & Design (textbook excerpt) — carries NSR as a cross-page definition: intro p.92, formula p.93, interpretive remark p.94 (§2.3.5). This is the case that justifies section-aware chunking over fixed token windows — the whole definition must stay inside one semantic unit.

Known open risk: GLiNER multi-v2.1's Indonesian performance is unverified, and the BUMA standard is majority Bahasa Indonesia. This is the standing "Indonesian degradation" question, now with a concrete test case. Verify before trusting recall numbers.

Flagged as scope creep, proposed not decided: ensembling GLiNER + LangExtract on every ingest. Cheap in API terms, but it adds a reconciliation/dedup step between two candidate sets that may disagree — new engineering surface for uncertain gain.

---

8. Integration guidance for this repo

Deliberately shape-agnostic — decide the specifics against CLAUDE.md and the existing subsystem patterns, not against this doc.

• Both halves are new subsystems, not modifications to the existing document/catalog pipelines. The current unstructured path (Tesseract OCR → chunk → pgvector) stays as it is; the MinerU path is additive and, at least initially, gated.
• Follow the repo's existing multi-stage precedent. The query subsystem (IR → validator → compiler → executor, orchestrated from a service facade, triggered by thin entry points) is the closest structural analogue to what both halves need.
• Respect the hard boundaries. Go owns the dedorch schema — any new table needs a Harry-ready DDL handoff, never DDL executed from Python. Any new endpoint on the live surface needs a contract-doc entry, and an admin ingest surface is a different risk class than the read-only chat surface — raise the authentication question rather than assuming.
• Build the offline path first. Because this runs a few times a year and is triggered by an admin, a script-driven run over a fixture document is the honest first milestone; HTTP endpoints are a convenience layer over it, not the other way round.
• Fixtures over live parsing in tests. A committed parsed-artifact fixture from one of the two sample documents lets the extraction half be tested without MinerU installed at all — which is the seam doing its job.