[NOTICKET] docs(knowledge-parsing): module README + refresh the parsing/seam statuses

README follows the per-module convention already used by src/catalog, src/query,
src/retrieval and src/security, and records the three things that are easy to get
wrong and expensive to discover: text stays verbatim, page_idx is 0-based, and
markup never goes in text (with the recall numbers behind that last one).

KNOWLEDGE_PIPELINE_TODO.md — parsing (B) and seam (S) rows only; P/X/D/E/F and
S7/S8 are untouched:

- B6, B8, S2, S3, S4 -> done. Contract landed, envelope carries parser identity
read back from MinerU's _middle.json, section-structure claim corrected against
MinerU's source, source_wording added.
- B4, B5, S1, S5, S6 -> in progress, deliberately NOT marked done:
B5's acceptance test needs the Open Pit textbook (S7) before the NSR
cross-page case can be claimed; S5 is satisfied on the parsing side but is
Rifqi's to close; B4's language question is answered but text quality is not
yet judged. Marking unproven work done is what makes a checklist useless.
- S6: (c) chunk size, (d) chapters-as-list and (e) pydantic are closed. (b) page
indexing stays open as a joint call. (a) is open with its premise corrected —
text_format is emitted by both backends and is not a formatting signal, and
MinerU carries no emphasis information anywhere, so that ranker signal has no
source short of a separate PDF pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KNOWLEDGE_PIPELINE_TODO.md

@@ -75,7 +75,12 @@ either port scaffolding you didn't want or discard measurements you can't recrea
 
 ## 3. The seam — shared with Sofhia
 
-**Status: under active discussion.** Sofhia circulated a draft `contracts.py` on 2026-08-19,
+**Status update 2026-08-19 (Sofhia).** The contract has landed in
+`src/knowledge_parsing/contracts.py` and every open item raised against the draft is closed except
+the two that need a joint decision (S6a formatting, S6b page indexing). Rows below updated
+accordingly; the extraction-side rows (S7, S8) are left as Rifqi has them.
+
+**Original note — under active discussion.** Sofhia circulated a draft `contracts.py` on 2026-08-19,
 derived from real MinerU output rather than designed on paper, and it is adopted as the base. Three
 of its choices improve on the prototype: a `kind` discriminator (text/table/chart/equation),
 `source_items` index back-references into MinerU's `content_list.json`, and an explicit
@@ -84,12 +89,12 @@ verbatim-text warning that independently identifies the same failure mode as the
 
 | # | Task | Owner | Status | Description |
 |---|---|---|---|---|
-| **S1** | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | Under discussion. Everything else in this section is an open item against her draft |
-| **S2** | Add a document-level envelope | Sofhia + Rifqi | 🔄 | The draft is `list[Chunk]` with no wrapper, so there is nowhere for `content_hash`, `n_pages`, `source_path`, or **`version`** — and a versioned artifact is a plan requirement. Proposed: `ParsedDocument{doc_id, source_path, content_hash, version, n_pages, parser_name, parser_version, chunks}`. Parser identity matters *because* the seam exists to make parsers swappable: when extraction output shifts, we must be able to tell a parser change from a prompt change |
-| **S3** | Correct the section-structure claim | Rifqi → Sofhia | 🔄 | The draft states MinerU does not supply section structure (`text_level` on 1 of 84 items). **On the BUMA standard it does:** the full 9-page parse carries `text_level` on **29 of 118 items**, cleanly numbered `1.` → `2.2.5`, including `2.1.3. Physical of Availability (PA)`. Her sample appears to be the handbook (its `doc_id` is `handbook`, and it has item types `page_number`/`header` absent from the BUMA parse), which genuinely has no numbered headings. Keeping both fields Optional is correct; the stated *reason* is not, and acting on it would starve section-aware chunking on exactly the documents that carry numbering — i.e. every client standard |
-| **S4** | Add `source_wording` while the contract is open | Rifqi → Sofhia | 🔄 | Cheap moment to fix the defect in §7. With `heading` carried verbatim, a `source_wording` field span-checked against it makes *"Physical **of** Availability"* survive instead of being silently normalised |
-| **S5** | Keep the seam off the file path | Sofhia + Rifqi | ⬜ | The prototype's boundary was `Parser.parse(path) -> list[Page]` — the one shape the plan forbids. v2 must not reproduce it. Must be settled **before** the MinerU parser is built against it, or it becomes a rewrite of Sofhia's work |
-| **S6** | Minor contract items (deferred to review) | Rifqi → Sofhia | ⏸️ | Raised but not blocking: **(a)** the formatting evidence signal — ranking uses six signals including formatting, read from `bold_spans`; the draft rightly excludes it as backend-specific, so either derive it from markdown emphasis or drop it deliberately, not by silence. **(b)** page indexing — draft is 0-based "as MinerU reports", prototype is 1-based, and page numbers are shown to the expert; name it `page_idx` if 0-based. **(c)** chunk-size policy — the sample chunk spans 32 source items across 2 pages; the prototype capped ~1,500 tokens. **(d)** `chapter` is lossy when a chunk crosses a chapter boundary. **(e)** pydantic over dataclass, matching this repo. **(f)** point `TermRecord` at the prototype's richer entry rather than re-deriving it |
+| **S1** | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | **Adopted as the base and landed** in `src/knowledge_parsing/contracts.py` (pydantic, per S6e). Everything raised against the draft is closed except the two joint decisions in S6 (a: formatting signal, b: page indexing). |
+| **S2** | Add a document-level envelope | Sofhia + Rifqi | ✅ | **Done 2026-08-19.** `ParsedDocument{doc_id, source_path, content_hash, version, n_pages, parser_name, parser_version, chunks}` as proposed, plus three fields worth having: **`parser_backend`** read back from MinerU's `_middle.json` so the artifact records what *actually* ran rather than what was configured; `parser_config` (fingerprint of the settings that affect output); and `raw_output_dir`, pointing at the untouched MinerU output kept in the parse cache. |
+| **S3** | Correct the section-structure claim | Rifqi → Sofhia | ✅ | **Corrected 2026-08-19.** Checked against MinerU's source: `pipeline` and `vlm` run *identical* `text_level` logic and both emit `bbox`, so heading availability is document-dependent, not backend-dependent — the original claim came from sampling the handbook, which has no numbered headings. The normalizer now **reads** the hierarchy into a `heading_path` breadcrumb instead of deriving it, with numbering as fallback. Fields stay Optional. |
+| **S4** | Add `source_wording` while the contract is open | Rifqi → Sofhia | ✅ | **Done 2026-08-19.** `source_wording` added to `TermRecord`, and `heading` is carried verbatim — so "Physical **of** Availability" survives instead of being silently normalised (R1). |
+| **S5** | Keep the seam off the file path | Sofhia + Rifqi | 🔄 | **Satisfied on the parsing side.** `ParsedDocument` never carries a file handle; `source_path` is provenance only, and extraction receives the artifact. Left open for Rifqi to confirm from his side — the prototype's `Parser.parse(path)` boundary is the shape this must not reproduce. |
+| **S6** | Minor contract items (deferred to review) | Rifqi → Sofhia | 🔄 | **Four of six closed 2026-08-19; two need a joint call.** **(b) page indexing — OPEN:** the artifact uses `page_idx`/`page_idxs`, **0-based**, no conversion anywhere, so it always matches the raw MinerU output kept beside it; converting to 1-based is the UI's job, done once at display. Say if you want a 1-based `page_no` added instead, since this number reaches Mas Beta. **(a) formatting signal — OPEN, but the premise was wrong:** `text_format` is emitted by *both* backends and is not a formatting signal at all — it is the constant `'latex'` on equation blocks. Checked further: MinerU carries **no emphasis information anywhere** (zero `**` in its markdown, no style/font/span keys in `middle.json`). So the signal has no source in MinerU; it would need a separate pass reading the PDF directly (PyMuPDF exposes per-span bold flags). Recommend dropping it from the ranker for now — one of six signals, and adding a PDF pass buys nothing measured yet. Your ranker, your call. **(c) chunk size — done:** ~1500-token cap, matching calibration §4; splits at item boundaries so text stays verbatim. **(d) `chapter` lossy — done:** now `chapters: list[str]`, one per page the chunk spans. **(e) pydantic — done.** **(f) `TermRecord`** — left to you; the declaration in `contracts.py` is a placeholder, not a claim. |
 | **S7** | Commit fixture artifacts | Rifqi | ⬜ | Both source PDFs exist (BUMA standard + Open Pit textbook). Promote a parsed artifact of each into the target repo as committed fixtures, so the extraction half is testable **with MinerU not installed at all** — the seam doing its job |
 | **S8** | Write the seam rule into the repo docs | Rifqi | ⬜ | Extraction never takes a file path; neither half imports the other's internals; parsing stays swappable behind the artifact. The prototype drifted from this, so the rule needs to outlive our memory of it |
 
@@ -105,11 +110,11 @@ detection with type labels. Presented to the execs 13 Aug as the prerequisite fo
 | **B1** | MinerU cost justification (written) | Sofhia | ⬜ | Owed for the budget proposal, from the 13 Aug exec review |
 | **B2** | MinerU install + run on the sample docs | Sofhia | ✅ | Complete 9-page parse of the BUMA standard delivered 2026-08-19: **118 items** (90 text, 14 table, 11 equation, 3 chart), pages 0–8. Includes the rule-of-thumb grounding content (Other Activity, Controllable/Uncontrollable, Standby), so the fixture exercises all four extraction branches. Supersedes an earlier 5-page run |
 | **B3** | Evaluate MinerU quality vs. the current path | Sofhia | ⬜ | Compare on tables, the p.2 legend block, formulas and figures — against the PyMuPDF baseline the prototype used, which has real numbers rather than a guess |
-| **B4** | Indonesian check on **parsing** | Sofhia | ⬜ | Distinct from E1, which measured GLiNER, not the parser. Confirm MinerU's text and layout quality hold on Bahasa Indonesia |
-| **B5** | Section-aware chunking | Sofhia | ⬜ | Split on numbered headings rather than fixed token windows. Acceptance test: the NSR definition in the Open Pit textbook (intro p.92, formula p.93, remark p.94) stays inside one semantic unit. The prototype's chunker solved this for the PyMuPDF path and is worth reading before rebuilding — it already handles the breadcrumb problem below |
-| **B6** | Emit the agreed artifact | Sofhia | ⬜ | The parser writes our schema, not MinerU's native shape. This is what keeps MinerU swappable. Gated on §3 |
+| **B4** | Indonesian check on **parsing** | Sofhia | 🔄 | Distinct from E1, which measured GLiNER, not the parser. Partly answered 2026-08-19: MinerU has **no `id` language code** — Indonesian documents use `ch`, whose model covers Latin script (`config.periksa()` now rejects a wrong code before models load). `lang` only applies to the `pipeline` backend and only when OCR actually runs, so it is moot under `vlm`. Text/layout **quality** on Bahasa Indonesia still to be judged against B3 |
+| **B5** | Section-aware chunking | Sofhia | 🔄 | **Built 2026-08-19** in `normalize.py`. Splits on MinerU's `text_level` hierarchy into a `heading_path` breadcrumb, with numbering as fallback. Three real-document traps handled, two of them from §4 of the calibration doc: repeated breadcrumbs no longer reopen a section, **only numbered headings** may open one (so `Keterangan:` keeps a legend with its figure), and a ~1500-token cap guards documents with no detected headings. **Acceptance test still open** — needs the Open Pit textbook (S7) to confirm the NSR definition survives pp. 92–94 intact |
+| **B6** | Emit the agreed artifact | Sofhia | ✅ | **Done 2026-08-19** (`5840d201`). `src/knowledge_parsing/` writes `ParsedDocument`, pydantic, not MinerU's native shape. `parse.py` keeps a content-addressed cache of the untouched MinerU output beside it, and `parser_backend` is read back from `_middle.json` so the artifact records what actually ran |
 | **B7** | Fallback path behind the same artifact | Sofhia | ⬜ | Tesseract / Azure Document Intelligence emits the *same* artifact, so extraction never knows which parser ran |
-| **B8** | Keep the parsing subsystem additive + gated | Sofhia | ⬜ | The existing unstructured path (`src/knowledge/`, Tesseract OCR → chunk → pgvector) stays untouched. MinerU is a new flag-gated path beside it |
+| **B8** | Keep the parsing subsystem additive + gated | Sofhia | ✅ | **Done 2026-08-19.** `src/knowledge/` is untouched; `src/knowledge_parsing/` sits beside it. MinerU is an **optional extra** (`knowledge-parsing`), not a main dependency, and is imported inside `parse.py` at call time — verified that importing the package leaves `mineru` and `torch` out of `sys.modules`, so the deployed Space neither builds nor ships them |
 
 **Findings from the real MinerU output that affect B5/B6:**
 

src/knowledge_parsing/README.md

@@ -0,0 +1,76 @@
+# knowledge_parsing
+
+Document parsing — the first half of the knowledge pipeline. PDF/DOCX in, a
+versioned `ParsedDocument` out.
+
+The second half (`src/knowledge_extraction/`) consumes that artifact and never
+the parser itself, which is what keeps MinerU swappable: a Tesseract or Azure
+Document Intelligence path emits the same artifact and extraction never learns
+which one ran.
+
+Additive and flag-gated. The existing unstructured path (`src/knowledge/`,
+Tesseract OCR → chunk → pgvector) is untouched and keeps running as-is.
+
+## Files
+
+- `contracts.py` — **the seam.** `Chunk`, `ParsedDocument`, and the declared
+  `Mention` / `TermRecord` handoff shapes. The one file both halves must agree on
+- `config.py` — all settings, and the single place the GPU backend flips
+- `parse.py` — MinerU wrapper with a content-addressed parse cache
+- `normalize.py` — MinerU's flat item list → meaningful chunks
+- `render.py` — LaTeX/HTML → readable prose for `Chunk.text`
+- `checks.py` — quality warnings, including silent-corruption detection
+- `manifest.py` / `report.py` — per-run record and throughput extrapolation
+- `run.py` — batch CLI: one folder in, artifacts + manifest out
+
+## Use
+
+```bash
+python -m src.knowledge_parsing.run --input data/knowledge_docs/
+python -m src.knowledge_parsing.report --scale 6800
+```
+
+MinerU is an optional extra, not a main dependency — the agent service never
+parses documents at request time, so the deployed Space does not ship it:
+
+```bash
+uv sync --extra knowledge-parsing
+```
+
+Importing this package does not import MinerU or torch; only `parse.py` does, at
+call time.
+
+## Three things that are easy to get wrong
+
+**`Chunk.text` must stay verbatim.** Extraction's guardrail locates quoted spans
+literally inside it. Reflowing or normalising the text makes the lookup fail and
+the field goes silently null — which presents as a bad model, not as a parser
+bug.
+
+**`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion
+anywhere in the pipeline. Converting to 1-based is the UI's job, done once at
+display time, so the artifact always matches the raw MinerU output kept beside
+it in the cache.
+
+**Markup never goes in `text`.** The term filter is an NER model reading prose;
+MinerU writes formulas character-spaced (`P u r c h a s i n g ~ c o s t s`) and
+tables as HTML, and neither yields a single mention. Measured on one document and
+gold set, changing only the parse: raw markup scored recall 0.7561 against 0.8537
+for plain text; rendering it back recovered 0.8293. The markup is preserved in
+`Chunk.latex` and `Chunk.table_html`, because the formula branch needs that form.
+
+## Output layout
+
+```
+data/knowledge_cache/parse/<hash>-<config>-<version>/   untouched MinerU output
+data/knowledge_runs/<run_id>/
+├── manifest.json     backend, versions, timings, quality warnings
+├── failures.jsonl    documents that failed, with traces
+└── <doc>/chunks.json the artifact
+```
+
+Both are gitignored — they are data, not code.
+
+See `KNOWLEDGE_PIPELINE_TODO.md` (root) for the task breakdown and the seam
+discussion, and `KNOWLEDGE_PIPELINE_CALIBRATION.md` §4 for the chunking
+constants this module follows.