[NOTICKET] docs(knowledge-parsing): output contract for the parsing half

Companion to KNOWLEDGE_OUTPUT_CONTRACT.md, which covers extraction. Values are
literal from run 20260824-100720 on the 9-page BUMA standard, not illustrative.

Worth noting for the transport discussion: the envelope the extraction contract
proposes as "not built, for discussion" already exists here. chunks.json is a
self-describing object carrying parser name, version, backend, config
fingerprint and source hash — reusing it beats inventing a second shape.

Three gaps recorded rather than smoothed over:

- chunk_id disagrees across the seam. This side emits "doc::0003"; the
extraction contract shows "doc#0007". Provenance back-references break if a
consumer splits on the wrong separator. Needs one joint call.
- kind="equation" is declared but never emitted; all 11 equations arrive as
latex on text chunks. Consumers must not branch on it today.
- heading_path carries page-number pollution — its root is frequently "1. 7",
page 7's number caught as a numbered heading, in 7 of 14 distinct paths.
heading_path feeds the evidence ranker, so this is not cosmetic. Open
against B5.

Also records that the chunk count is now 31, against the 13 quoted in the
extraction contract's reference run: that run predates the section-aware
chunker. Anything calibrated on 13 needs rechecking.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md

@@ -0,0 +1,203 @@
+# Knowledge Pipeline — Parsing Output Contract
+
+**Source of truth:** `src/knowledge_parsing/contracts.py`
+**Producer:** `python -m src.knowledge_parsing.run --input data/knowledge_docs/`
+**Reference run:** `20260824-100720` · BUMA `STD_2026_006_MNO` · 9 pages · MinerU `3.4.4`, backend `pipeline`
+**Status:** offline artifacts only — no endpoint, no table. Values below are **literal from that run**, not illustrative.
+
+Companion to `KNOWLEDGE_OUTPUT_CONTRACT.md` (extraction). This is the half **upstream** of it:
+parsing emits `ParsedDocument`, extraction consumes it and never touches the parser.
+
+---
+
+## Transport reality — read this first
+
+There is **no endpoint and no table yet.** The CLI writes one directory per run:
+
+```
+data/knowledge_runs/<run_id>/
+├── manifest.json                    run record: config, timings, quality warnings
+├── failures.jsonl                   documents that failed, with tracebacks
+└── <doc_id>/
+    ├── chunks.json                  ← the artifact: one ParsedDocument
+    └── sumber-mineru.txt            pointer to the untouched MinerU output
+```
+
+**Unlike the extraction payloads, this artifact is not a bare array.** `chunks.json` is a single
+self-describing object that carries its own provenance — parser name, version, backend, config
+fingerprint, source hash. The envelope proposed at the end of the extraction contract already exists
+here, and is worth reusing rather than inventing a second shape.
+
+Three conventions hold:
+
+- **`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion anywhere. Converting to
+  1-based is the UI's job, done once at display, so the artifact always matches the raw MinerU output
+  kept beside it in the cache.
+- **`text` is VERBATIM.** Extraction's span check locates quoted spans literally inside it. Reflowing
+  or normalising it makes the lookup fail and the field goes silently null — which presents as a bad
+  model, not as a parser bug.
+- **Markup never goes in `text`.** Formulas and tables are rendered to readable prose for `text`, and
+  preserved in their original form in `latex` / `table_html`.
+
+---
+
+## 1 · `chunks.json` → `ParsedDocument` *(single object)*
+
+```json
+{
+  "doc_id": "STD_2026_006_MNO",
+  "chunks": [ "… Chunk[] …" ],
+  "source_path": "data\\knowledge_docs\\STD_2026_006_MNO.pdf",
+  "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
+  "n_pages": 9,
+  "parser_name": "mineru",
+  "parser_version": "3.4.4",
+  "parser_backend": "pipeline",
+  "parser_config": "b367868551b9",
+  "version": 1,
+  "schema_version": "0.2.0",
+  "created_at": "2026-08-24T10:07:21",
+  "raw_output_dir": "…/data/knowledge_cache/parse/f10807178b20531e-b367868551b9-3.4.4"
+}
+```
+
+| Field | Type | Req | Notes |
+|---|---|---|---|
+| `doc_id` | string | ✅ | Source filename stem |
+| `chunks` | Chunk[] | ✅ | 31 on the reference run |
+| `source_path` | string | ✅ | |
+| `content_hash` | string | ✅ | sha256 of the **source file** — a changed document forces a new artifact |
+| `n_pages` | int | ✅ | |
+| `parser_name` | string | ✅ | Default `"mineru"`. The field exists so a Tesseract / Azure DI path is distinguishable |
+| `parser_version` | string \| null | — | e.g. `"3.4.4"` |
+| `parser_backend` | string \| null | — | `pipeline` · `vlm` · `hybrid`. **Read back from MinerU's `_middle.json`** — what actually ran, not what was requested |
+| `parser_config` | string \| null | — | 12-char fingerprint of the settings that affect output |
+| `version` | int | ✅ | Version of **this artifact for this document**; bumped on re-parse |
+| `schema_version` | string | ✅ | Versions **the contract**. Currently `0.2.0` |
+| `created_at` | string \| null | — | ISO 8601, seconds precision |
+| `raw_output_dir` | string \| null | — | Untouched MinerU output, kept content-addressed and never copied |
+
+> **Why provenance is mandatory rather than nice-to-have:** without
+> `parser_version` / `parser_backend`, a MinerU upgrade and a prompt change are
+> indistinguishable when extraction results shift.
+
+---
+
+## 2 · `Chunk`
+
+```json
+{
+  "chunk_id": "STD_2026_006_MNO::0003",
+  "doc_id": "STD_2026_006_MNO",
+  "kind": "text",
+  "text": "Adalah serangkaian kegiatan yang dilakukan untuk menciptakan nilai tambah terhadap sumber daya …",
+  "page_idx": 1,
+  "page_idxs": [1],
+  "section_no": "2.1.1",
+  "heading": "Production",
+  "chapters": [],
+  "heading_path": ["2.1.1. Production"],
+  "has_formula": true,
+  "is_tabular": false,
+  "source_items": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
+  "bbox": [122, 293, 275, 307],
+  "images": ["images/76503982720c45fc9d387accb74d5f5c5083f4f3109ed7fbf6a4da345bc20e0f.jpg"],
+  "latex": ["$$\nP r o d u c t i o n = M O H H x Q t y x P A x U A x P t y\n$$"],
+  "table_html": null
+}
+```
+
+| Field | Type | Req | Notes |
+|---|---|---|---|
+| `chunk_id` | string | ✅ | **Format `<doc_id>::<seq>`**, `seq` zero-padded to 4. Note the `::` separator — see *Known gaps* |
+| `doc_id` | string | ✅ | |
+| `kind` | enum | ✅ | `text` · `table` · `chart` · `equation` |
+| `text` | string | ✅ | **VERBATIM.** Excludes the section heading line — that lives in `heading`. Consumers needing both compose them |
+| `page_idx` | int | ✅ | **0-based.** First page of the chunk |
+| `page_idxs` | int[] | ✅ | Every page the chunk spans. Default `[]` |
+| `section_no` | string \| null | — | e.g. `"2.1.3"`; null when the document is unnumbered |
+| `heading` | string \| null | — | This chunk's own section title, **verbatim as written** — `"Physical of Availability (PA)"`, not the tidied `"Physical Availability"` |
+| `chapters` | string[] | ✅ | Running header per page spanned. Default `[]` |
+| `heading_path` | string[] | ✅ | Breadcrumb of enclosing headings, outermost first |
+| `has_formula` | bool | ✅ | Cheap downstream filter |
+| `is_tabular` | bool | ✅ | |
+| `source_items` | int[] | ✅ | Indices into MinerU's `content_list.json` — the trace back |
+| `bbox` | int[] \| null | — | `[x0, y0, x1, y1]` of the first source item. Carried for a curation UI; **nothing in the pipeline reasons about it** — ordering uses item sequence, never coordinates |
+| `images` | string[] | ✅ | Paths relative to `raw_output_dir`. Formula images, table/chart crops |
+| `latex` | string[] | ✅ | Source markup for formulas, kept beside the rendered prose |
+| `table_html` | string \| null | — | Source markup for tables |
+
+### Why `text` is rendered but markup is kept
+
+The term filter downstream is an NER model reading prose. MinerU writes formulas character-spaced
+(`P u r c h a s i n g ~ c o s t s`) and tables as HTML, and neither yields a single mention.
+Measured on the same document and gold set, changing only the parse:
+
+| `text` contains | Recall |
+|---|---|
+| raw markup | 0.7561 |
+| plain text | 0.8537 |
+| markup rendered back to prose | 0.8293 |
+
+The markup is not discarded, because the formula branch needs exactly that form.
+
+---
+
+## 3 · Reference run — what 9 pages actually produced
+
+| | |
+|---|---|
+| MinerU items | 118 — 90 text, 14 table, 11 equation, 3 chart |
+| Chunks emitted | **31** — 15 `text`, 13 `table`, 3 `chart`, **0 `equation`** |
+| Images | 27 unique (**3.0 per page**) |
+| Quality warnings | 3 |
+| Parse time | 385.5 s (**42.8 s/page**, CPU) |
+
+> The chunk count differs from the `13` quoted in the extraction contract's reference run
+> (`20260813-145132`). That run predates the section-aware chunker built 2026-08-19; **31 is
+> current.** Anything calibrated against 13 chunks should be re-checked.
+
+### Quality warnings — `manifest.json`
+
+Warnings never fail a document; they are recorded so quality cannot drop silently across hundreds of
+documents. The reference run's three:
+
+| `jenis` | Page | Detail |
+|---|---|---|
+| `latex_tanpa_angka` | 1 | `P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y` — symbolic, so no digits |
+| `tabel_kosong` | 6 | 1 of 14 tables detected but empty |
+| `chart_tanpa_konteks` | 7 | Chart with neither content nor caption |
+
+The check that matters most — **every number in a formula cross-checked against the source PDF's own
+text layer** — returned clean. It exists because a CPU-pipeline parse once wrote `5600` as `55600`
+with no error and a perfectly valid-looking formula.
+
+---
+
+## Known gaps — do not design around these without asking
+
+**`chunk_id` separator disagrees with the extraction contract.** This artifact emits
+`STD_2026_006_MNO::0003`. `KNOWLEDGE_OUTPUT_CONTRACT.md` shows `STD_2026_006_MNO#0007`. Provenance
+back-references break if a consumer parses on the wrong separator. **Needs one call, then both docs
+updated.**
+
+**`kind="equation"` is declared but never emitted.** All 11 equations arrive as `latex` entries on
+`text` chunks with `has_formula=true`. Either the enum member goes, or the chunker should emit them —
+but consumers must not branch on `kind == "equation"` today.
+
+**`heading_path` carries page-number pollution.** On the reference run its root is frequently
+`"1. 7"` — page 7's number caught as a numbered heading — appearing in 7 of 14 distinct paths. And
+`"2. PENJELASAN PARAMETER"` never appears as a root although it should. `heading_path` feeds the
+evidence ranker, so this is not cosmetic. Open against B5.
+
+**`chapters` is empty on every chunk** of the reference run. The field is populated only when MinerU
+reports running headers; this document has none detected.
+
+---
+
+## Caveat on the sample values
+
+Field shapes, types and defaults are read from `src/knowledge_parsing/contracts.py`. Unlike the
+extraction contract, **the values here are literal** — taken from
+`data/knowledge_runs/20260824-100720/STD_2026_006_MNO/chunks.json`, with long strings truncated and
+marked `…`.