docs/knowledge/KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md

Knowledge Pipeline — Parsing Output Contract

🔄 LIVE — the seam. Output contract of the parsing half, consumed by src/knowledge_extraction/adapter.py. Source of truth is src/knowledge_parsing/contracts.py; this doc describes it.

Source of truth: src/knowledge_parsing/contracts.py Producer: python -m src.knowledge_parsing.run --input data/knowledge_docs/ Reference run: 20260824-100720 · BUMA STD_2026_006_MNO · 9 pages · MinerU 3.4.4, backend pipeline Status: offline artifacts only — no endpoint, no table. Values below are literal from that run, not illustrative.

Companion to KNOWLEDGE_OUTPUT_CONTRACT.md (extraction). This is the half upstream of it: parsing emits ParsedDocument, extraction consumes it and never touches the parser.

⚠️ schema_version 0.4.0 — asset pointers (2026-09-10)

A figure is no longer its own chunk. Its markdown reference is folded into the prose chunk at its reading-order position, and the figure travels as an Asset on that chunk. This is a MINOR bump, not a patch: chunk boundaries change, so a 0.3.x consumer would silently read a different document shape. adapter.check_schema_version rejects a newer minor by design — parsing and extraction ship together.

Why. Before 0.4.0 an image became its own chunk with an empty text, which made it unreachable by construction: no text → no term mentions → joins no cluster → never selected as evidence. Measured across three documents: 13 figures, every one described by a vision model we pay for, and ZERO appearing in any cluster's evidence. Meanwhile the prose that said "Figure 1.1" had its ![](…) reference stripped, so nothing connected the sentence to the picture.

What changed

	0.3.3	0.4.0
A figure	its own chunk, text="", unreachable	folded into the prose chunk as , carried in Chunk.assets
A table	its own chunk	unchanged — its linearised text earns its evidence slot (74.6% of the Komatsu manual's top-3 evidence is tabular). Gains an Asset + two-way pointers
Asset bytes	returned by the API, written nowhere; images[] pointed at files that did not exist	content-addressed and persisted; Asset.storage_key
Asset identity	the parser's page-local filename	sha256(bytes)[:16]
A caption	landed in a neighbouring prose chunk, unlinked	attached to the Asset and left in the prose text (it is verbatim document content)
A description	stored raw, JSON braces included	unwrapped; description_source records what produced it

Measured effect (same three documents, no re-parse — the cache fingerprint is unchanged): chunks 32→28, 85→80, 59→44 · image chunks 2/3/8 → 0/0/0 · empty-text chunks → 0 · 13 assets persisted · 0 dangling references · 0 descriptions leaked into text · captions still 17/17 in text · recall unmoved.

The invariant that must not break: a model-written description never enters Chunk.text. text is the haystack the span check searches, so generated prose in there would let a fabrication pass the control built to catch it.

Current state (2026-09-02) — additive changes since the reference run below. The default parser is now Mistral OCR (parser_name: "mistral-ocr", parser_backend: "mistral-ocr"), API-based and needing no GPU. The artifact shape is identical whichever parser produced it — extraction never learns which ran, which is what made the swap possible at all.

2026-09-07: MinerU was DROPPED 2026-09-07 (Rifqi): commented out of the knowledge-parsing extra and installed nowhere — not the Space, not the dev venv (verified). It is not a fallback; there is one parser, Mistral OCR. Its code paths in src/knowledge_parsing/ are untouched (comment-out-don't-delete) and every MinerU measurement in these docs stands as record. Dropping it was also forced: mineru==3.4.4 pinned against this project's python-docx and pypdf, which made uv lock fail for the whole project. The parser_name / parser_backend fields stay in the contract: committed artifacts record mineru, and a reader must keep distinguishing them. schema_version is now 0.3.3, all changes additive (null/derived on older artifacts): source_title (0.3.2) and normaliser_version (0.3.1) at the document level, and a computed 1-based page_no (0.3.3) on each Chunk. The reference JSON below is still the literal 2026-08-24 MinerU run, so it predates these fields.

---

Transport reality — read this first

There is no endpoint and no table yet. The CLI writes one directory per run:

data/knowledge_runs/<run_id>/
├── manifest.json                    run record: config, timings, quality warnings
├── failures.jsonl                   documents that failed, with tracebacks
└── <doc_id>/
    ├── chunks.json                  ← the artifact: one ParsedDocument
    └── mineru-source.txt            pointer to the untouched MinerU output

Unlike the extraction payloads, this artifact is not a bare array. chunks.json is a single self-describing object that carries its own provenance — parser name, version, backend, config fingerprint, source hash. The envelope proposed at the end of the extraction contract already exists here, and is worth reusing rather than inventing a second shape.

Three conventions hold:

• page_idx is 0-based, exactly as MinerU reports it, with no conversion anywhere. Converting to 1-based is the UI's job, done once at display, so the artifact always matches the raw MinerU output kept beside it in the cache.
• text is VERBATIM. Extraction's span check locates quoted spans literally inside it. Reflowing or normalising it makes the lookup fail and the field goes silently null — which presents as a bad model, not as a parser bug.
• Markup never goes in text. Formulas and tables are rendered to readable prose for text, and preserved in their original form in latex / table_html.

---

1 · chunks.json → ParsedDocument (single object)

{
  "doc_id": "STD_2026_006_MNO",
  "chunks": [ "… Chunk[] …" ],
  "source_path": "data\\knowledge_docs\\STD_2026_006_MNO.pdf",
  "content_hash": "f10807178b20531e86237a2640ff4125d259aa68e5e9d9f910c0c2365cd92833",
  "n_pages": 9,
  "parser_name": "mineru",
  "parser_version": "3.4.4",
  "parser_backend": "pipeline",
  "parser_config": "b367868551b9",
  "version": 1,
  "schema_version": "0.3.0",
  "created_at": "2026-08-24T10:07:21",
  "raw_output_dir": "…/data/knowledge_cache/parse/f10807178b20531e-b367868551b9-3.4.4"
}

Field	Type	Req	Notes
doc_id	string	✅	Source filename stem
chunks	Chunk[]	✅	31 on the reference run
source_path	string	✅
content_hash	string	✅	sha256 of the source file — a changed document forces a new artifact
n_pages	int	✅
parser_name	string	✅	Default "mineru". The field exists so a Tesseract / Azure DI path is distinguishable
parser_version	string | null	—	e.g. "3.4.4"
parser_backend	string | null	—	pipeline · vlm · hybrid. Read back from MinerU's _middle.json — what actually ran, not what was requested
parser_config	string | null	—	12-char fingerprint of the settings that affect output
source_title	string | null	—	(0.3.2) The document's human-readable name. Equals doc_id today; a separate field so the two can diverge once doc_id becomes a content hash
normaliser_version	string | null	—	(0.3.1) Fingerprint of the normalisation code that produced these chunks. parser_config covers the parser's settings; this covers our half
version	int	✅	Version of this artifact for this document; bumped on re-parse
schema_version	string	✅	Versions the contract. 0.3.0 on this reference run; current 0.4.0 (see the banner above)
created_at	string | null	—	ISO 8601, seconds precision
raw_output_dir	string | null	—	Untouched MinerU output, kept content-addressed and never copied

Why provenance is mandatory rather than nice-to-have: without parser_version / parser_backend, a MinerU upgrade and a prompt change are indistinguishable when extraction results shift.

---

2 · Chunk

{
  "chunk_id": "STD_2026_006_MNO::0003",
  "doc_id": "STD_2026_006_MNO",
  "kind": "text",
  "text": "Adalah serangkaian kegiatan yang dilakukan untuk menciptakan nilai tambah terhadap sumber daya …",
  "page_idx": 1,
  "page_idxs": [1],
  "section_no": "2.1.1",
  "heading": "Production",
  "chapters": [],
  "heading_path": ["2.1.1. Production"],
  "has_formula": true,
  "is_tabular": false,
  "source_items": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
  "bbox": [122, 293, 275, 307],
  "images": ["images/76503982720c45fc9d387accb74d5f5c5083f4f3109ed7fbf6a4da345bc20e0f.jpg"],
  "latex": ["$$\nP r o d u c t i o n = M O H H x Q t y x P A x U A x P t y\n$$"],
  "table_html": null
}

Field	Type	Req	Notes
chunk_id	string	✅	Format <doc_id>::<seq>, seq zero-padded to 4. Note the :: separator — see Known gaps
doc_id	string	✅
kind	enum	✅	text · table · chart · equation · image. Backends disagree on the label — a figure pipeline calls a chart, vlm calls an image. Branch on both, never on one
text	string	✅	VERBATIM. Excludes the section heading line — that lives in heading. Consumers needing both compose them
page_idx	int	✅	0-based. First page of the chunk
page_idxs	int[]	✅	Every page the chunk spans. Default []
page_no	int	✅	(0.3.3) 1-based, computed page_idx + 1. The number a human / review UI reads; derived, never stored, so it cannot drift from page_idx
section_no	string | null	—	e.g. "2.1.3"; null when the document is unnumbered
heading	string | null	—	This chunk's own section title, verbatim as written — "Physical of Availability (PA)", not the tidied "Physical Availability"
chapters	string[]	✅	Running header per page spanned. Default []
heading_path	string[]	✅	Breadcrumb of enclosing headings, outermost first
has_formula	bool	✅	Cheap downstream filter
is_tabular	bool	✅
source_items	int[]	✅	Indices into MinerU's content_list.json — the trace back
bbox	int[] | null	—	[x0, y0, x1, y1] of the first source item. Carried for a curation UI; nothing in the pipeline reasons about it — ordering uses item sequence, never coordinates
images	string[]	✅	⚠️ DEPRECATED 0.4.0 — superseded by assets. Still populated for one version so a 0.3.x reader degrades rather than breaking. Holds the parser's own filenames, which are not reliable: Mistral reported image_id: img-2.jpeg on a block whose markdown said img-1.jpeg, so two figures collided and one image's bytes were lost. Read assets
assets	Asset[]	✅	New 0.4.0. Figures and tables this chunk contains or points at. Default []
referenced_by	string[]	✅	New 0.4.0. chunk_ids pointing AT this chunk. Populated on table chunks. Default []
latex	string[]	✅	Source markup for formulas, kept beside the rendered prose. ⚠️ LOAD-BEARING — extraction's formula guard validates against this field. Reflowing it or leaving it empty weakens the check with no error. See below
table_html	string | null	—	Source markup for tables
generated_description	string | null	—	⚠️ MODEL-WRITTEN, NOT FROM THE DOCUMENT. MinerU's description of an image or chart. Never span-checkable — treat as a model claim about a picture, never as evidence. See below

2b · Asset (new at 0.4.0)

{
  "asset_id": "d2e7b4ffe31006ec",
  "kind": "figure",
  "storage_key": "assets/d2e7b4ffe31006ec.jpeg",
  "page_idx": 2,
  "page_no": 3,
  "bbox": [93, 562, 434, 907],
  "caption": "Figure 1.1. The relationship between exploration information, resources and reserves (SME, 1991).",
  "description": "The diagram illustrates the classification of mineral resources and reserves based on increasing levels of geological knowledge…",
  "description_source": "mistral-ocr/bbox_annotation"
}

Field	Type	Req	Notes
asset_id	string	✅	sha256(bytes)[:16]. Not the parser's id — see the images row. Content addressing makes a re-parse idempotent and de-duplicates a figure that appears twice (Open Pit has one such pair)
kind	enum	✅	figure · chart · diagram · table — one vocabulary for both cases
storage_key	string | null	—	Where the bytes live. null for a table (no bytes of its own) and whenever persistence was skipped or failed — writing assets is a never-throw seam
page_idx	int	✅	0-based, as the parser reports
page_no	int	✅	Computed page_idx + 1. The number a human reads; derived so it cannot drift
bbox	int[] | null	—	[x0,y0,x1,y1] of the asset, so a UI can highlight the figure. Distinct from the host chunk's bbox, which stays where the text is
caption	string | null	—	VERBATIM from the document's own caption block. Quotable as evidence, span-checkable. Also left in the prose text, where it already landed — the asset gets a copy, not a move
description	string | null	—	⚠️ MODEL-WRITTEN. Never span-checkable, never evidence, and never in text
description_source	string | null	—	e.g. mistral-ocr/bbox_annotation, so a later model swap is traceable rather than silently changing what the field means

A figure's reference appears inline in its host chunk's text, at the reading-order position the document put it:

…so closely and the geological character is so well defined that size, shape, depth
and mineral content of the resource are well established.

![](asset://d2e7b4ffe31006ec)

Figure 1.1. The relationship between exploration information, resources and reserves (SME, 1991).

– Indicated. Quantity and grade…

Two consumer obligations:

• Resolve asset://<id> against that chunk's own assets. Every reference is guaranteed to resolve locally (0 dangling across the three test documents).
• Switch on the caption/description distinction. A caption may be quoted and used as a provenance span; a description may not, and a span taken from one is rejected by the validator.

---

Why text is rendered but markup is kept

The term filter downstream is an NER model reading prose. MinerU writes formulas character-spaced (P u r c h a s i n g ~ c o s t s) and tables as HTML, and neither yields a single mention. Measured on the same document and gold set, changing only the parse:

text contains	Recall
raw markup	0.7561
plain text	0.8537
markup rendered back to prose	0.8293

The markup is not discarded, because the formula branch needs exactly that form.

latex is load-bearing — changing it weakens a guardrail silently

As of 2026-08-26 this field carries weight it did not carry before. Extraction's formula guard validates a model's formula_latex claim against Chunk.latex — not against text, which never contains markup. Until that landed, latex was carried but unread; a parser change touching it would have broken nothing. That is no longer true.

The failure mode is the dangerous kind. If this field is reflowed, "tidied", or left unpopulated, the guard does not error — it quietly has less to check against, and formula claims it can no longer verify are recorded as unverifiable rather than caught. Fewer things fail; nothing looks wrong.

Two obligations follow, and they run opposite to the rule for text:

• Do not clean it. MinerU's character-spacing (P r o d u c t i o n), its \mathrm{} wrappers and its ~ spacing are all expected — extraction canonicalises before comparing, so cosmetic differences are already tolerated. Pre-cleaning here removes information the canonicaliser was built to absorb.
• Do not drop it. An artifact with no latex leaves every formula claim unverifiable. That is handled gracefully, but it is a guard switched off.

What extraction tolerates is notation; what it will not tolerate is structure. Presentational wrappers, \cdot versus \times, and brace style are all erased before comparison. A missing \frac is not — that is a different formula, and catching it is the point of the guard.

Worked evidence. The pipeline backend writes multiplication as a bare letter x, with no \times anywhere. Extraction cannot then tell a multiplication sign from a variable, so every product formula from such an artifact comes back latex_verification: "unverified_operator" — correctly, since the source genuinely cannot express the operator. This is a real state, not a hypothetical: every artifact produced without a GPU is a pipeline artifact. It is one more reason the production backend is hybrid/high (see KNOWLEDGE_PIPELINE_CALIBRATION.md §13).

Cheap health check, no paid run needed. latex_verification is set during validation, not by a model call, so counting verified against unverified_operator in formulas.json measures directly whether an artifact's markup is rich enough to support the guard.

What decides a chunk boundary — not the page

A page break is a printing artifact, not a meaning boundary. Cutting there would sever a definition that happens to straddle one, and the evidence ranker downstream reasons about sections. So chunking follows the document's own structure, in this priority:

	Opens a new chunk
1	A numbered section heading	the primary boundary
2	A table or a chart	one MinerU item → one chunk
3	6 000 characters (~1 500 tokens)	safety net only for documents where no heading is detected; without it one chunk could swallow a whole document, blunting evidence ranking and inflating the summary branch's token share. Splits at an item boundary, so text stays verbatim

Measured on the reference run — the spread is content-driven, not fixed:

Chunks per page	1 to 7 (page 0 → 1, page 5 → 7)
Chunks spanning >1 page	0 of 31 here, though page_idxs is a list precisely because it can happen
text length	median 48 words, max 444, one empty (the caption-less chart)

Rule 3 therefore almost never fires on a well-structured standard — the headings do the work. It matters for the mid-chapter textbook pages, which carry no headings at all.

---

3 · Reference run — what 9 pages actually produced

MinerU items	118 — 90 text, 14 table, 11 equation, 3 chart
Chunks emitted	31 — 15 text, 13 table, 3 chart, 0 equation
Images	27 unique (3.0 per page)
Quality warnings	3
Parse time	385.5 s (42.8 s/page, CPU)

The chunk count differs from the 13 quoted in the extraction contract's reference run (20260813-145132). That run predates the section-aware chunker built 2026-08-19; 31 is current. Anything calibrated against 13 chunks should be re-checked.

Quality warnings — manifest.json

Warnings never fail a document; they are recorded so quality cannot drop silently across hundreds of documents. The reference run's three:

jenis	Page	Detail
latex_tanpa_angka	1	P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y — symbolic, so no digits
tabel_kosong	6	1 of 14 tables detected but empty
chart_tanpa_konteks	7	Chart with neither content nor caption

The check that matters most — every number in a formula cross-checked against the source PDF's own text layer — returned clean. It exists because a CPU-pipeline parse once wrote 5600 as 55600 with no error and a perfectly valid-looking formula.

---

generated_description — why it is not in text

text is the haystack the extraction guardrail searches to prove a quote is verbatim. Generated prose placed there would let a hallucinated sentence pass the very check built to catch hallucination — silently, and with no code change to blame. A caption printed in the document is verbatim and stays in text; a description invented about the picture does not.

Whether it is populated at all depends on the backend, read from MinerU's own source (is_image_analysis_option_visible):

Backend	Describes images
pipeline	never — content is always empty
hybrid + effort=medium	disabled outright
hybrid + effort=high	yes
vlm	yes, always

So an artifact from pipeline carries null here throughout. That is the backend's behaviour, not a parsing failure — and one more reason parser_backend has to travel with the data.

---

Known gaps — do not design around these without asking

chunk_id separator disagrees with the extraction contract. This artifact emits STD_2026_006_MNO::0003. KNOWLEDGE_OUTPUT_CONTRACT.md shows STD_2026_006_MNO#0007. Provenance back-references break if a consumer parses on the wrong separator. Needs one call, then both docs updated.

kind="equation" is declared but never emitted. All 11 equations arrive as latex entries on text chunks with has_formula=true. Either the enum member goes, or the chunker should emit them — but consumers must not branch on kind == "equation" today.

heading_path carries page-number pollution. On the reference run its root is frequently "1. 7" — page 7's number caught as a numbered heading — appearing in 7 of 14 distinct paths. And "2. PENJELASAN PARAMETER" never appears as a root although it should. heading_path feeds the evidence ranker, so this is not cosmetic. Open against B5.

chapters is empty on every chunk of the reference run. The field is populated only when MinerU reports running headers; this document has none detected.

---

Caveat on the sample values

Field shapes, types and defaults are read from src/knowledge_parsing/contracts.py. Unlike the extraction contract, the values here are literal — taken from data/knowledge_runs/20260824-100720/STD_2026_006_MNO/chunks.json, with long strings truncated and marked ….