src/knowledge_parsing/README.md

knowledge_parsing

Document parsing — the first half of the knowledge pipeline. PDF/DOCX in, a versioned ParsedDocument out.

The second half (src/knowledge_extraction/) consumes that artifact and never the parser itself, which is what keeps MinerU swappable: a Tesseract or Azure Document Intelligence path emits the same artifact and extraction never learns which one ran.

Additive and flag-gated. The existing unstructured path (src/knowledge/, Tesseract OCR → chunk → pgvector) is untouched and keeps running as-is.

Current (2026-09-02) — the default parser is now Mistral OCR. --backend mistral (the config default) is an API with no GPU: run --backend mistral produces the artifact in ~8 s. The MinerU backends (hybrid/vlm/pipeline, everything below about GPU) are now the fallback — both emit the identical ParsedDocument. Mistral needs MISTRAL_API_KEY (read from the repo .env); its normalizer is normalize_mistral.py, chunking section-level by default. Recall on the BUMA fixture: section 0.878 (eval/parsing/README.md).

2026-09-24 — paddleocr backend, v2 document processing only. paddle_ocr.py calls Baidu AI Studio's PaddleOCR-VL job API (multipart submit → poll state until done → JSONL result.layoutParsingResults[], one per page); figures come back as hosted URLs and are downloaded at parse time into markdown.images_base64, so the cache is self-contained. normalize_paddle.py turns the layout blocks into the same ParsedDocument. Selected by the app setting DOCUMENT_PARSER_BACKEND; the default here and the CLI's --backend choices are unchanged, so offline knowledge runs never reach it. PDF and images only. Needs PADDLEOCR_ACCESS_TOKEN.

Files

• contracts.py — the seam. Chunk, ParsedDocument, and the declared Mention / TermRecord handoff shapes. The one file both halves must agree on
• config.py — all settings, and the single place the GPU backend flips
• parse.py — parser wrapper (Mistral or MinerU) with a content-addressed parse cache
• mistral_ocr.py — the Mistral OCR API call (upload → ocr.process); no GPU
• normalize.py — MinerU's flat item list → meaningful chunks
• normalize_mistral.py — Mistral's block stream → chunks (the Mistral counterpart of normalize.py)
• render.py — LaTeX/HTML → readable prose for Chunk.text (shared by both normalizers)
• checks.py — quality warnings, including silent-corruption detection
• manifest.py / report.py — per-run record and throughput extrapolation
• run.py — batch CLI: one folder in, artifacts + manifest out

Use

py scripts/parse.py                     # Windows
python3 scripts/parse.py                # Linux
py scripts/parse.py data/other          # a different folder
py scripts/parse.py --backend vlm       # force a specific backend

The script sets the env vars that are required and picks a backend. Calling the module directly works too, but then those env vars are yours to set — see scripts/parse.py for the list.

The wrapper is Python, not shell, deliberately. Bash and PowerShell versions were written first and inside twenty minutes hit two traps that are specific to shells: CRLF line endings making bash fail on its very first line, and a trailing backslash on a path swallowing the argument AFTER it. subprocess takes a list of arguments and never goes through a shell, so neither is possible.

This is a development aid. Production calls the module directly — parsing is an admin-triggered offline batch, not something a person runs from a terminal.

A successful run looks like this:

>> No GPU detected — falling back to pipeline
   (no image descriptions; multiplication copied as the letter 'x')
Run 20260826-131546 | backend=pipeline | 2 documents

[1/2] Maintenance Engineering Handbook (1)-1 … ok (cache) | 1 chunk
[2/2] STD_2026_006_MNO … ok 385.5s | 31 chunks | ⚠ 3 warnings

--- summary ---
  ok / failed     : 2 / 0
  total pages     : 10
  ⚠ quality warnings: 3 — see manifest.json

How to read it:

• ok (cache) means that document was not re-parsed. The cache key is document content + settings + MinerU version, so only what genuinely changed gets redone. Cached documents are excluded from the seconds/page average.
• ⚠ n warnings is NORMAL, not a failure. The document is still processed. Warnings flag quality worth a human look — formulas with no numbers, empty tables, charts with no context, and most importantly numbers that are absent from the source PDF. Details land in manifest.json.
• ok / failed is the line to watch. Failed documents are recorded in full, with their traces, in failures.jsonl, and one failure does not stop the rest.

Throughput numbers and their extrapolation:

uv run --no-sync python -m src.knowledge_parsing.report --scale 6800

Backend

These are the MinerU backends — now the fallback (see the Mistral note at the top). They still apply whenever --backend hybrid/vlm/pipeline is chosen.

Among the MinerU backends the default is hybrid + effort=high, decided 2026-08-24 after all four were measured on the same GPU over the same 29 pages:

Backend	s/page	Image descriptions	Multiplication sign
hybrid/high ⭐	4.85	yes	\times
hybrid/medium	1.88	force-disabled by MinerU	\times
pipeline	2.67	never	the letter x — fuses tokens
vlm	6.96	yes	\times

vlm and hybrid/high produce 100% identical output, but vlm is slower. pipeline is the only one that copies the multiplication sign as a literal x, which fuses MOHH x Qty x PA into a single token and takes five terms with it.

Determinism: settled 2026-08-26. Two genuine re-parses produced 0 differences across 50 chunks — image descriptions included, character for character. Frozen baselines do not drift. Parse time varies (model loading); the output does not.

Running on a laptop with no GPU

hybrid and vlm need a GPU. Without one the script falls back to pipeline on its own and says so. That output is valid but lower quality — no image descriptions, and the multiplication sign copied as a letter. Do not use it as a reference measurement.

Calling the module directly without --backend stops with a message naming the command to use instead, rather than an opaque torch error.

Installing MinerU

MinerU is an extra, not a core dependency — the agent service never parses a document during a request, so the deployed Space does not carry it. Importing this package does not import MinerU or torch; only parse.py does, when called.

UV_CACHE_DIR=D:/uv-cache uv pip install --python .venv mineru==3.4.4

⚠️ Not uv sync --extra knowledge-parsing. That fails: mineru needs python-docx>=1.2.0 while this project pins 1.1.2. Locking it means raising the shared pin, and that is a team decision — not taken yet.

So mineru is absent from uv.lock. Two consequences:

• A plain uv run uninstalls MinerU — it syncs the venv to the lock first. Use uv run --no-sync, this repo's standing convention (see CLAUDE.md); scripts/parse.py already does.
• If anyone runs uv sync, MinerU disappears. Reinstall with the command above; scripts/parse.py prints that command itself when it happens.

Where the large files live

All kept off the nearly-full C: drive:

Model weights	D:/mineru-models/hf, D:/mineru-models/modelscope
Temp files	D:/mineru-models/tmp
uv cache	D:/uv-cache
Parse cache + run output	data/knowledge_cache/, data/knowledge_runs/ (gitignored)

Naming documents — a convention, and why it is temporary

The file name is currently a primary key, not a label. doc_id is the file name stem; chunk_id is built from doc_id; and term_id, formula_id, rule_id and domain_id are all digests of doc_id (knowledge_extraction/ids.py). Renaming a file after ingestion therefore changes every identifier derived from it, and an expert's approve/reject decisions are keyed to those identifiers.

Until that changes, name documents like this:

<DOC-CODE>_Rev<N.N>.pdf

STD-2026-006-MNO_Rev0.0.pdf
MEH-2019_Ch03.pdf

• Stable prefix, varying suffix. The doc code stays fixed across revisions so they can eventually be recognised as versions of one document; _Rev1.0 is the part that moves, and it is machine-readable rather than only human-readable.
• No spaces. They force quoting in every command, and a trailing backslash on a quoted path has already swallowed an argument once in this repo.
• Never (1) / (2). That suffix means "downloaded twice", not "version 2". Ingested as-is it becomes a second legitimate document with its own full set of identifiers. If the content is identical, delete one — run.py warns when two inputs share a hash. If the content genuinely differs, it is a revision: give it a real _Rev<N>.
• Never rename after ingestion.

The long human title is not lost by any of this — it stays inside the document, and normally appears as the first chunk's heading.

Why this is a stopgap, not a design

The convention holds only because we control the file names while parsing is an admin-triggered batch. The moment this sits behind an upload endpoint — which is planned, after eval and the playground — it stops holding: people will upload Copy of SOP final FIX (2).pdf, and permanent knowledge-base identifiers should never have depended on what someone typed into a Save dialog.

The fix is to stop deriving identity from the name. ParsedDocument already carries content_hash, a SHA-256 of the file, so the identifier exists — it is simply not the one being used:

doc_id        content_hash      key, stable, machine-facing
source_title  the file name     label, may change freely

That is a seam change (doc_id reaches chunk_id, Provenance and ids.py), so it needs agreeing rather than doing. It is also cheapest now: the migration cost rises with every document ingested and every expert decision recorded, because re-keying orphans all of them. One consequence to handle when it lands — Provenance currently shows doc_id to the reviewer, so a display title has to travel with it or experts will be reading hashes.

parse._mineru_name exists for the same reason and is deliberately minimal: once doc_id is a hash, every path is short by construction and that function is deleted.

Three things that are easy to get wrong

Chunk.text must stay verbatim. Extraction's guardrail locates quoted spans literally inside it. Reflowing or normalising the text makes the lookup fail and the field goes silently null — which presents as a bad model, not as a parser bug.

page_idx is 0-based, exactly as MinerU reports it, with no conversion anywhere in the pipeline. Converting to 1-based is the UI's job, done once at display time, so the artifact always matches the raw MinerU output kept beside it in the cache.

Markup never goes in text. The term filter is an NER model reading prose; MinerU writes formulas character-spaced (P u r c h a s i n g ~ c o s t s) and tables as HTML, and neither yields a single mention. Measured on one document and gold set, changing only the parse: raw markup scored recall 0.7561 against 0.8537 for plain text; rendering it back recovered 0.8293. The markup is preserved in Chunk.latex and Chunk.table_html, because the formula branch needs that form.

Output layout

data/knowledge_cache/parse/<hash>-<config>-<version>/   untouched MinerU output
data/knowledge_runs/<run_id>/
├── manifest.json     backend, versions, timings, quality warnings
├── failures.jsonl    documents that failed, with traces
└── <doc>/chunks.json the artifact

Both are gitignored — they are data, not code.

See docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md for the task breakdown and the seam discussion, and docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md §4 for the chunking constants this module follows.