# knowledge_parsing Document parsing — the first half of the knowledge pipeline. PDF/DOCX in, a versioned `ParsedDocument` out. The second half (`src/knowledge_extraction/`) consumes that artifact and never the parser itself, which is what keeps MinerU swappable: a Tesseract or Azure Document Intelligence path emits the same artifact and extraction never learns which one ran. Additive and flag-gated. The existing unstructured path (`src/knowledge/`, Tesseract OCR → chunk → pgvector) is untouched and keeps running as-is. > **Current (2026-09-02) — the default parser is now Mistral OCR.** `--backend mistral` (the config > default) is an API with **no GPU**: `run --backend mistral` produces the artifact in ~8 s. The > MinerU backends (`hybrid`/`vlm`/`pipeline`, everything below about GPU) are now the **fallback** — > both emit the identical `ParsedDocument`. Mistral needs `MISTRAL_API_KEY` (read from the repo > `.env`); its normalizer is `normalize_mistral.py`, chunking **section**-level by default. Recall on > the BUMA fixture: section **0.878** (`eval/parsing/README.md`). > **2026-09-24 — `paddleocr` backend, v2 document processing only.** `paddle_ocr.py` calls > Baidu AI Studio's PaddleOCR-VL job API (multipart submit → poll `state` until `done` → > JSONL `result.layoutParsingResults[]`, one per page); figures come back as hosted URLs and are > downloaded at parse time into `markdown.images_base64`, so the cache is self-contained. > `normalize_paddle.py` turns the layout blocks into the same `ParsedDocument`. Selected by the > app setting `DOCUMENT_PARSER_BACKEND`; the default here and the CLI's `--backend` choices are > unchanged, so offline knowledge runs never reach it. PDF and images only. Needs > `PADDLEOCR_ACCESS_TOKEN`. ## Files - `contracts.py` — **the seam.** `Chunk`, `ParsedDocument`, and the declared `Mention` / `TermRecord` handoff shapes. The one file both halves must agree on - `config.py` — all settings, and the single place the GPU backend flips - `parse.py` — parser wrapper (Mistral or MinerU) with a content-addressed parse cache - `mistral_ocr.py` — the Mistral OCR API call (upload → `ocr.process`); no GPU - `normalize.py` — MinerU's flat item list → meaningful chunks - `normalize_mistral.py` — Mistral's block stream → chunks (the Mistral counterpart of `normalize.py`) - `render.py` — LaTeX/HTML → readable prose for `Chunk.text` (shared by both normalizers) - `checks.py` — quality warnings, including silent-corruption detection - `manifest.py` / `report.py` — per-run record and throughput extrapolation - `run.py` — batch CLI: one folder in, artifacts + manifest out ## Use ```bash py scripts/parse.py # Windows python3 scripts/parse.py # Linux py scripts/parse.py data/other # a different folder py scripts/parse.py --backend vlm # force a specific backend ``` The script sets the env vars that are required and picks a backend. Calling the module directly works too, but then those env vars are yours to set — see `scripts/parse.py` for the list. > The wrapper is **Python, not shell**, deliberately. Bash and PowerShell > versions were written first and inside twenty minutes hit two traps that are > specific to shells: CRLF line endings making bash fail on its very first line, > and a trailing backslash on a path swallowing the argument AFTER it. > `subprocess` takes a list of arguments and never goes through a shell, so > neither is possible. This is a development aid. **Production calls the module directly** — parsing is an admin-triggered offline batch, not something a person runs from a terminal. A successful run looks like this: ``` >> No GPU detected — falling back to pipeline (no image descriptions; multiplication copied as the letter 'x') Run 20260826-131546 | backend=pipeline | 2 documents [1/2] Maintenance Engineering Handbook (1)-1 … ok (cache) | 1 chunk [2/2] STD_2026_006_MNO … ok 385.5s | 31 chunks | ⚠ 3 warnings --- summary --- ok / failed : 2 / 0 total pages : 10 ⚠ quality warnings: 3 — see manifest.json ``` How to read it: - **`ok (cache)`** means that document was not re-parsed. The cache key is document content + settings + MinerU version, so only what genuinely changed gets redone. Cached documents are **excluded** from the seconds/page average. - **`⚠ n warnings` is NORMAL, not a failure.** The document is still processed. Warnings flag quality worth a human look — formulas with no numbers, empty tables, charts with no context, and most importantly **numbers that are absent from the source PDF**. Details land in `manifest.json`. - **`ok / failed`** is the line to watch. Failed documents are recorded in full, with their traces, in `failures.jsonl`, and one failure does not stop the rest. Throughput numbers and their extrapolation: ```bash uv run --no-sync python -m src.knowledge_parsing.report --scale 6800 ``` ### Backend > These are the **MinerU** backends — now the **fallback** (see the Mistral note at the top). > They still apply whenever `--backend hybrid/vlm/pipeline` is chosen. Among the MinerU backends the default is **`hybrid` + `effort=high`**, decided 2026-08-24 after all four were measured on the same GPU over the same 29 pages: | Backend | s/page | Image descriptions | Multiplication sign | |---|---|---|---| | `hybrid/high` ⭐ | 4.85 | **yes** | `\times` | | `hybrid/medium` | 1.88 | force-disabled by MinerU | `\times` | | `pipeline` | 2.67 | never | the letter `x` — **fuses tokens** | | `vlm` | 6.96 | yes | `\times` | `vlm` and `hybrid/high` produce **100% identical** output, but `vlm` is slower. `pipeline` is the only one that copies the multiplication sign as a literal `x`, which fuses `MOHH x Qty x PA` into a single token and takes five terms with it. **Determinism: settled 2026-08-26.** Two genuine re-parses produced 0 differences across 50 chunks — image descriptions included, character for character. Frozen baselines do not drift. Parse *time* varies (model loading); the output does not. ### Running on a laptop with no GPU `hybrid` and `vlm` need a GPU. Without one the script falls back to `pipeline` on its own and says so. That output is **valid but lower quality** — no image descriptions, and the multiplication sign copied as a letter. Do not use it as a reference measurement. Calling the module directly without `--backend` stops with a message naming the command to use instead, rather than an opaque torch error. ### Installing MinerU MinerU is an **extra**, not a core dependency — the agent service never parses a document during a request, so the deployed Space does not carry it. Importing this package does not import MinerU or torch; only `parse.py` does, when called. ```bash UV_CACHE_DIR=D:/uv-cache uv pip install --python .venv mineru==3.4.4 ``` ⚠️ **Not `uv sync --extra knowledge-parsing`.** That fails: `mineru` needs `python-docx>=1.2.0` while this project pins `1.1.2`. Locking it means raising the shared pin, and that is a team decision — not taken yet. So `mineru` is **absent from `uv.lock`**. Two consequences: - **A plain `uv run` uninstalls MinerU** — it syncs the venv to the lock first. Use `uv run --no-sync`, this repo's standing convention (see CLAUDE.md); `scripts/parse.py` already does. - If anyone runs `uv sync`, MinerU disappears. Reinstall with the command above; `scripts/parse.py` prints that command itself when it happens. ### Where the large files live All kept off the nearly-full `C:` drive: | | | |---|---| | Model weights | `D:/mineru-models/hf`, `D:/mineru-models/modelscope` | | Temp files | `D:/mineru-models/tmp` | | `uv` cache | `D:/uv-cache` | | Parse cache + run output | `data/knowledge_cache/`, `data/knowledge_runs/` (gitignored) | ## Naming documents — a convention, and why it is temporary **The file name is currently a primary key, not a label.** `doc_id` is the file name stem; `chunk_id` is built from `doc_id`; and `term_id`, `formula_id`, `rule_id` and `domain_id` are all digests of `doc_id` (`knowledge_extraction/ids.py`). Renaming a file after ingestion therefore changes every identifier derived from it, and an expert's approve/reject decisions are keyed to those identifiers. Until that changes, name documents like this: ``` _Rev.pdf STD-2026-006-MNO_Rev0.0.pdf MEH-2019_Ch03.pdf ``` - **Stable prefix, varying suffix.** The doc code stays fixed across revisions so they can eventually be recognised as versions of one document; `_Rev1.0` is the part that moves, and it is machine-readable rather than only human-readable. - **No spaces.** They force quoting in every command, and a trailing backslash on a quoted path has already swallowed an argument once in this repo. - **Never `(1)` / `(2)`.** That suffix means "downloaded twice", not "version 2". Ingested as-is it becomes a second legitimate document with its own full set of identifiers. If the content is identical, delete one — `run.py` warns when two inputs share a hash. If the content genuinely differs, it is a revision: give it a real `_Rev`. - **Never rename after ingestion.** The long human title is not lost by any of this — it stays inside the document, and normally appears as the first chunk's `heading`. ### Why this is a stopgap, not a design The convention holds only because *we* control the file names while parsing is an admin-triggered batch. The moment this sits behind an upload endpoint — which is planned, after eval and the playground — it stops holding: people will upload `Copy of SOP final FIX (2).pdf`, and permanent knowledge-base identifiers should never have depended on what someone typed into a Save dialog. The fix is to stop deriving identity from the name. `ParsedDocument` already carries `content_hash`, a SHA-256 of the file, so the identifier exists — it is simply not the one being used: ``` doc_id content_hash key, stable, machine-facing source_title the file name label, may change freely ``` That is a seam change (`doc_id` reaches `chunk_id`, `Provenance` and `ids.py`), so it needs agreeing rather than doing. It is also cheapest now: the migration cost rises with every document ingested and every expert decision recorded, because re-keying orphans all of them. One consequence to handle when it lands — `Provenance` currently shows `doc_id` to the reviewer, so a display title has to travel with it or experts will be reading hashes. `parse._mineru_name` exists for the same reason and is deliberately minimal: once `doc_id` is a hash, every path is short by construction and that function is deleted. ## Three things that are easy to get wrong **`Chunk.text` must stay verbatim.** Extraction's guardrail locates quoted spans literally inside it. Reflowing or normalising the text makes the lookup fail and the field goes silently null — which presents as a bad model, not as a parser bug. **`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion anywhere in the pipeline. Converting to 1-based is the UI's job, done once at display time, so the artifact always matches the raw MinerU output kept beside it in the cache. **Markup never goes in `text`.** The term filter is an NER model reading prose; MinerU writes formulas character-spaced (`P u r c h a s i n g ~ c o s t s`) and tables as HTML, and neither yields a single mention. Measured on one document and gold set, changing only the parse: raw markup scored recall 0.7561 against 0.8537 for plain text; rendering it back recovered 0.8293. The markup is preserved in `Chunk.latex` and `Chunk.table_html`, because the formula branch needs that form. ## Output layout ``` data/knowledge_cache/parse/--/ untouched MinerU output data/knowledge_runs// ├── manifest.json backend, versions, timings, quality warnings ├── failures.jsonl documents that failed, with traces └── /chunks.json the artifact ``` Both are gitignored — they are data, not code. See [`docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md`](../../docs/knowledge/KNOWLEDGE_PIPELINE_TODO.md) for the task breakdown and the seam discussion, and [`docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md`](../../docs/knowledge/KNOWLEDGE_PIPELINE_CALIBRATION.md) §4 for the chunking constants this module follows.