| # knowledge_parsing |
| |
| Document parsing — the first half of the knowledge pipeline. PDF/DOCX in, a |
| versioned `ParsedDocument` out. |
| |
| The second half (`src/knowledge_extraction/`) consumes that artifact and never |
| the parser itself, which is what keeps MinerU swappable: a Tesseract or Azure |
| Document Intelligence path emits the same artifact and extraction never learns |
| which one ran. |
|
|
| Additive and flag-gated. The existing unstructured path (`src/knowledge/`, |
| Tesseract OCR → chunk → pgvector) is untouched and keeps running as-is. |
|
|
| ## Files |
|
|
| - `contracts.py` — **the seam.** `Chunk`, `ParsedDocument`, and the declared |
| `Mention` / `TermRecord` handoff shapes. The one file both halves must agree on |
| - `config.py` — all settings, and the single place the GPU backend flips |
| - `parse.py` — MinerU wrapper with a content-addressed parse cache |
| - `normalize.py` — MinerU's flat item list → meaningful chunks |
| - `render.py` — LaTeX/HTML → readable prose for `Chunk.text` |
| - `checks.py` — quality warnings, including silent-corruption detection |
| - `manifest.py` / `report.py` — per-run record and throughput extrapolation |
| - `run.py` — batch CLI: one folder in, artifacts + manifest out |
|
|
| ## Use |
|
|
| ```bash |
| python -m src.knowledge_parsing.run --input data/knowledge_docs/ |
| python -m src.knowledge_parsing.report --scale 6800 |
| ``` |
|
|
| MinerU is an optional extra, not a main dependency — the agent service never |
| parses documents at request time, so the deployed Space does not ship it: |
|
|
| ```bash |
| uv sync --extra knowledge-parsing |
| ``` |
|
|
| Importing this package does not import MinerU or torch; only `parse.py` does, at |
| call time. |
|
|
| ## Three things that are easy to get wrong |
|
|
| **`Chunk.text` must stay verbatim.** Extraction's guardrail locates quoted spans |
| literally inside it. Reflowing or normalising the text makes the lookup fail and |
| the field goes silently null — which presents as a bad model, not as a parser |
| bug. |
|
|
| **`page_idx` is 0-based**, exactly as MinerU reports it, with no conversion |
| anywhere in the pipeline. Converting to 1-based is the UI's job, done once at |
| display time, so the artifact always matches the raw MinerU output kept beside |
| it in the cache. |
| |
| **Markup never goes in `text`.** The term filter is an NER model reading prose; |
| MinerU writes formulas character-spaced (`P u r c h a s i n g ~ c o s t s`) and |
| tables as HTML, and neither yields a single mention. Measured on one document and |
| gold set, changing only the parse: raw markup scored recall 0.7561 against 0.8537 |
| for plain text; rendering it back recovered 0.8293. The markup is preserved in |
| `Chunk.latex` and `Chunk.table_html`, because the formula branch needs that form. |
| |
| ## Output layout |
| |
| ``` |
| data/knowledge_cache/parse/<hash>-<config>-<version>/ untouched MinerU output |
| data/knowledge_runs/<run_id>/ |
| ├── manifest.json backend, versions, timings, quality warnings |
| ├── failures.jsonl documents that failed, with traces |
| └── <doc>/chunks.json the artifact |
| ``` |
| |
| Both are gitignored — they are data, not code. |
| |
| See `KNOWLEDGE_PIPELINE_TODO.md` (root) for the task breakdown and the seam |
| discussion, and `KNOWLEDGE_PIPELINE_CALIBRATION.md` §4 for the chunking |
| constants this module follows. |
| |
