Commit History
-
[NOTICKET] feat(knowledge-parsing): wire Mistral OCR backend end-to-end
Makes Mistral OCR the primary parser (MinerU kept as a fallback), runnable as `python -m src.knowledge_parsing.run --backend mistral` with no GPU. - config: backend="mistral" default + Mistral settings (include_blocks, bbox_annotation, table_format). fingerprint() adds the Mistral settings ONLY on that backend, so existing MinerU cache keys stay byte-identical. - mistral_ocr.py (new): the API call — upload -> ocr.process, key from env/.env, file_id with a signed-url fallback, optional per-image descriptions. - parse.py: _parse_mistral caches the response like the MinerU path; cache_key uses the model id as the parser version for Mistral. - run.py: dispatches to normalise_mistral for the Mistral backend (skips the MinerU-only content-drift check and word-boundary vocabulary), stamps parser_name/normaliser_version per backend. - pyproject: mistralai in the knowledge-parsing extra. NOTE: it needs pydantic>=2.11.2, which the pinned pydantic==2.10.3 does not satisfy; the repo…6856ea8
-
[NOTICKET] fix(knowledge-parsing): long names, name clashes, and a GPU check in the wrong place
Four defects, all found by asking what happens when a real person uploads a real file rather than the two well-named documents in the test folder. **Long file names failed the parse.** MinerU names its output folder after `pdf_file_names` and then writes images into it under a full SHA-256 filename. Measured on this repo the fixed part of the path costs 217 characters, leaving 43 for the name — and the BUMA standard's own name is 56. `_mineru_name` now passes a bounded name instead. Head AND tail are kept, because the names that collide in practice share a long prefix and differ at the end ("… and ECA" versus "… and ECA (2)"); keeping only the head would render exactly those indistinguishable. Verified with a real parse of the 56-character name: 239 characters, 31 chunks, where before it died on MAX_PATH. Uniqueness does not depend on that name — the parent folder is the cache key, which is the full SHA-256 of the file contents. It is cosmetic, exists so the cache can be browsed whil…9083dd1
-
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments, manifest keys and console output. Everything else in the repo is English, and this module is the newest work, so it is the one to bring into line rather than the older code around it. Renamed across the module: parse_dokumen -> parse_document, normalisasi -> normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak -> trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen -> DocumentRecord, and the locals with them. Docstrings and comments were REWRITTEN rather than translated word for word. Several carry the reasoning behind a decision — why `\frac` is unwrapped before the braced wrappers, why a heading never enters `text`, why the digit threshold is 3 — and a mechanical translation would have flattened exactly the part worth keeping. Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan -> warnings, jenis -> kind, galat -> error. That is a data change, …7f153d6
-
[NOTICKET] fix(knowledge-parsing): send MinerU a backend name it accepts
The GPU flip config.py advertises did not work. MinerU dispatches on `backend == "pipeline"`, then `startswith("vlm-")` and `startswith("hybrid-")`. A bare "vlm" matches none of them, so MinerU did nothing at all, silently — and the failure surfaced here as "content_list not found", which points at the wrong place entirely. Valid names are pipeline / vlm-engine / hybrid-engine. The short names stay in the config because they are part of the cache key and read better; the translation happens at the boundary, just before MinerU is touched. Nobody would have caught this until the day a GPU arrived and everyone was waiting on the result. Also: the temp directory used with_suffix(), which REPLACES the last suffix rather than appending. Cache folders end in the MinerU version ("...-3.4.4"), so ".4" was thrown away and two versions collided on one temp directory that the next line deletes without asking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>8e93c60
-
[NOTICKET] feat(knowledge-parsing): document parsing half of the knowledge pipeline
Implements §4 B2/B5/B6 and closes the four open seam items (S2, S3, S4, S6b/c/d) against the contracts.py draft circulated 2026-08-19. Additive and flag-gated: src/knowledge/ (Tesseract -> pgvector) is untouched. Seam — src/knowledge_parsing/contracts.py, pydantic per S6e: - ParsedDocument envelope: content_hash, n_pages, version, parser_name/version/ backend/config, schema_version, raw_output_dir. parser_backend is read from MinerU's _middle.json, so it records what actually ran, not what was configured — a MinerU change and a prompt change stay distinguishable (S2). - Heading structure is READ, not derived. Verified in MinerU's source: pipeline and vlm run identical text_level logic, and bbox is emitted by both. Heading availability is document-dependent, not backend-dependent — the earlier claim came from sampling the McGraw-Hill handbook, which has no numbered headings (S3). - source_wording on TermRecord; heading carried verbatim (S4). - page_idx/page_idxs, 0-based, n…5840d20