Commit History
-
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments, manifest keys and console output. Everything else in the repo is English, and this module is the newest work, so it is the one to bring into line rather than the older code around it. Renamed across the module: parse_dokumen -> parse_document, normalisasi -> normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak -> trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen -> DocumentRecord, and the locals with them. Docstrings and comments were REWRITTEN rather than translated word for word. Several carry the reasoning behind a decision — why `\frac` is unwrapped before the braced wrappers, why a heading never enters `text`, why the digit threshold is 3 — and a mechanical translation would have flattened exactly the part worth keeping. Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan -> warnings, jenis -> kind, galat -> error. That is a data change, …7f153d6
-
[NOTICKET] fix(knowledge-parsing): read word boundaries back from the source PDF
MinerU writes formulas one character at a time ("T o t a l H o u r s"), and once those are rejoined the word boundaries are gone: "Total Hours" and "TotalHours" are no longer distinguishable from the LaTeX alone. Everything fuses into a single token. The multiplication written as a letter "x" is not a separate bug. It is the same one — an "x" between terms is just another spaced character, so "MOHH x Qty x PA x UA x Pty" collapses to "MOHHxQtyxPAxUAxPty". Two obvious fixes were tried and rejected: - Rewriting a lone "x" to a multiplication sign guesses, and guesses wrong on any document with a variable named x, silently. It also leaves "TotalHours" untouched, which is the larger half of the damage. - Emitting a warning fires on 7 of 11 equations. A warning that lights up on 64% of items is not actionable, and recovers nothing. The source PDF still holds the boundaries: its text layer reads "MOHH x Qty x PA x UA x Pty", properly spaced. So the answer is read from t…e047f7a -
[NOTICKET] fix(knowledge-parsing): render_latex was deleting division (X21)
Rifqi's three faults, all reproduced against the document's own 11 equations and all fixed. Two of his numbers were off, and the corrections matter: - It hit 4 of 11 equations, not 2 — PA, PA_ij, UA_ij and PTY_ij. Not just PA and UA: every composite form was affected too. - The nested braces come from mathrm{} and sum _{ij}, NOT from text{}, which this standard never uses. A fix written against text{} would have repaired the synthetic example and no real equation. (C), the serious one: the frac rewrite needed a brace-free argument, could not match a nested numerator, and the unmatched frac then fell through to the generic "drop leftover commands" pass. The division was deleted, leaving a formula that reads as a product and states something the document does not. The ordering decision this needed: braced wrappers are unwrapped BEFORE frac, and the unwrap consumes its braces. The old pass removed only the command name and left { } behind — those leftover braces were wh…a356e3c
-
[NOTICKET] fix(knowledge-parsing): render markup to prose in text; only numbered headings split sections
Both defects were measured by the extraction side on the same document and gold set, changing only the parse: PyMuPDF (plain text) recall 0.8537 35/41 MinerU, raw markup in text recall 0.7561 31/41 MinerU, formula + table rendered recall 0.8293 34/41 1. Chunk.text now carries readable prose, not markup. The term filter is an NER model reading prose. MinerU writes formulas with every character spaced out — "{ \mathrm { P u r c h a s i n g ~ c o s t s } }" — and tables as HTML; neither yields a single mention for terms the same model finds easily in plain text. render.py rejoins spelled-out words and numbers, folds subscripts, maps operators to non-letter glyphs (a literal "x" for \times would merge into the token beside it), and flattens tables to " | "-separated rows. The markup is kept, not discarded: Chunk.latex and Chunk.table_html hold it verbatim, because the formula branch needs exactly that form. Found while verifying: table cells carr…2cb1336