[NOTICKET] fix(knowledge-parsing): render markup to prose in text; only numbered headings split sections

Both defects were measured by the extraction side on the same document and gold
set, changing only the parse:

PyMuPDF (plain text) recall 0.8537 35/41
MinerU, raw markup in text recall 0.7561 31/41
MinerU, formula + table rendered recall 0.8293 34/41

1. Chunk.text now carries readable prose, not markup.

The term filter is an NER model reading prose. MinerU writes formulas with every
character spaced out — "{ \mathrm { P u r c h a s i n g ~ c o s t s } }" — and
tables as HTML; neither yields a single mention for terms the same model finds
easily in plain text. render.py rejoins spelled-out words and numbers, folds
subscripts, maps operators to non-letter glyphs (a literal "x" for \times would
merge into the token beside it), and flattens tables to " | "-separated rows.

The markup is kept, not discarded: Chunk.latex and Chunk.table_html hold it
verbatim, because the formula branch needs exactly that form.

Found while verifying: table cells carry inline LaTeX of their own, so stripping
HTML alone still left markup in the prose. Inline math inside cells is rendered
too.

2. Only NUMBERED headings open a section.

text_level alone is not sufficient evidence of a section boundary: on the BUMA
standard MinerU also marks "Keterangan:" and "Keterangan grafik:" as headings.
Treating those as boundaries separates a legend from the figure it explains, and
"Other Activity" and "Uncontrollable" go missing even though they sit as prose in
a chunk the filter already processed. text_level is still used for the hierarchy
level; numbering decides whether a line is a boundary at all.

Verified: no markup leaks into text on the sample document; heading-less
"Keterangan" lines stay with their section; ruff clean; import main OK; importing
the package still leaves mineru and torch out of sys.modules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/knowledge_parsing/contracts.py

@@ -108,6 +108,19 @@ class Chunk(BaseModel):
     # Non-text attachments (formula images, table/chart crops)
     images: list[str] = Field(default_factory=list)
 
+    # Source markup, kept verbatim beside the rendered prose in `text`.
+    #
+    # `text` carries a readable rendering because the term filter is an NER
+    # model reading prose: MinerU writes formulas character-spaced
+    # ("P u r c h a s i n g ~ c o s t s") and tables as HTML, and neither
+    # produces a single mention. Measured on the same document and gold set,
+    # only the parse differing: raw markup in `text` scored recall 0.7561 against
+    # 0.8537 for plain text; rendering it back recovered 0.8293.
+    #
+    # The markup is not discarded — the formula branch needs exactly this form.
+    latex: list[str] = Field(default_factory=list)
+    table_html: str | None = None
+
 
 class ParsedDocument(BaseModel):
     """The artifact itself — one parsed document, self-describing.

src/knowledge_parsing/normalize.py

@@ -37,6 +37,7 @@ from pathlib import Path
 from typing import Any
 
 from .contracts import Chunk
+from .render import render_latex, render_table
 
 # "2.1.3 Judul" / "2.1.3. Judul" / "4 Judul"
 _POLA_NOMOR = re.compile(r"^(\d+(?:\.\d+)*)\.?\s+(\S.*)$")
@@ -59,10 +60,18 @@ MAKS_KARAKTER_CHUNK = 6000
 def _judul(item: dict[str, Any]) -> tuple[str | None, str | None, int | None]:
     """Kembalikan (nomor_section, judul, level) kalau item ini judul section.
 
-    `text_level` dari MinerU adalah sinyal utama — dipakai kalau ada, karena itu
-    hasil deteksi judul MinerU sendiri dan membawa tingkat hierarkinya
-    (2 / 2.1 / 2.1.1 -> level 2 / 3 / 4). Pola penomoran cuma cadangan untuk
-    dokumen yang judulnya tidak terdeteksi MinerU.
+    ⭐ HANYA judul BERNOMOR yang membuka section baru.
+
+    `text_level` dari MinerU tidak cukup dijadikan syarat tunggal: pada standar
+    BUMA, MinerU juga menandai "Keterangan:" dan "Keterangan grafik:" sebagai
+    judul. Kalau itu dianggap batas section, legend-nya terpisah dari gambar yang
+    dijelaskannya — dan istilah seperti "Other Activity" dan "Uncontrollable"
+    hilang, padahal ada sebagai prosa di chunk yang sudah lewat filter. Itu sisa
+    selisih recall terhadap jalur teks polos.
+
+    Jadi `text_level` dipakai untuk TINGKAT hierarkinya, tapi penomoranlah yang
+    menentukan apakah sebuah baris benar-benar batas section. Baris ber-
+    `text_level` tanpa nomor tetap ikut sebagai isi chunk yang sedang berjalan.
     """
     if item.get("type") not in {"text", "title"}:
         return None, None, None
@@ -70,23 +79,27 @@ def _judul(item: dict[str, Any]) -> tuple[str | None, str | None, int | None]:
     if not teks or len(teks) > _MAKS_PANJANG_JUDUL:
         return None, None, None
 
-    level = item.get("text_level")
-    if level:                              # MinerU yakin ini judul
-        m = _POLA_NOMOR.match(teks)
-        return (m.group(1), m.group(2), int(level)) if m else (None, teks, int(level))
-
     m = _POLA_NOMOR.match(teks)
-    if m and not teks.endswith((".", ":", ";")):
-        # Tanpa text_level, tingkat diperkirakan dari kedalaman penomoran:
-        # "2" -> 1, "2.1" -> 2, "2.1.1" -> 3
-        return m.group(1), m.group(2), m.group(1).count(".") + 1
-    return None, None, None
+    if not m or teks.endswith((".", ":", ";")):
+        return None, None, None
+
+    level = item.get("text_level")
+    if level:
+        return m.group(1), m.group(2), int(level)
+    # Tanpa text_level, tingkat diperkirakan dari kedalaman penomoran:
+    # "2" -> 1, "2.1" -> 2, "2.1.1" -> 3
+    return m.group(1), m.group(2), m.group(1).count(".") + 1
 
 
 def _teks_tabel(item: dict[str, Any]) -> str:
+    """Caption + isi tabel sebagai teks terbaca.
+
+    HTML mentahnya TIDAK ditaruh di sini — disimpan terpisah di
+    `Chunk.table_html`. Lihat alasan terukurnya di render.py.
+    """
     bagian = list(item.get("table_caption") or [])
     if item.get("table_body"):
-        bagian.append(item["table_body"])
+        bagian.append(render_table(item["table_body"]))
     bagian += list(item.get("table_footnote") or [])
     return "\n\n".join(b for b in bagian if b)
 
@@ -149,6 +162,7 @@ def normalisasi(items: list[dict[str, Any]], doc_id: str) -> list[Chunk]:
             c = buka("table", page)
             c.text = _teks_tabel(item)
             c.is_tabular = True
+            c.table_html = item.get("table_body") or None
             c.source_items = [i]
             c.bbox = item.get("bbox")
             if item.get("img_path"):
@@ -178,7 +192,10 @@ def normalisasi(items: list[dict[str, Any]], doc_id: str) -> list[Chunk]:
             if item.get("img_path"):
                 berjalan.images.append(item["img_path"])
             if latex:
-                potongan.append(latex)
+                berjalan.latex.append(latex)          # mentah, untuk cabang formula
+                terbaca = render_latex(latex)
+                if terbaca:
+                    potongan.append(terbaca)          # prosa, supaya NER menemukannya
             continue
 
         # sisanya: teks

src/knowledge_parsing/render.py

@@ -0,0 +1,122 @@
+"""Render MinerU's markup back into readable prose for `Chunk.text`.
+
+Why this exists — measured, not assumed. Running the same extraction pipeline,
+same document, same gold set, changing only the parse:
+
+    PyMuPDF (plain text)                recall 0.8537   35/41
+    MinerU, raw markup in text          recall 0.7561   31/41
+    MinerU, formula + table rendered     recall 0.8293   34/41
+
+The term filter is an NER model reading prose. MinerU writes formulas with every
+character spaced out — `{ \\mathrm { P u r c h a s i n g ~ c o s t s } }` — and
+tables as HTML. Neither yields a single mention, even for terms the same model
+finds easily in plain text.
+
+The markup is not discarded: `Chunk.latex` and `Chunk.table_html` keep it
+verbatim, because the formula branch needs exactly that form.
+"""
+
+from __future__ import annotations
+
+import html
+import re
+
+# --- LaTeX ---------------------------------------------------------------
+
+_DROP_COMMANDS = re.compile(
+    r"\\(?:qquad|quad|left|right|displaystyle|limits|nolimits)\b|\\[!,;:]"
+)
+_WRAPPERS = re.compile(r"\\(?:mathrm|mathbf|mathit|mathsf|text|textrm|operatorname)\s*")
+# Operators map to non-letter glyphs on purpose. A letter here (e.g. "x" for
+# \times) would itself look like a spelled-out single character and get merged
+# into the token beside it — "C_p x T" becoming "C_p xT".
+_SYMBOLS = {
+    r"\times": "×", r"\cdot": "·", r"\div": "/", r"\pm": "±",
+    r"\leq": "≤", r"\geq": "≥", r"\neq": "≠", r"\approx": "≈",
+    r"\%": "%", r"\$": "$", r"\&": "&",
+}
+
+# "P u r c h a s i n g" -> "Purchasing", "1 0 0" -> "100". Runs of >= 2 single
+# characters; letters and digits are joined separately so "C 5" is left alone.
+# Subscripts are folded first (below), so a real variable like `C_p` is already
+# one token and never gets swallowed into a neighbouring word.
+_SPACED_RUN = re.compile(
+    r"(?<!\S)(?:[A-Za-z]\s+){1,}[A-Za-z](?!\S)"
+    r"|(?<!\S)(?:\d\s+){1,}\d(?!\S)"
+)
+
+# `~` is a non-breaking SPACE in LaTeX, i.e. a real word boundary. It has to
+# survive the whitespace pass, otherwise "c o s t s" after it merges into the
+# preceding word and "Purchasing costs" becomes "Purchasingcosts".
+_WORD_GAP = "\x00"
+
+
+def render_latex(latex: str) -> str:
+    """LaTeX -> a readable line. Best effort: it feeds a term filter, not a parser."""
+    s = latex.replace("$$", " ").replace("$", " ")
+    s = s.replace("~", _WORD_GAP)
+
+    # array/matrix scaffolding carries no meaning for a term filter
+    s = re.sub(r"\\(?:begin|end)\s*\{[^{}]*\}", " ", s)
+    s = re.sub(r"\{\s*(?:[rlc|]\s*){1,8}\}", " ", s)   # column spec, e.g. { r l }
+    s = s.replace("\\\\", " ; ").replace("&", " ")
+
+    # fold sub/superscripts first, so `C _ { p }` becomes one token `C_p`
+    s = re.sub(r"\s*_\s*\{\s*([^{}]*?)\s*\}", r"_\1", s)
+    s = re.sub(r"\s*\^\s*\{\s*([^{}]*?)\s*\}", r"^\1", s)
+    s = re.sub(r"\s*_\s*([A-Za-z0-9])", r"_\1", s)
+    s = re.sub(r"\s*\^\s*([A-Za-z0-9])", r"^\1", s)
+
+    # \frac{a}{b} -> (a) / (b); repeat for nesting
+    for _ in range(4):
+        baru = re.sub(r"\\d?frac\s*\{([^{}]*)\}\s*\{([^{}]*)\}", r"(\1) / (\2)", s)
+        if baru == s:
+            break
+        s = baru
+
+    s = re.sub(r"\\sqrt\s*\{([^{}]*)\}", r"sqrt(\1)", s)
+    for k, v in _SYMBOLS.items():
+        s = s.replace(k, f" {v} ")
+    s = _WRAPPERS.sub(" ", s)
+    s = _DROP_COMMANDS.sub(" ", s)
+    s = re.sub(r"\\[A-Za-z]+", " ", s)     # any command left over
+    s = s.replace("{", " ").replace("}", " ").replace("\\", " ")
+    s = re.sub(r"[ \t\r\n]+", " ", s).strip()
+
+    # now that spacing is uniform, rejoin the spelled-out words and numbers
+    s = _SPACED_RUN.sub(lambda m: m.group(0).replace(" ", ""), s)
+
+    s = s.replace(_WORD_GAP, " ")
+    return re.sub(r"\s+", " ", s).strip()
+
+
+# --- HTML tables ---------------------------------------------------------
+
+_CELL_END = re.compile(r"</\s*(?:td|th)\s*>", re.I)
+_ROW_END = re.compile(r"</\s*tr\s*>", re.I)
+_TAG = re.compile(r"<[^>]+>")
+
+# Table cells routinely carry inline LaTeX ("$\frac{kVA}{...}$"). Stripping the
+# HTML alone leaves that markup sitting in the prose, which is the same problem
+# the formula rendering exists to solve — found in a real table on the first run.
+_INLINE_MATH = re.compile(r"\$([^$]{1,400})\$")
+
+
+def render_table(table_html: str) -> str:
+    """HTML table -> plain rows, cells separated by ' | '.
+
+    Keeps the reading order a person would use, so a term appearing only in a
+    table header is still a mention.
+    """
+    s = _ROW_END.sub("\n", table_html)
+    s = _CELL_END.sub(" | ", s)
+    s = _TAG.sub(" ", s)
+    s = html.unescape(s)
+    s = _INLINE_MATH.sub(lambda m: " " + render_latex(m.group(1)) + " ", s)
+
+    baris = []
+    for raw in s.split("\n"):
+        bersih = re.sub(r"[ \t]+", " ", raw).strip().strip("|").strip()
+        if bersih:
+            baris.append(re.sub(r"\s*\|\s*", " | ", bersih))
+    return "\n".join(baris)