[NOTICKET] fix(knowledge-parsing): render_latex was deleting division (X21)

Rifqi's three faults, all reproduced against the document's own 11 equations
and all fixed. Two of his numbers were off, and the corrections matter:

- It hit 4 of 11 equations, not 2 — PA, PA_ij, UA_ij and PTY_ij. Not just
PA and UA: every composite form was affected too.
- The nested braces come from mathrm{} and sum _{ij}, NOT from text{}, which
this standard never uses. A fix written against text{} would have repaired
the synthetic example and no real equation.

(C), the serious one: the frac rewrite needed a brace-free argument, could not
match a nested numerator, and the unmatched frac then fell through to the
generic "drop leftover commands" pass. The division was deleted, leaving a
formula that reads as a product and states something the document does not.

The ordering decision this needed: braced wrappers are unwrapped BEFORE frac,
and the unwrap consumes its braces. The old pass removed only the command name
and left { } behind — those leftover braces were what blocked frac all along.
frac now uses a balanced-brace scanner; a regex cannot express nesting and
fails silently rather than raising.

(A) and (B) were run-boundary faults: boundaries are now word-character
classes, and subscript contents are joined at fold time.

Division lost 4 -> 0. Six regression cases unchanged.

Still open, deliberately, because it changes interpretation and not mechanics:
after MinerU spaces every character, a multiplication "x" cannot be told apart
from the letter. The standard's flagship formula therefore still renders as
MOHHxQtyxPAxUAxPty, costing the term filter five mentions. The retraction in
the previous diagnosis claimed that formula "renders perfectly" — it does not;
its raw source carries a literal "x", no times and no mathrm.

Docs corrected to match, including escaping damage that had turned frac and
text into control characters in the diagnosis itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KNOWLEDGE_PIPELINE_CALIBRATION.md

@@ -356,44 +356,84 @@ Fix direction: `rule.txt` should exclude bare equations explicitly ("an equation
 branch's job; a rule is prose that constrains behaviour"). That is a prompt change and therefore owes
 an eval run per CLAUDE.md §7B — **not** to be done on vibes.
 
-### Cause 3 — three distinct `render_latex` faults (corrected 2026-08-24)
-
-> **Correction.** An earlier version of this section said `render_latex` "eats the first character",
-> with a three-line reproduction. **That was wrong** — the repro was mangled by shell escaping on the
-> way in, not by her code. `Production = MOHH × Qty × PA × UA × Pty` renders perfectly. The real
-> faults are these three, reproduced against the document's own 11 equations
-> (`scratchpad/repro_render.py`, no escaping involved).
-
-**(A) A spaced run touching a bracket is only half-collapsed.** `_SPACED_RUN` is bounded by
-`(?<!\S)` / `(?!\S)`, so a run cannot match if it touches `(` or `)`. The `rac{a}{b}` →
-`(a) / (b)` rewrite inserts exactly those brackets, which strands the character at **each** end:
+### Cause 3 — three distinct `render_latex` faults (FIXED 2026-08-24)
+
+> **Two corrections, both from re-measuring against the document's own 11 equations.**
+>
+> An earlier version of this section said `render_latex` "eats the first character". That was
+> retracted — the repro was mangled by shell escaping, not by the code.
+>
+> The retraction then over-corrected. It claimed `Production = MOHH × Qty × PA × UA × Pty`
+> "renders perfectly". **It does not.** The document's raw `content_list.json` contains no
+> `\times` and no `\mathrm` — just a literal `x`, spaced like every other character:
+>
+> ```
+> 'P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y'
+>   ->  'Production = MOHHxQtyxPAxUAxPty'
+> ```
+>
+> MOHH, Qty, PA, UA and Pty — the five terms the glossary is built around — fuse into one
+> token. See *Still open* below.
+
+**(A) A spaced run touching a bracket is only half-collapsed.** `_SPACED_RUN` was bounded by
+`(?<!\S)` / `(?!\S)`, so a run could not match if it touched `(` or `)`. The `\frac{a}{b}` →
+`(a) / (b)` rewrite inserts exactly those brackets, stranding a character at **each** end:
 
 ```
 'I N P R H o u r s'      ->  'INPRHours'      # correct, no brackets
 '(I N P R H o u r s)'    ->  '(I NPRHour s)'  # first and last stranded
 ```
 
-**(B) The subscript fold runs before the collapse and splits the run.** `Q t y _ {a c t i v i t y}`
-becomes `Qt y_a ctivity` — the `_` is glued mid-run, so the collapse sees three broken fragments
-instead of two words.
+**(B) The subscript fold ran before the collapse and split the run.** `Q t y _ {a c t i v i t y}`
+became `Qt y_a ctivity` — the `_` glued mid-run, so the collapse saw broken fragments.
 
-**(C) The serious one — fraction structure is silently dropped.**
+**(C) The serious one — fraction structure was silently dropped.**
 
 ```
-\mathrm{PA} = rac {	ext {Total Hours} - 	ext {Breakdown}}{	ext {Total Hours}} 	imes 100\%
-->  'PA = Total Hours - Breakdown Total Hours × 100 %'
+\mathrm { P A } = { \frac { \mathrm { T o t a l } ~ H o u r s - B r e a k d o w n }
+                          { \mathrm { T o t a l } ~ H o u r s } } \mathrm { x } 1 0 0 \%
+  ->  'PA = Total Hours - BreakdownTotal Hoursx 100 %'
 ```
 
-The `rac` rewrite needs `\{([^{}]*)\}`, which cannot match a numerator containing nested
-`	ext{…}` braces, so the division is dropped rather than rendered. **This is not a spacing
-blemish — the formula now reads as a product and states something the document does not.** It
-affects the PA and UA formulas, i.e. the two most important in the standard.
+The `\frac` rewrite needed `\{([^{}]*)\}`, which cannot match a numerator containing nested
+braces. The unmatched `\frac` then fell through to the generic "drop any leftover command" pass,
+so **the division was deleted rather than rendered.** This is not a spacing blemish — the formula
+reads as a product and states something the document does not.
+
+> **The nesting comes from `\mathrm{…}` and `\sum _{ij}`, not `\text{…}`.** The BUMA standard
+> never uses `\text{}`. A fix written against `\text{}` would repair a synthetic example and no
+> real equation.
+
+**Measured tally, 11 equations:**
+
+| | Before | After |
+|---|---|---|
+| Division lost | **4** — eq 6, 7, 9, 11 (`PA`, `PA_ij`, `UA_ij`, `PTY_ij`) | **0** |
+| Stranded characters | 4 | 0 |
+| Clean | 5 | 9 + 2 carrying only the document's own literal `x` |
+
+Not two equations and not only PA/UA: **every composite (`_ij`) form was affected too.**
+
+**The fix** (`src/knowledge_parsing/render.py`):
+
+1. **Braced wrappers are unwrapped before `\frac`, consuming their braces.** The old `_WRAPPERS`
+   deleted only the command name and left `{ }` behind — those leftover braces were what stopped
+   `\frac` from matching. This is the ordering decision the diagnosis asked for.
+2. **`\frac` uses a balanced-brace scanner**, not a regex. `[^{}]*` can never handle a nested
+   argument, and it fails silently rather than raising.
+3. **Run boundaries are word-character classes**, and subscript contents are joined at fold time
+   (`_ { i j }` → `_ij`).
+
+Six regression cases hold unchanged: `C_p`, `C 5`, `C_p × T`, `\frac{a}{b}`, `Purchasing costs`,
+and a bare spaced run.
 
-Of the document's 11 equations, 5 render cleanly, 4 carry stranded characters, and the 2 with
-`	ext{}` numerators lose their division.
+**Still open — a decision, not a defect.** After MinerU spaces every character, a multiplication
+`x` is indistinguishable from the letter `x`: no `\times`, no `\mathrm`, no signal at all. Treating
+it as an operator is right for engineering formulas and wrong for a variable named `x`. The
+`_SYMBOLS` comment shows the author already weighed this trade for `\times`. Left unfixed
+deliberately, because it changes interpretation rather than mechanics.
 
-Impact: corrupted formula text sits in `Chunk.text`, so a span quoting a formula cannot be verified;
-`formula_latex` carries the damage into the Skill Registry; and a term occurring only inside a
-formula can be split away from itself.
+Impact of what remains: a term occurring only inside `Production = …` cannot be separated from its
+neighbours, which costs the term filter five mentions on the standard's flagship formula.
 
-**Owner: Sofhia** — `src/knowledge_parsing/render.py`. Not fixed here: it is the parsing half.
+**Owner: Sofhia** — `src/knowledge_parsing/render.py`.

KNOWLEDGE_PIPELINE_TODO.md

@@ -169,7 +169,8 @@ envelope shape proposed in §3 — so S1 can settle either way without touching
 | **X18** | Full-document run through both halves | Rifqi | ✅ | **Done 2026-08-24** → `eval/knowledge/results/v2_full_document_2026-08-24_093051.json`. Parsing module built the artifact (31 chunks), extraction ran all four branches: 77 calls, 80% cache hit. **E3 = 0.90, PASS** — the prototype's failing experiment (0.75 vs a 0.80 line) now clears it, on a 10-entry scoreable base |
 | **X19** | Rule-branch recall regression | Rifqi | 🔎 | **Diagnosed 2026-08-24 — calibration §12.** Three causes: (a) 2 of 3 losses are a *scoring artifact* (gold written `QtyA`, rendered output `Qty_A`; `_` is a word char so the substring match fails) — true comparison is 4/15 → 3/15; (b) the one real loss is rendered formulas competing with prose — 4 of 8 v2 rule statements are bare equations vs 0 of 7 in the prototype, and the formula branch already captures them; (c) a `render_latex` bug (X21) |
 | **X20** | Exclude bare equations in `rule.txt` | Rifqi | ⬜ | The fix for X19(b). A prompt change → owes an eval run per §7B; do not ship on vibes. The formula branch already covers equations, so this is duplicated spend as well as a miss |
-| **X21** | `render_latex` — three faults, one serious | Sofhia | ⛔ | **Found 2026-08-24, corrected same day.** Not "eats the first character" (that repro was mangled by my shell escaping — retracted). Three real faults: (A) a spaced run touching a `(` inserted by the `rac` rewrite is only half-collapsed → `(I NPRHour s)`; (B) the subscript fold splits the run → `Qt y_a ctivity`; **(C) a `rac` with `	ext{}` in the numerator loses its division entirely** → `PA = Total Hours - Breakdown Total Hours × 100 %`, which states something the document does not, and it hits the PA and UA formulas. Repro: `scratchpad/repro_render.py`, calibration §12 |
+| **X21** | `render_latex` — three faults, one serious | Sofhia | ✅ | **Found and FIXED 2026-08-24.** (A) a spaced run touching a `(` inserted by the `\frac` rewrite was only half-collapsed → `(I NPRHour s)`; (B) the subscript fold split the run → `Qt y_a ctivity`; **(C) a `\frac` whose numerator contains nested braces lost its division entirely** → `PA = Total Hours - BreakdownTotal Hours`, which states something the document does not. **Two corrections on re-measurement:** it hit **4 of 11 equations, not 2** — `PA`, `PA_ij`, `UA_ij`, `PTY_ij`, so every composite form as well; and the nesting comes from `\mathrm{}` / `\sum _{ij}`, **not `\text{}`**, which this standard never uses — a fix written against `\text{}` would have repaired nothing real. Fix: unwrap braced wrappers **before** `\frac`, balanced-brace scanner instead of `[^{}]*`, word-class run boundaries. Division lost 4 → 0; six regression cases unchanged. **Still open, as a decision not a defect:** after character-spacing a multiplication `x` is indistinguishable from the letter, so `Production = MOHH x Qty x PA x UA x Pty` renders as one fused token and costs the term filter five mentions. Calibration §12 |
+rac` rewrite is only half-collapsed → `(I NPRHour s)`; (B) the subscript fold splits the run → `Qt y_a ctivity`; **(C) a `rac` with `	ext{}` in the numerator loses its division entirely** → `PA = Total Hours - Breakdown Total Hours × 100 %`, which states something the document does not, and it hits the PA and UA formulas. Repro: `scratchpad/repro_render.py`, calibration §12 |
 | **X22** | Gold/scorer notation decision | Rifqi | ⬜ | Whether to normalise `_` in the scorer or rewrite the gold's `statement_contains`. Either edits the frozen measuring instrument, so it needs a deliberate call rather than a silent fix |
 | **X13** | Conflict detection | 🔎 | ✅ | `validate/conflict.py`, token overlap (explainable to the reviewer, unlike embeddings); never picks a winner. Unit-tested both ways; still **0 conflicts on real data** — one consistent standard gives it nothing to find. Works only because clustering puts all evidence in one call |
 | **X14** | Diff vs. active glossary version | 🔎 | ✅ | `diff/glossary_diff.py`. The baseline is passed in explicitly (`--active-glossary`) rather than read from wherever the last run wrote, so the duplicate and conflicting paths are reachable — all three verified by test |

src/knowledge_parsing/render.py

@@ -40,9 +40,24 @@ _SYMBOLS = {
 # characters; letters and digits are joined separately so "C 5" is left alone.
 # Subscripts are folded first (below), so a real variable like `C_p` is already
 # one token and never gets swallowed into a neighbouring word.
+#
+# The boundaries are word-character classes, not `\S`. Two reasons, both found
+# on real equations:
+#   - `\frac` rewriting INTRODUCES parentheses, and `(?<!\S)` refuses to match a
+#     run touching one, stranding a letter at each end:
+#     "(I N P R H o u r s)" -> "(I NPRHour s)".
+#   - the trailing side lets `_`/`^` follow, so a run whose last letter carries a
+#     subscript is not cut short: "Q t y_activity" -> "Qty_activity".
 _SPACED_RUN = re.compile(
-    r"(?<!\S)(?:[A-Za-z]\s+){1,}[A-Za-z](?!\S)"
-    r"|(?<!\S)(?:\d\s+){1,}\d(?!\S)"
+    r"(?<![A-Za-z0-9_])(?:[A-Za-z]\s+){1,}[A-Za-z](?![A-Za-z0-9])"
+    r"|(?<![A-Za-z0-9_])(?:\d\s+){1,}\d(?![A-Za-z0-9])"
+)
+
+# Wrappers that take a braced argument. Handled by brace matching rather than
+# `_WRAPPERS` below, because merely deleting the command name leaves its braces
+# behind — and those braces are what stopped `\frac` from matching.
+_BRACED_WRAPPER = re.compile(
+    r"\\(?:mathrm|mathbf|mathit|mathsf|mathtt|text|textrm|textbf|operatorname)\s*(?=\{)"
 )
 
 # `~` is a non-breaking SPACE in LaTeX, i.e. a real word boundary. It has to
@@ -51,6 +66,62 @@ _SPACED_RUN = re.compile(
 _WORD_GAP = "\x00"
 
 
+def _rapatkan(teks: str) -> str:
+    """Gabungkan deret karakter yang dieja satu-satu: 'i j' -> 'ij'."""
+    return _SPACED_RUN.sub(lambda m: m.group(0).replace(" ", ""), teks)
+
+
+def _argumen(s: str, i: int) -> tuple[str | None, int]:
+    """Isi kurung SEIMBANG yang dimulai di `s[i] == '{'`.
+
+    Regex `\\{([^{}]*)\\}` tidak bisa dipakai di sini: argumen nyata mengandung
+    kurung bersarang (`\\mathrm{...}`, `\\sum _{ij}`), dan pola itu diam-diam
+    gagal cocok, bukan memberi galat.
+    """
+    dalam = 0
+    for j in range(i, len(s)):
+        if s[j] == "{":
+            dalam += 1
+        elif s[j] == "}":
+            dalam -= 1
+            if dalam == 0:
+                return s[i + 1:j], j + 1
+    return None, i
+
+
+def _buka_pembungkus(s: str) -> str:
+    r"""`\mathrm{X}` -> `X`, IKUT memakan kurungnya."""
+    for _ in range(8):
+        m = _BRACED_WRAPPER.search(s)
+        if not m:
+            break
+        isi, akhir = _argumen(s, m.end())
+        if isi is None:
+            break
+        s = s[:m.start()] + " " + isi + " " + s[akhir:]
+    return s
+
+
+def _render_frac(s: str) -> str:
+    r"""`\frac{a}{b}` -> `(a) / (b)`, sadar kurung bersarang, rekursif."""
+    for _ in range(8):
+        m = re.search(r"\\d?frac\s*(?=\{)", s)
+        if not m:
+            break
+        pembilang, lanjut = _argumen(s, s.index("{", m.end() - 1))
+        if pembilang is None:
+            break
+        mulai_penyebut = s.find("{", lanjut)
+        if mulai_penyebut == -1:
+            break
+        penyebut, akhir = _argumen(s, mulai_penyebut)
+        if penyebut is None:
+            break
+        s = (s[:m.start()] + " (" + _render_frac(pembilang) + ") / ("
+             + _render_frac(penyebut) + ") " + s[akhir:])
+    return s
+
+
 def render_latex(latex: str) -> str:
     """LaTeX -> a readable line. Best effort: it feeds a term filter, not a parser."""
     s = latex.replace("$$", " ").replace("$", " ")
@@ -61,30 +132,41 @@ def render_latex(latex: str) -> str:
     s = re.sub(r"\{\s*(?:[rlc|]\s*){1,8}\}", " ", s)   # column spec, e.g. { r l }
     s = s.replace("\\\\", " ; ").replace("&", " ")
 
-    # fold sub/superscripts first, so `C _ { p }` becomes one token `C_p`
-    s = re.sub(r"\s*_\s*\{\s*([^{}]*?)\s*\}", r"_\1", s)
-    s = re.sub(r"\s*\^\s*\{\s*([^{}]*?)\s*\}", r"^\1", s)
+    # ORDER MATTERS, and this is the ordering decision: unwrap the braced
+    # wrappers BEFORE touching `\frac`.
+    #
+    # `\frac`'s arguments in real documents are not brace-free — the BUMA
+    # standard writes `\frac { \mathrm { T o t a l } ~ H o u r s - ... } { ... }`.
+    # With the wrappers still in place the old `\{([^{}]*)\}` pattern could not
+    # match, so the `\frac` was never rewritten; it then fell through to the
+    # generic "drop any leftover command" pass and the DIVISION WAS SILENTLY
+    # DELETED. `PA = (Total - Breakdown) / Total` came out as
+    # `PA = Total Hours - BreakdownTotal Hours`, which reads as a product and
+    # states something the document does not. It hit 4 of 11 equations —
+    # PA, PA_ij, UA_ij, PTY_ij.
+    s = _buka_pembungkus(s)
+
+    # fold sub/superscripts, so `C _ { p }` becomes one token `C_p`. The content
+    # is joined too: `_ { i j }` -> `_ij`, otherwise the space inside survives
+    # and splits the token later ("_i j").
+    s = re.sub(r"\s*_\s*\{\s*([^{}]*?)\s*\}", lambda m: "_" + _rapatkan(m.group(1)), s)
+    s = re.sub(r"\s*\^\s*\{\s*([^{}]*?)\s*\}", lambda m: "^" + _rapatkan(m.group(1)), s)
     s = re.sub(r"\s*_\s*([A-Za-z0-9])", r"_\1", s)
     s = re.sub(r"\s*\^\s*([A-Za-z0-9])", r"^\1", s)
 
-    # \frac{a}{b} -> (a) / (b); repeat for nesting
-    for _ in range(4):
-        baru = re.sub(r"\\d?frac\s*\{([^{}]*)\}\s*\{([^{}]*)\}", r"(\1) / (\2)", s)
-        if baru == s:
-            break
-        s = baru
+    s = _render_frac(s)
 
     s = re.sub(r"\\sqrt\s*\{([^{}]*)\}", r"sqrt(\1)", s)
     for k, v in _SYMBOLS.items():
         s = s.replace(k, f" {v} ")
-    s = _WRAPPERS.sub(" ", s)
+    s = _WRAPPERS.sub(" ", s)      # sisa pembungkus tanpa kurung, mis. `\mathrm x`
     s = _DROP_COMMANDS.sub(" ", s)
     s = re.sub(r"\\[A-Za-z]+", " ", s)     # any command left over
     s = s.replace("{", " ").replace("}", " ").replace("\\", " ")
     s = re.sub(r"[ \t\r\n]+", " ", s).strip()
 
     # now that spacing is uniform, rejoin the spelled-out words and numbers
-    s = _SPACED_RUN.sub(lambda m: m.group(0).replace(" ", ""), s)
+    s = _rapatkan(s)
 
     s = s.replace(_WORD_GAP, " ")
     return re.sub(r"\s+", " ", s).strip()