Agentic-Service-Data-Eyond-Catalog

Sleeping

App Files Files Community
Agentic-Service-Data-Eyond-Catalog / docs /knowledge /KNOWLEDGE_PIPELINE_CALIBRATION.md
ishaq101
/fix parsing and term extract (#21)
f07443e about 1 month ago
preview code
Raw History Blame Contribute Delete
61.3 kB
# Knowledge Pipeline — Calibration Reference

<!-- doc-status -->
> 🔄 **LIVE — tuned constants and the reason behind each.** Read before changing any constant in `src/knowledge_extraction/settings.py`.
> **Note (2026-09-02):** the primary parser is now **Mistral OCR** (no GPU). The MinerU-specific *parsing* sections here (§10 quality, §12 `render_latex`, §13 backend choice) are **historical as of 2026-09-07** — MinerU was dropped and is installed nowhere, so those sections record how the choice was made rather than a path you can run; the *extraction* constants (GLiNER, clustering, ranking, thresholds) are unaffected and current. See `eval/parsing/README.md`.

**Harvested from the `kex` prototype, 2026-08-19.** Every number below cost a run, a bug, or a
measurement to find, and **none of it is in the spec** — it lived only as constants and comments in
a codebase that is not being ported.

This file exists because clean rewrites drop accidental knowledge and then rediscover it as
production defects. When building extraction v2, take the value **and** the reason. If you change
one, change it deliberately and record why.

Companions: [KNOWLEDGE_PIPELINE_TODO.md](KNOWLEDGE_PIPELINE_TODO.md) (plan) ·
[eval/knowledge/](../../eval/knowledge/) (gold set, scorer, frozen baseline) ·
[knowledge_pipeline_context.md](knowledge_pipeline_context.md) (intent).

---

## 1. Term filter — GLiNER

| Constant | Value | Why this value |
|---|---|---|
| Model | `urchade/gliner_multi-v2.1` | CPU, ~400 MB. Cheap enough to run unconditionally on every chunk, which is what lets us reject a standalone relevance gate |
| Score threshold | **0.25** | Not 0.35. The sweep measured recall **0.854 @ 0.25** vs **0.658 @ 0.35** on the `broad` label set. Precision falls (0.41 vs 0.50) and that is accepted: the filter is meant to be over-inclusive, clustering and evidence ranking absorb the noise, and **a term the filter never proposes can never be recovered downstream** |
| Label variant | **`broad`** | Two earlier variants both missed the same class — mining activities and materials (coal mining, land clearing, waste removal, joint survey). `broad` adds `mining activity`, `material or commodity`, `data source or system`, `work category`, `analysis method` to cover it |
| Window size | **130 words**, overlap **30** (re-confirmed 2026-09-10: 256 truncates 46 of 99 Open Pit windows, 180 truncates 4; 130 is the only size measured safe on both English documents. One BUMA window exceeds the cap at EVERY size — pre-existing) | Not 220. GLiNER truncates past ~384 of its own tokens and **warns rather than failing** — silently dropping the tail of every long chunk. Indonesian technical prose subword-tokenises at roughly **2.5×**, so a 220-word window still tripped the cap. Chunks are fed as overlapping word windows with offsets remapped |
| Span token cap | **12** (GLiNER's own) | Compound terms ("UA plan grouping (composite) actual") exceed it. Candidates that hit the cap are **flagged, not silently truncated**, so a recall shortfall can be attributed to the cap |

**Label sets** (`labels.yaml`) — three variants were tried and A/B'd without touching code. Keep
that property: label phrasing is the main recall lever and GLiNER is very sensitive to it. The
Indonesian-phrasing variant (`istilah produksi tambang`, …) did **not** win; `broad` did.

---

## 2. Clustering

| Constant | Value | Why this value |
|---|---|---|
| Match order | exact (normalised) → abbreviation↔expansion → fuzzy | Cheapest and most certain first |
| ~~Fuzzy threshold~~ | ~~**92**~~ | ⛔ **RETIRED 2026-09-10.** `token_set_ratio` returns exactly 100 when one token set is a SUBSET of the other, so this threshold was **inert** and a head noun always absorbed its compounds. 131 of 140 merges were subset matches; the 6 correct ones were plurals, now handled by `normalize(lemma=True)`. The pass is commented out at its call site; `_fuzzy_match` is parked. Subset merges went to 0 on all three test documents |
| **Fuzzy disabled below 5 characters** | `min(len(a), len(b)) < 5` → skip | **The one to not lose.** `PA` vs `UA` scores high on `token_set_ratio`. Below 5 chars only exact matching is allowed |
| Canonical surface | shortest variant ≥ 2 chars | That is how a reader looks a term up — `PA`, not `Physical Availability (PA) untuk …` |

**Over-merging is worse than under-merging, and the asymmetry is not close.** An under-merge costs
one extra LLM call and one extra review row. A wrong merge silently destroys a distinct term and
**the expert never sees it** — there is no downstream mechanism that recovers it.

**Legend extraction must run before clustering.** Without the abbreviation index, `PA` and
`Physical Availability` cluster as two unrelated terms.

**Stop surfaces** — dropped as whole surface forms only (never as substrings, so no term is lost):
`unit · type · class · equipment · equipment unit · parameter · activity · data · nilai · proses ·
hasil · total`.

**Normalisation** (clustering only, never for span checking): NFKC → casefold → `-`/`_` → space →
strip `.’'` → non-word to space → collapse whitespace → strip ` ()/`.

---

## 3. Evidence ranking

Six signals plus one penalty. The full ranked list is retained, not just the top K — escalation
needs the tail.

| Signal | Weight |
|---|---|
| Definitional cue within 100 chars of the mention | **+5.0** |
| Term appears in the chunk heading | **+4.0** |
| Mention sits inside a legend block | **+3.5** |
| Chunk contains a formula | **+2.0** |
| Mention is bold/italic | **+1.5** |
| First occurrence of the cluster | **+1.0** |
| Chunk is tabular | **−3.0** |

| Constant | Value | Why |
|---|---|---|
| `evidence_k` | **5** (raised from 3, 2026-09-10) | Chunks fed per extraction round |
| Cue proximity | **100 chars** | Distance from mention to a definitional cue |
| `evidence_link_partners` | **1** (added 2026-09-11) | Linked partner chunks pulled into the window on top of the ranked K |

### Linked partners (2026-09-11)

Schema 0.4.0 pairs a table chunk with the prose that introduces it, via `referenced_by` in both
directions. `top_k` now pulls **at most one** partner into the evidence window.

**Why it is not redundant — the measurement that decided it.** Ranking only, on the three v2
artifacts and their recorded filter payloads (free; no GLiNER, no spend):

| | clusters | top-5 touches a linked pair | **both sides already there** | only one side |
|---|---|---|---|---|
| BUMA | 83 | 31 | **0** | 31 |
| Komatsu | 93 | 55 | **0** | 55 |
| Open Pit | 427 | 70 | **0** | 70 |

**Zero clusters out of 603 saw both sides.** The ranker never co-selects a table and the prose that
names it — and that prose is frequently the table's only title (*"Troubleshooting of engine
(S-mode)"* above a bare `S-1 | 6` grid). Cost at the cap: evidence read **+14.5% / +24.5% / +9.9%**,
which is **+1.6% / +4.1% / +4.2%** of total prompt spend; Open Pit moves 71% → ~74% of the 2M ceiling.

**Two placement rules, both load-bearing.** The expansion lives in `top_k`, *not* `rank_evidence`:

1. The **ranked list is never mutated**, so everything reading `evidence_chunk_ids` sees the number
   it saw before.
2. `evidence_block` (the prompt) and `evidence_text` (the span check) both take their ids from the
   same `top_k` call, so *what the model reads is exactly what the span check searches* stays true.
   Expanding in the prompt renderer instead would null every quote from a partner as unlocatable.
   Verified across all 603 clusters: **0 parity failures**.

**Scope, stated honestly.** The document this helps most (Komatsu, 55 of 93 clusters) is the one that
returns 0 definitions for *genre* reasons. This is groundwork for the table branch (Step 8) rather
than a glossary win, and it has **not** been measured against a paid run — with this corpus's noise a
single run could not have shown anything either way. Set `EVIDENCE_LINK_PARTNERS = 0` to disable.

**Heading matching is word-boundary, never substring.** `PA` is a substring of `Parameter`, `pada`
and `composite` — substring matching handed the +4.0 heading bonus to almost every Indonesian
heading and pushed real definition sections *below* formula tables.

**Why this filter is legitimate when the relevance gate was rejected:** no term is dropped (only its
*evidence* is narrowed), every mention stays in provenance so a bad top-3 is visible, and the full
ranked list is retained so the tail stays reachable. The rejected gate had none of those three
properties. Preserve all three if ranking is ever revisited. (The third property used to be phrased
as "the escalation loop self-corrects"; escalation was retired on 2026-09-11 with
`MAX_ESCALATION_ROUNDS = 0`, but retaining the tail is what made the property true and it still
holds.)

---

## 4. Chunking

> **Chunking stays section-based.** Page-based grouping was tested on 2026-09-02 and **did not
> improve term recall** — it lowered it, and the terms it recovered were offset by ones it lost to
> span-overlap competition. Full numbers and the reasoning in §7. Do not re-propose it without new
> evidence; the lever it was reaching for is in `filters/span_filter.py`, not here.
| Constant | Value | Why |
|---|---|---|
| Max chunk tokens | **1500** | With paragraph-boundary splitting; sentence boundaries as fallback for a single over-budget paragraph |
| Max heading length | **90 chars** | Longer lines are sentences or formula rows, not headings |
| Boilerplate frequency | **≥ 0.6 of pages** | A line repeating on 60%+ of pages is a running header/footer. Detected by frequency with **digits normalised to `#`** so page counters collapse — never by a hardcoded document-specific string |
| Tabular detection | short lines > 0.6 **and** numeric lines > 0.4 | Feeds the −3.0 ranking penalty |
**Three things the real document forced, all of which corrupt every downstream number if missed:**
1. **Running headers/footers** — every page repeats the title block and "Confidential".
2. **Breadcrumb headings** — pages re-print `2. PENJELASAN PARAMETER / 2.1. Production Parameter / …`
   at the top. A naive splitter re-opens the section and shatters it. Rule used: *a heading that is
   the current section or an ancestor of it, already seen, is a breadcrumb* — the section continues
   and `page_end` extends.
3. **Colon-continuation** — legend rows arrive as `MOHH` then `: Machine on Hand Hours` on the next
   line. Rejoined before the legend filter sees them.

Points 1 and 2 apply to MinerU output too: its `title` blocks repeat the same breadcrumbs on pages
2–8 of the standard.

---

## 5. Extraction (the paid stage)

| Constant | Value | Why |
|---|---|---|
| Temperature | **0.0** | And still not deterministic — see below |
| Seed | 7 | |
| **Cache minimum** | **1024 prompt tokens** | OpenAI-family prompt caching **does not engage at all** below this. The glossary prefix is padded past it *on purpose*. Measured hit rate **54%** — 125,184 of 155,313 prompt tokens |
| Structured output | `json_schema`, fallback `json_object` + validate-retry | Which mode was used is recorded in the run manifest |
| API version | `2024-12-01-preview` | `json_schema` needs ≥ `2024-08-01-preview` |

**Prompts live in files, not code, and the fixed prefix must stay byte-identical across calls** —
any drift and caching stops engaging, silently, at ~10× the input cost.

**`temperature=0` is not determinism.** Consecutive runs of the identical pipeline scored 0.75 and
0.625 on schema fill. Never report a single run as a measurement on a small base.

---

## 6. Validation

| Constant | Value | Why |
|---|---|---|
| Span normalisation | **whitespace only** (+ NFKC) | Not case, not punctuation, not diacritics. Every additional normalisation is a hole a fabrication fits through |
| LaTeX normalisation | **notation-aware canonical form** (+ NFKC) | Whitespace removed entirely, LaTeX spacing commands (`~`, `\,`, `\;`, `\quad`, `\ `) dropped, presentational wrappers (`\mathrm{}`, `\text{}`, `\mathbf{}`, `\operatorname{}`) unwrapped, `\cdot`→`\times`, `\%`→`%`, `\frac{A}{B}`→`frac(A)(B)`, remaining braces dropped. A different notion of equality, not a relaxation — the prompt asks the model to *transcribe into* LaTeX, which is a translation, so identity is the wrong test. **Structure is preserved on purpose:** dropping braces outright would make `\frac{a}{bc}` and `\frac{ab}{c}` normalise alike. Applies **only** to `formula_latex` against `Chunk.latex` — never to prose, where deleting whitespace would make “no data” match “nodata” |
| Escalation rounds | **max 2** | Then keep the entry flagged `no_definition_found` and pass it to review anyway — a term we found but could not define is still useful; dropping it hides a known unknown |
| Conflict overlap threshold | **0.4** token overlap | Token overlap, not embeddings: cheaper, needs no model, and **explainable to the reviewer who has to act on it** |
| Duplicate overlap threshold | **0.8** | Above it, a differing definition is a duplicate; below, a conflict |

**Guarded fields** (span-checked; failure sets the field to `None` and logs it):

| Branch | Fields |
|---|---|
| glossary | `definition`, `full_name`, `formula_latex`, `interpretation` |
| rule | `statement`, `condition`, `consequence` |
| formula | `formula_latex` — checked against **`Chunk.latex`**, not `text` |
| summary | `title`, `purpose_verbatim` — **changed 2026-09-02**, was *(none)* |

**If the provenance span itself is not verbatim, every guarded field on the entry is rejected** —
the entry's only evidence link is broken, so nothing on it can be trusted.

**The summary branch became guardable by changing the question, not the model** (2026-09-02). It was
the one exception here: it asked for a *summary*, and a summary cannot be located in the source by
construction, so guarding it would have rejected every correct answer. It now asks the model to
**locate** the document's own statement of purpose — `purpose` became `purpose_verbatim` — and a
quote is checkable like any other transcription. Both fields are in `_TRANSCRIBED` as well as
`GUARDED_FIELDS`; the `formula_latex` story below is why that distinction matters. Measured on the
reference document: 0 rejections on a real run, and a deliberately fabricated purpose is refused.
This also weakens the case that the branch needs a larger tier (D3 / risk R5) — locating a sentence
is a much smaller ask than composing one.

**`formula_latex` is checked against the source markup, not the rendered text** (2026-08-26). It was
previously guarded in name only: it sat in `GUARDED_FIELDS` but not in `_TRANSCRIBED`, so the LaTeX
value itself was never searched — only its `provenance.span` was. A structurally wrong transcription
with a valid prose span passed, which is exactly the shape X21 produced (a `\frac` that lost its
division reads as a product and states something the document does not).
The fix could not simply add the field to `_TRANSCRIBED`: `text` carries rendered prose and can never
contain LaTeX, so that would have nulled **every** formula entry and presented as a model failure.
`Chunk.latex` — which the parsing half has always emitted, and which extraction's adapter used to
drop on the floor — now crosses the seam and is the haystack instead, compared with whitespace
removed entirely (see the normalisation row above). Found by Sofhia; verified on the real MinerU
shapes.

**An artifact carrying no `latex` fails open**, keeping the old provenance-only guard rather than
rejecting. “Cannot verify” must not present as “failed verification” — the alternative silently voids
every formula entry on every artifact written before this landed. The consequence to know: on such an
artifact `formula_latex` carries the **weaker** guarantee, and nothing in the output says which one
applied.

**Identity was still the wrong test — corrected 2026-08-26.** Sofhia ran the first version against real
artifacts and found `Production = MOHH \times Qty \times …` rejected. Probing further: on a source that
*does* carry `\times`, the guard also rejected a model that dropped `\mathrm{}`, used `\cdot`, or wrote
idiomatic LaTeX — **4 of 5 plausible correct outputs**. It passed only when the model mirrored MinerU's
notation character for character. The first version's own passing test passed only because the fixture
was hand-written to mirror the source.

That is the same category error as the original defect, one level up: first the haystack was the wrong
*format*, then it was the wrong *notion of correctness*. The fix is the notation-aware form in the table
above. It still catches X21 — a lost `\frac` is a structural difference, not a notational one.

**The `pipeline` backend cannot express multiplication, and that is reported rather than guessed.**
`pipeline` writes `Production = MOHH x Qty x PA` — a bare letter `x`, no `\times`, no `\mathrm`, no
signal at all (§12, *Still open*). A claim spelling out the operator is therefore **unprovable rather
than wrong**, and takes the same fail-open path as an empty `latex`. Without that, every product formula
on every `pipeline` artifact would be nulled silently and read as a bad model — and every artifact on
Sofhia's machine is `pipeline`, because she has no GPU.

Resolving `x` to multiplication inside the guard was rejected: it would silently settle a decision §12
deliberately leaves open, and would let `Ma\times Capacity` match a source reading `MaxCapacity`. The
narrow exception covers a *missing operator* only — a claim that also loses its division is still
rejected, and there is a test for exactly that.

`\mathrm{x}` **is** treated as a multiplication sign, before the wrapper unwrap that would destroy the
distinction. That is grounded: `\mathrm{x}100\%` is the real PA formula's tail in this document.

**Measured against real markup, 2026-08-26.** Sofhia supplied the same 11 equations as MinerU
transcribed them under four backends (44 strings, committed at
`eval/knowledge/fixtures/latex_backends.json`). Agreement with `hybrid/high`, the production backend:

| Backend | Whitespace-only | Notation-aware | After the evidence-led additions |
|---|---|---|---|
| `vlm/medium` | 9/11 | **11/11** | 11/11 |
| `hybrid/medium` | — | 10/11 | **11/11** |
| `pipeline` | 5/11 | 8/11 | **10/11** |

The additions came from the inventory, not from guesswork: `\mathsf` (equation 7) and
`\displaystyle` (equation 4) occur in this corpus and were missing; the array **alignment spec**
(`{l}` from one backend, `{r}` from another) and a **trailing row separator** before `\end{array}`
are presentational. A separator BETWEEN rows stays structural. `\mathbf`, `\mathit`,
`\textrm` and `\operatorname` were guesses and **never occur** — kept because they are the
same class and cost nothing, but their presence is not evidence of anything.

The single residual, `pipeline` equation 1, is `Production = MOHH x Qty x PA x UA x Pty`: the
operator is genuinely absent from the source, so it is the marker's job, not the normaliser's. It is
reported `unverified_operator`, which is the designed behaviour.

### X20 same-day A/B: it costs a rule (2026-08-27)

The first controlled comparison in this project. Arms alternated within one session so any drift hit
both equally; arm B reverts X20's three edits and **keeps X23's synthetic examples**, so the only
variable is X20. Rule branch only, 15 calls per run, 5 rounds each. Served by
`gpt-5.4-nano-2026-03-17`.

| Arm | Runs | Result |
|---|---|---|
| **A** — X20 in | 5 | **5/14 every time** |
| **B** — X20 reverted | 5 | **6/14 every time** |

Zero variance in both arms, and exactly one rule differs: **`GAINLOSS_QTY_ZERO_PLAN`, 0% with X20 and
100% without it.** X20 is a **net −1** as written.
**The mechanism is identifiable, and it is X20's own wording.** That gold rule's fragments are
`QtyP(M) = 0` and `QtyA(M) > 0` — a genuine conditional whose CONDITION is written symbolically
(*“Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka: …”*). X20 closes with *“a statement that is mostly
symbols, `=` and numbers is an equation”*, which instructs the model to null precisely this. The
heuristic cannot tell a **computation** from a **symbolic condition**.
**And X20 is not buying what it was written to buy — today.** `GAINLOSS_UNITS`, the rule it targets,
fires **0% in BOTH arms**. Yesterday it fired 6/7 with X20 in place. So X20's apparent win was the
day, not the prompt (see the cross-day section below), and with today's model X20 has no measurable
upside and one measurable cost.
**REVERTED 2026-08-27, on Rifqi's decision.** The carve-out idea (`jika` / `apabila` / `maka` → a
rule however symbolic its condition) was considered and not taken: X20 had a measured cost and no
measured benefit, and refining it would spend more runs defending a change that has yet to earn any.
The intent — stop the rule branch duplicating the formula branch — remains valid and is available to
anyone who wants to attempt it properly.

**The shipped prompt is byte-identical to arm B**, verified by hash. That matters for provenance: the
state now in `main` is the exact state measured at 6/14 across 5 same-day runs on
`gpt-5.4-nano-2026-03-17`, not a reconstruction of it. No re-run was needed and none was done.

**Scope of that number:** arm B measured the **rule branch only** (15 calls). It does not carry E1,
E2 or E3, and **no full-run result file reflects the reverted prompt** — every committed full run on
26–27 Aug had X20 in. Do not read one of those as the current configuration's baseline.

**Cache floor survived the revert:** `rule.txt` is 1097 estimated tokens, still clear of 1024. X20 was
what first pushed it over, so this was worth checking rather than assuming.

**Method note worth keeping:** rule-branch-only runs cost 15 calls against 82 for a full run and take
~35s. A/Bs on this branch are cheap enough that no prompt change to `rule.txt` should ever again be
argued about instead of measured.

### ⚠️ Scores are not stable across DAYS (2026-08-27)

The most important measurement fact found so far, and it invalidates casual cross-day comparison.

`GAINLOSS_UNITS` scored **6/7 on 2026-08-26** and **0/12 on 2026-08-27** — 3 eval runs, 1 CLI run and
8 isolated repeats of the single call. Every input was proven identical:

* only `::0016` contains the gold fragments (`BCM`/`ton`/`jam`), so no other chunk could ever have
  scored it;
* that chunk's candidate is **byte-identical** with and without the day's cue change — same cue
  (`Catatan`), same offset (255), same snippet, verified by building both candidate sets;
* `rule.txt` was untouched between the two days;
* 0 span rejections, so the field is a genuine model abstention, not a guard kill.

Same input, same prompt, same deployment name, `temperature=0` — 6/7 one day, 0/12 the next. The cause
is outside this repo (a server-side model or deployment change is the obvious candidate) and cannot be
determined from here.

**What follows from it, and it is not optional:**

1. **A committed result file is a measurement of a day, not of the pipeline.** Comparing today's run
   to a file from last week silently mixes in whatever the model did in between.
2. **Any A/B must re-run BOTH arms on the same day.** Yesterday's median is not a control.
3. **`temperature=0` was never determinism** (§5 already said so within a day); it is not even
   stability across days.
4. Borderline judgements move most. `::0016` is a tabular chunk whose only prose sits in a `Catatan:`
   line — exactly the case `rule.txt` asks the model to weigh formula against prose. Stable rules
   (`ECA_OTHER_ACTIVITY`, `QTY_ACTUAL_ZERO_PLAN_ALT`) did not move at all.

**This retracts yesterday's “X20 confirmed” reading.** X20 was called confirmed because
`GAINLOSS_UNITS` — the rule it names — fired 6/7. That rule now fires 0/12 with X20 still in place, so
the 6/7 measured a day, not a fix. X20 may still be correct; it is no longer *demonstrated*, and
showing it needs a same-day A/B against a reverted `rule.txt`.

### Rule-branch noise band, and what it revealed (2026-08-26)

Seven runs of the identical pipeline on the same artifact, `temperature=0`. Run 1 carried the
latex-guard defect; it is included because that defect cannot touch the rule branch.

| Rule recall | Runs |
|---|---|
| 3/15 | 1 |
| **4/15** | **5** |
| 6/15 | 1 |

min 3 · median **4** · max 6 · mean 4.14. **The branch is far less noisy than one pair of runs
suggested** — 5 of 7 land on exactly 4/15. Never quote a single run; quote the median and the band.

**Per-rule stability is the useful view, not the total.** Three rules fire in 7/7 runs
(`GAINLOSS_QTY_ZERO_PLAN`, `QTY_ACTUAL_ZERO_PLAN_ALT`, `ECA_OTHER_ACTIVITY`). Two fire in 1/7
(`QTY_PER_ACTIVITY`, `UA_BY_ACTIVITY`) and are the whole of the variance. Nine never fire at all.

**X20 is demonstrated, not assumed.** `GAINLOSS_UNITS` — the single rule X20 was written to recover,
the units sentence that shared a chunk with a Gain/Loss equation — fires in **6 of 7** runs against
**0** at the 3/15 baseline. That is a stable +1, outside the noise, and it is the rule the fix names.

**X23's gold question is answered by measurement.** `WATERFALL_TWO_METHODS` fires **0/7** with the
few-shot that forbade it removed. The prompt was never the binding constraint: the model judges that
sentence descriptive on its own. The gold entry is unreachable with this prompt and model, so the
remaining decision is about the gold, not about the prompt.

**The real ceiling is upstream, and it is not the prompt.** Traced properly on 2026-08-27, which
**corrected a wrong first reading** — an earlier version of this section blamed tabular chunks. It is
not tables: **5 of the 8 candidates already come from tabular chunks**, so the cue regex reaches them
fine. Three separate causes, and only one is the prompt's business:

| # | Cause | Costs | Fix lives in |
|---|---|---|---|
| 1 | **Two gold rules have no matching text in ANY chunk** — `PTY_PRODUCTION_SOURCE`, `PTY_WORKING_HOURS_DEFINITION` | 2 rules, permanently | The gold set, or the parse. Not extraction |
| 2 | **One call returns ONE rule, but a chunk can hold several.** `::0006` alone holds 4 gold rules | 3 rules | Schema/call unit — a list instead of a single object |
| 3 | **Four prose chunks are never proposed as candidates** (`::0008`, `::0011`, `::0017`, `::0024`) | 4 rules | The discourse-cue regex, on prose |

Arithmetic: 13 reachable gold rules live in **9 chunks**, and one-rule-per-call caps the branch at
**9/15** even with perfect candidates and perfect extraction. Against a median of 4/15, the headroom
is 5 rules — and **none of it is reachable by tuning `rule.txt`.**

**E3 was 1.00 in all seven runs**, on a scoreable base of 6–7. Zero variance: when nano answers, it
answers correctly; the open question is how rarely it answers, not whether to trust it.

### What the LaTeX normalisation assumes, and where it stops

Audited 2026-08-26 against cases the corpus does **not** contain, because every fixture here is BUMA
and a suite built only on BUMA cannot catch a rule that is really a fact about one document.

**Held, and these are the ones that matter.** Different operands, precedence (`\frac{a+b}{c}` vs
`a+\frac{b}{c}`), operand order, sub- versus superscript, and `+` versus `-` all stay distinct. The
guard's discrimination follows from preserving structure, not from anything about this corpus.

**Two deliberate over-normalisations. Neither is safe to assume elsewhere.**

| Assumption | Why it is accepted here | When it breaks |
|---|---|---|
| `\mathrm{x}` is a multiplication sign | It is how MinerU transcribes `×` in this corpus. Applied to **both** sides, so a genuine roman `x` converts identically in claim and source and they still match | A corpus where `x` is a real variable name — algebra, statistics. Residual risk is a contrived false *acceptance*, never a false rejection |
| Whitespace inside `\text{}` is deleted | In text mode a space is a real word boundary, so `\text{Total Hours}` and `\text{TotalHours}` compare equal | Two labels differing only by internal spacing. Tolerated because a fabrication differing from the truth by one space is not a threat worth a mode-aware scanner — but it IS lost discrimination, not a neutral simplification |

**`\cdot` is deliberately NOT folded into `\times`.** It was, speculatively — it appears nowhere in
the 44 real strings — and on vectors the two are different operations. The cost is real and pinned by a
test: a model writing `\cdot` for scalar multiplication is now rejected. If that shows up in a live
run, report it the way `unverified_operator` is reported; do not re-fold the two.

**Known incompleteness, all in the fail-safe direction** (a false rejection, never a false acceptance):
plain-TeX `{a \over b}` is infix and unhandled; `\mathbf`, `\mathit`, `\textrm`,
`\operatorname`, `\quad` and the thin-space commands are handled but **never observed**, so their
presence in the code is not evidence that this corpus emits them.

**One boundary bug worth remembering:** `\limits\b` never matched before a subscript, because `_`
is a word character. That is the *same* defect X22 found in the eval scorer, in a different file. Where
a LaTeX command can be followed by `_`, `\b` is the wrong boundary — use `(?![A-Za-z])`.

**Which guarantee applied is recorded on the entry** (`latex_verification`, added 2026-08-26). Three
paths leave `formula_latex` populated — located in the markup, no markup to check against, operator
not recoverable — and they are not equally strong. Until this field existed the output could not tell
them apart, so a verified formula and an unverifiable one reached the expert as identical rows, and on
a `pipeline` artifact most product formulas are the unverifiable kind. Set by the span check, never by
the model; `null` when there is no claim to characterise. Values and per-field notes are in the output
contract §7.

**Never repair a failed span.** A repaired span is an unfalsifiable claim, which is exactly what the
control exists to prevent.

**Conflict detection never picks a winner.** The expert decides.

---

## 7. Negative findings — do not re-derive these

| Finding | Detail |
|---|---|
| **nano is not sufficient for schema fill** | 0.75 precision against a 0.80 kill line. Decision 2026-08-19: **stay on nano anyway** for now; the `__54n` env quad is provisioned. Revisit if quality blocks the demo |
| **Escalation is unreachable on small documents** | 54 of 66 clusters had exactly **1** evidence chunk against K=3, so `rounds_available()` correctly returned 0 and the loop never fired. `escalated = 0` is not a bug. It stays unverified until a larger corpus runs |
| **Conflict detection had nothing to find** | 0 conflicts on a single internally-consistent standard. Untested against real disagreement |
| **85% abstention** | 56 of 66 entries returned no definition. Correct behaviour, but it means the review queue is mostly *"term found, no definition in document"*. Whether that is useful or noise is a review-experience question for Mas Beta |
| **Diff only ever ran the empty-baseline path** | The prototype diffed against the file it then overwrote, so every entry came back `new`. v2 needs a real active-version pointer |
| **Literal source wording was normalised away** | The standard's heading reads *"Physical **of** Availability (PA)"*; the entry carried `full_name: "Physical Availability"`. The gold set records the literal form correctly, so **gold was right and the pipeline was wrong**. Mechanism: the heading is a separate field from the chunk text, and the chunk text does not contain the phrase, so the literal wording never reached the model. Fix: feed the heading verbatim + a span-checked `source_wording` field |
| **A `glob('*.pdf')` picked the wrong document once** | Scored the textbook against the standard's gold set → recall 0.05. Document selection must be explicit and error on ambiguity |
| **Page-based chunking does not improve term recall** (2026-09-02) | Tested directly: the 31 section chunks of the BUMA fixture regrouped into 9 page chunks, everything else held constant. It fixed the structural symptom — chunks under GLiNER's 130-word window went **28/31 → 2/9** — and **E1 fell 0.8293 → 0.8049** (34/41 → 33/41). The net −1 hides the real result, which is a **swap of 7 terms**: page-grouping *recovered* `Fleet management`, `Mineplan`, `Uncontrollable` (the terms the short-context theory predicts) but *lost* `Coal barging`, `Model Unit`, `Standby`, `Total Hours`, and the filter's `dropped_overlapping` rose **7 → 16**. So chunk size is not the binding constraint — **span-overlap resolution is**, and that lives in `filters/span_filter.py`, not in the parser. Read this as *no evidence page-based helps* rather than *page-based is worse*: n=1 document, 41 gold terms, and a 1-term delta is inside the noise band §6 documents. Runs: `results/v2_filters_2026-09-02_092638.json` (control) and `_092714.json` (experiment) |
| **torch wheels install corrupt** | `torchgen` missing, version reports `None`. Fix is delete `site-packages/torch*` and reinstall. Needs **torch ≥ 2.6**. Documented so nobody debugs it as a GLiNER problem |

---

## 8. What was measured, so v2 can be compared

Frozen in `eval/knowledge/results/baseline_prototype_2026-08-13_145132.json`.

Funnel on the 9-page standard: **9 pages → 13 chunks → 169 mentions** (195 before noise filtering)
**→ 66 clusters → 83 LLM calls → 66 entries → 66 review-queue items.**

E1 recall **0.854** · E2 compression **2.56×** · E3 schema-fill precision **0.75** (FAIL) ·
E4 fabrication rejection **1.00** with **0.00** false rejections. Cost **$0.0069**, 242 s wall
clock, 3 fields rejected by span check.

**When quoting E2 externally:** the context doc's **6.2×** is a 1,000-page projection; **2.56×** is
the 9-page measurement. Compression grows as terms repeat across a corpus. Both are true — use the
right one for the audience.

---

## 9. Deliberate deviations from the prototype (v2)

Changes made while rebuilding, with the reason. Each one alters behaviour, so
each is listed rather than folded silently into the tables above.

| # | Change | Why | Measured? |
|---|---|---|---|
| **V1** | **A heading that names the term counts as a mention at offset 0** for cue-proximity scoring | Indonesian standards are written *"2.1.3 Physical of Availability (PA)"* / *"Adalah ketersediaan fisik…"* — the heading names the term and the body opens with the definitional cue **without repeating it**. The prototype therefore awarded no cue bonus on the definition section itself. Measured on the BUMA standard: PA's own definition section (§2.1.3) scored 7.5 and ranked **third**, below a computation section (§2.2.4, 8.5) that merely mentions PA often. After the fix, all five terms rank their own definition section first — PA→2.1.3, UA→2.1.4, Qty→2.1.2, Pty→2.1.5, MOHH→2.1.1 | Structurally verified on the 9-page standard. **Not yet scored** — needs the paid stage to show up in E3 |
| **V2** | **`first_occurrence` follows document order**, and ranking ties break on document order | The prototype awarded the +1.0 to whichever chunk a `set` iteration yielded first, making the bonus arbitrary and the ranking unreproducible across runs. "First occurrence" should mean the term's first appearance in the document | Determinism locked by test; verified stable across `PYTHONHASHSEED` 0/1/7/42/99 |
| **V3** | **`source_wording` field added** to the glossary entry | The control for the literal-source-wording defect (§7). Not yet populated — needs the seam to carry the heading verbatim and the extraction prompt to fill it | No |

**Both V1 and V2 change which chunks reach the paid stage**, so they can move E3
when the extraction branch lands. Re-score against
`eval/knowledge/results/baseline_prototype_2026-08-13_145132.json` at that point
and record the delta rather than assuming the improvement carries.

**V1 and the R1 literal-wording defect share a root cause:** the heading is a
separate field from the chunk text, so anything that reasons over "the term and
its definition together" has to be told to look at both.

| **V4** | **`source_wording` is taken from the section heading, not from the model** | Asked to quote the source wording, the model returned "Physical Availability (PA)" — a genuine verbatim quote from the page-1 intro, but not the §2.1.3 heading "Physical **of** Availability (PA)". Both occur in the document; only the heading form reveals the disagreement. Taking it deterministically means it cannot be normalised away | Live-verified on the pilot: the entry now carries both forms and the queue flags the mismatch |
| **V5** | **`evidence_text` includes chunk headings** | The heading is part of the source document and is often the only place a term is formally named. Excluding it rejected a correct verbatim quotation of the document's own section title (2 rejections on a 3-entry pilot, both spurious) | Live-verified: 0 rejections after the change |
| **V6** | **`full_name` and `source_wording` are span-checked against the source themselves** | Both claim to be literal transcriptions. The provenance span passing does not prove the transcription is faithful, and a normalised "full name" is exactly the silent correction this pipeline exists to surface | Unit-tested |

**Cache-floor finding (measured on the real prompts):** only the **glossary**
prefix clears the 1024-token floor at **1,401 tokens**. `rule` (731), `formula`
(459) and `summary` (288) are **not cacheable** and cache nothing today. Glossary
is also the branch with by far the most calls (66 of 83 on the reference
document), so the padding is on the branch that matters — but padding the other
three is free money if their call counts ever grow.

---

## 10. Parser comparison — MinerU vs PyMuPDF (2026-08-19)

First run of the extraction half on **MinerU-parsed input**, same document, same
gold set, same pipeline code. Result:
`eval/knowledge/results/v2_parser_comparison_2026-08-19_152842.json`.

| Input | E1 recall | Terms found |
|---|---|---|
| PyMuPDF (what the frozen baseline used) | **0.8537** | 35 / 41 |
| MinerU, raw markup | **0.7561** | 31 / 41 |
| MinerU, formulas + tables rendered to prose | **0.8293** | 34 / 41 |

**The span model reads prose, not markup.** MinerU's structure is an asset for
the formula branch and a liability for the filter unless it is rendered first:

- Terms wrapped in LaTeX are invisible. `\text {Total Hours} - \text {Breakdown}`
  yields no mentions for either term, though both are gold terms and both are
  found in the plain-text parse. Unwrapping `\text{}`/`\mathrm{}`/`\frac{}{}`
  and collapsing MinerU's character-spaced output recovers them.
- Raw `<table>` HTML in the chunk text is noise the model has to read past. Eight
  of fourteen chunks carried it before rendering.

**The residual ~2.4 points are chunking, not content.** MinerU marks non-section
labels — `Keterangan:`, `Keterangan grafik:` — as `text_level` titles. Opening a
new section at each one splits a legend away from the figure it explains, and
`Other Activity` / `Uncontrollable` are then missed **despite being present as
prose in a chunk the filter processed**. Only **numbered** headings should open a
section, which is consistent with §9's finding that MinerU flattens every heading
to `level: 2` — numbering is the only reliable section signal it gives us.

**Caveat on attribution:** the MinerU artifacts here were built by a scratch
converter, because section-aware chunking belongs to the parsing half. These are
findings about what that chunker must handle, not a measurement of MinerU
itself.

### Re-measured through the production module — B3 (2026-08-26)

The caveat above is now discharged: `src/knowledge_parsing` produced the
artifact, not a scratch converter. `eval/parsing/run_eval.py`, result in
`eval/parsing/results/`.

| Input | E1 recall | Terms |
|---|---|---|
| PyMuPDF — frozen baseline | 0.8537 | 35 / 41 |
| **MinerU `hybrid/high`, production module** | **0.8293** | **34 / 41** |
| scratch converter (above) | 0.8293 | 34 / 41 |

**Parity with the scratch converter, from code that ships.** The miss lists differ
by one term in each direction — the module finds `Other Activity`, which the
converter missed, and misses `Breakdown`, which it found. The other six are
identical.

Getting there required fixing a real defect. The first run scored **0.7073**, and
the cause was not MinerU: `normalize.py` **silently dropped every `type: "list"`
item.** MinerU keeps list content in `list_items` and gives those items no `text`
field, so they fell through the generic text path, came back empty, and were
skipped with no warning — the same shape of bug the `chart`/`image` branch was
written to prevent. Two dropped blocks took five gold terms with them
(`Joint survey`, `Truck count`, `EWH`, `Controllable`,
`Hierarki Level Analysis`). Fixing it moved recall **+0.1220**.

Note what this says about §10's own conclusion: the scratch converter *did* handle
list blocks, so `Uncontrollable` really was "present as prose but missed" in that
run, exactly as recorded. §10's attribution was right. The production module had
regressed against it, and B3 is what surfaced that.

**The residual 0.0244 is not parsing's to close.** `Breakdown` appears three times
in the text and sits in the *same chunks* as `Total Hours`, which the filter does
find — same chunk, same sentence, `( Total Hours - Breakdown )`, one tagged and
one not. Chunk arrangement cannot explain it, so it is filter behaviour. Chunk
counts confirm there is no over-splitting to blame: 31 chunks = 15 prose (11
headed, 9 distinct sections) + 14 tables + 2 images, against the converter's 14
prose chunks. On prose granularity the two agree.

**Trap for anyone re-running this.** A harness that re-normalises a cached
`content_list` must pass the page vocabulary, as `run.py` does. Without it,
word-boundary recovery has no arbiter, `INPR Hours` fuses into `INPRHours`, and
the scorer's word-boundary match fails — costing a full term. That mis-measurement
(0.8049) was produced and discarded during B3; the module was never at fault.

---

## 11. Full-document run, both halves (2026-08-24)

First run with the parsing module building the artifact and extraction running every branch.
Result: `eval/knowledge/results/v2_full_document_2026-08-24_093051.json`.

| Metric | v2 | Prototype | Verdict |
|---|---|---|---|
| E1 term-filter recall | 0.8049 | 0.8537 | PASS (kill 0.70) |
| **E3 schema fill** | **0.90** | 0.75 | **PASS (kill 0.80) — the prototype FAILED this** |
| Rule-branch recall | 0.0667 | 0.2667 | **REGRESSION** |

77 calls, 139,283 prompt tokens with **110,848 cached (80%)**, 2 fields rejected by the span check,
48 of 58 entries abstained.

**Read E3 with the same care the prototype's number needed.** The scoreable base is 10 entries, so
one adjudication moves it ~10 points, and nano is not deterministic at `temperature=0`. The single
failure — `produksi` — is the *same* one the prototype recorded, and looks like a gold-adjudication
case rather than a model error: the document carries a broader definition than the gold set names.

**E1 is below the baseline because the chunking differs, not because the filter regressed.** The
parsing module splits to a token budget (31 chunks) where the earlier comparison used 14, so the
span model sees different windows. Same code, same gold set.

**The rule branch went backwards and is not yet diagnosed** (checklist X19). Candidate generation is
not the cause: 8 cue passages against the prototype's 9.

### Bug this run surfaced — blank lines closed the legend block

The legend filter returned **zero** abbreviation pairs on the parsing half's output. Cause: that
module separates blocks with a blank line — ordinary prose formatting — and `find_legend_lines`
treated the first blank as the end of the block.

The consequence is exactly what the filter exists to prevent: `PA` and `Physical Availability`
clustered as **two unrelated terms** and reached the expert's queue as two rows. Fixed by skipping
blank lines and closing the block on two consecutive blanks.

| | Before | After |
|---|---|---|
| Abbreviation pairs | 0 | 5 |
| Clusters | 61 | 58 |
| LLM calls | 81 | 77 |
| Cache hit rate | 57% | 80% |
| `PA` rows in the queue | **2** | 1 |

**The lesson generalises:** the two halves agree on the *contract*, and this still slipped through —
a formatting choice that is completely reasonable on one side silently broke a regex on the other.
Whitespace is part of the interface whether or not the schema says so.

---

## 12. Rule-branch regression — diagnosed (2026-08-24)

The full run scored the rule branch at **1/15**, against the prototype's 4/15. Diagnosed to three
separate causes, only one of which is an actual quality loss.

### Cause 1 — two of the three losses are a scoring artifact, not a regression

The gold set's `statement_contains` strings were written against the PyMuPDF-era notation, which ran
subscripts together: `QtyA`, `QtyP`. The rendered LaTeX writes them with an underscore: `Qty_A`,
`Qty_P`. The scorer normalises punctuation but **`_` is a word character**, so `qty_a` never contains
`qtya` and the match fails on notation alone.
| Scoring | Rules matched |
|---|---|
| As committed | **1/15** |
| Collapsing `_` to nothing | **3/15** |

`GAINLOSS_QTY_ZERO_PLAN` and `QTY_ACTUAL_ZERO_PLAN_ALT` are both correct extractions scored as
misses. **The true comparison is 4/15 → 3/15, a loss of one rule.** Fixing this means changing either
the scorer or the gold set — both are the frozen measuring instrument, so it is Rifqi's call, not a
silent edit.

### Cause 2 — the real loss: rendered formulas now compete with prose

**4 of 8 v2 rule statements are bare equations. In the prototype, 0 of 7 were.**

| Run | Formula statements | Prose statements |
|---|---|---|
| v2 | 4 | 4 |
| Prototype | 0 | 7 |

Before rendering, formulas reached the model as mangled character-spaced LaTeX and did not read as
statements. Rendered to clean prose they read exactly like prescriptive rules, and the model takes
them. `rule.txt` never excludes them — it defines a rule as *"a statement that constrains or directs
behaviour"*, which a formula arguably is.

The lost rule shows the mechanism exactly: chunk `::0015` contains **both** the units sentence
(*"Gain/Loss menggunakan satuan BCM atau ton…"* — gold `GAINLOSS_UNITS`) and a Gain/Loss equation.
The chunk **was** a cue candidate, so the model saw it, and returned the equation.

This is pure loss, because **the formula branch already captures those equations** — `Qty = (I
NPRHour s) / (T otalHour s)` appears in *both* branches of the same run. The rule branch is spending
calls duplicating the formula branch and dropping the operational prose it exists for.

Fix direction: `rule.txt` should exclude bare equations explicitly ("an equation is the formula
branch's job; a rule is prose that constrains behaviour"). That is a prompt change and therefore owes
an eval run per CLAUDE.md §7B — **not** to be done on vibes.

#### X20 — the fix, applied 2026-08-26 (`296cdd7`). NOT YET EVALUATED.

Three edits to `rule.txt`:

1. **A rule is now *prose*,** not "a statement", with a paragraph stating that an equation belongs
   to the formula branch and that "calculation policy" means prose saying **how, when or on what
   basis** a calculation applies — never the calculation itself.
2. **New correctness requirement 3:** evidence whose only prescriptive content is a formula returns
   `statement: null`.
3. **The load-bearing clause:** evidence carrying **both** a formula and constraining prose yields
   the **prose**. This is the one that targets the measured loss — chunk `::0015` held both the
   units sentence (gold `GAINLOSS_UNITS`) and a Gain/Loss equation, and the model returned the
   equation.

**Worked example 3 is deliberately synthetic** (MTTR, absent from all 15 gold rules). Using the real
`GAINLOSS_UNITS` case would have leaked a gold answer into the prompt and made the eval score it
trivially. Worth knowing: **worked example 1 already is gold rule `PTY_PRODUCTION_SOURCE` verbatim**,
so that contamination pre-exists — the new example was kept clean rather than adding to it.

**Committed without the eval §7B requires**, on Rifqi's explicit call, because the dev box has no
source document, no parsed artifact and no MinerU install (`data/knowledge_docs/`,
`data/knowledge_cache/`, `data/knowledge_runs/` are gitignored and absent; there is also no
`eval/knowledge/run_eval.py` — the 2026-08-24 run was assembled ad-hoc). Restoring one parsed
artifact unblocks the measurement. **The measurement is still owed.**

**Compare against 3/15, not 1/15.** X22 landed on 2026-08-24 (`abae344`) *after* this regression was
diagnosed: the scorer now deletes underscores, so the notation artifact in Cause 1 is gone. The
`1/15` in `v2_full_document_2026-08-24_093051.json` was measured with the old instrument and is no
longer the number to beat.

**Side effect to verify on the first paid run.** `rule.txt` grew from ~731 to ~1140 *estimated*
tokens and crossed the 1024 prompt-cache floor, so `cacheable('rule')` now returns `True` for the
first time — the "free money" §9 predicted. But `est_tokens` is `len/3.6`; only the API's
`cached_tokens` proves a hit.

**Found while doing this, not actioned — the rule branch cannot score better than 14/15 today.**
Worked example 2 teaches that *"Waterfall Analysis dapat di-review melalui dua metode"* is **not** a
rule and must return `null`; the gold set records that exact sentence as `WATERFALL_TWO_METHODS`.
The prompt and the frozen gold contradict each other, so one gold rule is unwinnable by
construction. Both are measuring instruments — Rifqi's call, tracked as X23.

### Cause 3 — three distinct `render_latex` faults (FIXED 2026-08-24)
> **Two corrections, both from re-measuring against the document's own 11 equations.**
>
> An earlier version of this section said `render_latex` "eats the first character". That was
> retracted — the repro was mangled by shell escaping, not by the code.
>
> The retraction then over-corrected. It claimed `Production = MOHH × Qty × PA × UA × Pty`
> "renders perfectly". **It does not.** The document's raw `content_list.json` contains no
> `\times` and no `\mathrm` — just a literal `x`, spaced like every other character:
>
> ```
> 'P r o d u c t i o n = M O H H x Q t y x P A x U A x P t y'
>   ->  'Production = MOHHxQtyxPAxUAxPty'
> ```
>
> MOHH, Qty, PA, UA and Pty — the five terms the glossary is built around — fuse into one
> token. See *Still open* below.
**(A) A spaced run touching a bracket is only half-collapsed.** `_SPACED_RUN` was bounded by
`(?<!\S)` / `(?!\S)`, so a run could not match if it touched `(` or `)`. The `\frac{a}{b}` →
`(a) / (b)` rewrite inserts exactly those brackets, stranding a character at **each** end:
```
'I N P R H o u r s'      ->  'INPRHours'      # correct, no brackets
'(I N P R H o u r s)'    ->  '(I NPRHour s)'  # first and last stranded
```
**(B) The subscript fold ran before the collapse and split the run.** `Q t y _ {a c t i v i t y}`
became `Qt y_a ctivity` — the `_` glued mid-run, so the collapse saw broken fragments.

**(C) The serious one — fraction structure was silently dropped.**

```
\mathrm { P A } = { \frac { \mathrm { T o t a l } ~ H o u r s - B r e a k d o w n }
                          { \mathrm { T o t a l } ~ H o u r s } } \mathrm { x } 1 0 0 \%
  ->  'PA = Total Hours - BreakdownTotal Hoursx 100 %'
```

The `\frac` rewrite needed `\{([^{}]*)\}`, which cannot match a numerator containing nested
braces. The unmatched `\frac` then fell through to the generic "drop any leftover command" pass,
so **the division was deleted rather than rendered.** This is not a spacing blemish — the formula
reads as a product and states something the document does not.

> **The nesting comes from `\mathrm{…}` and `\sum _{ij}`, not `\text{…}`.** The BUMA standard
> never uses `\text{}`. A fix written against `\text{}` would repair a synthetic example and no
> real equation.
**Measured tally, 11 equations:**
| | Before | After |
|---|---|---|
| Division lost | **4** — eq 6, 7, 9, 11 (`PA`, `PA_ij`, `UA_ij`, `PTY_ij`) | **0** |
| Stranded characters | 4 | 0 |
| Clean | 5 | 9 + 2 carrying only the document's own literal `x` |

Not two equations and not only PA/UA: **every composite (`_ij`) form was affected too.**
**The fix** (`src/knowledge_parsing/render.py`):
1. **Braced wrappers are unwrapped before `\frac`, consuming their braces.** The old `_WRAPPERS`
   deleted only the command name and left `{ }` behind — those leftover braces were what stopped
   `\frac` from matching. This is the ordering decision the diagnosis asked for.
2. **`\frac` uses a balanced-brace scanner**, not a regex. `[^{}]*` can never handle a nested
   argument, and it fails silently rather than raising.
3. **Run boundaries are word-character classes**, and subscript contents are joined at fold time
   (`_ { i j }` → `_ij`).

Six regression cases hold unchanged: `C_p`, `C 5`, `C_p × T`, `\frac{a}{b}`, `Purchasing costs`,
and a bare spaced run.

**Still open — a decision, not a defect.** After MinerU spaces every character, a multiplication
`x` is indistinguishable from the letter `x`: no `\times`, no `\mathrm`, no signal at all. Treating
it as an operator is right for engineering formulas and wrong for a variable named `x`. The
`_SYMBOLS` comment shows the author already weighed this trade for `\times`. Left unfixed
deliberately, because it changes interpretation rather than mechanics.

Impact of what remains: a term occurring only inside `Production = …` cannot be separated from its
neighbours, which costs the term filter five mentions on the standard's flagship formula.

**Owner: Sofhia** — `src/knowledge_parsing/render.py`.

---

## 13. Backend comparison — all four measured (2026-08-24)

Every backend, one Colab GPU, the same two documents (BUMA standard 9 pages +
McGraw-Hill handbook 20 pages = 29 pages). Until this run, `hybrid` had never
been measured at all, and every `effort` figure in circulation came from an
older Colab session on different hardware.

| Backend | s/page | Image descriptions | Multiplication sign |
|---|---|---|---|
| **hybrid / high** ⭐ | 4.85 | **3 · 1** | `\times` |
| hybrid / medium | **1.88** | 0 · 0 — force-disabled | `\times` |
| pipeline | 2.67 | 0 · 0 — never analyses | **letter `x`** |
| vlm | 6.96 | 3 · 1 | `\times` |

**Formula transcription overlap**, across the standard's 11 equations:

| Pair | Identical |
|---|---|
| `pipeline` vs everything else | **0%** |
| `vlm` vs `hybrid/high` | **100%** |
| `medium` vs `high` | 70% |

Two findings decide the backend. `vlm` and `hybrid/high` are the *same output* —
same LaTeX, same descriptions — so `vlm` buys nothing for its extra time. And
`pipeline` is the only backend that copies the multiplication sign as a literal
`x`, fusing `MOHH x Qty x PA` into one token and taking five glossary terms with
it; it is also slower than `hybrid/medium`, so it loses on both counts.

The choice therefore narrows to `hybrid/medium` vs `hybrid/high`, i.e. whether
image descriptions are wanted. Across the whole 6,800-page BUMA corpus that is
$2.67 against $6.89 — a $4 difference, too small to trade the feature for.
**Decided: `hybrid` + `effort=high`.**

> Rifqi's claim that `medium` and `high` capture only ~30% of the same formulas
> **did not reproduce** — measured at 70%.

### Image descriptions are gated by `sub_type`, and coverage is partial
MinerU only describes a picture it has first classified. The pattern held 6 of 6
across both documents:
| `sub_type` | `content` |
|---|---|
| `line` · `bar` · `flowchart` | filled |
| absent | **empty** |

Coverage was **4 of 6 (67%)**, and the miss is not explained by image quality:
`Gambar 2.1` is a plainly legible waterfall chart with seven labelled bars, and
got no `sub_type` at all. Meanwhile `Gambar 2.2` was labelled `flowchart` while
the description it produced calls it *"This waterfall chart…"* — so the
classifier is shaky in both directions.

**Consequence for planning:** the parsing stage cannot fully replace a dedicated
image-description pipeline. It supplies some descriptions free, but a clear chart
can be skipped with no warning. A fallback path is easy to target, though —
chunks with `images` populated and `generated_description` null are exactly the
gap.

### Determinism — settled (2026-08-26)

`hybrid/high` **is deterministic.** Two runs over the same two documents, with the
parse cache moved aside in between so both genuinely re-parsed (`from_cache:
false` on all four records), produced **0 differences across 50 chunks** — every
one of `chunk_id`, `kind`, `page_idx`, `section_no`, `heading`, `text`, `latex`,
`table_html` and `generated_description` identical.

That result is not vacuous: **all four image descriptions were present in both
runs and character-for-character identical** (192, 256, 77 and 146 chars). Those
are the only model-written content in the artifact and were the likeliest thing
to wobble; they did not.

This closes the strongest objection to calibrating on the VLM family — that a
frozen baseline would drift between runs and no eval number could be trusted.
It does not. **Existing baselines stay valid** and need no re-measurement on this
account, and the backend decision is final rather than provisional.

Parse *time* does vary — 167.4 s against 65.2 s for the same 20-page document,
which is model loading on the first run. The output does not.

---

## 14. Word-boundary recovery — reading the answer from the source (2026-08-24)

MinerU writes formulas one character at a time (`T o t a l   H o u r s`). Once
those are joined the boundaries are gone: `Total Hours` and `TotalHours` become
indistinguishable, and a multiplication written as the letter `x` fuses in too,
because it looks like any other spaced character.

Two obvious fixes were tried and rejected. Rewriting a lone `x` to a
multiplication sign **guesses**, and guesses wrong — silently — on any document
with a variable named `x`; it also leaves `TotalHours` untouched, which is the
larger half of the damage. Emitting a warning instead fires on 7 of 11 equations:
a warning lighting up on 64% of items is not actionable and recovers nothing.

The source PDF still holds the boundaries — its text layer reads
`MOHH x Qty x PA x UA x Pty`, properly spaced. So the answer is read from the
document. A token not already a word on that page is split only if EVERY piece is
a word on that page, and only if exactly one shortest such split exists.
Ambiguity is left alone; so is anything unsegmentable. It fails closed.

NFKC normalisation is required — formulas use a mathematical font, so the page
literally contains U+1D440 and friends, and without it nothing matches.

**Measured across three documents, two of them an English textbook with nothing
in common with the BUMA standard:**

| Document | Text-layer quality | Equations repaired |
|---|---|---|
| BUMA standard, 9 pages | 99.6% | 6 of 11 |
| Handbook, 20 pages | 96.0% | 1 of 3 |
| Handbook, 5 pages | 98.1% | 8 of 20 |

```
MOHHxQtyxPAxUAxPty  ->  MOHH x Qty x PA x UA x Pty
TotalHours          ->  Total Hours
newwrenchtime       ->  new wrench time
areaofroom, sqft    ->  area of room , sq ft
```

Glossary terms readable as whole tokens went **320 -> 397** on the standard,
across 9 of 31 chunks. The second test document was unchanged — as it should be,
having no such damage.

### The guard that makes it safe on documents nobody has seen

Trusting the text layer is safe when it is intact and dangerous when it is HALF
there: a legitimate word starts looking unknown and gets split. So each page has
to prove itself — how many of MinerU's own prose words on that page are actually
found in its text layer. A page scoring below 90%, or carrying too little prose
to judge, loses its vocabulary and skips recovery: **damaged but honest, rather
than repaired into something the document never said.**

That replaces a constant tuned against one document with a measure that assesses
each page on its own evidence. A scanned PDF with no text layer is already safe —
no vocabulary, no recovery. The untested territory is a text layer well below
96%, which none of the three documents provides.

Threshold measured rather than picked: a minimum token length of 6, 4 and 3 give
identical results on equations, and 3 additionally recovers `MOHHx` / `xPA_A`
inside table cells. **2 starts doing damage**, splitting `PA_ij` into `PA_i j`
because `i` and `j` are both valid words on that page.