KNOWLEDGE_PIPELINE_CALIBRATION.md

# Knowledge Pipeline — Calibration Reference

**Harvested from the `kex` prototype, 2026-08-19.** Every number below cost a run, a bug, or a
measurement to find, and **none of it is in the spec** — it lived only as constants and comments in
a codebase that is not being ported.

This file exists because clean rewrites drop accidental knowledge and then rediscover it as
production defects. When building extraction v2, take the value **and** the reason. If you change
one, change it deliberately and record why.

Companions: [KNOWLEDGE_PIPELINE_TODO.md](KNOWLEDGE_PIPELINE_TODO.md) (plan) ·
[eval/knowledge/](eval/knowledge/) (gold set, scorer, frozen baseline) ·
[knowledge_pipeline_context.md](knowledge_pipeline_context.md) (intent).

---

## 1. Term filter — GLiNER

| Constant | Value | Why this value |
|---|---|---|
| Model | `urchade/gliner_multi-v2.1` | CPU, ~400 MB. Cheap enough to run unconditionally on every chunk, which is what lets us reject a standalone relevance gate |
| Score threshold | **0.25** | Not 0.35. The sweep measured recall **0.854 @ 0.25** vs **0.658 @ 0.35** on the `broad` label set. Precision falls (0.41 vs 0.50) and that is accepted: the filter is meant to be over-inclusive, clustering and evidence ranking absorb the noise, and **a term the filter never proposes can never be recovered downstream** |
| Label variant | **`broad`** | Two earlier variants both missed the same class — mining activities and materials (coal mining, land clearing, waste removal, joint survey). `broad` adds `mining activity`, `material or commodity`, `data source or system`, `work category`, `analysis method` to cover it |
| Window size | **130 words**, overlap **30** | Not 220. GLiNER truncates past ~384 of its own tokens and **warns rather than failing** — silently dropping the tail of every long chunk. Indonesian technical prose subword-tokenises at roughly **2.5×**, so a 220-word window still tripped the cap. Chunks are fed as overlapping word windows with offsets remapped |
| Span token cap | **12** (GLiNER's own) | Compound terms ("UA plan grouping (composite) actual") exceed it. Candidates that hit the cap are **flagged, not silently truncated**, so a recall shortfall can be attributed to the cap |

**Label sets** (`labels.yaml`) — three variants were tried and A/B'd without touching code. Keep
that property: label phrasing is the main recall lever and GLiNER is very sensitive to it. The
Indonesian-phrasing variant (`istilah produksi tambang`, …) did **not** win; `broad` did.

---

## 2. Clustering

| Constant | Value | Why this value |
|---|---|---|
| Match order | exact (normalised) → abbreviation↔expansion → fuzzy | Cheapest and most certain first |
| Fuzzy threshold | **92** (`rapidfuzz.token_set_ratio`) | Conservative on purpose |
| **Fuzzy disabled below 5 characters** | `min(len(a), len(b)) < 5` → skip | **The one to not lose.** `PA` vs `UA` scores high on `token_set_ratio`. Below 5 chars only exact matching is allowed |
| Canonical surface | shortest variant ≥ 2 chars | That is how a reader looks a term up — `PA`, not `Physical Availability (PA) untuk …` |

**Over-merging is worse than under-merging, and the asymmetry is not close.** An under-merge costs
one extra LLM call and one extra review row. A wrong merge silently destroys a distinct term and
**the expert never sees it** — there is no downstream mechanism that recovers it.

**Legend extraction must run before clustering.** Without the abbreviation index, `PA` and
`Physical Availability` cluster as two unrelated terms.

**Stop surfaces** — dropped as whole surface forms only (never as substrings, so no term is lost):
`unit · type · class · equipment · equipment unit · parameter · activity · data · nilai · proses ·
hasil · total`.

**Normalisation** (clustering only, never for span checking): NFKC → casefold → `-`/`_` → space →
strip `.’'` → non-word to space → collapse whitespace → strip ` ()/`.

---

## 3. Evidence ranking

Six signals plus one penalty. The full ranked list is retained, not just the top K — escalation
needs the tail.

| Signal | Weight |
|---|---|
| Definitional cue within 100 chars of the mention | **+5.0** |
| Term appears in the chunk heading | **+4.0** |
| Mention sits inside a legend block | **+3.5** |
| Chunk contains a formula | **+2.0** |
| Mention is bold/italic | **+1.5** |
| First occurrence of the cluster | **+1.0** |
| Chunk is tabular | **−3.0** |

| Constant | Value | Why |
|---|---|---|
| `evidence_k` | **3** | Chunks fed per extraction round |
| Cue proximity | **100 chars** | Distance from mention to a definitional cue |

**Heading matching is word-boundary, never substring.** `PA` is a substring of `Parameter`, `pada`
and `composite` — substring matching handed the +4.0 heading bonus to almost every Indonesian
heading and pushed real definition sections *below* formula tables.

**Why this filter is legitimate when the relevance gate was rejected:** no term is dropped (only its
*evidence* is narrowed), every mention stays in provenance so a bad top-3 is visible, and the
escalation loop self-corrects. The rejected gate had none of those three properties. Preserve all
three if ranking is ever revisited.

---

## 4. Chunking

| Constant | Value | Why |
|---|---|---|
| Max chunk tokens | **1500** | With paragraph-boundary splitting; sentence boundaries as fallback for a single over-budget paragraph |
| Max heading length | **90 chars** | Longer lines are sentences or formula rows, not headings |
| Boilerplate frequency | **≥ 0.6 of pages** | A line repeating on 60%+ of pages is a running header/footer. Detected by frequency with **digits normalised to `#`** so page counters collapse — never by a hardcoded document-specific string |
| Tabular detection | short lines > 0.6 **and** numeric lines > 0.4 | Feeds the −3.0 ranking penalty |

**Three things the real document forced, all of which corrupt every downstream number if missed:**

1. **Running headers/footers** — every page repeats the title block and "Confidential".
2. **Breadcrumb headings** — pages re-print `2. PENJELASAN PARAMETER / 2.1. Production Parameter / …`
   at the top. A naive splitter re-opens the section and shatters it. Rule used: *a heading that is
   the current section or an ancestor of it, already seen, is a breadcrumb* — the section continues
   and `page_end` extends.
3. **Colon-continuation** — legend rows arrive as `MOHH` then `: Machine on Hand Hours` on the next
   line. Rejoined before the legend filter sees them.

Points 1 and 2 apply to MinerU output too: its `title` blocks repeat the same breadcrumbs on pages
2–8 of the standard.

---

## 5. Extraction (the paid stage)

| Constant | Value | Why |
|---|---|---|
| Temperature | **0.0** | And still not deterministic — see below |
| Seed | 7 | |
| **Cache minimum** | **1024 prompt tokens** | OpenAI-family prompt caching **does not engage at all** below this. The glossary prefix is padded past it *on purpose*. Measured hit rate **54%** — 125,184 of 155,313 prompt tokens |
| Structured output | `json_schema`, fallback `json_object` + validate-retry | Which mode was used is recorded in the run manifest |
| API version | `2024-12-01-preview` | `json_schema` needs ≥ `2024-08-01-preview` |

**Prompts live in files, not code, and the fixed prefix must stay byte-identical across calls** —
any drift and caching stops engaging, silently, at ~10× the input cost.

**`temperature=0` is not determinism.** Consecutive runs of the identical pipeline scored 0.75 and
0.625 on schema fill. Never report a single run as a measurement on a small base.

---

## 6. Validation

| Constant | Value | Why |
|---|---|---|
| Span normalisation | **whitespace only** (+ NFKC) | Not case, not punctuation, not diacritics. Every additional normalisation is a hole a fabrication fits through |
| Escalation rounds | **max 2** | Then keep the entry flagged `no_definition_found` and pass it to review anyway — a term we found but could not define is still useful; dropping it hides a known unknown |
| Conflict overlap threshold | **0.4** token overlap | Token overlap, not embeddings: cheaper, needs no model, and **explainable to the reviewer who has to act on it** |
| Duplicate overlap threshold | **0.8** | Above it, a differing definition is a duplicate; below, a conflict |

**Guarded fields** (span-checked; failure sets the field to `None` and logs it):

| Branch | Fields |
|---|---|
| glossary | `definition`, `full_name`, `formula_latex`, `interpretation` |
| rule | `statement`, `condition`, `consequence` |
| formula | `formula_latex` |
| summary | *(none — generation cannot be span-checked at all)* |

**If the provenance span itself is not verbatim, every guarded field on the entry is rejected** —
the entry's only evidence link is broken, so nothing on it can be trusted.

**Never repair a failed span.** A repaired span is an unfalsifiable claim, which is exactly what the
control exists to prevent.

**Conflict detection never picks a winner.** The expert decides.

---

## 7. Negative findings — do not re-derive these

| Finding | Detail |
|---|---|
| **nano is not sufficient for schema fill** | 0.75 precision against a 0.80 kill line. Decision 2026-08-19: **stay on nano anyway** for now; the `__54n` env quad is provisioned. Revisit if quality blocks the demo |
| **Escalation is unreachable on small documents** | 54 of 66 clusters had exactly **1** evidence chunk against K=3, so `rounds_available()` correctly returned 0 and the loop never fired. `escalated = 0` is not a bug. It stays unverified until a larger corpus runs |
| **Conflict detection had nothing to find** | 0 conflicts on a single internally-consistent standard. Untested against real disagreement |
| **85% abstention** | 56 of 66 entries returned no definition. Correct behaviour, but it means the review queue is mostly *"term found, no definition in document"*. Whether that is useful or noise is a review-experience question for Mas Beta |
| **Diff only ever ran the empty-baseline path** | The prototype diffed against the file it then overwrote, so every entry came back `new`. v2 needs a real active-version pointer |
| **Literal source wording was normalised away** | The standard's heading reads *"Physical **of** Availability (PA)"*; the entry carried `full_name: "Physical Availability"`. The gold set records the literal form correctly, so **gold was right and the pipeline was wrong**. Mechanism: the heading is a separate field from the chunk text, and the chunk text does not contain the phrase, so the literal wording never reached the model. Fix: feed the heading verbatim + a span-checked `source_wording` field |
| **A `glob('*.pdf')` picked the wrong document once** | Scored the textbook against the standard's gold set → recall 0.05. Document selection must be explicit and error on ambiguity |
| **torch wheels install corrupt** | `torchgen` missing, version reports `None`. Fix is delete `site-packages/torch*` and reinstall. Needs **torch ≥ 2.6**. Documented so nobody debugs it as a GLiNER problem |

---

## 8. What was measured, so v2 can be compared

Frozen in `eval/knowledge/results/baseline_prototype_2026-08-13_145132.json`.

Funnel on the 9-page standard: **9 pages → 13 chunks → 169 mentions** (195 before noise filtering)
**→ 66 clusters → 83 LLM calls → 66 entries → 66 review-queue items.**

E1 recall **0.854** · E2 compression **2.56×** · E3 schema-fill precision **0.75** (FAIL) ·
E4 fabrication rejection **1.00** with **0.00** false rejections. Cost **$0.0069**, 242 s wall
clock, 3 fields rejected by span check.

**When quoting E2 externally:** the context doc's **6.2×** is a 1,000-page projection; **2.56×** is
the 9-page measurement. Compression grows as terms repeat across a corpus. Both are true — use the
right one for the audience.

---

## 9. Deliberate deviations from the prototype (v2)

Changes made while rebuilding, with the reason. Each one alters behaviour, so
each is listed rather than folded silently into the tables above.

| # | Change | Why | Measured? |
|---|---|---|---|
| **V1** | **A heading that names the term counts as a mention at offset 0** for cue-proximity scoring | Indonesian standards are written *"2.1.3 Physical of Availability (PA)"* / *"Adalah ketersediaan fisik…"* — the heading names the term and the body opens with the definitional cue **without repeating it**. The prototype therefore awarded no cue bonus on the definition section itself. Measured on the BUMA standard: PA's own definition section (§2.1.3) scored 7.5 and ranked **third**, below a computation section (§2.2.4, 8.5) that merely mentions PA often. After the fix, all five terms rank their own definition section first — PA→2.1.3, UA→2.1.4, Qty→2.1.2, Pty→2.1.5, MOHH→2.1.1 | Structurally verified on the 9-page standard. **Not yet scored** — needs the paid stage to show up in E3 |
| **V2** | **`first_occurrence` follows document order**, and ranking ties break on document order | The prototype awarded the +1.0 to whichever chunk a `set` iteration yielded first, making the bonus arbitrary and the ranking unreproducible across runs. "First occurrence" should mean the term's first appearance in the document | Determinism locked by test; verified stable across `PYTHONHASHSEED` 0/1/7/42/99 |
| **V3** | **`source_wording` field added** to the glossary entry | The control for the literal-source-wording defect (§7). Not yet populated — needs the seam to carry the heading verbatim and the extraction prompt to fill it | No |

**Both V1 and V2 change which chunks reach the paid stage**, so they can move E3
when the extraction branch lands. Re-score against
`eval/knowledge/results/baseline_prototype_2026-08-13_145132.json` at that point
and record the delta rather than assuming the improvement carries.

**V1 and the R1 literal-wording defect share a root cause:** the heading is a
separate field from the chunk text, so anything that reasons over "the term and
its definition together" has to be told to look at both.

| **V4** | **`source_wording` is taken from the section heading, not from the model** | Asked to quote the source wording, the model returned "Physical Availability (PA)" — a genuine verbatim quote from the page-1 intro, but not the §2.1.3 heading "Physical **of** Availability (PA)". Both occur in the document; only the heading form reveals the disagreement. Taking it deterministically means it cannot be normalised away | Live-verified on the pilot: the entry now carries both forms and the queue flags the mismatch |
| **V5** | **`evidence_text` includes chunk headings** | The heading is part of the source document and is often the only place a term is formally named. Excluding it rejected a correct verbatim quotation of the document's own section title (2 rejections on a 3-entry pilot, both spurious) | Live-verified: 0 rejections after the change |
| **V6** | **`full_name` and `source_wording` are span-checked against the source themselves** | Both claim to be literal transcriptions. The provenance span passing does not prove the transcription is faithful, and a normalised "full name" is exactly the silent correction this pipeline exists to surface | Unit-tested |

**Cache-floor finding (measured on the real prompts):** only the **glossary**
prefix clears the 1024-token floor at **1,401 tokens**. `rule` (731), `formula`
(459) and `summary` (288) are **not cacheable** and cache nothing today. Glossary
is also the branch with by far the most calls (66 of 83 on the reference
document), so the padding is on the branch that matters — but padding the other
three is free money if their call counts ever grow.

---

## 10. Parser comparison — MinerU vs PyMuPDF (2026-08-19)

First run of the extraction half on **MinerU-parsed input**, same document, same
gold set, same pipeline code. Result:
`eval/knowledge/results/v2_parser_comparison_2026-08-19_152842.json`.

| Input | E1 recall | Terms found |
|---|---|---|
| PyMuPDF (what the frozen baseline used) | **0.8537** | 35 / 41 |
| MinerU, raw markup | **0.7561** | 31 / 41 |
| MinerU, formulas + tables rendered to prose | **0.8293** | 34 / 41 |

**The span model reads prose, not markup.** MinerU's structure is an asset for
the formula branch and a liability for the filter unless it is rendered first:

- Terms wrapped in LaTeX are invisible. `\text {Total Hours} - \text {Breakdown}`
  yields no mentions for either term, though both are gold terms and both are
  found in the plain-text parse. Unwrapping `\text{}`/`\mathrm{}`/`\frac{}{}`
  and collapsing MinerU's character-spaced output recovers them.
- Raw `<table>` HTML in the chunk text is noise the model has to read past. Eight
  of fourteen chunks carried it before rendering.

**The residual ~2.4 points are chunking, not content.** MinerU marks non-section
labels — `Keterangan:`, `Keterangan grafik:` — as `text_level` titles. Opening a
new section at each one splits a legend away from the figure it explains, and
`Other Activity` / `Uncontrollable` are then missed **despite being present as
prose in a chunk the filter processed**. Only **numbered** headings should open a
section, which is consistent with §9's finding that MinerU flattens every heading
to `level: 2` — numbering is the only reliable section signal it gives us.

**Caveat on attribution:** the MinerU artifacts here were built by a scratch
converter, because section-aware chunking belongs to the parsing half. These are
findings about what that chunker must handle, not a measurement of MinerU
itself.