Knowledge Pipeline — Calibration Reference
Harvested from the kex prototype, 2026-08-19. Every number below cost a run, a bug, or a
measurement to find, and none of it is in the spec — it lived only as constants and comments in
a codebase that is not being ported.
This file exists because clean rewrites drop accidental knowledge and then rediscover it as production defects. When building extraction v2, take the value and the reason. If you change one, change it deliberately and record why.
Companions: KNOWLEDGE_PIPELINE_TODO.md (plan) · eval/knowledge/ (gold set, scorer, frozen baseline) · knowledge_pipeline_context.md (intent).
1. Term filter — GLiNER
| Constant | Value | Why this value |
|---|---|---|
| Model | urchade/gliner_multi-v2.1 |
CPU, ~400 MB. Cheap enough to run unconditionally on every chunk, which is what lets us reject a standalone relevance gate |
| Score threshold | 0.25 | Not 0.35. The sweep measured recall 0.854 @ 0.25 vs 0.658 @ 0.35 on the broad label set. Precision falls (0.41 vs 0.50) and that is accepted: the filter is meant to be over-inclusive, clustering and evidence ranking absorb the noise, and a term the filter never proposes can never be recovered downstream |
| Label variant | broad |
Two earlier variants both missed the same class — mining activities and materials (coal mining, land clearing, waste removal, joint survey). broad adds mining activity, material or commodity, data source or system, work category, analysis method to cover it |
| Window size | 130 words, overlap 30 | Not 220. GLiNER truncates past ~384 of its own tokens and warns rather than failing — silently dropping the tail of every long chunk. Indonesian technical prose subword-tokenises at roughly 2.5×, so a 220-word window still tripped the cap. Chunks are fed as overlapping word windows with offsets remapped |
| Span token cap | 12 (GLiNER's own) | Compound terms ("UA plan grouping (composite) actual") exceed it. Candidates that hit the cap are flagged, not silently truncated, so a recall shortfall can be attributed to the cap |
Label sets (labels.yaml) — three variants were tried and A/B'd without touching code. Keep
that property: label phrasing is the main recall lever and GLiNER is very sensitive to it. The
Indonesian-phrasing variant (istilah produksi tambang, …) did not win; broad did.
2. Clustering
| Constant | Value | Why this value |
|---|---|---|
| Match order | exact (normalised) → abbreviation↔expansion → fuzzy | Cheapest and most certain first |
| Fuzzy threshold | 92 (rapidfuzz.token_set_ratio) |
Conservative on purpose |
| Fuzzy disabled below 5 characters | min(len(a), len(b)) < 5 → skip |
The one to not lose. PA vs UA scores high on token_set_ratio. Below 5 chars only exact matching is allowed |
| Canonical surface | shortest variant ≥ 2 chars | That is how a reader looks a term up — PA, not Physical Availability (PA) untuk … |
Over-merging is worse than under-merging, and the asymmetry is not close. An under-merge costs one extra LLM call and one extra review row. A wrong merge silently destroys a distinct term and the expert never sees it — there is no downstream mechanism that recovers it.
Legend extraction must run before clustering. Without the abbreviation index, PA and
Physical Availability cluster as two unrelated terms.
Stop surfaces — dropped as whole surface forms only (never as substrings, so no term is lost):
unit · type · class · equipment · equipment unit · parameter · activity · data · nilai · proses · hasil · total.
Normalisation (clustering only, never for span checking): NFKC → casefold → -/_ → space →
strip .’' → non-word to space → collapse whitespace → strip ()/.
3. Evidence ranking
Six signals plus one penalty. The full ranked list is retained, not just the top K — escalation needs the tail.
| Signal | Weight |
|---|---|
| Definitional cue within 100 chars of the mention | +5.0 |
| Term appears in the chunk heading | +4.0 |
| Mention sits inside a legend block | +3.5 |
| Chunk contains a formula | +2.0 |
| Mention is bold/italic | +1.5 |
| First occurrence of the cluster | +1.0 |
| Chunk is tabular | −3.0 |
| Constant | Value | Why |
|---|---|---|
evidence_k |
3 | Chunks fed per extraction round |
| Cue proximity | 100 chars | Distance from mention to a definitional cue |
Heading matching is word-boundary, never substring. PA is a substring of Parameter, pada
and composite — substring matching handed the +4.0 heading bonus to almost every Indonesian
heading and pushed real definition sections below formula tables.
Why this filter is legitimate when the relevance gate was rejected: no term is dropped (only its evidence is narrowed), every mention stays in provenance so a bad top-3 is visible, and the escalation loop self-corrects. The rejected gate had none of those three properties. Preserve all three if ranking is ever revisited.
4. Chunking
| Constant | Value | Why |
|---|---|---|
| Max chunk tokens | 1500 | With paragraph-boundary splitting; sentence boundaries as fallback for a single over-budget paragraph |
| Max heading length | 90 chars | Longer lines are sentences or formula rows, not headings |
| Boilerplate frequency | ≥ 0.6 of pages | A line repeating on 60%+ of pages is a running header/footer. Detected by frequency with digits normalised to # so page counters collapse — never by a hardcoded document-specific string |
| Tabular detection | short lines > 0.6 and numeric lines > 0.4 | Feeds the −3.0 ranking penalty |
Three things the real document forced, all of which corrupt every downstream number if missed:
- Running headers/footers — every page repeats the title block and "Confidential".
- Breadcrumb headings — pages re-print
2. PENJELASAN PARAMETER / 2.1. Production Parameter / …at the top. A naive splitter re-opens the section and shatters it. Rule used: a heading that is the current section or an ancestor of it, already seen, is a breadcrumb — the section continues andpage_endextends. - Colon-continuation — legend rows arrive as
MOHHthen: Machine on Hand Hourson the next line. Rejoined before the legend filter sees them.
Points 1 and 2 apply to MinerU output too: its title blocks repeat the same breadcrumbs on pages
2–8 of the standard.
5. Extraction (the paid stage)
| Constant | Value | Why |
|---|---|---|
| Temperature | 0.0 | And still not deterministic — see below |
| Seed | 7 | |
| Cache minimum | 1024 prompt tokens | OpenAI-family prompt caching does not engage at all below this. The glossary prefix is padded past it on purpose. Measured hit rate 54% — 125,184 of 155,313 prompt tokens |
| Structured output | json_schema, fallback json_object + validate-retry |
Which mode was used is recorded in the run manifest |
| API version | 2024-12-01-preview |
json_schema needs ≥ 2024-08-01-preview |
Prompts live in files, not code, and the fixed prefix must stay byte-identical across calls — any drift and caching stops engaging, silently, at ~10× the input cost.
temperature=0 is not determinism. Consecutive runs of the identical pipeline scored 0.75 and
0.625 on schema fill. Never report a single run as a measurement on a small base.
6. Validation
| Constant | Value | Why |
|---|---|---|
| Span normalisation | whitespace only (+ NFKC) | Not case, not punctuation, not diacritics. Every additional normalisation is a hole a fabrication fits through |
| Escalation rounds | max 2 | Then keep the entry flagged no_definition_found and pass it to review anyway — a term we found but could not define is still useful; dropping it hides a known unknown |
| Conflict overlap threshold | 0.4 token overlap | Token overlap, not embeddings: cheaper, needs no model, and explainable to the reviewer who has to act on it |
| Duplicate overlap threshold | 0.8 | Above it, a differing definition is a duplicate; below, a conflict |
Guarded fields (span-checked; failure sets the field to None and logs it):
| Branch | Fields |
|---|---|
| glossary | definition, full_name, formula_latex, interpretation |
| rule | statement, condition, consequence |
| formula | formula_latex |
| summary | (none — generation cannot be span-checked at all) |
If the provenance span itself is not verbatim, every guarded field on the entry is rejected — the entry's only evidence link is broken, so nothing on it can be trusted.
Never repair a failed span. A repaired span is an unfalsifiable claim, which is exactly what the control exists to prevent.
Conflict detection never picks a winner. The expert decides.
7. Negative findings — do not re-derive these
| Finding | Detail |
|---|---|
| nano is not sufficient for schema fill | 0.75 precision against a 0.80 kill line. Decision 2026-08-19: stay on nano anyway for now; the __54n env quad is provisioned. Revisit if quality blocks the demo |
| Escalation is unreachable on small documents | 54 of 66 clusters had exactly 1 evidence chunk against K=3, so rounds_available() correctly returned 0 and the loop never fired. escalated = 0 is not a bug. It stays unverified until a larger corpus runs |
| Conflict detection had nothing to find | 0 conflicts on a single internally-consistent standard. Untested against real disagreement |
| 85% abstention | 56 of 66 entries returned no definition. Correct behaviour, but it means the review queue is mostly "term found, no definition in document". Whether that is useful or noise is a review-experience question for Mas Beta |
| Diff only ever ran the empty-baseline path | The prototype diffed against the file it then overwrote, so every entry came back new. v2 needs a real active-version pointer |
| Literal source wording was normalised away | The standard's heading reads "Physical of Availability (PA)"; the entry carried full_name: "Physical Availability". The gold set records the literal form correctly, so gold was right and the pipeline was wrong. Mechanism: the heading is a separate field from the chunk text, and the chunk text does not contain the phrase, so the literal wording never reached the model. Fix: feed the heading verbatim + a span-checked source_wording field |
A glob('*.pdf') picked the wrong document once |
Scored the textbook against the standard's gold set → recall 0.05. Document selection must be explicit and error on ambiguity |
| torch wheels install corrupt | torchgen missing, version reports None. Fix is delete site-packages/torch* and reinstall. Needs torch ≥ 2.6. Documented so nobody debugs it as a GLiNER problem |
8. What was measured, so v2 can be compared
Frozen in eval/knowledge/results/baseline_prototype_2026-08-13_145132.json.
Funnel on the 9-page standard: 9 pages → 13 chunks → 169 mentions (195 before noise filtering) → 66 clusters → 83 LLM calls → 66 entries → 66 review-queue items.
E1 recall 0.854 · E2 compression 2.56× · E3 schema-fill precision 0.75 (FAIL) · E4 fabrication rejection 1.00 with 0.00 false rejections. Cost $0.0069, 242 s wall clock, 3 fields rejected by span check.
When quoting E2 externally: the context doc's 6.2× is a 1,000-page projection; 2.56× is the 9-page measurement. Compression grows as terms repeat across a corpus. Both are true — use the right one for the audience.
9. Deliberate deviations from the prototype (v2)
Changes made while rebuilding, with the reason. Each one alters behaviour, so each is listed rather than folded silently into the tables above.
| # | Change | Why | Measured? |
|---|---|---|---|
| V1 | A heading that names the term counts as a mention at offset 0 for cue-proximity scoring | Indonesian standards are written "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik…" — the heading names the term and the body opens with the definitional cue without repeating it. The prototype therefore awarded no cue bonus on the definition section itself. Measured on the BUMA standard: PA's own definition section (§2.1.3) scored 7.5 and ranked third, below a computation section (§2.2.4, 8.5) that merely mentions PA often. After the fix, all five terms rank their own definition section first — PA→2.1.3, UA→2.1.4, Qty→2.1.2, Pty→2.1.5, MOHH→2.1.1 | Structurally verified on the 9-page standard. Not yet scored — needs the paid stage to show up in E3 |
| V2 | first_occurrence follows document order, and ranking ties break on document order |
The prototype awarded the +1.0 to whichever chunk a set iteration yielded first, making the bonus arbitrary and the ranking unreproducible across runs. "First occurrence" should mean the term's first appearance in the document |
Determinism locked by test; verified stable across PYTHONHASHSEED 0/1/7/42/99 |
| V3 | source_wording field added to the glossary entry |
The control for the literal-source-wording defect (§7). Not yet populated — needs the seam to carry the heading verbatim and the extraction prompt to fill it | No |
Both V1 and V2 change which chunks reach the paid stage, so they can move E3
when the extraction branch lands. Re-score against
eval/knowledge/results/baseline_prototype_2026-08-13_145132.json at that point
and record the delta rather than assuming the improvement carries.
V1 and the R1 literal-wording defect share a root cause: the heading is a separate field from the chunk text, so anything that reasons over "the term and its definition together" has to be told to look at both.
| V4 | source_wording is taken from the section heading, not from the model | Asked to quote the source wording, the model returned "Physical Availability (PA)" — a genuine verbatim quote from the page-1 intro, but not the §2.1.3 heading "Physical of Availability (PA)". Both occur in the document; only the heading form reveals the disagreement. Taking it deterministically means it cannot be normalised away | Live-verified on the pilot: the entry now carries both forms and the queue flags the mismatch |
| V5 | evidence_text includes chunk headings | The heading is part of the source document and is often the only place a term is formally named. Excluding it rejected a correct verbatim quotation of the document's own section title (2 rejections on a 3-entry pilot, both spurious) | Live-verified: 0 rejections after the change |
| V6 | full_name and source_wording are span-checked against the source themselves | Both claim to be literal transcriptions. The provenance span passing does not prove the transcription is faithful, and a normalised "full name" is exactly the silent correction this pipeline exists to surface | Unit-tested |
Cache-floor finding (measured on the real prompts): only the glossary
prefix clears the 1024-token floor at 1,401 tokens. rule (731), formula
(459) and summary (288) are not cacheable and cache nothing today. Glossary
is also the branch with by far the most calls (66 of 83 on the reference
document), so the padding is on the branch that matters — but padding the other
three is free money if their call counts ever grow.
10. Parser comparison — MinerU vs PyMuPDF (2026-08-19)
First run of the extraction half on MinerU-parsed input, same document, same
gold set, same pipeline code. Result:
eval/knowledge/results/v2_parser_comparison_2026-08-19_152842.json.
| Input | E1 recall | Terms found |
|---|---|---|
| PyMuPDF (what the frozen baseline used) | 0.8537 | 35 / 41 |
| MinerU, raw markup | 0.7561 | 31 / 41 |
| MinerU, formulas + tables rendered to prose | 0.8293 | 34 / 41 |
The span model reads prose, not markup. MinerU's structure is an asset for the formula branch and a liability for the filter unless it is rendered first:
- Terms wrapped in LaTeX are invisible.
\text {Total Hours} - \text {Breakdown}yields no mentions for either term, though both are gold terms and both are found in the plain-text parse. Unwrapping\text{}/\mathrm{}/\frac{}{}and collapsing MinerU's character-spaced output recovers them. - Raw
<table>HTML in the chunk text is noise the model has to read past. Eight of fourteen chunks carried it before rendering.
The residual ~2.4 points are chunking, not content. MinerU marks non-section
labels — Keterangan:, Keterangan grafik: — as text_level titles. Opening a
new section at each one splits a legend away from the figure it explains, and
Other Activity / Uncontrollable are then missed despite being present as
prose in a chunk the filter processed. Only numbered headings should open a
section, which is consistent with §9's finding that MinerU flattens every heading
to level: 2 — numbering is the only reliable section signal it gives us.
Caveat on attribution: the MinerU artifacts here were built by a scratch converter, because section-aware chunking belongs to the parsing half. These are findings about what that chunker must handle, not a measurement of MinerU itself.