[NOTICKET] feat(eval): knowledge-extraction gold set, scorer + frozen prototype baseline

P2 + P3 + P4 from the knowledge-pipeline plan. Lifts the two things out of the
kex prototype that could not be recreated if the machine died — the calibration
knowledge and the measurement baseline — without porting any pipeline code.

eval/knowledge/ (new, follows the eval/{intent,help,readiness,planner} convention):
- knowledge_gold.yaml 41 terms + 15 rules, verbatim from the prototype. Still
provisional (not expert-reviewed); accepted as good enough
so the rebuild is not blocked on expert time.
- score.py ported; only change is dropping the prototype's sys.path
bootstrap. Pipeline-independent: scores plain lists, so it
works against the prototype's artifacts and v2 alike.
- results/baseline_prototype_2026-08-13_145132.json
FROZEN. The numbers v2 must match or beat.
- README.md provenance, what the scorer refuses to conflate, and how
to read E3 on an 8-entry scoreable base.

Migration verified faithful: the ported scorer re-derives the baseline exactly
from the prototype's artifacts — E1 recall 0.8537, E3 precision 0.75, scoreable
base 8, abstentions 56. All MATCH.

KNOWLEDGE_PIPELINE_CALIBRATION.md (new): every tuned constant with its reason —
GLiNER 0.25 threshold and the 130-word window, rapidfuzz 92 with fuzzy off below
5 chars (PA/UA collide), the six-signal weight table, chunking traps, the
1024-token cache floor, span/escalation/conflict thresholds — plus §7, the eight
negative findings v2 must not re-derive.

Decisions recorded: prototype stays as-is unversioned (risk accepted, now much
reduced); all four branches stay on nano via the __54n env quad; provisional gold
is good enough for now.

Verification: ruff clean on eval/knowledge/; PYTHONPATH=. python -c "import main" OK.
No dependency, schema or endpoint change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KNOWLEDGE_PIPELINE_CALIBRATION.md

@@ -0,0 +1,187 @@
+# Knowledge Pipeline — Calibration Reference
+
+**Harvested from the `kex` prototype, 2026-08-19.** Every number below cost a run, a bug, or a
+measurement to find, and **none of it is in the spec** — it lived only as constants and comments in
+a codebase that is not being ported.
+
+This file exists because clean rewrites drop accidental knowledge and then rediscover it as
+production defects. When building extraction v2, take the value **and** the reason. If you change
+one, change it deliberately and record why.
+
+Companions: [KNOWLEDGE_PIPELINE_TODO.md](KNOWLEDGE_PIPELINE_TODO.md) (plan) ·
+[eval/knowledge/](eval/knowledge/) (gold set, scorer, frozen baseline) ·
+[knowledge_pipeline_context.md](knowledge_pipeline_context.md) (intent).
+
+---
+
+## 1. Term filter — GLiNER
+
+| Constant | Value | Why this value |
+|---|---|---|
+| Model | `urchade/gliner_multi-v2.1` | CPU, ~400 MB. Cheap enough to run unconditionally on every chunk, which is what lets us reject a standalone relevance gate |
+| Score threshold | **0.25** | Not 0.35. The sweep measured recall **0.854 @ 0.25** vs **0.658 @ 0.35** on the `broad` label set. Precision falls (0.41 vs 0.50) and that is accepted: the filter is meant to be over-inclusive, clustering and evidence ranking absorb the noise, and **a term the filter never proposes can never be recovered downstream** |
+| Label variant | **`broad`** | Two earlier variants both missed the same class — mining activities and materials (coal mining, land clearing, waste removal, joint survey). `broad` adds `mining activity`, `material or commodity`, `data source or system`, `work category`, `analysis method` to cover it |
+| Window size | **130 words**, overlap **30** | Not 220. GLiNER truncates past ~384 of its own tokens and **warns rather than failing** — silently dropping the tail of every long chunk. Indonesian technical prose subword-tokenises at roughly **2.5×**, so a 220-word window still tripped the cap. Chunks are fed as overlapping word windows with offsets remapped |
+| Span token cap | **12** (GLiNER's own) | Compound terms ("UA plan grouping (composite) actual") exceed it. Candidates that hit the cap are **flagged, not silently truncated**, so a recall shortfall can be attributed to the cap |
+
+**Label sets** (`labels.yaml`) — three variants were tried and A/B'd without touching code. Keep
+that property: label phrasing is the main recall lever and GLiNER is very sensitive to it. The
+Indonesian-phrasing variant (`istilah produksi tambang`, …) did **not** win; `broad` did.
+
+---
+
+## 2. Clustering
+
+| Constant | Value | Why this value |
+|---|---|---|
+| Match order | exact (normalised) → abbreviation↔expansion → fuzzy | Cheapest and most certain first |
+| Fuzzy threshold | **92** (`rapidfuzz.token_set_ratio`) | Conservative on purpose |
+| **Fuzzy disabled below 5 characters** | `min(len(a), len(b)) < 5` → skip | **The one to not lose.** `PA` vs `UA` scores high on `token_set_ratio`. Below 5 chars only exact matching is allowed |
+| Canonical surface | shortest variant ≥ 2 chars | That is how a reader looks a term up — `PA`, not `Physical Availability (PA) untuk …` |
+
+**Over-merging is worse than under-merging, and the asymmetry is not close.** An under-merge costs
+one extra LLM call and one extra review row. A wrong merge silently destroys a distinct term and
+**the expert never sees it** — there is no downstream mechanism that recovers it.
+
+**Legend extraction must run before clustering.** Without the abbreviation index, `PA` and
+`Physical Availability` cluster as two unrelated terms.
+
+**Stop surfaces** — dropped as whole surface forms only (never as substrings, so no term is lost):
+`unit · type · class · equipment · equipment unit · parameter · activity · data · nilai · proses ·
+hasil · total`.
+
+**Normalisation** (clustering only, never for span checking): NFKC → casefold → `-`/`_` → space →
+strip `.’'` → non-word to space → collapse whitespace → strip ` ()/`.
+
+---
+
+## 3. Evidence ranking
+
+Six signals plus one penalty. The full ranked list is retained, not just the top K — escalation
+needs the tail.
+
+| Signal | Weight |
+|---|---|
+| Definitional cue within 100 chars of the mention | **+5.0** |
+| Term appears in the chunk heading | **+4.0** |
+| Mention sits inside a legend block | **+3.5** |
+| Chunk contains a formula | **+2.0** |
+| Mention is bold/italic | **+1.5** |
+| First occurrence of the cluster | **+1.0** |
+| Chunk is tabular | **−3.0** |
+
+| Constant | Value | Why |
+|---|---|---|
+| `evidence_k` | **3** | Chunks fed per extraction round |
+| Cue proximity | **100 chars** | Distance from mention to a definitional cue |
+
+**Heading matching is word-boundary, never substring.** `PA` is a substring of `Parameter`, `pada`
+and `composite` — substring matching handed the +4.0 heading bonus to almost every Indonesian
+heading and pushed real definition sections *below* formula tables.
+
+**Why this filter is legitimate when the relevance gate was rejected:** no term is dropped (only its
+*evidence* is narrowed), every mention stays in provenance so a bad top-3 is visible, and the
+escalation loop self-corrects. The rejected gate had none of those three properties. Preserve all
+three if ranking is ever revisited.
+
+---
+
+## 4. Chunking
+
+| Constant | Value | Why |
+|---|---|---|
+| Max chunk tokens | **1500** | With paragraph-boundary splitting; sentence boundaries as fallback for a single over-budget paragraph |
+| Max heading length | **90 chars** | Longer lines are sentences or formula rows, not headings |
+| Boilerplate frequency | **≥ 0.6 of pages** | A line repeating on 60%+ of pages is a running header/footer. Detected by frequency with **digits normalised to `#`** so page counters collapse — never by a hardcoded document-specific string |
+| Tabular detection | short lines > 0.6 **and** numeric lines > 0.4 | Feeds the −3.0 ranking penalty |
+
+**Three things the real document forced, all of which corrupt every downstream number if missed:**
+
+1. **Running headers/footers** — every page repeats the title block and "Confidential".
+2. **Breadcrumb headings** — pages re-print `2. PENJELASAN PARAMETER / 2.1. Production Parameter / …`
+   at the top. A naive splitter re-opens the section and shatters it. Rule used: *a heading that is
+   the current section or an ancestor of it, already seen, is a breadcrumb* — the section continues
+   and `page_end` extends.
+3. **Colon-continuation** — legend rows arrive as `MOHH` then `: Machine on Hand Hours` on the next
+   line. Rejoined before the legend filter sees them.
+
+Points 1 and 2 apply to MinerU output too: its `title` blocks repeat the same breadcrumbs on pages
+2–8 of the standard.
+
+---
+
+## 5. Extraction (the paid stage)
+
+| Constant | Value | Why |
+|---|---|---|
+| Temperature | **0.0** | And still not deterministic — see below |
+| Seed | 7 | |
+| **Cache minimum** | **1024 prompt tokens** | OpenAI-family prompt caching **does not engage at all** below this. The glossary prefix is padded past it *on purpose*. Measured hit rate **54%** — 125,184 of 155,313 prompt tokens |
+| Structured output | `json_schema`, fallback `json_object` + validate-retry | Which mode was used is recorded in the run manifest |
+| API version | `2024-12-01-preview` | `json_schema` needs ≥ `2024-08-01-preview` |
+
+**Prompts live in files, not code, and the fixed prefix must stay byte-identical across calls** —
+any drift and caching stops engaging, silently, at ~10× the input cost.
+
+**`temperature=0` is not determinism.** Consecutive runs of the identical pipeline scored 0.75 and
+0.625 on schema fill. Never report a single run as a measurement on a small base.
+
+---
+
+## 6. Validation
+
+| Constant | Value | Why |
+|---|---|---|
+| Span normalisation | **whitespace only** (+ NFKC) | Not case, not punctuation, not diacritics. Every additional normalisation is a hole a fabrication fits through |
+| Escalation rounds | **max 2** | Then keep the entry flagged `no_definition_found` and pass it to review anyway — a term we found but could not define is still useful; dropping it hides a known unknown |
+| Conflict overlap threshold | **0.4** token overlap | Token overlap, not embeddings: cheaper, needs no model, and **explainable to the reviewer who has to act on it** |
+| Duplicate overlap threshold | **0.8** | Above it, a differing definition is a duplicate; below, a conflict |
+
+**Guarded fields** (span-checked; failure sets the field to `None` and logs it):
+
+| Branch | Fields |
+|---|---|
+| glossary | `definition`, `full_name`, `formula_latex`, `interpretation` |
+| rule | `statement`, `condition`, `consequence` |
+| formula | `formula_latex` |
+| summary | *(none — generation cannot be span-checked at all)* |
+
+**If the provenance span itself is not verbatim, every guarded field on the entry is rejected** —
+the entry's only evidence link is broken, so nothing on it can be trusted.
+
+**Never repair a failed span.** A repaired span is an unfalsifiable claim, which is exactly what the
+control exists to prevent.
+
+**Conflict detection never picks a winner.** The expert decides.
+
+---
+
+## 7. Negative findings — do not re-derive these
+
+| Finding | Detail |
+|---|---|
+| **nano is not sufficient for schema fill** | 0.75 precision against a 0.80 kill line. Decision 2026-08-19: **stay on nano anyway** for now; the `__54n` env quad is provisioned. Revisit if quality blocks the demo |
+| **Escalation is unreachable on small documents** | 54 of 66 clusters had exactly **1** evidence chunk against K=3, so `rounds_available()` correctly returned 0 and the loop never fired. `escalated = 0` is not a bug. It stays unverified until a larger corpus runs |
+| **Conflict detection had nothing to find** | 0 conflicts on a single internally-consistent standard. Untested against real disagreement |
+| **85% abstention** | 56 of 66 entries returned no definition. Correct behaviour, but it means the review queue is mostly *"term found, no definition in document"*. Whether that is useful or noise is a review-experience question for Mas Beta |
+| **Diff only ever ran the empty-baseline path** | The prototype diffed against the file it then overwrote, so every entry came back `new`. v2 needs a real active-version pointer |
+| **Literal source wording was normalised away** | The standard's heading reads *"Physical **of** Availability (PA)"*; the entry carried `full_name: "Physical Availability"`. The gold set records the literal form correctly, so **gold was right and the pipeline was wrong**. Mechanism: the heading is a separate field from the chunk text, and the chunk text does not contain the phrase, so the literal wording never reached the model. Fix: feed the heading verbatim + a span-checked `source_wording` field |
+| **A `glob('*.pdf')` picked the wrong document once** | Scored the textbook against the standard's gold set → recall 0.05. Document selection must be explicit and error on ambiguity |
+| **torch wheels install corrupt** | `torchgen` missing, version reports `None`. Fix is delete `site-packages/torch*` and reinstall. Needs **torch ≥ 2.6**. Documented so nobody debugs it as a GLiNER problem |
+
+---
+
+## 8. What was measured, so v2 can be compared
+
+Frozen in `eval/knowledge/results/baseline_prototype_2026-08-13_145132.json`.
+
+Funnel on the 9-page standard: **9 pages → 13 chunks → 169 mentions** (195 before noise filtering)
+**→ 66 clusters → 83 LLM calls → 66 entries → 66 review-queue items.**
+
+E1 recall **0.854** · E2 compression **2.56×** · E3 schema-fill precision **0.75** (FAIL) ·
+E4 fabrication rejection **1.00** with **0.00** false rejections. Cost **$0.0069**, 242 s wall
+clock, 3 fields rejected by span check.
+
+**When quoting E2 externally:** the context doc's **6.2×** is a 1,000-page projection; **2.56×** is
+the 9-page measurement. Compression grows as terms repeat across a corpus. Both are true — use the
+right one for the audience.

KNOWLEDGE_PIPELINE_TODO.md

@@ -66,10 +66,10 @@ either port scaffolding you didn't want or discard measurements you can't recrea
 
 | # | Task | Owner | Status | Description |
 |---|---|---|---|---|
-| **P1** | **Freeze the prototype** | Rifqi | ⬜ | It has no `.git`. ~2,763 LOC, both gold sets and all four experiment results exist on exactly one machine. Snapshot it read-only — zip, orphan branch or private repo, whichever is least ceremony. It is an archive, not a codebase to develop |
-| **P2** | **Harvest the tuned constants** | Rifqi | ⬜ | Every number that took runs to find, written down with its reason, **before** v2 code. Known set: rapidfuzz threshold **92**; fuzzy matching **off below 5 characters** (because `PA` and `UA` score highly against each other); evidence **K=3**; the six-signal evidence weight table; the **1024-token prefix pad** (below that threshold OpenAI-family caching does not engage at all — worth the measured 54% hit rate); `labels.yaml` GLiNER label set; `cues.yaml` discourse cues; the 4 prompt files. None of this is in the spec; it lives only as constants and comments, and a clean rewrite will rediscover them as production defects |
-| **P3** | **Migrate the evidence into `eval/knowledge/`** | Rifqi | ⬜ | Gold sets (41 terms + 15 rules) + scorer + the four experiment runners. Follows the existing convention exactly — `eval/{intent,help,readiness,planner}/` each carry `README.md`, `<area>_dataset.json`, `run_eval.py`, and `results/<name>_result_YYYY-MM-DD_HHMMSS.json` files that are never overwritten. **Do this before any v2 pipeline code**: without a standing baseline, "new and improved" is unfalsifiable, and given E3 already failed, quality is exactly what this work will be judged on |
-| **P4** | **Record the negative findings** | Rifqi | ⬜ | Things v2 must not re-derive: nano fails schema fill; escalation is unreachable on small documents (54 of 66 clusters had exactly 1 evidence chunk against K=3, so the loop correctly never fired); 85% of entries returned no definition; conflict detection had nothing to find in a single internally-consistent standard |
+| **P1** | Freeze the prototype | Rifqi | ✅ | **Resolved 2026-08-19: kept as-is, in place, unversioned — Rifqi's call.** The machine-failure risk is accepted knowingly. It is materially reduced anyway now that P2 and P3 have lifted the two things that could not be recreated (the calibration knowledge and the gold set + baseline) into this repo |
+| **P2** | Harvest the tuned constants | Rifqi | ✅ | **Done 2026-08-19 → [KNOWLEDGE_PIPELINE_CALIBRATION.md](KNOWLEDGE_PIPELINE_CALIBRATION.md).** 8 sections, every value with its reason: GLiNER threshold 0.25 (recall 0.854 vs 0.658 @ 0.35) and the 130-word window (Indonesian subword-tokenises ~2.5×, so 220 still tripped GLiNER's 384 cap); rapidfuzz 92 with fuzzy disabled below 5 chars (`PA`/`UA`); the six-signal weight table + word-boundary heading matching; chunking's 1500-token cap, 0.6 boilerplate frequency and the three real-document traps; the 1024-token cache floor; span/escalation/conflict thresholds and the guarded-field table |
+| **P3** | Migrate the evidence into `eval/knowledge/` | Rifqi | ✅ | **Done 2026-08-19.** `eval/knowledge/` now carries `knowledge_gold.yaml` (41 terms + 15 rules, verbatim), `score.py` (ported; only the prototype's `sys.path` bootstrap removed), `README.md`, and the frozen `results/baseline_prototype_2026-08-13_145132.json`. **Migration verified faithful:** the ported scorer re-derives the baseline exactly from the prototype's artifacts — E1 recall 0.8537, E3 precision 0.75, scoreable base 8, abstentions 56, all MATCH. `ruff` clean; `import main` OK. No `run_eval.py` yet — it lands with the first v2 stage that produces scoreable output |
+| **P4** | Record the negative findings | Rifqi | ✅ | **Done 2026-08-19 — calibration doc §7.** Eight entries: nano insufficient for schema fill; escalation unreachable on small documents; conflict detection had nothing to find; 85% abstention; diff only ever ran the empty-baseline path; literal source wording normalised away (gold was right, pipeline wrong); a `glob('*.pdf')` once scored the wrong document (recall 0.05); torch wheels install corrupt and need ≥ 2.6 |
 
 ---
 
@@ -166,10 +166,10 @@ name (`src/kex/`, `src/knowledge_extraction/`, TBD in §6).
 |---|---|---|---|---|
 | **D1** | New dependencies | Rifqi (asks) | ⬜ | GLiNER + torch (heavy) for extraction; MinerU for parsing. `pyproject.toml` changes need sign-off. Prototype needed **torch ≥ 2.6** and hit a corrupt-wheel failure worth not re-debugging |
 | **D2** | New tables (DDL handoff) | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns the dedorch schema — Python never executes DDL. One consolidated Harry-ready handoff beats three |
-| **D3** | Model tier per branch | Rifqi | ⛔ | **Half-answered: nano FAILS at 0.75 vs 0.80.** Blocked on a **mini deployment being provisioned**. Until then all four branches run on nano — including rule and summary, whose failure mode is least detectable, since a plausible summary cannot be span-checked |
+| **D3** | Model tier per branch | Rifqi | ✅ | **Decided 2026-08-19: stay on nano for all four branches.** The `.env` carries the quad `azureai__{api_key,endpoint__url,deployment__name,api__version}__54n`, mirroring the existing `__54m` scheme. Accepted knowingly: nano measured 0.75 against a 0.80 line, and `rule`/`summary` run on the tier whose failure mode is least detectable. Revisit if extraction quality blocks the demo. **`src/config/settings.py` does not expose the `__54n` quad yet** — four `Field(alias=…)` entries, first build step |
 | **D4** | Admin ingest surface | Rifqi | ⬜ | Whether it is HTTP at all, and if so its auth posture. A write surface triggered by an admin is a different risk class from the current unauthenticated read-only chat surface — raise it, don't inherit the posture by default. Only after the offline path works |
 | **D5** | v2 module name | Rifqi | ⬜ | `src/knowledge/` is taken by the existing OCR→pgvector service. Pick a non-colliding name before the first commit |
-| **D6** | Expert review of the gold set | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is self-bootstrapped and provisional. E1 and E3 both move when it is reviewed, and E3's scoreable base is 8 entries — each adjudication shifts it ~12 points. Now *more* important than before: it is the baseline v2 gets measured against |
+| **D6** | Expert review of the gold set | Rifqi → Mas Beta | ⏸️ | **Deferred 2026-08-19: provisional gold is good enough for now** so the rebuild is not blocked on expert time. Standing caveat: E1 and E3 both move when it is reviewed, and E3's scoreable base is 8 entries, so each adjudication shifts it ~12 points. Every E3 failure is listed verbatim in the frozen baseline for when he does review it |
 | **D7** | GLiNER + LangExtract ensembling | — | ⏸️ | Proposed, not decided; flagged as scope creep. Cheap in API terms but adds a reconciliation step between two candidate sets that may disagree. Not in scope for the test integration |
 
 ---

eval/knowledge/README.md

@@ -0,0 +1,84 @@
+# eval/knowledge — knowledge-extraction eval harness
+
+Scores the knowledge-extraction pipeline against a gold set built from
+**BUMA `STD/2026/006/MNO Rev.0.0` — Production Parameter & ECA** (9 pages, majority Bahasa
+Indonesia).
+
+Migrated from the `kex` prototype on 2026-08-19. **The prototype itself is not being ported** — it
+answered the design questions and stays where it is. This directory carries the parts that survive
+it: the gold set, the scorer, and a frozen baseline. Plan: [KNOWLEDGE_PIPELINE_TODO.md](../../KNOWLEDGE_PIPELINE_TODO.md),
+tuned constants: [KNOWLEDGE_PIPELINE_CALIBRATION.md](../../KNOWLEDGE_PIPELINE_CALIBRATION.md).
+
+## Why this landed before any v2 pipeline code
+
+Without a standing baseline, "new and improved" is unfalsifiable — and one of the four experiments
+already **failed** (E3, schema fill). Quality is what this work gets judged on, so the yardstick
+ships first and every v2 stage is measured from its first commit rather than retroactively.
+
+## Contents
+
+| Path | What it is |
+|---|---|
+| `knowledge_gold.yaml` | 41 terms + 15 rules. **Provisional** — bootstrapped by reading the source PDF, *not* expert-reviewed |
+| `score.py` | Precision / recall / F1 per branch. Pipeline-independent: takes plain lists of surfaces and entry dicts, so it scores the prototype's artifacts and v2 alike |
+| `results/baseline_prototype_2026-08-13_145132.json` | **Frozen.** The prototype's measured numbers. Never regenerate — add new timestamped files beside it |
+
+No `run_eval.py` yet: there is no v2 pipeline to run. It lands with the first stage that produces
+scoreable output, following the house convention (module mode, timestamped results, never
+overwritten) used by `eval/{intent,help,readiness,planner}/`.
+
+## The baseline to beat
+
+Measured by the prototype on the document above, run `20260813-145132`, deployment `gpt-5.4-nano`:
+
+| Experiment | Question | Result | Kill line | Verdict |
+|---|---|---|---|---|
+| **E1** | GLiNER recall on Bahasa Indonesia technical prose | **0.854** | 0.70 | **PASS** |
+| **E2** | Does clustering cut expert review burden? | **2.56×** (169 → 66) | 2.0× | **PASS** |
+| **E3** | Is nano sufficient for schema fill? | **0.75** | 0.80 | **FAIL** |
+| **E4** | Does verbatim-span validation catch fabrication? | **1.00** (false-rejection 0.00) | 0.90 | **PASS** |
+
+Funnel: 9 pages → 13 chunks → 169 mentions → 66 clusters → 83 LLM calls → 66 entries → 66 queue
+items. Cost $0.0069; 125,184 of 155,313 prompt tokens served from cache.
+
+## Two things the scorer refuses to do
+
+**It never conflates term-filter recall with extraction precision.** They are different failure
+modes with different fixes — recall is fixed at the filter stage (GLiNER labels), precision at the
+extraction stage (model tier, prompt). E1 is the recall number specifically.
+
+**It never counts abstention as an error.** For a term the document does not define, `null` is the
+correct answer. Scoring is restricted to the *scoreable* subset: entries that produced a definition,
+whose term is in the gold set, and whose gold record carries `definition_contains` to check against.
+Counting the rest as errors would measure gold coverage while claiming to measure model accuracy.
+Coverage is reported separately in `Score.as_dict()`.
+
+## Reading E3 carefully
+
+The scoreable base is **8 entries**, so each adjudication moves the number ~12 points. nano is also
+not deterministic at `temperature=0` — consecutive runs of the identical pipeline scored 0.75 and
+0.625. Treat E3 as a signal, not a measurement, until the gold set is reviewed.
+
+Every failure is listed verbatim under `experiments.E3.scoreable_basis.failures` in the baseline
+file, for Mas Beta to adjudicate. Some are genuine extraction errors; others are cases where the
+document carries two valid definitions and the provisional gold names only one. **Gold was not
+edited after seeing output**, so the figure is a lower bound.
+
+## Gold-set conventions (keep these when extending it)
+
+- `term` is the string a reader would look up.
+- `full_name` is the **literal source wording**, never normalised. The standard writes *"Physical of
+  Availability (PA)"* in the heading and *"Physical Availability"* in the legend — both are recorded
+  as variants. This is deliberate: the pipeline is required to surface that discrepancy to the
+  expert rather than silently correct it.
+- `definition_contains` are substrings that **must** appear in a correct definition. Substring
+  matching, not exact, so a correct-but-differently-worded extraction is not scored as a miss.
+- The gold set is **partial by design** — the expert is the labelling bottleneck, so scoring reports
+  coverage rather than blocking on a complete file.
+
+## Status of the gold set
+
+`status: provisional`, `labelled_by: claude-bootstrap`, `labelled_on: 2026-08-13`. It has **not**
+been reviewed by Mas Beta. Every score derived from it carries that caveat, and both E1 and E3 will
+move when it is reviewed. Accepted as good enough for now (2026-08-19, Rifqi) so the rebuild is not
+blocked on expert time.

eval/knowledge/knowledge_gold.yaml

@@ -0,0 +1,312 @@
+# PROVISIONAL gold set — bootstrapped by reading the source PDF, NOT yet expert-reviewed.
+# Reviewer: Mas Beta. Until reviewed, every score derived from this file carries that caveat.
+#
+# Labelling rules used:
+#  - `term` is the string a reader would look up.
+#  - `full_name` is the LITERAL source wording, not a normalised or corrected form.
+#    (e.g. the STD writes "Physical of Availability (PA)" in the heading and
+#     "Physical Availability" in the legend — both are recorded as variants.)
+#  - `definition_contains` are substrings that MUST appear in a correct definition.
+#    Substring matching, not exact match, so a correct-but-differently-worded
+#    extraction is not scored as a miss (spec §5).
+
+doc_id: STD_2026_006_MNO
+status: provisional
+labelled_by: claude-bootstrap
+labelled_on: 2026-08-13
+source_pages: 9
+
+terms:
+  - term: Production
+    full_name: Production
+    variants: ["Production", "produksi"]
+    definition_contains: ["nilai tambah", "sumber daya"]
+    formula_present: true
+    page: 2
+  - term: MOHH
+    full_name: Machine on Hand Hours
+    variants: ["MOHH"]
+    definition_contains: ["Machine on Hand Hours"]
+    formula_present: false
+    page: 2
+  - term: Qty
+    full_name: Quantity (Qty)
+    variants: ["Qty", "Quantity", "quantity"]
+    definition_contains: ["jumlah equipment", "periode"]
+    formula_present: true
+    page: 2
+  - term: PA
+    full_name: Physical of Availability (PA)
+    variants: ["PA", "Physical Availability", "Physical of Availability"]
+    definition_contains: ["ketersediaan fisik", "available"]
+    formula_present: true
+    page: 4
+  - term: UA
+    full_name: Utilization of Availability (UA)
+    variants: ["UA", "Utilization of Availability"]
+    definition_contains: ["efektifitas penggunaan", "working hours"]
+    formula_present: true
+    page: 4
+  - term: Pty
+    full_name: Productivity (Pty)
+    variants: ["Pty", "PTY", "Productivity", "Produktivitas"]
+    definition_contains: ["efektifitas penggunaan sumber daya", "output"]
+    formula_present: true
+    page: 5
+  - term: ECA
+    full_name: Equipment Capacity Analysis (ECA)
+    variants: ["ECA", "Equipment Capacity Analysis"]
+    definition_contains: ["ketidaktercapaian", "kapasitas produksi"]
+    formula_present: false
+    page: 6
+  - term: INPR
+    full_name: in operation
+    variants: ["INPR", "in operation", "INPR Hours"]
+    definition_contains: ["in operation"]
+    formula_present: false
+    page: 2
+  - term: Total Hours
+    full_name: Total Hours
+    variants: ["Total Hours", "total jam"]
+    definition_contains: ["total jam"]
+    formula_present: true
+    page: 2
+  - term: Breakdown
+    full_name: Breakdown
+    variants: ["Breakdown", "breakdown"]
+    definition_contains: []
+    formula_present: true
+    page: 4
+  - term: Working Hours
+    full_name: Working Hours
+    variants: ["Working Hours", "working hours"]
+    definition_contains: ["waktu kerja"]
+    formula_present: true
+    page: 4
+  - term: EWH
+    full_name: Effective Working Hours (EWH)
+    variants: ["EWH", "Effective Working Hours", "productive working hours"]
+    definition_contains: ["waktu kerja efektif"]
+    formula_present: false
+    page: 5
+  - term: Land clearing
+    full_name: land clearing
+    variants: ["land clearing"]
+    definition_contains: []
+    formula_present: false
+    page: 2
+  - term: Overburden removal
+    full_name: pengupasan lapisan penutup (overburden removal)
+    variants: ["overburden removal", "pengupasan lapisan penutup"]
+    definition_contains: []
+    formula_present: false
+    page: 2
+  - term: Coal mining
+    full_name: penambangan batubara (coal mining)
+    variants: ["coal mining", "Coal Mining", "CM", "penambangan batubara"]
+    definition_contains: []
+    formula_present: false
+    page: 2
+  - term: Coal hauling
+    full_name: pengangkutan batubara (coal hauling/coal transporting)
+    variants: ["coal hauling", "coal transporting", "Coal Transport", "CT"]
+    definition_contains: []
+    formula_present: false
+    page: 2
+  - term: Coal barging
+    full_name: coal barging atau ship loading
+    variants: ["coal barging", "ship loading", "pengapalan"]
+    definition_contains: []
+    formula_present: false
+    page: 2
+  - term: Waste Removal
+    full_name: Waste Removal (WR)
+    variants: ["Waste Removal", "WR", "waste removal"]
+    definition_contains: []
+    formula_present: false
+    page: 3
+  - term: General Work
+    full_name: General Work
+    variants: ["General Work"]
+    definition_contains: []
+    formula_present: false
+    page: 3
+  - term: Rental
+    full_name: Rental
+    variants: ["Rental"]
+    definition_contains: []
+    formula_present: false
+    page: 3
+  - term: Grouping (Composite)
+    full_name: Grouping (Composite)
+    variants: ["Grouping (Composite)", "grouping (composite)", "Composite"]
+    definition_contains: ["type", "class"]
+    formula_present: true
+    page: 3
+  - term: EX2500
+    full_name: EX2500
+    variants: ["EX2500"]
+    definition_contains: []
+    formula_present: false
+    page: 3
+  - term: Time Performance MCD Application
+    full_name: raw data Time Performance MCD Application
+    variants: ["Time Performance MCD Application", "MCD Application", "MCD"]
+    definition_contains: []
+    formula_present: false
+    page: 3
+  - term: Joint survey
+    full_name: joint survey
+    variants: ["joint survey"]
+    definition_contains: []
+    formula_present: false
+    page: 5
+  - term: Truck count
+    full_name: truck count
+    variants: ["truck count"]
+    definition_contains: []
+    formula_present: false
+    page: 5
+  - term: Weight average
+    full_name: rata-rata tertimbang (weight average)
+    variants: ["weight average", "rata-rata tertimbang", "pembobotan"]
+    definition_contains: []
+    formula_present: false
+    page: 4
+  - term: Gain/Loss
+    full_name: Gain/Loss
+    variants: ["Gain/Loss", "Gain / Loss"]
+    definition_contains: ["plan", "actual"]
+    formula_present: true
+    page: 6
+  - term: Plan
+    full_name: rencana (plan)
+    variants: ["plan", "Plan", "rencana", "P(M)", "P(G)"]
+    definition_contains: []
+    formula_present: false
+    page: 6
+  - term: Actual
+    full_name: realisasi (actual)
+    variants: ["actual", "Actual", "realisasi", "A(M)", "A(G)"]
+    definition_contains: []
+    formula_present: false
+    page: 6
+  - term: Model Unit
+    full_name: Model Unit
+    variants: ["Model Unit", "model unit", "(M)", "Model"]
+    definition_contains: []
+    formula_present: true
+    page: 6
+  - term: Standby
+    full_name: Delay/Standby
+    variants: ["Standby", "standby", "Delay/Standby", "Gap Standby"]
+    definition_contains: []
+    formula_present: true
+    page: 6
+  - term: BCM
+    full_name: BCM
+    variants: ["BCM"]
+    definition_contains: []
+    formula_present: false
+    page: 6
+  - term: Waterfall Analysis
+    full_name: Waterfall Analysis
+    variants: ["Waterfall Analysis", "grafik Waterfall"]
+    definition_contains: ["visualisasi", "Gain/Loss"]
+    formula_present: false
+    page: 8
+  - term: Other Activity
+    full_name: Other Activity
+    variants: ["Other Activity"]
+    definition_contains: ["tidak sesuai", "direncanakan"]
+    formula_present: false
+    page: 8
+  - term: Loader
+    full_name: loader
+    variants: ["loader", "PA Loader"]
+    definition_contains: []
+    formula_present: false
+    page: 8
+  - term: Hauler
+    full_name: hauler
+    variants: ["hauler"]
+    definition_contains: []
+    formula_present: false
+    page: 8
+  - term: Hierarki Level Analysis
+    full_name: Hierarki Level Analyisis
+    variants: ["Hierarki Level Analyisis", "Hierarki Level Analysis"]
+    definition_contains: ["level"]
+    formula_present: false
+    page: 9
+  - term: Uncontrollable
+    full_name: Uncontrollable
+    variants: ["Uncontrollable", "uncontrollable"]
+    definition_contains: ["level 1"]
+    formula_present: false
+    page: 9
+  - term: Controllable
+    full_name: controllable
+    variants: ["controllable", "UA Controllable"]
+    definition_contains: ["level 1"]
+    formula_present: false
+    page: 9
+  - term: Fleet management
+    full_name: fleet management
+    variants: ["fleet management"]
+    definition_contains: []
+    formula_present: false
+    page: 9
+  - term: Mineplan
+    full_name: mineplan
+    variants: ["mineplan"]
+    definition_contains: []
+    formula_present: false
+    page: 9
+
+rules:
+  - rule_id: QTY_PER_ACTIVITY
+    statement_contains: ["Qty", "kategori aktivitas", "terpisah"]
+    page: 3
+  - rule_id: QTY_GROUPING_SUM
+    statement_contains: ["Qty Grouping", "menjumlahkan"]
+    page: 3
+  - rule_id: ACTIVITY_CLASSIFICATION_SOURCE
+    statement_contains: ["status equipment", "Time Performance MCD"]
+    page: 3
+  - rule_id: PA_COMPOSITE_WEIGHTED
+    statement_contains: ["weight average", "Quantity", "pembobot"]
+    page: 4
+  - rule_id: UA_BY_ACTIVITY
+    statement_contains: ["UA", "kategori aktivitas"]
+    page: 4
+  - rule_id: PTY_PRODUCTION_SOURCE
+    statement_contains: ["joint survey", "truck count"]
+    page: 5
+  - rule_id: PTY_WORKING_HOURS_DEFINITION
+    statement_contains: ["Effective Working Hours", "waktu kerja efektif"]
+    page: 5
+  - rule_id: PTY_COMPOSITE_WEIGHTED
+    statement_contains: ["Pty", "pembobotan", "Quantity"]
+    page: 5
+  - rule_id: GAINLOSS_QTY_ZERO_PLAN
+    statement_contains: ["QtyP(M) = 0", "QtyA(M) > 0"]
+    page: 6
+  - rule_id: GAINLOSS_COMPOSITE_SUM
+    statement_contains: ["grouping", "penjumlahan", "model unit"]
+    page: 6
+  - rule_id: GAINLOSS_UNITS
+    statement_contains: ["BCM", "ton", "jam"]
+    page: 6
+  - rule_id: QTY_ACTUAL_ZERO_PLAN_ALT
+    statement_contains: ["QtyA", "QtyP", "0"]
+    page: 7
+  - rule_id: QTY_PHYSICAL_REPORTING
+    statement_contains: ["fisik unit", "INPR"]
+    page: 7
+  - rule_id: ECA_OTHER_ACTIVITY
+    statement_contains: ["alokasi", "tidak sesuai", "direncanakan"]
+    page: 8
+  - rule_id: WATERFALL_TWO_METHODS
+    statement_contains: ["Waterfall", "dua metode"]
+    page: 8

eval/knowledge/results/baseline_prototype_2026-08-13_145132.json

@@ -0,0 +1,157 @@
+{
+  "_note": "FROZEN BASELINE — measured by the `kex` prototype, not by this repo. Extraction v2 must match or beat these numbers on the same document and the same gold set. Copied from the prototype's eval/out/metrics.json on 2026-08-19; never regenerate this file, add a new timestamped one.",
+  "_source": {
+    "repo": "kex prototype (outside version control)",
+    "file": "eval/out/metrics.json",
+    "copied_on": "2026-08-19"
+  },
+  "provenance": {
+    "run_id": "20260813-145132",
+    "config_hash": "89545a78c702",
+    "created_at": "2026-08-13T14:51:32",
+    "doc_id": "STD_2026_006_MNO",
+    "client_mode": "azure",
+    "deployments": {
+      "glossary": "gpt-5.4-nano",
+      "rule": "gpt-5.4-nano",
+      "formula": "gpt-5.4-nano",
+      "summary": "gpt-5.4-nano"
+    },
+    "api_version": "2024-12-01-preview",
+    "gold_status": "provisional — not yet expert-reviewed"
+  },
+  "funnel": {
+    "status": "measured",
+    "stages": [
+      {
+        "stage": "pages",
+        "count": 9
+      },
+      {
+        "stage": "chunks",
+        "count": 13
+      },
+      {
+        "stage": "mentions",
+        "count": 169
+      },
+      {
+        "stage": "clusters",
+        "count": 66
+      },
+      {
+        "stage": "llm_calls",
+        "count": 83
+      },
+      {
+        "stage": "glossary_entries",
+        "count": 66
+      },
+      {
+        "stage": "review_queue",
+        "count": 66
+      }
+    ],
+    "raw_mentions_before_noise_filter": 195
+  },
+  "experiments": {
+    "E1": {
+      "question": "GLiNER recall on Bahasa Indonesia technical prose",
+      "status": "measured",
+      "value": 0.8537,
+      "kill_line": 0.7,
+      "verdict": "PASS",
+      "detail": "broad @ threshold 0.25",
+      "n_gold": 41
+    },
+    "E2": {
+      "question": "Does clustering cut expert review burden?",
+      "status": "measured",
+      "value": 2.561,
+      "kill_line": 2.0,
+      "verdict": "PASS",
+      "detail": "169 mentions -> 66 clusters",
+      "llm_calls_avoided": 103
+    },
+    "E3": {
+      "question": "Is nano sufficient for schema fill?",
+      "status": "measured_nano_only",
+      "value": 0.75,
+      "kill_line": 0.8,
+      "verdict": "FAIL",
+      "detail": "nano only — no mini deployment provisioned, so the spec's nano-vs-mini A/B could not be run",
+      "glossary_precision": 0.75,
+      "glossary_recall": 0.3333,
+      "glossary_f1": 0.4615,
+      "rule_precision": 0.5714,
+      "rule_recall": 0.2667,
+      "n_gold_terms": 18,
+      "scoreable_basis": {
+        "entries_total": 66,
+        "entries_with_definition": 10,
+        "abstained_null_definition": 56,
+        "scoreable": 8,
+        "unscoreable_term_not_in_gold": 2,
+        "unscoreable_gold_has_no_criteria": 0,
+        "gold_terms_with_criteria": 18,
+        "failures": [
+          {
+            "term": "produksi",
+            "gold_requires": [
+              "nilai tambah",
+              "sumber daya"
+            ],
+            "extracted": "Dalam konteks industri pertambangan, produksi merupakan seluruh rangkaian kegiatan operasional penambangan yang dimulai dari land clearing, pengupasan lapisan penutup (overburden removal), penambangan batubara (coal mining), pengangkutan ba"
+          },
+          {
+            "term": "Production",
+            "gold_requires": [
+              "nilai tambah",
+              "sumber daya"
+            ],
+            "extracted": "Dalam konteks industri pertambangan, produksi merupakan seluruh rangkaian kegiatan operasional penambangan yang dimulai dari land clearing, pengupasan lapisan penutup (overburden removal), penambangan batubara (coal mining), pengangkutan ba"
+          }
+        ]
+      },
+      "abstention_note": "entries where nano returned null are NOT counted as errors — for a term the document never defines, null is the correct answer",
+      "adjudication_pending": "The scoreable base is small, so each failure moves the number by ~12 points. Every failure is listed verbatim under scoreable_basis.failures for Mas Beta to adjudicate: some are genuine extraction errors, others are cases where the document carries two valid definitions and the provisional gold names only one. Gold was NOT edited after seeing output, so this figure is a lower bound.",
+      "stability_warning": "nano is not fully deterministic at temperature=0: consecutive runs of the identical pipeline produced precision 0.75 and 0.625. On a base this small, treat E3 as a signal, not a measurement."
+    },
+    "E4": {
+      "question": "Does verbatim-span validation catch fabrication?",
+      "status": "measured",
+      "value": 1.0,
+      "kill_line": 0.9,
+      "verdict": "PASS",
+      "false_rejection_rate": 0.0,
+      "detail": "span_check is deterministic; measures the control itself"
+    }
+  },
+  "quality": {
+    "status": "measured",
+    "glossary_entries": 66,
+    "rule_entries": 7,
+    "formula_entries": 7,
+    "fields_rejected_by_span_check": 3,
+    "no_definition_found": 56,
+    "escalated": 0,
+    "definition_conflicts": 0
+  },
+  "efficiency": {
+    "status": "measured + stated assumption",
+    "llm_calls_avoided_by_clustering": 103,
+    "review_queue_size": 66,
+    "assumed_minutes_per_term_review": 3.0,
+    "assumed_minutes_per_term_manual": 12.0,
+    "est_review_hours": 3.3,
+    "est_manual_hours": 13.2,
+    "est_hours_saved": 9.9
+  },
+  "cost_measured": {
+    "prompt": 155313,
+    "cached": 125184,
+    "completion": 11862,
+    "usd": 0.006877,
+    "latency_s": 242.4
+  }
+}

eval/knowledge/score.py

@@ -0,0 +1,227 @@
+"""Precision / recall / F1 per branch of the knowledge-extraction pipeline.
+
+Migrated verbatim (bar the import bootstrap) from the `kex` prototype, 2026-08-19.
+The prototype is not being ported; this scorer is, because it is the only thing
+that can show extraction v2 matches or beats the measured baseline in
+`results/baseline_prototype_2026-08-13_145132.json`.
+
+Pipeline-independent by design: it scores plain lists of surfaces / entry dicts,
+so it works against the prototype's artifacts and against v2 alike.
+
+The one thing this module refuses to do is conflate **term-filter recall** with
+**extraction precision** (spec §5). They are different failure modes with
+different fixes: recall is fixed at stage 2 (GLiNER labels), precision is fixed
+at stage 3 (model tier / prompt). E1 is the recall number specifically.
+
+Gold sets are treated as partial by design — Mas Beta is the labelling
+bottleneck, so scoring reports coverage rather than blocking on a complete file.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+import yaml
+
+GOLD_PATH = Path(__file__).resolve().parent / "knowledge_gold.yaml"
+
+
+def norm(s: str) -> str:
+    s = unicodedata.normalize("NFKC", s).casefold()
+    s = re.sub(r"[^\w\s]", " ", s)
+    return re.sub(r"\s+", " ", s).strip()
+
+
+@dataclass
+class Score:
+    label: str
+    n_gold: int
+    n_pred: int
+    true_positives: int
+    precision: float
+    recall: float
+    f1: float
+    misses: list[str]
+
+    def as_dict(self) -> dict:
+        d = asdict(self)
+        if hasattr(self, "coverage"):
+            d["coverage"] = self.coverage
+        return d
+
+
+def load_gold(path: Path) -> dict:
+    with open(path, encoding="utf-8") as fh:
+        return yaml.safe_load(fh)
+
+
+def _prf(tp: int, n_pred: int, n_gold: int) -> tuple[float, float, float]:
+    precision = tp / n_pred if n_pred else 0.0
+    recall = tp / n_gold if n_gold else 0.0
+    f1 = (
+        2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
+    )
+    return round(precision, 4), round(recall, 4), round(f1, 4)
+
+
+def score_term_filter(gold: dict, surfaces: list[str]) -> Score:
+    """E1. A gold term counts as recalled if ANY of its variants appears among
+    the filter's mention surfaces (substring match both ways, so 'PA' inside
+    'Physical Availability (PA)' counts).
+
+    Precision is reported but is NOT E1's criterion: the filter is deliberately
+    over-inclusive, and clustering plus evidence ranking absorb the noise.
+    """
+    normed = {norm(s) for s in surfaces if norm(s)}
+    blob = " | ".join(sorted(normed))
+    tp, misses = 0, []
+    gold_terms = gold.get("terms", [])
+    for entry in gold_terms:
+        variants = [entry["term"]] + list(entry.get("variants", []))
+        if entry.get("full_name"):
+            variants.append(entry["full_name"])
+        hit = False
+        for v in variants:
+            nv = norm(v)
+            if not nv:
+                continue
+            if nv in normed or re.search(rf"(?<![\w]){re.escape(nv)}(?![\w])", blob):
+                hit = True
+                break
+        if hit:
+            tp += 1
+        else:
+            misses.append(entry["term"])
+    precision, recall, f1 = _prf(tp, len(normed), len(gold_terms))
+    return Score(
+        label="term_filter_recall(E1)",
+        n_gold=len(gold_terms),
+        n_pred=len(normed),
+        true_positives=tp,
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        misses=misses,
+    )
+
+
+def score_glossary(gold: dict, entries: list[dict]) -> Score:
+    """E3: when nano fills the schema, is it right?
+
+    Scoring is restricted to the SCOREABLE subset: entries that produced a
+    definition AND whose term is in the gold set AND whose gold record carries
+    `definition_contains` to check against.
+
+    Why not simply tp/len(entries): the term filter is deliberately
+    over-inclusive and the gold set is deliberately partial, so most entries are
+    for terms gold says nothing about. Counting those as errors would measure
+    gold coverage while claiming to measure nano's accuracy — precisely the
+    conflation spec §5 forbids. Coverage is reported separately in as_dict().
+
+    Substring matching, not exact — exact match would under-report
+    correct-but-differently-worded extractions (spec §5).
+    """
+    gold_by_term: dict[str, dict] = {}
+    for entry in gold.get("terms", []):
+        for v in [entry["term"], *entry.get("variants", [])]:
+            gold_by_term.setdefault(norm(v), entry)
+
+    checkable_gold = [
+        g for g in gold.get("terms", []) if g.get("definition_contains")
+    ]
+    n_checkable_gold = len(checkable_gold)
+
+    correct, incorrect = 0, 0
+    unscoreable_no_gold, unscoreable_no_criteria = 0, 0
+    matched_gold, wrong, failures = set(), [], []
+
+    for pred in entries:
+        if not (pred.get("definition") or "").strip():
+            continue  # abstention is scored separately, not as an error
+        g = gold_by_term.get(norm(pred.get("term", "")))
+        if not g:
+            unscoreable_no_gold += 1
+            continue
+        required = [norm(x) for x in g.get("definition_contains", [])]
+        if not required:
+            unscoreable_no_criteria += 1
+            continue
+        definition = norm(pred.get("definition") or "")
+        if all(r in definition for r in required):
+            correct += 1
+            matched_gold.add(g["term"])
+        else:
+            incorrect += 1
+            wrong.append(f"{pred.get('term')} (definition did not match gold)")
+            failures.append(
+                {
+                    "term": pred.get("term"),
+                    "gold_requires": g.get("definition_contains"),
+                    "extracted": (pred.get("definition") or "")[:240],
+                }
+            )
+
+    misses = [g["term"] for g in checkable_gold if g["term"] not in matched_gold]
+    n_scoreable = correct + incorrect
+    precision, recall, f1 = _prf(correct, n_scoreable, n_checkable_gold)
+
+    score = Score(
+        label="glossary_schema_fill(E3)",
+        n_gold=n_checkable_gold,
+        n_pred=n_scoreable,
+        true_positives=correct,
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        misses=misses + wrong,
+    )
+    score.coverage = {  # type: ignore[attr-defined]
+        "entries_total": len(entries),
+        "entries_with_definition": sum(
+            1 for e in entries if (e.get("definition") or "").strip()
+        ),
+        "abstained_null_definition": sum(
+            1 for e in entries if not (e.get("definition") or "").strip()
+        ),
+        "scoreable": n_scoreable,
+        "unscoreable_term_not_in_gold": unscoreable_no_gold,
+        "unscoreable_gold_has_no_criteria": unscoreable_no_criteria,
+        "gold_terms_with_criteria": n_checkable_gold,
+        "failures": failures,
+    }
+    return score
+
+
+def score_rules(gold: dict, entries: list[dict]) -> Score:
+    gold_rules = gold.get("rules", [])
+    pred_blobs = [
+        norm(
+            " ".join(
+                str(v)
+                for v in (e.get("statement"), e.get("condition"), e.get("consequence"))
+                if v
+            )
+        )
+        for e in entries
+    ]
+    tp, misses = 0, []
+    for rule in gold_rules:
+        required = [norm(x) for x in rule.get("statement_contains", [])]
+        if any(all(r in blob for r in required) for blob in pred_blobs):
+            tp += 1
+        else:
+            misses.append(rule["rule_id"])
+    precision, recall, f1 = _prf(tp, len(entries), len(gold_rules))
+    return Score(
+        label="rule",
+        n_gold=len(gold_rules),
+        n_pred=len(entries),
+        true_positives=tp,
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        misses=misses,
+    )