eval/knowledge/README.md

eval/knowledge — knowledge-extraction eval harness

Scores the knowledge-extraction pipeline against a gold set built from BUMA STD/2026/006/MNO Rev.0.0 — Production Parameter & ECA (9 pages, majority Bahasa Indonesia).

Migrated from the kex prototype on 2026-08-19. The prototype itself is not being ported — it answered the design questions and stays where it is. This directory carries the parts that survive it: the gold set, the scorer, and a frozen baseline. Plan: KNOWLEDGE_PIPELINE_TODO.md, tuned constants: KNOWLEDGE_PIPELINE_CALIBRATION.md.

Why this landed before any v2 pipeline code

Without a standing baseline, "new and improved" is unfalsifiable — and one of the four experiments already failed (E3, schema fill). Quality is what this work gets judged on, so the yardstick ships first and every v2 stage is measured from its first commit rather than retroactively.

Contents

Path	What it is
knowledge_gold.yaml	41 terms + 15 rules. Provisional — bootstrapped by reading the source PDF, not expert-reviewed
score.py	Precision / recall / F1 per branch. Pipeline-independent: takes plain lists of surfaces and entry dicts, so it scores the prototype's artifacts and v2 alike
results/baseline_prototype_2026-08-13_145132.json	Frozen. The prototype's measured numbers. Never regenerate — add new timestamped files beside it

No run_eval.py yet: there is no v2 pipeline to run. It lands with the first stage that produces scoreable output, following the house convention (module mode, timestamped results, never overwritten) used by eval/{intent,help,readiness,planner}/.

The baseline to beat

Measured by the prototype on the document above, run 20260813-145132, deployment gpt-5.4-nano:

Experiment	Question	Result	Kill line	Verdict
E1	GLiNER recall on Bahasa Indonesia technical prose	0.854	0.70	PASS
E2	Does clustering cut expert review burden?	2.56× (169 → 66)	2.0×	PASS
E3	Is nano sufficient for schema fill?	0.75	0.80	FAIL
E4	Does verbatim-span validation catch fabrication?	1.00 (false-rejection 0.00)	0.90	PASS

Funnel: 9 pages → 13 chunks → 169 mentions → 66 clusters → 83 LLM calls → 66 entries → 66 queue items. Cost $0.0069; 125,184 of 155,313 prompt tokens served from cache.

Two things the scorer refuses to do

It never conflates term-filter recall with extraction precision. They are different failure modes with different fixes — recall is fixed at the filter stage (GLiNER labels), precision at the extraction stage (model tier, prompt). E1 is the recall number specifically.

It never counts abstention as an error. For a term the document does not define, null is the correct answer. Scoring is restricted to the scoreable subset: entries that produced a definition, whose term is in the gold set, and whose gold record carries definition_contains to check against. Counting the rest as errors would measure gold coverage while claiming to measure model accuracy. Coverage is reported separately in Score.as_dict().

Reading E3 carefully

The scoreable base is 8 entries, so each adjudication moves the number ~12 points. nano is also not deterministic at temperature=0 — consecutive runs of the identical pipeline scored 0.75 and 0.625. Treat E3 as a signal, not a measurement, until the gold set is reviewed.

Every failure is listed verbatim under experiments.E3.scoreable_basis.failures in the baseline file, for Mas Beta to adjudicate. Some are genuine extraction errors; others are cases where the document carries two valid definitions and the provisional gold names only one. Gold was not edited after seeing output, so the figure is a lower bound.

Gold-set conventions (keep these when extending it)

• term is the string a reader would look up.
• full_name is the literal source wording, never normalised. The standard writes "Physical of Availability (PA)" in the heading and "Physical Availability" in the legend — both are recorded as variants. This is deliberate: the pipeline is required to surface that discrepancy to the expert rather than silently correct it.
• definition_contains are substrings that must appear in a correct definition. Substring matching, not exact, so a correct-but-differently-worded extraction is not scored as a miss.
• The gold set is partial by design — the expert is the labelling bottleneck, so scoring reports coverage rather than blocking on a complete file.

Status of the gold set

status: provisional, labelled_by: claude-bootstrap, labelled_on: 2026-08-13. It has not been reviewed by Mas Beta. Every score derived from it carries that caveat, and both E1 and E3 will move when it is reviewed. Accepted as good enough for now (2026-08-19, Rifqi) so the rebuild is not blocked on expert time.