File size: 5,092 Bytes
60cf4a8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | # eval/knowledge — knowledge-extraction eval harness
Scores the knowledge-extraction pipeline against a gold set built from
**BUMA `STD/2026/006/MNO Rev.0.0` — Production Parameter & ECA** (9 pages, majority Bahasa
Indonesia).
Migrated from the `kex` prototype on 2026-08-19. **The prototype itself is not being ported** — it
answered the design questions and stays where it is. This directory carries the parts that survive
it: the gold set, the scorer, and a frozen baseline. Plan: [KNOWLEDGE_PIPELINE_TODO.md](../../KNOWLEDGE_PIPELINE_TODO.md),
tuned constants: [KNOWLEDGE_PIPELINE_CALIBRATION.md](../../KNOWLEDGE_PIPELINE_CALIBRATION.md).
## Why this landed before any v2 pipeline code
Without a standing baseline, "new and improved" is unfalsifiable — and one of the four experiments
already **failed** (E3, schema fill). Quality is what this work gets judged on, so the yardstick
ships first and every v2 stage is measured from its first commit rather than retroactively.
## Contents
| Path | What it is |
|---|---|
| `knowledge_gold.yaml` | 41 terms + 15 rules. **Provisional** — bootstrapped by reading the source PDF, *not* expert-reviewed |
| `score.py` | Precision / recall / F1 per branch. Pipeline-independent: takes plain lists of surfaces and entry dicts, so it scores the prototype's artifacts and v2 alike |
| `results/baseline_prototype_2026-08-13_145132.json` | **Frozen.** The prototype's measured numbers. Never regenerate — add new timestamped files beside it |
No `run_eval.py` yet: there is no v2 pipeline to run. It lands with the first stage that produces
scoreable output, following the house convention (module mode, timestamped results, never
overwritten) used by `eval/{intent,help,readiness,planner}/`.
## The baseline to beat
Measured by the prototype on the document above, run `20260813-145132`, deployment `gpt-5.4-nano`:
| Experiment | Question | Result | Kill line | Verdict |
|---|---|---|---|---|
| **E1** | GLiNER recall on Bahasa Indonesia technical prose | **0.854** | 0.70 | **PASS** |
| **E2** | Does clustering cut expert review burden? | **2.56×** (169 → 66) | 2.0× | **PASS** |
| **E3** | Is nano sufficient for schema fill? | **0.75** | 0.80 | **FAIL** |
| **E4** | Does verbatim-span validation catch fabrication? | **1.00** (false-rejection 0.00) | 0.90 | **PASS** |
Funnel: 9 pages → 13 chunks → 169 mentions → 66 clusters → 83 LLM calls → 66 entries → 66 queue
items. Cost $0.0069; 125,184 of 155,313 prompt tokens served from cache.
## Two things the scorer refuses to do
**It never conflates term-filter recall with extraction precision.** They are different failure
modes with different fixes — recall is fixed at the filter stage (GLiNER labels), precision at the
extraction stage (model tier, prompt). E1 is the recall number specifically.
**It never counts abstention as an error.** For a term the document does not define, `null` is the
correct answer. Scoring is restricted to the *scoreable* subset: entries that produced a definition,
whose term is in the gold set, and whose gold record carries `definition_contains` to check against.
Counting the rest as errors would measure gold coverage while claiming to measure model accuracy.
Coverage is reported separately in `Score.as_dict()`.
## Reading E3 carefully
The scoreable base is **8 entries**, so each adjudication moves the number ~12 points. nano is also
not deterministic at `temperature=0` — consecutive runs of the identical pipeline scored 0.75 and
0.625. Treat E3 as a signal, not a measurement, until the gold set is reviewed.
Every failure is listed verbatim under `experiments.E3.scoreable_basis.failures` in the baseline
file, for Mas Beta to adjudicate. Some are genuine extraction errors; others are cases where the
document carries two valid definitions and the provisional gold names only one. **Gold was not
edited after seeing output**, so the figure is a lower bound.
## Gold-set conventions (keep these when extending it)
- `term` is the string a reader would look up.
- `full_name` is the **literal source wording**, never normalised. The standard writes *"Physical of
Availability (PA)"* in the heading and *"Physical Availability"* in the legend — both are recorded
as variants. This is deliberate: the pipeline is required to surface that discrepancy to the
expert rather than silently correct it.
- `definition_contains` are substrings that **must** appear in a correct definition. Substring
matching, not exact, so a correct-but-differently-worded extraction is not scored as a miss.
- The gold set is **partial by design** — the expert is the labelling bottleneck, so scoring reports
coverage rather than blocking on a complete file.
## Status of the gold set
`status: provisional`, `labelled_by: claude-bootstrap`, `labelled_on: 2026-08-13`. It has **not**
been reviewed by Mas Beta. Every score derived from it carries that caveat, and both E1 and E3 will
move when it is reviewed. Accepted as good enough for now (2026-08-19, Rifqi) so the rebuild is not
blocked on expert time.
|