[NOTICKET] fix(knowledge_extraction): move runner into the package; fix two ranking defects

Runner relocated scripts/run_knowledge_extraction.py -> the package as
src/knowledge_extraction/cli.py. scripts/ is gitignored, and this runner is the
pipeline's operator entry point, so it has to ship with the module:

uv run --no-sync python -m src.knowledge_extraction.cli <artifact.json>

pyproject: narrow the T201 lint exemption to that one CLI file rather than all
of scripts/, so the ignore traces to this change and nothing else.

Two ranking defects found while verifying against the real 9-page BUMA standard
(both recorded as deviations V1/V2 in KNOWLEDGE_PIPELINE_CALIBRATION.md §9):

V1 — a heading that names the term now counts as a mention at offset 0 for
cue-proximity scoring. Indonesian standards are written "2.1.3 Physical of
Availability (PA)" / "Adalah ketersediaan fisik...": the heading names the term
and the body opens with the definitional cue WITHOUT repeating it, so the
definition section earned no cue bonus at all. Measured before the fix, PA's own
definition section scored 7.5 and ranked THIRD, behind a computation section
(8.5) that merely mentions PA often. After it, all five terms rank their own
definition section first: PA->2.1.3, UA->2.1.4, Qty->2.1.2, Pty->2.1.5,
MOHH->2.1.1. Same root cause as the literal-wording defect — the heading is a
separate field from the text, so anything reasoning over "term + its definition"
must be told to read both.

V2 — first_occurrence now follows document order, and ranking ties break on it.
Previously the +1.0 went to whichever chunk a set iteration happened to yield
first, which made the bonus arbitrary and the ranking unreproducible.

Both change which chunks reach the paid stage, so they can move schema-fill
precision when that branch lands; re-score against the frozen baseline then
rather than assuming the improvement carries.

Verification: ruff clean on all touched paths; full suite 470 passed, 7 skipped
(was 467 + 3 new local tests); ranking stable across PYTHONHASHSEED 0/1/7/42/99;
CLI green on the BUMA chunks and on Sofhia's draft artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KNOWLEDGE_PIPELINE_CALIBRATION.md

@@ -185,3 +185,25 @@ clock, 3 fields rejected by span check.
 **When quoting E2 externally:** the context doc's **6.2×** is a 1,000-page projection; **2.56×** is
 the 9-page measurement. Compression grows as terms repeat across a corpus. Both are true — use the
 right one for the audience.
+
+---
+
+## 9. Deliberate deviations from the prototype (v2)
+
+Changes made while rebuilding, with the reason. Each one alters behaviour, so
+each is listed rather than folded silently into the tables above.
+
+| # | Change | Why | Measured? |
+|---|---|---|---|
+| **V1** | **A heading that names the term counts as a mention at offset 0** for cue-proximity scoring | Indonesian standards are written *"2.1.3 Physical of Availability (PA)"* / *"Adalah ketersediaan fisik…"* — the heading names the term and the body opens with the definitional cue **without repeating it**. The prototype therefore awarded no cue bonus on the definition section itself. Measured on the BUMA standard: PA's own definition section (§2.1.3) scored 7.5 and ranked **third**, below a computation section (§2.2.4, 8.5) that merely mentions PA often. After the fix, all five terms rank their own definition section first — PA→2.1.3, UA→2.1.4, Qty→2.1.2, Pty→2.1.5, MOHH→2.1.1 | Structurally verified on the 9-page standard. **Not yet scored** — needs the paid stage to show up in E3 |
+| **V2** | **`first_occurrence` follows document order**, and ranking ties break on document order | The prototype awarded the +1.0 to whichever chunk a `set` iteration yielded first, making the bonus arbitrary and the ranking unreproducible across runs. "First occurrence" should mean the term's first appearance in the document | Determinism locked by test; verified stable across `PYTHONHASHSEED` 0/1/7/42/99 |
+| **V3** | **`source_wording` field added** to the glossary entry | The control for the literal-source-wording defect (§7). Not yet populated — needs the seam to carry the heading verbatim and the extraction prompt to fill it | No |
+
+**Both V1 and V2 change which chunks reach the paid stage**, so they can move E3
+when the extraction branch lands. Re-score against
+`eval/knowledge/results/baseline_prototype_2026-08-13_145132.json` at that point
+and record the delta rather than assuming the improvement carries.
+
+**V1 and the R1 literal-wording defect share a root cause:** the heading is a
+separate field from the chunk text, so anything that reasons over "the term and
+its definition together" has to be told to look at both.

pyproject.toml

@@ -126,9 +126,9 @@ ignore = [
 "tests/**" = ["S101", "S105", "S106", "S608"]
 # T201: eval/ scripts are CLIs — print() is their intended output channel.
 "eval/**" = ["T201"]
-# Same rule, same reason: scripts/ are operator-run CLIs. (The existing scripts
-# already relied on this; it was simply never declared.)
-"scripts/**" = ["T201"]
+# Same rule, same reason: this is an operator-run CLI, and its printed output IS
+# the deliverable.
+"src/knowledge_extraction/cli.py" = ["T201"]
 
 [tool.mypy]
 python_version = "3.12"

src/knowledge_extraction/cli.py

@@ -0,0 +1,148 @@
+"""Offline runner for the knowledge-extraction pipeline (free stages only).
+
+The pipeline runs a few times a year, triggered by an admin — so a script over a
+parsed artifact is the honest entry point, and any HTTP surface is a convenience
+layer over this, never the other way round.
+
+Takes a **parsed-document artifact**, never a PDF: extraction does not parse.
+Every stage writes its own JSON so a later stage can be re-run without repeating
+an earlier one — which matters because prompt iteration is the main development
+loop and re-running the span filter is the slow part.
+
+Currently covers the free stages: adapter -> cue/legend filters -> cluster ->
+evidence ranking. The paid extraction stage is not built yet.
+
+    uv run --no-sync python -m src.knowledge_extraction.cli <artifact.json>
+    uv run --no-sync python -m src.knowledge_extraction.cli <artifact.json> --out-dir out/
+
+Lives inside the package rather than in `scripts/`, which is gitignored: this
+runner is the pipeline's operator entry point and has to ship with the module.
+
+Add --mentions <file.json> to supply span-NER mentions from elsewhere; without
+it the runner clusters only what the legend filter found, which is enough to
+exercise the wiring but is not a recall measurement.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+from .adapter import parsed_doc_from_artifact
+from .cluster import cluster_mentions
+from .filters import abbrev_pairs, rule_candidates
+from .models import Mention
+from .rank import rank_evidence
+from .settings import EVIDENCE_K
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    parser.add_argument("artifact", type=Path, help="parsed-document artifact JSON")
+    parser.add_argument("--mentions", type=Path, help="span-NER mentions JSON")
+    parser.add_argument("--out-dir", type=Path, default=Path("out/knowledge"))
+    parser.add_argument("--doc-id", help="override the artifact's doc_id")
+    args = parser.parse_args(argv)
+
+    if not args.artifact.exists():
+        print(f"artifact not found: {args.artifact}", file=sys.stderr)
+        return 2
+
+    raw = json.loads(args.artifact.read_text(encoding="utf-8"))
+    doc = parsed_doc_from_artifact(
+        raw, doc_id=args.doc_id, source_ref=str(args.artifact)
+    )
+    print(
+        f"[parse ] {doc.doc_id}: {len(doc.chunks)} chunks, {doc.n_pages} pages, "
+        f"hash {doc.content_hash}, heading-split={doc.used_heading_split}"
+    )
+
+    pairs = abbrev_pairs(doc.chunks)
+    rules = rule_candidates(doc.chunks)
+    print(f"[filter] {len(pairs)} abbreviation pairs, {len(rules)} rule candidates")
+    for pair in pairs[:8]:
+        print(f"         {pair.abbrev} -> {pair.expansion}")
+
+    mentions = _load_mentions(args.mentions) if args.mentions else _from_pairs(doc, pairs)
+    print(f"[filter] {len(mentions)} mentions in")
+
+    clustered = cluster_mentions(mentions, pairs, doc.doc_id)
+    print(
+        f"[cluster] {clustered.n_mentions} mentions -> {clustered.n_clusters} clusters "
+        f"(compression {clustered.compression_ratio}x)"
+    )
+
+    rank_evidence(clustered.clusters, doc.chunks)
+    print(f"[rank  ] evidence ranked, K={EVIDENCE_K}")
+    for cluster in clustered.clusters[:10]:
+        top = cluster.evidence_chunk_ids[:EVIDENCE_K]
+        print(
+            f"         {cluster.canonical:<28} mentions={cluster.mention_count:<4} "
+            f"evidence={len(cluster.evidence_chunk_ids)} top={top}"
+        )
+
+    args.out_dir.mkdir(parents=True, exist_ok=True)
+    _dump(args.out_dir / f"{doc.doc_id}.chunks.json", doc.model_dump(mode="json"))
+    _dump(
+        args.out_dir / f"{doc.doc_id}.filters.json",
+        {
+            "doc_id": doc.doc_id,
+            "abbrev_pairs": [p.model_dump(mode="json") for p in pairs],
+            "rule_candidates": [r.model_dump(mode="json") for r in rules],
+        },
+    )
+    _dump(args.out_dir / f"{doc.doc_id}.clusters.json", clustered.model_dump(mode="json"))
+    print(f"[write ] {args.out_dir}")
+    return 0
+
+
+def _load_mentions(path: Path) -> list[Mention]:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    items = raw.get("mentions", raw) if isinstance(raw, dict) else raw
+    return [Mention.model_validate(m) for m in items]
+
+
+def _from_pairs(doc, pairs) -> list[Mention]:
+    """Stand-in mentions from legend abbreviations, so the wiring is runnable
+    before the span filter lands.
+
+    NOT a recall measurement — it only sees terms a legend block already named.
+
+    Word-boundary matching, never substring: "PA" occurs inside "parameter",
+    "pada", "capacity" and "composite", and substring matching produced 126
+    spurious PA mentions on a 9-page document (77x compression instead of the
+    measured 2.56x). Same trap the evidence ranker documents for headings.
+    """
+    surfaces = {p.abbrev for p in pairs} | {p.expansion for p in pairs}
+    patterns = [
+        (s, re.compile(rf"(?<!\w){re.escape(s)}(?!\w)", re.IGNORECASE))
+        for s in surfaces
+    ]
+    out: list[Mention] = []
+    for chunk in doc.chunks:
+        for surface, pattern in patterns:
+            for match in pattern.finditer(chunk.text):
+                out.append(
+                    Mention(
+                        surface=surface,
+                        chunk_id=chunk.chunk_id,
+                        char_start=match.start(),
+                        char_end=match.end(),
+                        label="legend",
+                        score=1.0,
+                    )
+                )
+    return out
+
+
+def _dump(path: Path, payload) -> None:
+    path.write_text(
+        json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
+    )
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

src/knowledge_extraction/rank/evidence.py

@@ -34,18 +34,26 @@ def rank_evidence(
     and `evidence_scores`. Keeps the FULL ranked list, not just the top K."""
     w = {**EVIDENCE_WEIGHTS, **(weights or {})}
     by_id = {c.chunk_id: c for c in chunks}
+    # Document order, so "first occurrence" means the term's first appearance in
+    # the document — not whichever element a set happened to yield first, which
+    # would hand the +1.0 to an arbitrary chunk and make ranking unreproducible.
+    doc_order = {c.chunk_id: i for i, c in enumerate(chunks)}
     first_seen: set[str] = set()
 
     for cluster in clusters:
+        chunk_ids = sorted(
+            {m.chunk_id for m in cluster.mentions},
+            key=lambda cid: (doc_order.get(cid, len(chunks)), cid),
+        )
         scored: list[tuple[float, str]] = []
-        for chunk_id in {m.chunk_id for m in cluster.mentions}:
+        for chunk_id in chunk_ids:
             chunk = by_id.get(chunk_id)
             if chunk is None:
                 continue
             positions = [m for m in cluster.mentions if m.chunk_id == chunk_id]
             scored.append((_score_chunk(chunk, cluster, positions, w, first_seen), chunk_id))
-        # Tie-break on chunk_id so ranking is stable across runs.
-        scored.sort(key=lambda t: (-t[0], t[1]))
+        # Tie-break on document order so ranking is stable across runs.
+        scored.sort(key=lambda t: (-t[0], doc_order.get(t[1], len(chunks)), t[1]))
         cluster.evidence_chunk_ids = [cid for _, cid in scored]
         cluster.evidence_scores = [round(s, 2) for s, _ in scored]
 
@@ -78,24 +86,38 @@ def _score_chunk(
 ) -> float:
     score = 0.0
 
-    cues = definitional_hits(chunk.text)
-    for mention in positions:
-        if any(
-            abs(cs - mention.char_start) <= CUE_PROXIMITY_CHARS
-            or abs(ce - mention.char_end) <= CUE_PROXIMITY_CHARS
-            for cs, ce in cues
-        ):
-            score += w["definitional_cue_near"]
-            break
-
     # Word-boundary match, NOT substring. "PA" is a substring of "Parameter",
     # "pada" and "composite", so substring matching handed this bonus to almost
     # every Indonesian heading and pushed real definition sections below
     # formula tables.
     heading = normalize(chunk.heading or "")
-    if heading and any(_word_match(normalize(v), heading) for v in cluster.variants):
+    term_in_heading = bool(heading) and any(
+        _word_match(normalize(v), heading) for v in cluster.variants
+    )
+    if term_in_heading:
         score += w["term_in_heading"]
 
+    # Cue proximity. When the heading names the term, the heading counts as a
+    # mention at offset 0.
+    #
+    # Indonesian standards are written "2.1.3 Physical of Availability (PA)" /
+    # "Adalah ketersediaan fisik..." — the heading names the term and the body
+    # opens with the cue WITHOUT repeating it. Measured on the BUMA standard:
+    # without this, the PA definition section scored no cue bonus at all and
+    # ranked below a computation section that merely mentions PA often. The
+    # section's own definition is the single most valuable piece of evidence
+    # there is, so losing the signal there defeats the ranking.
+    cues = definitional_hits(chunk.text)
+    anchors = [(m.char_start, m.char_end) for m in positions]
+    if term_in_heading:
+        anchors.append((0, 0))
+    if any(
+        abs(cs - start) <= CUE_PROXIMITY_CHARS or abs(ce - end) <= CUE_PROXIMITY_CHARS
+        for cs, ce in cues
+        for start, end in anchors
+    ):
+        score += w["definitional_cue_near"]
+
     legend_lines = find_legend_lines(chunk.text)
     if legend_lines:
         line_starts = _line_starts(chunk.text)