[NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking

First slice of extraction v2, rebuilt in this repo (the kex prototype is not
being ported). All stages here are free: zero API calls, CPU only.

New module src/knowledge_extraction/ (named to avoid colliding with
src/knowledge/, the existing OCR->pgvector service, which is untouched):

- models.py pydantic v2 contracts. Encodes the three invariants: content
fields Optional, subdomain_tags as an enum, provenance.span
mandatory. Adds source_wording so the literal source wording
survives instead of being normalised away.
- adapter.py THE ONLY file that knows the seam's shape. Stages depend on an
internal Chunk subset, so the artifact contract can settle
either way without touching a stage. Verified against both
live shapes: Sofhia's bare-list draft with its `kind`
discriminator, and the document-level envelope proposed in the
checklist. Never accepts a file path; never reflows text.
- settings.py every tuned constant, each with the reason it has that value.
- filters/ cue_filter (rule candidates + the ranker's cue signal),
legend_filter (abbreviation pairs; runs BEFORE clustering, or
PA and Physical Availability never meet).
- cluster/ normalise + cluster. Fuzzy 92, disabled below 5 chars because
PA and UA score highly against each other.
- rank/ six-signal evidence ranking; full ranked list retained because
escalation consumes the tail.

Config: labels.yaml + cues.yaml carried over verbatim, so label phrasing stays
tunable without a code change.

src/config/settings.py: expose the __54n Azure quad (all four branches route to
nano per the model-tier decision). Mirrors the existing __54m block.

Dependency: rapidfuzz>=3.14.5 (approved). gliner/torch deliberately deferred —
not needed until the span filter, and the heavy tail is worth delaying.

pyproject: declare the T201 per-file-ignore for scripts/ that eval/ already has,
same reason (operator CLIs; print is the output channel). The existing scripts
already relied on it.

Verification: ruff clean on all new paths; PYTHONPATH=. python -c "import main"
exits 0; full suite 467 passed, 7 skipped (was 456 + 11 new local tests).
Ranking verified against the real 9-page BUMA document — every term's top-ranked
evidence chunk is its own definition section (PA->2.1.3, UA->2.1.4, Qty->2.1.2,
Pty->2.1.5), and the legend filter extracts the p.2 block exactly, 5 of 5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KNOWLEDGE_PIPELINE_TODO.md

@@ -133,22 +133,26 @@ detection with type labels. Presented to the execs 13 Aug as the prerequisite fo
 it. *Build* = exists in this repo. The gap between them is the remaining engineering effort, and
 collapsing the two is how a rebuild gets under-estimated.
 
-**Structural target:** follow `src/query/` (typed models → validator → deterministic stages →
-service facade → thin entry points), with the offline runner in `scripts/`. **Naming collision:**
-`src/knowledge/` is already taken by the existing OCR→pgvector service — v2 needs its own module
-name (`src/kex/`, `src/knowledge_extraction/`, TBD in §6).
+**Module: `src/knowledge_extraction/`** (decided 2026-08-19 — `src/knowledge/` is the existing
+OCR→pgvector service and stays untouched). Offline runner: `scripts/run_knowledge_extraction.py`.
+Structure follows `src/query/`: typed models → deterministic stages → thin entry point.
+
+**The seam is absorbed by one file.** `adapter.py` is the only module that knows the artifact's
+shape; every stage depends on the internal `models.Chunk` subset instead. Verified against **both**
+live shapes — Sofhia's bare-list draft (`contoh-chunk.json`, `kind` discriminator) and the
+envelope shape proposed in §3 — so S1 can settle either way without touching a stage.
 
 | # | Stage | Design | Build | Note |
 |---|---|---|---|---|
-| **X1** | Candidate-entry schema | ✅ | ⬜ | Three invariants to re-encode: all content fields Optional (a required field is a fabrication invitation), `subdomain_tags` as an enum, `provenance.span` mandatory. Pydantic v2 |
-| **X2** | Offline runner over a fixture | ✅ | ⬜ | Script-driven run is the honest first milestone; HTTP is a convenience layer over it, never the reverse. Every stage reads/writes its own artifact so extraction re-runs without re-running GLiNER |
-| **X3** | GLiNER span filter | ✅ | ⬜ | Runs on **all** chunks unconditionally. No standalone relevance gate — "zero candidate spans" is the relevance signal, produced as a byproduct of looking rather than a judgment made before looking |
+| **X1** | Candidate-entry schema | ✅ | ✅ | `models.py`. All three invariants encoded: content fields Optional, `SubdomainEnum`, `Provenance.span` mandatory. Adds `source_wording` for R1. Pydantic v2 |
+| **X2** | Offline runner over a fixture | ✅ | ✅ | `scripts/run_knowledge_extraction.py`. Takes an artifact, **never a PDF**; each stage writes its own JSON. Runs green on the 9-page BUMA chunks and on Sofhia's draft |
+| **X3** | GLiNER span filter | ✅ | ⬜ | **Next.** Runs on all chunks unconditionally; "zero candidate spans" is the relevance signal. Deferred with the `gliner`+`torch` dependency (D1); the runner has a legend-only stand-in meanwhile, explicitly not a recall measurement |
 | **X4** | Indonesian performance | ✅ **0.854** | n/a | Answered by E1. Re-measure once on v2 to confirm no regression |
-| **X5** | Discourse-cue filter → rule-of-thumb | ✅ | ⬜ | Free CPU pass. Grounding case: Other Activity + the controllability hierarchy |
-| **X6** | Legend-block filter → formula vars | ✅ | ⬜ | Grounding case: the MOHH/Qty/PA/UA/Pty legend block. Needs the LaTeX normalisation noted in §4 |
+| **X5** | Discourse-cue filter → rule-of-thumb | ✅ | ✅ | `filters/cue_filter.py`, cues in `config/cues.yaml`. 9 rule candidates on the BUMA standard. Also supplies `definitional_hits` for the ranker's +5.0 signal |
+| **X6** | Legend-block filter → formula vars | ✅ | ✅ | `filters/legend_filter.py`. Extracts the p.2 legend block **exactly** — MOHH · Qty · PA · UA · Pty, 5/5 — and rejects equation rows. LaTeX normalisation (§4) still pending for MinerU input |
 | **X7** | Section pass → summary units | ✅ | ⬜ | The quiet cost risk: few calls but ~¼ of all input tokens, because summarisation cannot be filtered — it needs whole documents |
-| **X8** | Normalise + cluster mentions | ✅ **2.56×** | ⬜ | The LLM call unit is the **term cluster**. Carry the tuned constants from P2 |
-| **X9** | Evidence ranking → top-K | ✅ | ⬜ | Six signals: cue · heading · legend · formula · formatting · first-occurrence. No term is ever dropped — ranking narrows *evidence*, not terms, and the full ranked list is retained for escalation |
+| **X8** | Normalise + cluster mentions | ✅ **2.56×** | ✅ | `cluster/`. Constants carried from P2 with their reasons. Locked by tests: PA/UA never merge; abbreviation↔expansion merges only with legend pairs; noise surfaces dropped as whole forms only |
+| **X9** | Evidence ranking → top-K | ✅ | ✅ | `rank/evidence.py`, six signals + tabular penalty. **Verified on the real document: every term's top-ranked chunk is its own definition section** — PA→2.1.3, UA→2.1.4, Qty→2.1.2, Pty→2.1.5. Full ranked list retained; word-boundary heading matching locked by test |
 | **X10** | LLM extraction call | ✅ | ⬜ | One call per term cluster, plus one per cue chunk, per unique formula, per document. Keep prompts as files with a byte-identical cached prefix |
 | **X11** | Verbatim-span validation | ✅ **1.00** | ⬜ | A field whose span cannot be located is rejected, **never repaired** — a repaired span is an unfalsifiable claim, which is what the control exists to prevent |
 | **X12** | Null-definition escalation | 🔎 | ⬜ | Built in the prototype but **never fired**: 54 of 66 clusters had exactly 1 evidence chunk against K=3, so the loop correctly had nowhere to escalate. Unverified until a larger corpus runs |
@@ -164,11 +168,11 @@ name (`src/kex/`, `src/knowledge_extraction/`, TBD in §6).
 
 | # | Decision | Owner | Status | Description |
 |---|---|---|---|---|
-| **D1** | New dependencies | Rifqi (asks) | ⬜ | GLiNER + torch (heavy) for extraction; MinerU for parsing. `pyproject.toml` changes need sign-off. Prototype needed **torch ≥ 2.6** and hit a corrupt-wheel failure worth not re-debugging |
+| **D1** | New dependencies | Rifqi | 🔄 | **`rapidfuzz>=3.14.5` added 2026-08-19** (approved). `gliner` + `torch` still **deferred** — not needed until X3, and the heavy tail is worth delaying. `PyYAML` was already present transitively. Prototype needed **torch ≥ 2.6** and hit a corrupt-wheel failure worth not re-debugging |
 | **D2** | New tables (DDL handoff) | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns the dedorch schema — Python never executes DDL. One consolidated Harry-ready handoff beats three |
 | **D3** | Model tier per branch | Rifqi | ✅ | **Decided 2026-08-19: stay on nano for all four branches.** The `.env` carries the quad `azureai__{api_key,endpoint__url,deployment__name,api__version}__54n`, mirroring the existing `__54m` scheme. Accepted knowingly: nano measured 0.75 against a 0.80 line, and `rule`/`summary` run on the tier whose failure mode is least detectable. Revisit if extraction quality blocks the demo. **`src/config/settings.py` does not expose the `__54n` quad yet** — four `Field(alias=…)` entries, first build step |
 | **D4** | Admin ingest surface | Rifqi | ⬜ | Whether it is HTTP at all, and if so its auth posture. A write surface triggered by an admin is a different risk class from the current unauthenticated read-only chat surface — raise it, don't inherit the posture by default. Only after the offline path works |
-| **D5** | v2 module name | Rifqi | ⬜ | `src/knowledge/` is taken by the existing OCR→pgvector service. Pick a non-colliding name before the first commit |
+| **D5** | v2 module name | Rifqi | ✅ | **`src/knowledge_extraction/`** (2026-08-19). Offline runner in `scripts/`, per the repo's existing precedent |
 | **D6** | Expert review of the gold set | Rifqi → Mas Beta | ⏸️ | **Deferred 2026-08-19: provisional gold is good enough for now** so the rebuild is not blocked on expert time. Standing caveat: E1 and E3 both move when it is reviewed, and E3's scoreable base is 8 entries, so each adjudication shifts it ~12 points. Every E3 failure is listed verbatim in the frozen baseline for when he does review it |
 | **D7** | GLiNER + LangExtract ensembling | — | ⏸️ | Proposed, not decided; flagged as scope creep. Cheap in API terms but adds a reconciliation step between two candidate sets that may disagree. Not in scope for the test integration |
 

pyproject.toml

@@ -90,6 +90,7 @@ dependencies = [
     "pytesseract>=0.3.13",
     "pypdf2>=3.0.1",
     "pyarrow>=24.0.0",
+    "rapidfuzz>=3.14.5",
 ]
 
 [project.optional-dependencies]
@@ -125,6 +126,9 @@ ignore = [
 "tests/**" = ["S101", "S105", "S106", "S608"]
 # T201: eval/ scripts are CLIs — print() is their intended output channel.
 "eval/**" = ["T201"]
+# Same rule, same reason: scripts/ are operator-run CLIs. (The existing scripts
+# already relied on this; it was simply never declared.)
+"scripts/**" = ["T201"]
 
 [tool.mypy]
 python_version = "3.12"

src/config/settings.py

@@ -47,6 +47,13 @@ class Settings(BaseSettings):
     azureai_deployment_name_54m: str = Field(alias="azureai__deployment__name__54m", default="")
     azureai_api_version_54m: str = Field(alias="azureai__api__version__54m", default="")
 
+    # Azure OpenAI - GPT-5.4-nano (knowledge-extraction pipeline; all four
+    # branches route here — see KNOWLEDGE_PIPELINE_TODO.md D3)
+    azureai_api_key_54n: str = Field(alias="azureai__api_key__54n", default="")
+    azureai_endpoint_url_54n: str = Field(alias="azureai__endpoint__url__54n", default="")
+    azureai_deployment_name_54n: str = Field(alias="azureai__deployment__name__54n", default="")
+    azureai_api_version_54n: str = Field(alias="azureai__api__version__54n", default="")
+
     # Azure OpenAI - Embeddings
     azureai_api_key_embedding: str = Field(alias="azureai__api_key__embedding", default="")
     azureai_endpoint_url_embedding: str = Field(alias="azureai__endpoint__url__embedding", default="")

src/knowledge_extraction/__init__.py

@@ -0,0 +1,46 @@
+"""Knowledge-extraction pipeline: parsed document -> candidate knowledge entries.
+
+Distinct from `src/knowledge/`, which is the existing OCR -> chunk -> pgvector
+ingestion path for unstructured RAG. This package does not touch it.
+
+Stage order, and which stages cost money:
+    adapter   seam artifact -> internal Chunk          free
+    filters   cue / legend / span NER -> mentions      free (CPU)
+    cluster   normalise + cluster mentions             free
+    rank      evidence scoring, top-K selection        free
+    extract   one LLM call per TERM CLUSTER            PAID  (not yet built)
+    validate  verbatim span check, escalation          free  (not yet built)
+    diff      new / duplicate / conflicting            free  (not yet built)
+    queue     frequency-sorted review queue            free  (not yet built)
+
+Design rationale: knowledge_pipeline_context.md
+Calibrated constants and why: KNOWLEDGE_PIPELINE_CALIBRATION.md
+"""
+
+from .adapter import parsed_doc_from_artifact
+from .models import (
+    AbbrevPair,
+    Chunk,
+    ClusterResult,
+    FilterResult,
+    GlossaryEntry,
+    Mention,
+    ParsedDoc,
+    Provenance,
+    RuleCandidate,
+    TermCluster,
+)
+
+__all__ = [
+    "AbbrevPair",
+    "Chunk",
+    "ClusterResult",
+    "FilterResult",
+    "GlossaryEntry",
+    "Mention",
+    "ParsedDoc",
+    "Provenance",
+    "RuleCandidate",
+    "TermCluster",
+    "parsed_doc_from_artifact",
+]

src/knowledge_extraction/adapter.py

@@ -0,0 +1,117 @@
+"""Seam adapter: parsed-document artifact → the pipeline's internal `Chunk`.
+
+**This is the only file that knows the seam's shape.** Every stage downstream
+depends on `models.Chunk` alone, so when the artifact contract settles with
+Sofhia the change lands here and nowhere else.
+
+The seam is still under discussion (KNOWLEDGE_PIPELINE_TODO.md §3), so this
+reads defensively: it accepts either the draft's field names or the prototype's,
+takes plain dicts, and tolerates missing optional structure. It deliberately
+does **not** accept a file path — extraction never opens a document. That
+constraint is the point of the split, not an implementation detail.
+
+Two things it must never do:
+  - reflow, strip or whitespace-normalise `text`. Span validation locates quoted
+    spans literally inside it; cleaning the text makes the lookup fail and the
+    field go silently null, which presents as a bad model.
+  - infer a page number it was not given. A wrong page sends the reviewer to the
+    wrong part of the document, which is worse than no page at all.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from typing import Any
+
+from .models import Chunk, ParsedDoc
+
+# Draft field name → internal field name, where they differ.
+_PAGE_KEYS = ("page_start", "page", "page_idx")
+_PAGES_KEYS = ("pages", "page_list")
+
+
+def chunk_from_dict(raw: dict[str, Any], doc_id: str, ordinal: int = 0) -> Chunk:
+    """Map one artifact item onto the internal chunk.
+
+    `kind`/`is_tabular` are reconciled: the draft carries a `kind` discriminator
+    while the prototype carried booleans. Either is accepted.
+    """
+    kind = raw.get("kind")
+    pages = _first(raw, _PAGES_KEYS) or []
+    page_start = _first(raw, _PAGE_KEYS)
+    if page_start is None:
+        page_start = min(pages) if pages else 0
+    page_end = max(pages) if pages else raw.get("page_end", page_start)
+
+    return Chunk(
+        chunk_id=raw.get("chunk_id") or f"{doc_id}#{ordinal:04d}",
+        doc_id=raw.get("doc_id") or doc_id,
+        text=raw["text"],  # verbatim, never cleaned
+        page_start=int(page_start),
+        page_end=int(page_end),
+        ordinal=int(raw.get("ordinal", ordinal)),
+        section_no=raw.get("section_no"),
+        heading=raw.get("heading"),
+        has_formula=bool(raw.get("has_formula", kind == "equation")),
+        is_tabular=bool(raw.get("is_tabular", kind == "table")),
+        bold_spans=list(raw.get("bold_spans") or []),
+    )
+
+
+def parsed_doc_from_artifact(
+    artifact: Any,
+    doc_id: str | None = None,
+    source_ref: str = "",
+    parser_name: str = "unknown",
+    parser_version: str = "",
+) -> ParsedDoc:
+    """Build a `ParsedDoc` from either shape of the artifact.
+
+    Accepts a bare `list[chunk]` (the draft's current shape) or a mapping with a
+    `chunks` key (the shape proposed for the document-level envelope). When the
+    envelope lands, its `content_hash`/`n_pages`/`version` are preferred over
+    the values derived here.
+    """
+    if isinstance(artifact, dict):
+        items = artifact.get("chunks") or []
+        doc_id = doc_id or artifact.get("doc_id")
+        source_ref = source_ref or artifact.get("source_path") or artifact.get("source_ref") or ""
+        parser_name = artifact.get("parser_name") or parser_name
+        parser_version = artifact.get("parser_version") or parser_version
+        declared_hash = artifact.get("content_hash")
+        declared_pages = artifact.get("n_pages")
+    else:
+        items = list(artifact)
+        declared_hash, declared_pages = None, None
+
+    if not doc_id:
+        doc_id = (items[0].get("doc_id") if items else None) or "unknown"
+
+    chunks = [chunk_from_dict(raw, doc_id, i) for i, raw in enumerate(items)]
+    pages = {p for c in chunks for p in (c.page_start, c.page_end)}
+
+    return ParsedDoc(
+        doc_id=doc_id,
+        source_ref=source_ref,
+        content_hash=declared_hash or content_hash(chunks),
+        n_pages=int(declared_pages) if declared_pages else (max(pages) + 1 if pages else 0),
+        chunks=chunks,
+        parser_name=parser_name,
+        parser_version=parser_version,
+        used_heading_split=any(c.section_no for c in chunks),
+    )
+
+
+def content_hash(chunks: list[Chunk]) -> str:
+    """Stable hash of the chunk text, so a re-parse that changed nothing can be
+    detected and the expensive stages skipped."""
+    blob = json.dumps([c.text for c in chunks], ensure_ascii=False).encode()
+    return hashlib.sha256(blob).hexdigest()[:16]
+
+
+def _first(raw: dict[str, Any], keys: tuple[str, ...]) -> Any:
+    for key in keys:
+        if raw.get(key) is not None:
+            return raw[key]
+    return None

src/knowledge_extraction/cluster/__init__.py

@@ -0,0 +1,4 @@
+from .cluster import cluster_mentions
+from .normalize import AbbrevIndex, is_noise, normalize
+
+__all__ = ["AbbrevIndex", "cluster_mentions", "is_noise", "normalize"]

src/knowledge_extraction/cluster/cluster.py

@@ -0,0 +1,111 @@
+"""Mention[] → TermCluster[].
+
+Matching order, cheapest and most certain first:
+    1. exact match after normalisation
+    2. abbreviation ↔ expansion (from legend blocks)
+    3. conservative fuzzy (token_set_ratio >= 92, and only above 5 characters)
+
+**Over-merging is much worse than under-merging.** An under-merge costs one
+extra LLM call and one extra review-queue row. A wrong merge silently destroys a
+distinct term, and no downstream stage recovers it — the expert never learns the
+term existed. Every threshold here is set on that asymmetry.
+
+Clustering is also what makes conflict detection possible at all: contradictory
+definitions can only be compared if all evidence for a term reaches the same
+call.
+"""
+
+from __future__ import annotations
+
+from rapidfuzz import fuzz
+
+from ..models import AbbrevPair, ClusterResult, Mention, TermCluster
+from ..settings import FUZZY_MIN_LEN, FUZZY_THRESHOLD
+from .normalize import AbbrevIndex, is_noise
+
+
+def cluster_mentions(
+    mentions: list[Mention],
+    abbrev_pairs: list[AbbrevPair],
+    doc_id: str,
+    fuzzy_threshold: int = FUZZY_THRESHOLD,
+) -> ClusterResult:
+    index = AbbrevIndex(abbrev_pairs)
+    kept = [m for m in mentions if not is_noise(m.surface)]
+
+    buckets: dict[str, dict] = {}
+
+    for mention in kept:
+        key = index.canonical_key(mention.surface)
+
+        if key in buckets:
+            _add(buckets[key], mention, "exact")
+            continue
+
+        linked = next((k for k in buckets if index.linked(mention.surface, k)), None)
+        if linked:
+            _add(buckets[linked], mention, "abbrev")
+            continue
+
+        match = _fuzzy_match(key, buckets.keys(), fuzzy_threshold)
+        if match:
+            _add(buckets[match], mention, "fuzzy")
+            continue
+
+        buckets[key] = {
+            "surfaces": {mention.surface},
+            "mentions": [mention],
+            "reasons": set(),
+        }
+
+    clusters: list[TermCluster] = []
+    ordered = sorted(buckets.items(), key=lambda kv: -len(kv[1]["mentions"]))
+    for i, (key, data) in enumerate(ordered):
+        variants = sorted(data["surfaces"], key=lambda s: (len(s), s))
+        clusters.append(
+            TermCluster(
+                cluster_id=f"{doc_id}#c{i:03d}",
+                canonical=_canonical(variants, key),
+                variants=variants,
+                mentions=data["mentions"],
+                mention_count=len(data["mentions"]),
+                merge_reasons=sorted(data["reasons"]),
+            )
+        )
+
+    n_mentions, n_clusters = len(kept), len(clusters)
+    return ClusterResult(
+        doc_id=doc_id,
+        clusters=clusters,
+        n_mentions=n_mentions,
+        n_clusters=n_clusters,
+        compression_ratio=round(n_mentions / n_clusters, 3) if n_clusters else 0.0,
+    )
+
+
+def _add(bucket: dict, mention: Mention, reason: str) -> None:
+    bucket["surfaces"].add(mention.surface)
+    bucket["mentions"].append(mention)
+    bucket["reasons"].add(reason)
+
+
+def _fuzzy_match(key: str, existing, threshold: int) -> str | None:
+    best, best_score = None, 0.0
+    for other in existing:
+        # Short strings fuzzy-match far too easily: "PA" vs "UA" scores high on
+        # token_set_ratio. Below FUZZY_MIN_LEN only exact matching is allowed.
+        if min(len(key), len(other)) < FUZZY_MIN_LEN:
+            continue
+        score = fuzz.token_set_ratio(key, other)
+        if score >= threshold and score > best_score:
+            best, best_score = other, score
+    return best
+
+
+def _canonical(variants: list[str], key: str) -> str:
+    """Prefer the shortest non-trivial surface — usually the term as a reader
+    would look it up ("PA", not "Physical Availability (PA) untuk ...")."""
+    for v in variants:
+        if len(v) >= 2:
+            return v
+    return key

src/knowledge_extraction/cluster/normalize.py

@@ -0,0 +1,77 @@
+"""Surface normalisation and the abbreviation index used by clustering.
+
+**This normalisation is for clustering only.** Span validation normalises
+whitespace and nothing else — every additional normalisation there is a hole a
+fabrication can fit through. Do not reuse `normalize()` in that path.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+
+from ..models import AbbrevPair
+
+# Surfaces that carry no discriminating power on their own. A mention of just
+# "unit" or "parameter" is not a term. These are dropped as WHOLE surface forms
+# only, never as substrings — so no term containing them is ever lost.
+STOP_SURFACES = {
+    "unit",
+    "type",
+    "class",
+    "equipment",
+    "equipment unit",
+    "parameter",
+    "activity",
+    "data",
+    "nilai",
+    "proses",
+    "hasil",
+    "total",
+}
+
+
+def normalize(surface: str) -> str:
+    s = unicodedata.normalize("NFKC", surface).casefold()
+    s = s.replace("-", " ").replace("_", " ")
+    s = re.sub(r"[.’']", "", s)
+    s = re.sub(r"[^\w\s/()]", " ", s)
+    s = re.sub(r"\s+", " ", s)
+    return s.strip(" ()/")
+
+
+def is_noise(surface: str) -> bool:
+    n = normalize(surface)
+    if len(n) < 2:
+        return True
+    if n in STOP_SURFACES:
+        return True
+    return not re.search(r"[a-z]", n)  # pure numbers / symbols
+
+
+class AbbrevIndex:
+    """Bidirectional abbreviation ↔ expansion lookup built from legend blocks.
+
+    This is why the legend filter runs before clustering: without it, `PA` and
+    `Physical Availability` never meet.
+    """
+
+    def __init__(self, pairs: list[AbbrevPair]):
+        self.to_expansion: dict[str, str] = {}
+        self.to_abbrev: dict[str, str] = {}
+        for pair in pairs:
+            abbrev, expansion = normalize(pair.abbrev), normalize(pair.expansion)
+            if not abbrev or not expansion:
+                continue
+            self.to_expansion[abbrev] = expansion
+            self.to_abbrev[expansion] = abbrev
+
+    def canonical_key(self, surface: str) -> str:
+        """Map a surface to a shared key so an abbreviation and its expansion
+        collide into the same bucket."""
+        n = normalize(surface)
+        return self.to_abbrev.get(n, n)
+
+    def linked(self, a: str, b: str) -> bool:
+        na, nb = normalize(a), normalize(b)
+        return self.to_expansion.get(na) == nb or self.to_expansion.get(nb) == na

src/knowledge_extraction/config/cues.yaml

@@ -0,0 +1,51 @@
+# Discourse cues (definitional) and legend-block patterns.
+# Used by cue_filter (rule candidates) and evidence ranking (+5.0 signal).
+
+definitional_id:
+  - '\badalah kondisi ketika\b'
+  - '\badalah\b'
+  - '\bmerupakan\b'
+  - '\byaitu\b'
+  - '\bdidefinisikan sebagai\b'
+  - '\bdimaksud dengan\b'
+
+definitional_en:
+  - '\bis defined as\b'
+  - '\brefers to\b'
+  - '\bis the\b'
+  - '\bmeans\b'
+
+conditional_id:
+  - '\bapabila\b'
+  - '\bjika\b.{0,80}\bmaka\b'
+  - '\bbila\b'
+  - '\bharus\b'
+  - '\btidak boleh\b'
+  - '\bwajib\b'
+
+conditional_en:
+  - '\bif\b.{0,80}\bthen\b'
+  - '\bmust\b'
+  - '\bshall\b'
+  - '\bNote that\b'
+
+note_id:
+  - '^\s*Catatan\b'
+  - '^\s*Sebagai contoh\b'
+  - '^\s*Contoh\b'
+
+# Legend blocks: 'Keterangan:' / 'Dimana:' / 'where' introduce symbol tables.
+legend_headers:
+  - '^\s*Keterangan\s*[::]'
+  - '^\s*Dimana\s*[::]'
+  - '^\s*Di\s*mana\s*[::]'
+  - '^\s*[Ww]here\s*[::]'
+
+legend_entry: '^\s*([A-Za-z][A-Za-z0-9_\.]{0,12})\s*[:=]\s*(.+)$'
+
+# A chunk is 'formula present' if any of these appear.
+formula_markers:
+  - '='
+  - '×'
+  - '\bx\s*100\s*%'
+  - '/'

src/knowledge_extraction/config/labels.yaml

@@ -0,0 +1,44 @@
+# GLiNER entity labels. E1 tunes these WITHOUT touching code.
+# Two variants so E1 can A/B label phrasing (GLiNER is sensitive to label wording).
+
+default:
+  - mining production metric
+  - equipment availability term
+  - operational abbreviation
+  - machine or equipment type
+  - time category
+  - production activity
+  - measurement unit
+
+# Variant B: Indonesian-language label phrasing. GLiNER multi is multilingual;
+# whether ID label strings improve ID recall is exactly what E1 tests.
+id_phrasing:
+  - istilah produksi tambang
+  - parameter ketersediaan alat
+  - singkatan operasional
+  - jenis alat berat
+  - kategori waktu kerja
+  - aktivitas produksi
+  - satuan pengukuran
+
+# Variant C: the first two variants both missed the same class of terms —
+# mining activities and materials (coal mining, land clearing, waste removal,
+# joint survey). This variant adds labels for that class specifically.
+broad:
+  - mining production metric
+  - equipment availability term
+  - operational abbreviation
+  - machine or equipment type
+  - time category
+  - mining activity
+  - material or commodity
+  - measurement unit
+  - data source or system
+  - work category
+  - analysis method
+
+# 0.25, not 0.35: the E1 sweep measured recall 0.854 @ 0.25 vs 0.658 @ 0.35 on the
+# `broad` variant. Precision falls (0.41 vs 0.50) but the term filter is meant to be
+# over-inclusive — clustering and evidence ranking absorb the noise, and a term the
+# filter never proposes can never be recovered downstream.
+threshold: 0.25

src/knowledge_extraction/filters/__init__.py

@@ -0,0 +1,9 @@
+from .cue_filter import definitional_hits, rule_candidates
+from .legend_filter import abbrev_pairs, find_legend_lines
+
+__all__ = [
+    "abbrev_pairs",
+    "definitional_hits",
+    "find_legend_lines",
+    "rule_candidates",
+]

src/knowledge_extraction/filters/cue_filter.py

@@ -0,0 +1,76 @@
+"""Discourse-cue regex → rule-of-thumb candidates. Free: zero API calls.
+
+Also supplies `definitional_hits`, which evidence ranking uses for its strongest
+signal (+5.0 when a cue sits within 100 characters of a mention). Cue patterns
+live in `config/cues.yaml` so they can be tuned per language without a code
+change.
+"""
+
+from __future__ import annotations
+
+import re
+from functools import lru_cache
+
+from ..models import Chunk, RuleCandidate
+from ..settings import load_yaml
+
+SNIPPET_CHARS = 240
+
+
+@lru_cache(maxsize=8)
+def _compiled(group: str) -> tuple[re.Pattern[str], ...]:
+    cues = load_yaml("cues.yaml")
+    return tuple(
+        re.compile(p, re.IGNORECASE | re.MULTILINE) for p in cues.get(group, [])
+    )
+
+
+def _groups(*names: str) -> tuple[re.Pattern[str], ...]:
+    out: list[re.Pattern[str]] = []
+    for name in names:
+        out.extend(_compiled(name))
+    return tuple(out)
+
+
+def definitional_hits(text: str) -> list[tuple[int, int]]:
+    """(start, end) of every definitional cue. Consumed by evidence ranking."""
+    spans: list[tuple[int, int]] = []
+    for pattern in _groups("definitional_id", "definitional_en"):
+        spans.extend((m.start(), m.end()) for m in pattern.finditer(text))
+    return sorted(spans)
+
+
+def rule_candidates(chunks: list[Chunk]) -> list[RuleCandidate]:
+    """Passages a conditional or note cue marks as possibly stating a rule.
+
+    Over-inclusive on purpose — this is a candidate generator, and the paid
+    stage decides whether a rule is actually present. Nothing is dropped here.
+    """
+    patterns = _groups("conditional_id", "conditional_en", "note_id")
+    out: list[RuleCandidate] = []
+    for chunk in chunks:
+        seen: set[int] = set()
+        for pattern in patterns:
+            for match in pattern.finditer(chunk.text):
+                start = match.start()
+                # One candidate per sentence-ish region: several cues in the
+                # same clause would otherwise produce near-identical rows.
+                bucket = start // 200
+                if bucket in seen:
+                    continue
+                seen.add(bucket)
+                out.append(
+                    RuleCandidate(
+                        chunk_id=chunk.chunk_id,
+                        cue=match.group(0).strip(),
+                        char_start=start,
+                        char_end=match.end(),
+                        snippet=_snippet(chunk.text, start),
+                    )
+                )
+    return out
+
+
+def _snippet(text: str, start: int) -> str:
+    lo = max(0, start - SNIPPET_CHARS // 4)
+    return text[lo : lo + SNIPPET_CHARS].strip()

src/knowledge_extraction/filters/legend_filter.py

@@ -0,0 +1,98 @@
+"""Legend blocks → abbreviation pairs + formula variables. Free: zero API calls.
+
+A legend block is the `Keterangan:` / `Dimana:` / `where` table that follows a
+formula and expands its symbols:
+
+    Keterangan:
+    MOHH : Machine on Hand Hours
+    Qty  : Quantity
+
+This runs **before clustering** and that ordering is load-bearing: without the
+abbreviation pairs it produces, `PA` and `Physical Availability` cluster as two
+unrelated terms and the expert sees the same concept twice.
+
+`find_legend_lines` additionally feeds evidence ranking (+3.5 when a mention
+sits inside a legend block).
+"""
+
+from __future__ import annotations
+
+import re
+from functools import lru_cache
+
+from ..models import AbbrevPair, Chunk
+from ..settings import load_yaml
+
+# How far below a legend header the block is assumed to run before a blank line
+# or a non-entry line closes it.
+MAX_BLOCK_LINES = 30
+
+
+@lru_cache(maxsize=2)
+def _headers() -> tuple[re.Pattern[str], ...]:
+    cues = load_yaml("cues.yaml")
+    return tuple(re.compile(p) for p in cues.get("legend_headers", []))
+
+
+@lru_cache(maxsize=2)
+def _entry() -> re.Pattern[str]:
+    cues = load_yaml("cues.yaml")
+    return re.compile(cues["legend_entry"])
+
+
+def find_legend_lines(text: str) -> set[int]:
+    """Zero-based line numbers that fall inside a legend block."""
+    lines = text.split("\n")
+    inside: set[int] = set()
+    for i, line in enumerate(lines):
+        if not any(h.search(line) for h in _headers()):
+            continue
+        for j in range(i + 1, min(i + 1 + MAX_BLOCK_LINES, len(lines))):
+            if not lines[j].strip():
+                break
+            if not _entry().match(lines[j]):
+                break
+            inside.add(j)
+    return inside
+
+
+def abbrev_pairs(chunks: list[Chunk]) -> list[AbbrevPair]:
+    """Extract `abbrev : expansion` rows from every legend block.
+
+    Precision matters more than recall here: a wrong pair merges two distinct
+    terms into one cluster, which destroys a term silently. So only lines
+    inside a detected block are read, never `X : Y` anywhere in the text.
+    """
+    out: list[AbbrevPair] = []
+    seen: set[tuple[str, str]] = set()
+    for chunk in chunks:
+        lines = chunk.text.split("\n")
+        for line_no in sorted(find_legend_lines(chunk.text)):
+            match = _entry().match(lines[line_no])
+            if not match:
+                continue
+            abbrev, expansion = match.group(1).strip(), match.group(2).strip()
+            if not _plausible(abbrev, expansion):
+                continue
+            key = (abbrev.casefold(), expansion.casefold())
+            if key in seen:
+                continue
+            seen.add(key)
+            out.append(
+                AbbrevPair(
+                    abbrev=abbrev, expansion=expansion, chunk_id=chunk.chunk_id
+                )
+            )
+    return out
+
+
+def _plausible(abbrev: str, expansion: str) -> bool:
+    """Reject rows that are formula lines rather than legend entries."""
+    if not abbrev or not expansion:
+        return False
+    if len(expansion) < 3 or len(expansion) > 120:
+        return False
+    # An expansion that is mostly digits/operators is the right-hand side of an
+    # equation, not a term ("Qty = 360/720").
+    letters = sum(c.isalpha() for c in expansion)
+    return letters >= max(3, len(expansion) // 3)

src/knowledge_extraction/models.py

@@ -0,0 +1,215 @@
+"""Pydantic contracts for the knowledge-extraction pipeline.
+
+Three invariants are encoded here rather than described in prose, because every
+one of them is a control that a later change could quietly remove:
+
+1. **All content fields are Optional.** A model that cannot answer null will
+   fabricate one. Abstention is correct behaviour, never an error.
+2. **`subdomain_tags` is an enum.** Classification, not generation.
+3. **`Provenance.span` is mandatory and verbatim-checked.** It is the primary
+   anti-hallucination control and the thing that makes expert review
+   finishable — the reviewer checks a quote against a page, not a claim
+   against their memory.
+
+`Chunk` here is the pipeline's **internal** unit, deliberately narrower than the
+parsed-document artifact being agreed with Sofhia (the seam). Stages depend only
+on this subset; `adapter.py` maps the seam type onto it, so seam churn lands in
+one file instead of seven. See KNOWLEDGE_PIPELINE_TODO.md §3.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+Branch = Literal["glossary", "rule", "formula", "summary"]
+ExtractionStatus = Literal["ok", "no_definition_found", "escalated"]
+DiffStatus = Literal["new", "duplicate", "conflicting"]
+
+
+class SubdomainEnum(str, Enum):
+    """Classification target. Extend deliberately — a new member changes what
+    the model is allowed to answer, which is a prompt change, not a data one."""
+
+    production = "production"
+    maintenance = "maintenance"
+    hauling = "hauling"
+    loading = "loading"
+    drilling_blasting = "drilling_blasting"
+    equipment = "equipment"
+    safety = "safety"
+    quality = "quality"
+    planning = "planning"
+    cost = "cost"
+    geology = "geology"
+    other = "other"
+
+
+# ── Stage 1: the chunk (internal view of the seam artifact) ─────────────
+
+
+class Chunk(BaseModel):
+    """One unit of a parsed document, as the extraction stages need it.
+
+    `text` must stay **verbatim** from the source document. Span validation
+    locates LLM-quoted spans literally inside this text; if it is ever reflowed
+    or whitespace-normalised the lookup fails and the field is silently set to
+    null. The failure presents as a bad model, but the cause would be here.
+    """
+
+    chunk_id: str
+    doc_id: str
+    text: str
+    page_start: int
+    page_end: int
+    ordinal: int = 0
+
+    # Structural context. Both Optional — many documents carry no numbering.
+    section_no: str | None = None
+    heading: str | None = None
+
+    # Cheap downstream filters / ranking signals
+    has_formula: bool = False
+    is_tabular: bool = False
+    bold_spans: list[str] = Field(default_factory=list)
+
+
+class ParsedDoc(BaseModel):
+    """A document's chunks plus the identity needed to version and cache them."""
+
+    doc_id: str
+    source_ref: str
+    content_hash: str
+    n_pages: int
+    chunks: list[Chunk]
+    parser_name: str = "unknown"
+    parser_version: str = ""
+    used_heading_split: bool = False
+
+
+# ── Stage 2: filters ────────────────────────────────────────────────────
+
+
+class Mention(BaseModel):
+    """One occurrence of a candidate term inside a chunk."""
+
+    surface: str
+    chunk_id: str
+    char_start: int
+    char_end: int
+    label: str = ""
+    score: float = 0.0
+    hit_span_cap: bool = False
+
+
+class RuleCandidate(BaseModel):
+    """A passage a discourse cue marks as possibly stating a rule of thumb."""
+
+    chunk_id: str
+    cue: str
+    char_start: int
+    char_end: int
+    snippet: str
+
+
+class AbbrevPair(BaseModel):
+    """`PA` ↔ `Physical Availability`, harvested from a legend block.
+
+    Legend extraction must run before clustering: without these, an
+    abbreviation and its expansion cluster as two unrelated terms.
+    """
+
+    abbrev: str
+    expansion: str
+    chunk_id: str
+
+
+class FilterResult(BaseModel):
+    doc_id: str
+    mentions: list[Mention] = Field(default_factory=list)
+    rule_candidates: list[RuleCandidate] = Field(default_factory=list)
+    abbrev_pairs: list[AbbrevPair] = Field(default_factory=list)
+
+
+# ── Stage 3: clusters ───────────────────────────────────────────────────
+
+
+class TermCluster(BaseModel):
+    """All mentions of one term. **The LLM call unit is the cluster**, not the
+    chunk and not the mention — that is what cuts expert review burden, and it
+    is also the only reason conflicting definitions can be detected at all
+    (they must arrive in the same call to be compared)."""
+
+    cluster_id: str
+    canonical: str
+    variants: list[str] = Field(default_factory=list)
+    mentions: list[Mention] = Field(default_factory=list)
+    mention_count: int = 0
+    merge_reasons: list[str] = Field(default_factory=list)
+
+    # Ranked best-first. The FULL list is kept, not just the top K —
+    # escalation consumes the tail.
+    evidence_chunk_ids: list[str] = Field(default_factory=list)
+    evidence_scores: list[float] = Field(default_factory=list)
+
+
+class ClusterResult(BaseModel):
+    doc_id: str
+    clusters: list[TermCluster] = Field(default_factory=list)
+    n_mentions: int = 0
+    n_clusters: int = 0
+    compression_ratio: float = 0.0
+
+
+# ── Stage 4+: extracted entries ─────────────────────────────────────────
+
+
+class Provenance(BaseModel):
+    """Where a claim came from. `span` is mandatory and must appear verbatim in
+    the evidence text; a field whose span cannot be located is rejected, never
+    repaired. A repaired span is an unfalsifiable claim."""
+
+    doc_id: str
+    span: str
+    page: int | None = None
+    section_no: str | None = None
+    chunk_id: str | None = None
+
+
+class GlossaryEntry(BaseModel):
+    term: str
+    full_name: str | None = None
+
+    # The literal wording as the document writes it, un-normalised. The BUMA
+    # standard heads its section "Physical of Availability (PA)" while the
+    # legend says "Physical Availability"; the discrepancy is surfaced to the
+    # expert rather than silently corrected.
+    source_wording: str | None = None
+
+    definition: str | None = None
+    formula_latex: str | None = None
+    interpretation: str | None = None
+    subdomain_tags: list[SubdomainEnum] = Field(default_factory=list)
+    domain: str | None = None
+    company: str | None = None
+    language: str | None = None
+
+    mention_count: int = 0
+    provenance: Provenance
+    extraction_status: ExtractionStatus = "ok"
+    diff_status: DiffStatus | None = None
+    definition_conflict: bool = False
+    conflict_variants: list[str] = Field(default_factory=list)
+
+
+class RejectedField(BaseModel):
+    """Audit row for a field the span check refused. Kept so a reviewer can see
+    what the control caught rather than only what it let through."""
+
+    entry_term: str
+    field: str
+    offending_value: str
+    reason: str
+    branch: Branch

src/knowledge_extraction/rank/__init__.py

@@ -0,0 +1,3 @@
+from .evidence import rank_evidence, rounds_available, top_k
+
+__all__ = ["rank_evidence", "rounds_available", "top_k"]

src/knowledge_extraction/rank/evidence.py

@@ -0,0 +1,146 @@
+"""Score and select the top-K evidence chunks per cluster.
+
+This is a filter, and it structurally resembles the relevance gate the team
+rejected. It is defensible only because of three properties, all of which must
+survive any future change here:
+
+1. **No term is ever dropped** — only its *evidence* is narrowed.
+2. **Every mention stays in the cluster**, so a bad top-3 is visible in
+   provenance rather than invisible.
+3. **The full ranked list is retained**, so escalation can take the next K and
+   self-correct.
+
+The rejected relevance gate had none of the three. If ranking is ever revisited,
+that is the test to apply.
+"""
+
+from __future__ import annotations
+
+import re
+
+from ..cluster.normalize import normalize
+from ..filters.cue_filter import definitional_hits
+from ..filters.legend_filter import find_legend_lines
+from ..models import Chunk, TermCluster
+from ..settings import CUE_PROXIMITY_CHARS, EVIDENCE_K, EVIDENCE_WEIGHTS
+
+
+def rank_evidence(
+    clusters: list[TermCluster],
+    chunks: list[Chunk],
+    weights: dict[str, float] | None = None,
+) -> None:
+    """Mutates clusters in place: sets `evidence_chunk_ids` (ranked, best first)
+    and `evidence_scores`. Keeps the FULL ranked list, not just the top K."""
+    w = {**EVIDENCE_WEIGHTS, **(weights or {})}
+    by_id = {c.chunk_id: c for c in chunks}
+    first_seen: set[str] = set()
+
+    for cluster in clusters:
+        scored: list[tuple[float, str]] = []
+        for chunk_id in {m.chunk_id for m in cluster.mentions}:
+            chunk = by_id.get(chunk_id)
+            if chunk is None:
+                continue
+            positions = [m for m in cluster.mentions if m.chunk_id == chunk_id]
+            scored.append((_score_chunk(chunk, cluster, positions, w, first_seen), chunk_id))
+        # Tie-break on chunk_id so ranking is stable across runs.
+        scored.sort(key=lambda t: (-t[0], t[1]))
+        cluster.evidence_chunk_ids = [cid for _, cid in scored]
+        cluster.evidence_scores = [round(s, 2) for s, _ in scored]
+
+
+def top_k(
+    cluster: TermCluster, k: int = EVIDENCE_K, round_index: int = 0
+) -> list[str]:
+    """Evidence for extraction round `round_index` (0 = best K, 1 = next K…)."""
+    start = round_index * k
+    return cluster.evidence_chunk_ids[start : start + k]
+
+
+def rounds_available(cluster: TermCluster, k: int, max_rounds: int) -> int:
+    """How many escalation rounds this cluster actually has evidence for.
+
+    Usually 0 on short documents — a cluster with a single evidence chunk has
+    nowhere to escalate, which is why an escalation count of zero is not by
+    itself a sign the loop is broken.
+    """
+    extra = max(0, len(cluster.evidence_chunk_ids) - k)
+    return min(max_rounds, -(-extra // k))  # ceil division
+
+
+def _score_chunk(
+    chunk: Chunk,
+    cluster: TermCluster,
+    positions: list,
+    w: dict[str, float],
+    first_seen: set[str],
+) -> float:
+    score = 0.0
+
+    cues = definitional_hits(chunk.text)
+    for mention in positions:
+        if any(
+            abs(cs - mention.char_start) <= CUE_PROXIMITY_CHARS
+            or abs(ce - mention.char_end) <= CUE_PROXIMITY_CHARS
+            for cs, ce in cues
+        ):
+            score += w["definitional_cue_near"]
+            break
+
+    # Word-boundary match, NOT substring. "PA" is a substring of "Parameter",
+    # "pada" and "composite", so substring matching handed this bonus to almost
+    # every Indonesian heading and pushed real definition sections below
+    # formula tables.
+    heading = normalize(chunk.heading or "")
+    if heading and any(_word_match(normalize(v), heading) for v in cluster.variants):
+        score += w["term_in_heading"]
+
+    legend_lines = find_legend_lines(chunk.text)
+    if legend_lines:
+        line_starts = _line_starts(chunk.text)
+        for mention in positions:
+            if _line_of(mention.char_start, line_starts) in legend_lines:
+                score += w["in_legend_block"]
+                break
+
+    if chunk.has_formula:
+        score += w["formula_present"]
+
+    normalised_variants = {normalize(v) for v in cluster.variants}
+    if any(normalize(b) in normalised_variants for b in chunk.bold_spans):
+        score += w["bold_or_italic"]
+
+    if cluster.cluster_id not in first_seen:
+        score += w["first_occurrence"]
+        first_seen.add(cluster.cluster_id)
+
+    if chunk.is_tabular:
+        score += w["tabular_penalty"]
+
+    return score
+
+
+def _word_match(needle: str, haystack: str) -> bool:
+    if not needle:
+        return False
+    return re.search(rf"(?<!\w){re.escape(needle)}(?!\w)", haystack) is not None
+
+
+def _line_starts(text: str) -> list[int]:
+    starts, pos = [0], 0
+    for line in text.split("\n")[:-1]:
+        pos += len(line) + 1
+        starts.append(pos)
+    return starts
+
+
+def _line_of(char_pos: int, line_starts: list[int]) -> int:
+    lo, hi = 0, len(line_starts) - 1
+    while lo < hi:
+        mid = (lo + hi + 1) // 2
+        if line_starts[mid] <= char_pos:
+            lo = mid
+        else:
+            hi = mid - 1
+    return lo

src/knowledge_extraction/settings.py

@@ -0,0 +1,93 @@
+"""Tunables for the knowledge-extraction pipeline.
+
+Every value here was calibrated on real documents and each one has a reason
+recorded in KNOWLEDGE_PIPELINE_CALIBRATION.md. Change them deliberately — most
+were arrived at by a measurement, and two of them (`FUZZY_MIN_LEN`,
+`CACHE_MIN_TOKENS`) fix bugs that are silent when reintroduced.
+
+Label and cue sets live in `config/*.yaml` so they can be tuned without a code
+change: label phrasing is the main recall lever and the filter is very sensitive
+to it.
+"""
+
+from __future__ import annotations
+
+from functools import lru_cache
+from pathlib import Path
+
+import yaml
+
+CONFIG_DIR = Path(__file__).resolve().parent / "config"
+
+# ── Term filter ─────────────────────────────────────────────────────────
+# Variant C beat both the English-default and Indonesian-phrasing label sets:
+# the other two missed the same class (mining activities and materials).
+LABELS_VARIANT = "broad"
+
+# 0.25, not 0.35: measured recall 0.854 @ 0.25 vs 0.658 @ 0.35 on `broad`.
+# Precision falls (0.41 vs 0.50) and that is the intended trade — the filter is
+# deliberately over-inclusive, clustering and ranking absorb the noise, and a
+# term the filter never proposes can never be recovered downstream.
+SPAN_SCORE_THRESHOLD = 0.25
+
+# The span NER model truncates past ~384 of its own tokens and *warns rather
+# than failing*, so a long chunk silently loses its tail. Indonesian technical
+# prose subword-tokenises at roughly 2.5x, so 220-word windows still tripped the
+# cap; 130 does not. Chunks are fed as overlapping windows with offsets remapped.
+WINDOW_WORDS = 130
+WINDOW_OVERLAP = 30
+SPAN_TOKEN_CAP = 12
+
+# ── Clustering ──────────────────────────────────────────────────────────
+FUZZY_THRESHOLD = 92
+
+# Below this length only exact matching is allowed: "PA" and "UA" score highly
+# against each other on token_set_ratio. Over-merging is far worse than
+# under-merging — an under-merge costs one extra call and one extra review row,
+# a wrong merge destroys a distinct term and the expert never sees it.
+FUZZY_MIN_LEN = 5
+
+# ── Evidence ranking ────────────────────────────────────────────────────
+EVIDENCE_K = 3
+CUE_PROXIMITY_CHARS = 100
+
+EVIDENCE_WEIGHTS: dict[str, float] = {
+    "definitional_cue_near": 5.0,
+    "term_in_heading": 4.0,
+    "in_legend_block": 3.5,
+    "formula_present": 2.0,
+    "bold_or_italic": 1.5,
+    "first_occurrence": 1.0,
+    "tabular_penalty": -3.0,
+}
+
+# ── Chunking ────────────────────────────────────────────────────────────
+MAX_CHUNK_TOKENS = 1500
+MAX_HEADING_LEN = 90
+BOILERPLATE_MIN_FRAC = 0.6
+
+# ── Extraction ──────────────────────────────────────────────────────────
+TEMPERATURE = 0.0
+
+# OpenAI-family prompt caching does not engage AT ALL below this many prompt
+# tokens, so a shorter fixed prefix caches nothing and costs ~10x on input. The
+# measured hit rate at/above it was 54%.
+CACHE_MIN_TOKENS = 1024
+
+# ── Validation ──────────────────────────────────────────────────────────
+MAX_ESCALATION_ROUNDS = 2
+CONFLICT_OVERLAP_THRESHOLD = 0.4
+DUPLICATE_OVERLAP_THRESHOLD = 0.8
+
+
+@lru_cache(maxsize=4)
+def load_yaml(name: str) -> dict:
+    with open(CONFIG_DIR / name, encoding="utf-8") as fh:
+        return yaml.safe_load(fh)
+
+
+def labels_for(variant: str = LABELS_VARIANT) -> tuple[list[str], float]:
+    """Returns (labels, threshold) for a label variant."""
+    cfg = load_yaml("labels.yaml")
+    labels = cfg.get(variant) or cfg.get(LABELS_VARIANT) or []
+    return list(labels), float(cfg.get("threshold", SPAN_SCORE_THRESHOLD))

uv.lock

@@ -1,5 +1,5 @@
 version = 1
-revision = 2
+revision = 3
 requires-python = "==3.12.*"
 resolution-markers = [
     "python_full_version >= '3.12.4'",
@@ -59,6 +59,7 @@ dependencies = [
     { name = "python-dotenv" },
     { name = "python-multipart" },
     { name = "rank-bm25" },
+    { name = "rapidfuzz" },
     { name = "redis" },
     { name = "sentence-transformers" },
     { name = "slowapi" },
@@ -149,6 +150,7 @@ requires-dist = [
     { name = "python-dotenv", specifier = "==1.0.1" },
     { name = "python-multipart", specifier = "==0.0.12" },
     { name = "rank-bm25", specifier = "==0.2.2" },
+    { name = "rapidfuzz", specifier = ">=3.14.5" },
     { name = "redis", specifier = "==5.2.1" },
     { name = "ruff", marker = "extra == 'dev'", specifier = "==0.8.4" },
     { name = "sentence-transformers", specifier = "==3.3.1" },
@@ -2770,6 +2772,25 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/2a/21/f691fb2613100a62b3fa91e9988c991e9ca5b89ea31c0d3152a3210344f9/rank_bm25-0.2.2-py3-none-any.whl", hash = "sha256:7bd4a95571adadfc271746fa146a4bcfd89c0cf731e49c3d1ad863290adbe8ae", size = 8584, upload-time = "2022-02-16T12:10:50.626Z" },
 ]
 
+[[package]]
+name = "rapidfuzz"
+version = "3.14.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2c/21/ef6157213316e85790041254259907eb722e00b03480256c0545d98acd33/rapidfuzz-3.14.5.tar.gz", hash = "sha256:ba10ac57884ce82112f7ed910b67e7fb6072d8ef2c06e30dc63c0f604a112e0e", size = 57901753, upload-time = "2026-04-07T11:16:31.931Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/d3/e3/574435c6aafb80254c191ef40d7aca2cb2bb97a095ec9395e9fa59ac307a/rapidfuzz-3.14.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0d3378f471ef440473a396ce2f8e97ee12f89a78b495540e0a5617bbfe895638", size = 1944601, upload-time = "2026-04-07T11:14:18.771Z" },
+    { url = "https://files.pythonhosted.org/packages/d0/1f/fbad3102a255ecc112ce9a7e779bacab7fd14398217be8868dc9082ba363/rapidfuzz-3.14.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1e910eebca9fd0eba245c0555e764597e8a0cccb673a92da2dc2397050725f48", size = 1164293, upload-time = "2026-04-07T11:14:20.534Z" },
+    { url = "https://files.pythonhosted.org/packages/88/37/a3eb7ff6121ed3a5f199a8c38cc86c8e481816f879cb0e0b738b078c9a7e/rapidfuzz-3.14.5-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:01550fe5f60fd176aa66b7611289d46dc4aa4b1b904874c7b6d1d54e581c5ec1", size = 1371999, upload-time = "2026-04-07T11:14:22.63Z" },
+    { url = "https://files.pythonhosted.org/packages/79/72/97a9728c711c7c1b06e107d3f0623880fb4ef90e147ed13c551a1730e7cc/rapidfuzz-3.14.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:48bee0b91bebfaec41e1081e351000659ab7570cc4598d617aa04d5bf827f9e6", size = 3145715, upload-time = "2026-04-07T11:14:24.508Z" },
+    { url = "https://files.pythonhosted.org/packages/ed/54/d5caabbea233ac90c286c87c260e49d7641467e87438a18d858e41c82e91/rapidfuzz-3.14.5-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:7e580cb04ad849ae9b786fa21383c6b994b6e6c1444ad1cb9f22392759d72741", size = 1456304, upload-time = "2026-04-07T11:14:26.515Z" },
+    { url = "https://files.pythonhosted.org/packages/fc/a7/2d1a81250ac8c01a0100c026018e76f0e7a097ff63e4c553e02a6938c6fb/rapidfuzz-3.14.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:09d6c9ba091854f07817055d795d604179c12a8f308ba4c7d56f3719dfea1646", size = 2389089, upload-time = "2026-04-07T11:14:28.635Z" },
+    { url = "https://files.pythonhosted.org/packages/65/0d/c47c3872203ae88e6506997c0b576ad731f5261daa25d559be09c9756658/rapidfuzz-3.14.5-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:1e989f86113be66574113b9c7bdf4793f3f863d248e47d911b355e05ca6b6b10", size = 2493404, upload-time = "2026-04-07T11:14:30.577Z" },
+    { url = "https://files.pythonhosted.org/packages/8f/2f/71e0a5a3130792146c8a200a2dd1e52aa16f7c1074012e17f2601eea9a90/rapidfuzz-3.14.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0ebd1a18e2e47bc0b292a07e6ed9c3642f8aaa672d12253885f599b50807a4f9", size = 4251709, upload-time = "2026-04-07T11:14:32.451Z" },
+    { url = "https://files.pythonhosted.org/packages/86/45/d39874901abacef325adb5b34ae416817c8486dfb4fb87c7a9b74ec5b072/rapidfuzz-3.14.5-cp312-cp312-win32.whl", hash = "sha256:9981d38a703b86f0e315a3cd229fd1906fe1d91c989ed121fb975b3c849f89f5", size = 1710069, upload-time = "2026-04-07T11:14:34.37Z" },
+    { url = "https://files.pythonhosted.org/packages/85/0b/f65572c53de8a1c704bda707f63a447b67bdbe95d7cdc70d18885e191df5/rapidfuzz-3.14.5-cp312-cp312-win_amd64.whl", hash = "sha256:d8375e3da319593389727c3187ccaf3e0e84199accc530866b8e0f2b79af05e9", size = 1540630, upload-time = "2026-04-07T11:14:36.287Z" },
+    { url = "https://files.pythonhosted.org/packages/5e/c3/143be3a578f989758cae516f3270d5cbb49783a7bfdf57cc27a670e00456/rapidfuzz-3.14.5-cp312-cp312-win_arm64.whl", hash = "sha256:478b59bb018a6780d73f33e38d0b3ec5e968a6c1ed42876b993dd456b7aa20e8", size = 813137, upload-time = "2026-04-07T11:14:38.289Z" },
+]
+
 [[package]]
 name = "redis"
 version = "5.2.1"