Autonomous round: equivalency trilogy, ENF 20 + fresh ENF 3, IRB Guidelines 2/3/8, parser fixes, typo bridge, new watches

Hill/Brannson/Li captured from CanLII and ingested (the FCA authorities the
s.36 equivalency suite applies); ENF 20 Detention + current ENF 3 edition
via filename overrides (old enf03-eng.pdf serves a stale copy); IRB
Chairperson's Guidelines 2/3/8 (227 chunks); Kinhnicki 1->37 chunks
(div-body fallback) and a footnote-sequence guard that fixed Li; trigram
typo bridge (prefix-guarded, unknown tokens only); refresh.py tariff-edition
probe. 25,551 vectors. Eval: legacy 0.79/0.93/0.99/1.00 MRR 0.87 (Hit@5/10
above the pre-growth baseline); typo slice 0.25->0.50; french Hit@3
0.40->0.80.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

RUNBOOK.md

@@ -91,17 +91,34 @@ git push mirror --all
 
 ## Known gaps / parked
 
-- Hill, Brannson and Li (the FCA equivalency trilogy) have curated
-  `CANLII_CASES` entries but need their page text captured manually from
-  CanLII (DataDome blocks scripts): open the URL each stub prints, save the
-  page text to the named `data/raw/caselaw/*.txt` file, re-run
+- ENF coverage (researched and URL-verified 2026-07-22): ENF 20 Detention
+  and the current ENF 3 edition live under non-standard filenames — handled
+  by `_URL_OVERRIDES` in canlex/enf.py (the old enf03-eng.pdf still serves a
+  STALE 2015-era copy; never remove the override). ENF 7 was never publicly
+  released; ENF 13/16/25 never existed; ENF 9 (Judicial Reviews) and ENF 11
+  (Verifying Departure) are retired with no successor (Wayback copies exist
+  but retired guidance is deliberately not ingested). ENF 27 (PR Card) was
+  replaced by the PDI section at
+  operational-bulletins-manuals/permanent-residence/card.html (~10 HTML
+  sub-pages) — a future small scraper. ENF 2 (Evaluating Inadmissibility)
+  and ENF 14/OP 19 (Criminal Rehabilitation) exist publicly only as
+  ATIP-released copies hosted by a private law firm — NOT ingested pending
+  the maintainer's call on unofficial sources; the Tran PDI page
+  (standard-requirements/tran.html) partially covers ENF 2's ground.
+- CanLII capture workflow (used for the Hill/Brannson/Li trilogy,
+  2026-07-22): open the URL a missing-file stub prints in a real browser,
+  copy the page text to the named `data/raw/caselaw/*.txt`, re-run
   `py -m canlex.caselaw && py -m canlex.embed`.
-- ENF chapters 2, 7, 9, 11, 13, 14, 16, 20 (detention), 25, 27 are not at
-  the legacy PDF URL pattern; the current-policy PDI HTML tree is the
-  eventual replacement source.
-- Dag (FCA) and Kinhnicki (FPSLREB) parse to 2/1 chunks (paragraph-detector
-  fallthrough on those cached pages) — known, low-impact.
+- Dag (FCA) parses to 2 chunks because it genuinely is a 9-paragraph bench
+  judgment — correct, not a bug.
+- A weekly citator sweep over the ~110 corpus decisions (negative-treatment
+  detection) was considered and deliberately skipped: ~6 minutes of
+  rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
+  a decision before relying on it instead.
 - French support is phase 1 only (accent-folded tokens + FR→EN synonym
   bridges; the `french` eval slice tracks it). Phase 2 = multilingual
   embedder via the `CANLEX_EMB_REPO` env hooks; phase 3 = parallel French
   corpus.
+- Typo robustness: unknown query tokens get a trigram-Dice fuzzy bridge
+  (0.6 floor, shared 2-char prefix, 0.7 score discount) in
+  `index._fuzzy_term`; the `typo` eval slice tracks it.

canlex/caselaw.py

@@ -441,23 +441,23 @@ CANLII_CASES = [
     # the raw text arrives via the manual browser-capture workflow and
     # ingestion skips gracefully until each file exists.
     {"file": "canlii-1981-brannson.txt",
-     "canlii_path": "en/ca/fca/doc/1980/1980canlii2735/1980canlii2735.html",
+     "canlii_path": "en/ca/fca/doc/1980/1980canlii4197/1980canlii4197.html",
      "short": "Brannson",
      "name": "Brannson v Canada (Minister of Employment and Immigration)",
      "cite": "[1981] 2 FC 141 (CA)",
-     "report": "",
-     "date": "1980-11-14",
+     "report": "1980 CanLII 4197",
+     "date": "1980-06-05",
      "court_name": "Federal Court of Appeal",
      "topic": "Foreign-conviction equivalency: compare the essential elements "
               "of the foreign and Canadian offences; the foreign offence's "
               "wording and the evidence may both be examined (first of the "
               "equivalency trilogy)"},
     {"file": "canlii-1987-hill.txt",
-     "canlii_path": "en/ca/fca/doc/1987/1987canlii5086/1987canlii5086.html",
+     "canlii_path": "en/ca/fca/doc/1987/1987canlii9881/1987canlii9881.html",
      "short": "Hill",
      "name": "Hill v Canada (Minister of Employment and Immigration)",
      "cite": "(1987), 73 NR 315 (FCA)",
-     "report": "",
+     "report": "1987 CanLII 9881; 1 Imm LR (2d) 1",
      "date": "1987-01-29",
      "court_name": "Federal Court of Appeal",
      "topic": "The governing three-branch equivalency test: (1) compare the "
@@ -469,8 +469,8 @@ CANLII_CASES = [
      "short": "Li",
      "name": "Li v Canada (Minister of Citizenship and Immigration)",
      "cite": "[1997] 1 FC 235 (CA)",
-     "report": "",
-     "date": "1996-10-11",
+     "report": "1996 CanLII 4086",
+     "date": "1996-08-07",
      "court_name": "Federal Court of Appeal",
      "topic": "Equivalency compares the essential elements of the offences, "
               "not procedural or evidentiary rules; defences available under "
@@ -589,6 +589,15 @@ def _paragraphs(soup):
               if "MsoFootnoteText" not in (p.get("class") or [])]
     texts = [p.get_text() for p in blocks]
 
+    # Some pre-2009 board pages (e.g. PSSRB 2003) carry the reasons in bare
+    # <div> blocks with no <p> markup at all, so the <p> strategy collects
+    # only header furniture. When it captured almost none of the body's
+    # text, hand the whole body to the plain-text parser instead -- the same
+    # one the CanLII paste captures use, with its own numbering detection
+    # and apparatus cutoff.
+    if sum(len(t) for t in texts) < 0.2 * len(body.get_text()):
+        return _text_paragraphs(body.get_text("\n"))
+
     # Drop the post-reasons apparatus (appended legislation, solicitors list);
     # it is not part of the reasons and would otherwise swell the last paragraph.
     for i, raw in enumerate(texts):
@@ -704,10 +713,13 @@ def _text_paragraphs(text):
                 openers[i] = n
                 expected = n + 1
 
-    if len(openers) < 5:
+    def _unnumbered():
         paras = [(str(j), _norm(ln)) for j, ln in enumerate(lines, start=1)]
         return False, [(n, t) for n, t in paras if len(t) > 1]
 
+    if len(openers) < 5:
+        return _unnumbered()
+
     paras, num, buf = [], None, []
     for i, ln in enumerate(lines):
         if i in openers:
@@ -719,7 +731,16 @@ def _text_paragraphs(text):
             buf.append(ln)
     if num is not None:
         paras.append((str(num), _norm(" ".join(buf))))
-    return True, [(n, t) for n, t in paras if t]
+    paras = [(n, t) for n, t in paras if t]
+
+    # A pre-2001 report-style decision has unnumbered prose reasons but a
+    # trailing footnote block whose own [1]..[N] sequence the detector will
+    # happily latch onto (Li: 26 footnotes, 4% of the text). If the
+    # "numbered paragraphs" cover only a sliver of the document, the real
+    # reasons are the unnumbered prose -- fall back to it.
+    if sum(len(t) for _, t in paras) < 0.2 * len(text):
+        return _unnumbered()
+    return True, paras
 
 
 def _canlii_chunks(case, text):

canlex/enf.py

@@ -26,6 +26,17 @@ BASE = ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
 RAW = RAW_DIR / "enf"
 _CHAPTERS = range(1, 41)      # probe; gaps and retired chapters just skip
 
+# Chapters whose current edition lives under a non-standard filename in the
+# same directory (verified 2026-07-22 against the OBM Active-manuals index).
+# ENF 3's old enf03-eng.pdf still returns 200 but is a STALE 2015-era copy;
+# ENF 20 (Detention) was renamed enf20a-en.pdf (Aug 2024 edition).
+_URL_OVERRIDES = {
+    3: ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
+        "resources/manuals/enf/enf-3-admissibility-(en)-final.pdf"),
+    20: ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
+         "resources/manuals/enf/enf20a-en.pdf"),
+}
+
 # Curated titles for the core chapters; anything else gets a parsed or
 # generic title. Only used for display -- retrieval works off the text.
 TITLES = {
@@ -130,7 +141,7 @@ def _norm_block(lines):
 def build(chapters=_CHAPTERS):
     all_chunks, skipped = [], []
     for n in chapters:
-        url = BASE.format(n=n)
+        url = _URL_OVERRIDES.get(n, BASE.format(n=n))
         dest = RAW / f"enf{n:02d}.pdf"
         try:
             fetch_cached(url, dest, powershell=True, pause=1.0)

canlex/index.py

@@ -214,6 +214,7 @@ class LegislationIndex:
         if not self.chunks:
             raise RuntimeError(
                 f"No processed legislation in {PROCESSED_DIR}. Run 'canlex.ingest' first.")
+        self._tri_index = None       # lazy; see _fuzzy_term
         self._build_bm25()
         self._build_note_tokens()
         self._build_xref()
@@ -332,6 +333,36 @@ class LegislationIndex:
             print(f"CanLex index: reranker disabled ({type(exc).__name__}: {exc}); "
                   f"using hybrid fusion order.", file=sys.stderr)
 
+    def _fuzzy_term(self, term):
+        """The closest vocabulary term for a token the corpus has never seen
+        -- a typo bridge ('detension' -> 'detention'), scored by character-
+        trigram Dice similarity with a strict 0.6 floor and a length guard.
+        Only ever consulted for UNKNOWN tokens, so correctly-spelled queries
+        are untouched. The trigram index is built lazily on first miss."""
+        if len(term) < 4:
+            return None
+        if self._tri_index is None:
+            tri = defaultdict(set)
+            for vocab_term in self.idf:
+                for i in range(len(vocab_term) - 2):
+                    tri[vocab_term[i:i + 3]].add(vocab_term)
+            self._tri_index = tri
+        tris = {term[i:i + 3] for i in range(len(term) - 2)}
+        counts = Counter()
+        for t in tris:
+            counts.update(self._tri_index.get(t, ()))
+        best, best_score = None, 0.6
+        for cand, n in counts.most_common(50):
+            # Typos rarely corrupt a word's opening letters; requiring a
+            # shared 2-char prefix kills drift like 'detens' -> 'pretens'
+            # while keeping 'seisur' -> 'seizur'.
+            if abs(len(cand) - len(term)) > 2 or cand[:2] != term[:2]:
+                continue
+            score = 2 * n / (len(tris) + len(cand) - 2)
+            if score > best_score:
+                best, best_score = cand, score
+        return best
+
     def _bm25_scores(self, query, allowed=None):
         """allowed: optional boolean list by chunk index. Filtered searches
         pass it so recall competes only within scope -- filtering after
@@ -340,14 +371,21 @@ class LegislationIndex:
         scores = defaultdict(float)
         for term in set(tokenize(query)):
             idf = self.idf.get(term)
+            weight = 1.0
             if idf is None:
-                continue
+                # Unknown token: try the typo bridge at a discount, so a
+                # misspelled query still reaches the provision while a real
+                # match always outranks a fuzzy one.
+                alt = self._fuzzy_term(term)
+                if alt is None:
+                    continue
+                term, idf, weight = alt, self.idf[alt], 0.7
             for idx, tf in self.postings[term]:
                 if allowed is not None and not allowed[idx]:
                     continue
                 dl = self.doc_len[idx]
                 denom = tf + K1 * (1 - B + B * dl / self.avgdl)
-                scores[idx] += idf * tf * (K1 + 1) / denom
+                scores[idx] += weight * idf * tf * (K1 + 1) / denom
         return scores
 
     def _semantic_ranking(self, query, allowed=None):

canlex/irb_guidelines.py

@@ -0,0 +1,127 @@
+"""Ingest the IRB Chairperson's Guidelines the border workflow actually meets.
+
+Guideline 2 (Detention) is what Immigration Division members apply at every
+48-hour/7-day/30-day review an officer attends; Guidelines 3 (child
+claimants) and 8 (vulnerable persons) govern how detained or referred minors
+and vulnerable people are handled. Soft law: issued under IRPA
+s. 159(1)(h), members are expected to apply them or explain why not --
+ingested as doc_type='memorandum' (guidance), like the D-memoranda.
+
+Pages are public HTML on irb-cisr.gc.ca (same host caselaw.py already
+parses for jurisprudential guides).
+
+    python -m canlex.irb_guidelines
+"""
+import json
+import re
+
+from bs4 import BeautifulSoup
+
+from ._common import fetch_cached, norm_ws as _norm, split_lines
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "irb_guidelines"
+
+GUIDELINES = [
+    {"num": "2", "file": "guideline2",
+     "url": "https://irb.gc.ca/en/legal-policy/policies/Pages/GuideDir02.aspx",
+     "title": "Detention"},
+    {"num": "3", "file": "guideline3",
+     "url": "https://irb.gc.ca/en/legal-policy/policies/Pages/GuideDir03.aspx",
+     "title": "Child Refugee Claimants: Procedural and Evidentiary Issues"},
+    {"num": "8", "file": "guideline8",
+     "url": "https://irb.gc.ca/en/legal-policy/policies/Pages/GuideDir08.aspx",
+     "title": "Accessibility to IRB Proceedings — Procedural Accommodations "
+              "and Substantive Considerations"},
+]
+
+# "Effective date: May 31, 2021" (or amended/revised) in the page prose.
+_EFFECTIVE = re.compile(
+    r"(?:effective|amended|revised)[^.:\n]*:?\s*((?:January|February|March|"
+    r"April|May|June|July|August|September|October|November|December)\s+"
+    r"\d{1,2},\s+\d{4})", re.I)
+_MONTHS = {m: i for i, m in enumerate(
+    ("January", "February", "March", "April", "May", "June", "July",
+     "August", "September", "October", "November", "December"), start=1)}
+
+
+def _iso(prose):
+    m = re.match(r"(\w+)\s+(\d{1,2}),\s+(\d{4})", prose or "")
+    if not m or m.group(1) not in _MONTHS:
+        return ""
+    return f"{m.group(3)}-{_MONTHS[m.group(1)]:02d}-{int(m.group(2)):02d}"
+
+
+def _sections(main):
+    """Yield (label, heading, text) per numbered heading in the page body."""
+    heads = main.find_all(["h2", "h3"])
+    for i, h in enumerate(heads):
+        heading = _norm(h.get_text())
+        m = re.match(r"(\d+(?:\.\d+)*)\.?\s+(.*)", heading)
+        label = m.group(1) if m else str(i + 1)
+        title = m.group(2) if m else heading
+        parts = []
+        for sib in h.next_siblings:
+            if getattr(sib, "name", None) in ("h2", "h3"):
+                break
+            if getattr(sib, "get_text", None):
+                parts.append(sib.get_text("\n"))
+        body = "\n".join(p for p in (_norm(x) for x in
+                                     "\n".join(parts).split("\n")) if p)
+        if len(body) > 80:
+            yield label, title, body
+
+
+def build():
+    chunks = []
+    for g in GUIDELINES:
+        try:
+            html = fetch_cached(g["url"], RAW / f"{g['file']}.html",
+                                pause=1.0).decode("utf-8", "replace")
+        except Exception as exc:
+            print(f"  !! Guideline {g['num']}: fetch failed "
+                  f"({type(exc).__name__}: {exc})")
+            continue
+        soup = BeautifulSoup(html, "html.parser")
+        main = soup.find("main") or soup
+        eff = _EFFECTIVE.search(main.get_text())
+        effective = _iso(eff.group(1)) if eff else ""
+        n = 0
+        for label, title, body in _sections(main):
+            pieces = split_lines(body, 1800) if len(body) > 2400 else [body]
+            total = len(pieces)
+            for k, piece in enumerate(pieces, start=1):
+                suffix = "" if total == 1 else f"-p{k}"
+                note = "" if total == 1 else f" (part {k} of {total})"
+                chunks.append({
+                    "id": f"irbg-{g['num']}-{label.replace('.', '-')}{suffix}",
+                    "doc_type": "memorandum",
+                    "act_code": f"IRB-G{g['num']}",
+                    "act_short": f"IRB Guideline {g['num']}",
+                    "act_name": f"IRB Chairperson's Guideline {g['num']} — "
+                                f"{g['title']}",
+                    "section": f"G{g['num']} s. {label}",
+                    "marginal_note": title,
+                    "part": g["title"],
+                    "division": "",
+                    "heading": "",
+                    "text": piece,
+                    "history": "",
+                    "last_amended": effective,
+                    "current_to": effective,
+                    "citation": f"Chairperson's Guideline {g['num']}, "
+                                f"s. {label} ({title}){note}",
+                    "source_url": g["url"],
+                })
+                n += 1
+        print(f"  Guideline {g['num']} ({g['title'][:40]}): {n} chunks, "
+              f"effective {effective or '?'}")
+    out = PROCESSED_DIR / "irb_guidelines.json"
+    out.write_text(json.dumps(chunks, ensure_ascii=False, indent=2),
+                   encoding="utf-8")
+    print(f"irb_guidelines: {len(chunks)} chunks -> {out.name}")
+    return chunks
+
+
+if __name__ == "__main__":
+    build()

canlex/refresh.py

@@ -122,6 +122,32 @@ def check_curated(today):
 _DMEMO_INDEX = "https://www.cbsa-asfc.gc.ca/publications/dm-md/d1-d23-eng.html"
 
 
+def check_tariff_edition(today):
+    """Probe whether CBSA has published a Customs Tariff edition newer than
+    the ingested one. The tariff pages live under a year-stamped path
+    (.../tariff-tarif/<year>/html/...), so probing the ingested-edition-plus-
+    one URL (and the current calendar year's) is a reliable new-edition
+    signal without parsing anything."""
+    from . import tariff_schedule
+    src = next(iter(tariff_schedule.SOURCES.values()))
+    have = int(src["edition"])
+    candidates = sorted({have + 1, int(today[:4])} - {have})
+    for year in candidates:
+        if year <= have:
+            continue
+        probe = src["url"].replace(f"/{src['edition']}/", f"/{year}/")
+        try:
+            req = urllib.request.Request(
+                probe, method="HEAD", headers={"User-Agent": BROWSER_UA})
+            with urllib.request.urlopen(req, timeout=30) as resp:
+                if resp.status == 200:
+                    return {"stored": str(have), "remote": str(year),
+                            "status": "stale", "error": ""}
+        except Exception:
+            continue                       # 404 = that edition doesn't exist yet
+    return {"stored": str(have), "remote": "", "status": "ok", "error": ""}
+
+
 def check_dmemo_index():
     """Newest 'Date modified' on the CBSA D-memo index vs stored currency.
     A newer date upstream means at least one memo changed since ingestion --
@@ -442,6 +468,7 @@ def run(as_json=False):
     nonxml = non_xml_currency()
     curated = check_curated(today)
     dmemo_idx = check_dmemo_index()
+    tariff = check_tariff_edition(today)
     pending = pending_in_force(today)
     stale = [r for r in leg if r["status"] in ("stale", "missing-local")]
     errors = [r for r in leg if r["status"] == "error"]
@@ -449,17 +476,19 @@ def run(as_json=False):
     dirs_stale = [r for r in dirs if r["status"] == "stale"]
     curated_flag = [r for r in curated if r["status"] in ("aging", "drift")]
     dmemo_stale = dmemo_idx["status"] == "stale"
+    tariff_stale = tariff["status"] == "stale"
 
     if as_json:
         print(json.dumps({"checked": today, "legislation": leg,
                           "delegation": deleg, "directives": dirs,
                           "non_xml": nonxml, "curated": curated,
-                          "dmemo_index": dmemo_idx,
+                          "dmemo_index": dmemo_idx, "tariff": tariff,
                           "pending_in_force": pending,
                           "stale_count": (len(stale) + len(deleg_stale)
                                           + len(dirs_stale)
                                           + len(curated_flag)
-                                          + (1 if dmemo_stale else 0)),
+                                          + (1 if dmemo_stale else 0)
+                                          + (1 if tariff_stale else 0)),
                           "error_count": len(errors)}, indent=2))
     else:
         print(f"CanLex corpus staleness check — {today}\n")
@@ -550,6 +579,14 @@ def run(as_json=False):
             print(f"\nD-memo index: ok (upstream {dmemo_idx['remote'] or '?'} "
                   f"<= stored {dmemo_idx['stored'] or '?'})")
 
+        if tariff["status"] == "stale":
+            print(f"\nCustoms Tariff edition: STALE — CBSA has published the "
+                  f"{tariff['remote']} edition (ingested: {tariff['stored']}); "
+                  f"bump the year in tariff_schedule.SOURCES and re-ingest")
+        else:
+            print(f"\nCustoms Tariff edition: ok ({tariff['stored']} is "
+                  f"current)")
+
         if pending:
             print("\nPending bills now in force (re-ingest the affected Acts):")
             for b in pending:
@@ -559,7 +596,7 @@ def run(as_json=False):
                 print(f"     run: py -m canlex.ingest --force {' '.join(b['affects'])}")
 
     return 1 if (stale or errors or deleg_stale or dirs_stale
-                 or curated_flag or dmemo_stale) else 0
+                 or curated_flag or dmemo_stale or tariff_stale) else 0
 
 
 def main():

data/processed/caselaw.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f64b399c2d461d8016af792160c868187afa2da17dce23cfb81196eeb6ed8954
-size 11504314
+oid sha256:d0a415257b35e02c821e4d8ea225f8c6c73a72acfef7bb2f5a9de99f7a7d0455
+size 11745635

data/processed/embeddings.npz

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f294b58f4c0f1f1dec6b6c103b38cda132f3aa5e855361a9ad70cb20fdddd12f
-size 44410370
+oid sha256:b827022dc504fa7bc636c50d3206fe90d38e5996420fe33950fd00ba193f16a7
+size 45276878