Typo repair v2, bilingual guidance corpus, full PDI trees

Misspelled words are now repaired before retrieval (trigram candidates,
Damerau-Levenshtein <= 2, prefix + frequency guards; ultra-rare stems get
the fix appended, never substituted) and the repaired text feeds the
embedder and reranker -- typo slice 0.00 -> 0.75 Hit@1. The D-memos (356
memos, 1,995 chunks) and all 11 NJC directives (1,135 chunks) gained their
French editions via --fr modes, lang-routed like the legislation; French
Hit@3 back to 0.80. pdi.py now crawls five border-relevant PDI category
trees breadth-first (242 pages, 2,068 chunks; caps logged, wipe-guards
kept). Weekly watch rows for all new files. 46,027 chunks / 51,756
vectors; legacy 0.74/0.94/0.98/0.99 MRR 0.84 (Hit@1 softened by guidance
competition, top-3 intact); holdout 0.77/1.00/1.00.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

RUNBOOK.md

@@ -9,15 +9,16 @@ Order matters — chunks feed embeddings feed the eval:
 ```
 py -m canlex.ingest [--force] [CODES]   # Justice Laws XML -> per-Act JSON (incl. Schedules)
 py -m canlex.ingest --fr                # parallel French corpus (same schema; SOR->DORS URLs)
-py -m canlex.pdi                        # scoped IRCC PDI pages (PR card / Tran)
+py -m canlex.pdi                        # IRCC PDI trees (TR / refugee / standard /
+                                        #   identity / PR-card; capped BFS, drops logged)
 py -m canlex.ciu                        # CIU/PSAC union documents + site VSSAs
-py -m canlex.dmemo                      # CBSA D-Memoranda
+py -m canlex.dmemo [--fr]               # CBSA D-Memoranda (EN; --fr = Mémorandums D)
 py -m canlex.enf                        # ENF operational manuals (canada.ca PDFs)
 py -m canlex.amps                       # AMPS Master Penalty Document
 py -m canlex.charter                    # Charter / Constitution HTML
 py -m canlex.caselaw                    # Lexum/Decisia courts + cached CanLII texts
 py -m canlex.delegation                 # CBSA/IRCC delegation instruments
-py -m canlex.directive                  # NJC directives
+py -m canlex.directive [--fr]           # NJC directives (EN; --fr = éditions françaises)
 py -m canlex.agreement                  # FB collective agreement
 py -m canlex.benefits                   # benefit-plan booklets
 py -m canlex.tariff_schedule            # Customs Tariff ch. 98/99 (CBSA edition)
@@ -150,12 +151,19 @@ git push mirror --all
   (Verifying Departure) are retired with no successor (Wayback copies exist
   but retired guidance is deliberately not ingested). ENF 27 (PR Card) was
   replaced by the PDI section at
-  operational-bulletins-manuals/permanent-residence/card.html (~10 HTML
-  sub-pages) — a future small scraper. ENF 2 (Evaluating Inadmissibility)
+  operational-bulletins-manuals/permanent-residence/card.html — ingested by
+  `canlex.pdi` (the `pdi-prcard` set). ENF 2 (Evaluating Inadmissibility)
   and ENF 14/OP 19 (Criminal Rehabilitation) exist publicly only as
   ATIP-released copies hosted by a private law firm — NOT ingested pending
-  the maintainer's call on unofficial sources; the Tran PDI page
-  (standard-requirements/tran.html) partially covers ENF 2's ground.
+  the maintainer's call on unofficial sources; the Tran PDI page now comes
+  in with the whole `standard-requirements` tree and partially covers
+  ENF 2's ground.
+- PDI scope: `canlex.pdi` crawls six category trees breadth-first
+  (`_MAX_DEPTH` 2, `_MAX_PAGES` 160 per tree). Deliberately out of scope:
+  citizenship, fees, office service delivery, and the year-by-year
+  operational-bulletin archives (mostly expired instructions). If a tree
+  ever hits the page cap the run PRINTS how many in-scope pages it dropped
+  — raise `_MAX_PAGES` rather than letting the silence read as coverage.
 - CanLII capture workflow (used for the Hill/Brannson/Li trilogy,
   2026-07-22): open the URL a missing-file stub prints in a real browser,
   copy the page text to the named `data/raw/caselaw/*.txt`, re-run
@@ -175,10 +183,48 @@ git push mirror --all
   regressed every slice and broke confidence calibration; don't retry
   without re-calibrating HEDGE_THRESHOLD. French guidance (D-memos are
   published bilingually) is the remaining bilingual frontier.
-- ENF 2 is ingested from its ATIP release (authorized 2026-07-23, bannered
-  UNOFFICIAL on every chunk). ENF 14/OP 19 (Criminal Rehabilitation) is
-  authorized but BLOCKED: its ATIP PDF is a 56-page image scan with no text
-  layer — needs an OCR pass (tesseract / Windows OCR) before ingestion.
-- Typo robustness: unknown query tokens get a trigram-Dice fuzzy bridge
-  (0.6 floor, shared 2-char prefix, 0.7 score discount) in
-  `index._fuzzy_term`; the `typo` eval slice tracks it.
+- ENF 2 and ENF 14/OP 19 are both ingested from their ATIP releases
+  (authorized 2026-07-23, bannered UNOFFICIAL on every chunk). ENF 14's
+  image-scan PDF was recovered by OCR (pymupdf rasterization + Windows OCR;
+  cached at data/raw/enf/enf14-ocr.txt — enf.py auto-falls-back to
+  `*-ocr.txt` when a PDF extracts under 500 chars).
+- Typo robustness, two layers: unknown STEMS get the trigram bridge in
+  `index._fuzzy_term` (BM25 recall only), and misspelled WORDS are repaired
+  by `index._correct_query` (trigram candidates, Damerau-Levenshtein <= 2,
+  shared 2-char prefix, frequency floor 5 so OCR debris can't win) -- the
+  repaired text feeds the semantic embedder and the reranker. Ultra-rare
+  stems (df <= 2) get the correction APPENDED, never substituted, so rare
+  legitimate terms keep their recall. The `typo` eval slice tracks it.
+- Bilingual guidance: the D-memos and NJC directives now carry their French
+  editions (`--fr` modes; dmemos-fr.json / directives-fr.json) alongside
+  the French legislation, all lang-routed. Remaining English-only: case
+  law, ENF/AMPS/PDI, benefits booklets, and the curated commentary.
+
+## Two retrieval invariants worth not re-breaking (2026-07-29)
+
+Ingesting the French guidance and the wider PDI trees exposed both of these.
+Each was measured question-by-question against the previous corpus, not
+guessed at — the harness is a per-question rank dump over two corpus
+variants, which is how a 5-question shuffle gets attributed instead of
+argued about.
+
+- **BM25 collection statistics are language-scoped.** `_build_bm25` keeps
+  `idf`/`avgdl` (whole corpus) AND `idf_en`/`avgdl_en` (English scope);
+  every English query scores with the English pair. An English query can
+  never retrieve the French twins, but a single corpus-wide idf let those
+  unreachable chunks set the rarity of every shared token (numbers, act
+  codes, folded cognates) and drag the average length: adding the French
+  D-memos and directives alone moved legacy Hit@1 0.767 → 0.750 without a
+  single French chunk being retrievable. French queries still use the
+  whole-corpus pair — they search both languages.
+- **The diversity cap keys memoranda on the parent DOCUMENT.** `_source_key`
+  used `section` for every memorandum, which is the document identity only
+  for the flat families (`_FLAT_MEMO_CODES`: one act_code across ~360
+  D-memos / ~1,000 AMPS contraventions). ENF chapters, PDI pages and IRB
+  guidelines carry a per-document act_code and use `section` to subdivide
+  it, so each ENF section and each PDI page was its own "source" and
+  neither RECALL_CAP nor SOURCE_CAP ever bound: the ~150-page PDI refugee
+  tree filled the candidate pool and pushed IRPA s. 112 out of the top 20
+  of "what is a pre-removal risk assessment" entirely. If a new corpus
+  family lands, check which field carries its document identity before
+  trusting the caps.

canlex/directive.py

@@ -151,11 +151,14 @@ def parse_directive(html, url, title, date):
 
 
 def _print_link(html):
-    """Find a 'Print Full Directive' / 'Print all FSDs' link on a TOC page."""
+    """Find a 'Print Full Directive' / 'Print all FSDs' link on a TOC page
+    (or its French label on the /fr editions)."""
     soup = BeautifulSoup(html, "html.parser")
     for a in soup.find_all("a", href=True):
         text = _norm(a.get_text(" ", strip=True)).lower()
-        if "print full directive" in text or "print all fsd" in text:
+        if ("print full directive" in text or "print all fsd" in text
+                or "imprimer la directive" in text
+                or "imprimer toutes" in text or "imprimez toutes" in text):
             return a["href"] if a["href"].startswith("http") else BASE + a["href"]
     return None
 
@@ -222,11 +225,71 @@ def ingest(force=False, limit=None, allow_shrink=False):
     return True
 
 
+OUT_FILE_FR = OUT_FILE.with_name("directives-fr.json")
+
+
+def ingest_fr(force=False, limit=None, allow_shrink=False):
+    """The French editions of the NJC directives, mirroring ingest().
+
+    The NJC publishes each directive at .../directive/<code>/fr with the
+    same markup as /en, so the parser is reused as-is. The EN index supplies
+    the directive list and identity (act_short stays the English title so
+    act filters behave one way regardless of language); chunks get
+    lang='fr' and a -fr id infix, and language-routed retrieval serves them
+    only to French queries."""
+    directives = directive_links(force=force)
+    if limit:
+        directives = directives[:limit]
+    print(f"Ingesting {len(directives)} directives du CNM (français)...")
+    all_chunks, failures = [], []
+    for url, title, date in directives:
+        code_match = _CODE.search(url)
+        code = code_match.group(1) if code_match else "x"
+        fr_url = re.sub(r"/en$", "/fr", url)
+        try:
+            html = _fetch(fr_url, DIRECTIVE_DIR / f"{code}-fr.html",
+                          force=force)
+            chunks = parse_directive(html, fr_url, title, date)
+            if not chunks:
+                print_url = _print_link(html)
+                if print_url:
+                    full = _fetch(print_url,
+                                  DIRECTIVE_DIR / f"{code}-fr-full.html",
+                                  force=force)
+                    chunks = parse_directive(full, fr_url, title, date)
+            if chunks:
+                all_chunks.extend(chunks)
+                print(f"  {title} (fr): {len(chunks)} chunks")
+            else:
+                failures.append((code, title, "no content parsed"))
+        except Exception as exc:
+            failures.append((code, title, f"{type(exc).__name__}: {exc}"))
+    for c in all_chunks:
+        c["id"] = c["id"].replace("directive-", "directive-fr-", 1)
+        c["lang"] = "fr"
+    if not limit:
+        all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                                stored_chunks(OUT_FILE_FR))
+        if preserved:
+            print(f"  kept the last-good copy of {len(preserved)} "
+                  f"unparseable directive(s)")
+    for _code, title, why in failures:
+        print(f"  FAILED {title} (fr): {why}")
+    if not write_corpus(OUT_FILE_FR, all_chunks, indent=2,
+                        allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} chunks from "
+          f"{len(directives) - len(failures)} directives -> "
+          f"{OUT_FILE_FR.name}")
+    return True
+
+
 def main():
     force = "--force" in sys.argv
     limit = next((int(a.split("=", 1)[1]) for a in sys.argv[1:]
                   if a.startswith("--limit=")), None)
-    ok = ingest(force=force, limit=limit,
+    runner = ingest_fr if "--fr" in sys.argv else ingest
+    ok = runner(force=force, limit=limit,
                 allow_shrink="--allow-shrink" in sys.argv)
     sys.exit(0 if ok else 1)
 

canlex/dmemo.py

@@ -32,10 +32,15 @@ DMEMO_DIR = RAW_DIR / "dmemos"
 OUT_FILE = PROCESSED_DIR / "dmemos.json"
 
 # <h2> headings that are page boilerplate rather than memo content.
-_SKIP_HEADINGS = {"contact us", "related links"}
-_MEMO_HREF = re.compile(r"/dm-md/d\d+/d[\d-]+-eng\.html")
-_URL_NUMBER = re.compile(r"/(d\d+-[\d-]+)-eng\.html")
+_SKIP_HEADINGS = {"contact us", "related links",
+                  "contactez-nous", "liens connexes"}
+# Bilingual: the French editions live at the same paths with -fra.html, so
+# one pattern serves both ingest passes.
+_MEMO_HREF = re.compile(r"/dm-md/d\d+/d[\d-]+-(?:eng|fra)\.html")
+_URL_NUMBER = re.compile(r"/(d\d+-[\d-]+)-(?:eng|fra)\.html")
 _DS_HREF = re.compile(r'href="([^"]+)"')
+DATASOURCE_URL_FR = DATASOURCE_URL.replace("-eng.json", "-fra.json")
+OUT_FILE_FR = PROCESSED_DIR / "dmemos-fr.json"
 
 
 def _fetch(url, dest, force=False):
@@ -46,7 +51,7 @@ def datasource_urls(raw):
     """Active D-memo URLs from the index's JSON datasource. Pure, for tests."""
     urls, seen = [], set()
     for row in json.loads(raw).get("data", []):
-        if _norm(row.get("STATUS", "")).lower() == "cancelled":
+        if _norm(row.get("STATUS", "")).lower() in ("cancelled", "annulé"):
             continue
         match = _DS_HREF.search(row.get("NUMBER", ""))
         if not match:
@@ -299,11 +304,62 @@ def ingest(force=False, limit=None, allow_shrink=False):
     return True
 
 
+def ingest_fr(force=False, limit=None, allow_shrink=False):
+    """The French editions (Mémorandums D), mirroring ingest().
+
+    Same pages at -fra.html, same WET markup, same parser; the chunks are
+    post-processed with lang='fr', a dmemo-fr- id prefix and the French
+    citation so language-routed retrieval (index.query_lang) serves them
+    only to French queries."""
+    try:
+        urls = datasource_urls(_fetch(DATASOURCE_URL_FR,
+                                      DMEMO_DIR / "_datasource-fra.json",
+                                      force=force))
+    except Exception as exc:
+        print(f"  FR datasource unavailable ({type(exc).__name__}: {exc})")
+        urls = []
+    if limit:
+        urls = urls[:limit]
+    print(f"Ingesting {len(urls)} Mémorandums D (français)...")
+    all_chunks, failures = [], []
+    for i, url in enumerate(urls, 1):
+        try:
+            html = _fetch(url, DMEMO_DIR / url.rsplit("/", 1)[-1], force=force)
+            chunks = parse_memo(html, url) or parse_pdf_memo(html, url)
+            if chunks:
+                all_chunks.extend(chunks)
+            else:
+                failures.append((url, "no content parsed"))
+        except Exception as exc:
+            failures.append((url, f"{type(exc).__name__}: {exc}"))
+        if i % 50 == 0:
+            print(f"  {i}/{len(urls)} ...")
+    for c in all_chunks:
+        c["id"] = c["id"].replace("dmemo-", "dmemo-fr-", 1)
+        c["lang"] = "fr"
+        c["citation"] = c["citation"].replace("Memorandum", "Mémorandum", 1)
+    if failures:
+        print(f"  {len(failures)} mémorandums with no content / errors "
+              f"(first: {failures[0][0].rsplit('/', 1)[-1]})")
+    if not limit:
+        all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                                stored_chunks(OUT_FILE_FR))
+        if preserved:
+            print(f"  kept the last-good copy of {len(preserved)} "
+                  f"unfetchable mémorandum(s)")
+    if not write_corpus(OUT_FILE_FR, all_chunks, allow_shrink=allow_shrink):
+        return False
+    print(f"  {len(all_chunks)} section-chunks from "
+          f"{len(urls) - len(failures)} mémorandums -> {OUT_FILE_FR.name}")
+    return True
+
+
 def main():
     force = "--force" in sys.argv
     limit = next((int(a.split("=", 1)[1]) for a in sys.argv[1:]
                   if a.startswith("--limit=")), None)
-    ok = ingest(force=force, limit=limit,
+    runner = ingest_fr if "--fr" in sys.argv else ingest
+    ok = runner(force=force, limit=limit,
                 allow_shrink="--allow-shrink" in sys.argv)
     sys.exit(0 if ok else 1)
 

canlex/index.py

@@ -71,6 +71,10 @@ APPENDIX_CAP = 3    # max referenced appendices co-surfaced into a result set
 # items are distinct provisions, so (like legislation) they are never collapsed
 # under the diversity cap.
 PRIMARY_DOC_TYPES = frozenset({"legislation", "agreement", "directive", "delegation"})
+# Memorandum families whose act_code is ONE shared constant over many
+# independent items, so the diversity cap keys on `section` instead (see
+# _source_key): a D-memo number, an AMPS contravention code.
+_FLAT_MEMO_CODES = frozenset({"D-Memo", "AMPS"})
 
 _TOKEN = re.compile(r"[a-z0-9]+")
 # A chunk id ending '-p<k>' marks one embeddable piece of a long section that
@@ -237,6 +241,8 @@ class LegislationIndex:
             raise RuntimeError(
                 f"No processed legislation in {PROCESSED_DIR}. Run 'canlex.ingest' first.")
         self._tri_index = None       # lazy; see _fuzzy_term
+        self._word_vocab = None      # lazy; see _correct_word (typo repair)
+        self._word_tri = None
         # Cached recall mask for the common case (English query, no
         # filters): everything but the French twins.
         self._en_mask = [c.get("lang", "en") != "fr" for c in self.chunks]
@@ -248,9 +254,22 @@ class LegislationIndex:
         self._load_reranker()
 
     def _build_bm25(self):
+        """BM25 postings plus TWO sets of collection statistics: one over the
+        whole corpus, one over the English scope alone.
+
+        An English query never retrieves the French twins (see _en_mask), yet
+        a single corpus-wide idf/avgdl let those unreachable chunks set the
+        rarity of every shared token (numbers, act codes, proper nouns, folded
+        cognates) and drag the average document length. Ingesting the French
+        D-memos and NJC directives moved legacy Hit@1 0.767 -> 0.750 that way,
+        without a single French chunk ever being retrievable -- the statistics
+        must be scoped to what the query can actually reach.
+        """
         self.doc_len = []
         self.postings = defaultdict(list)  # term -> [(doc_idx, term_frequency), ...]
         df = defaultdict(int)
+        df_en = defaultdict(int)
+        en_docs = en_total_len = 0
         for idx, c in enumerate(self.chunks):
             # The topical title is repeated to weight it above body text;
             # the Act name, code and section are indexed too, so an Act's own
@@ -265,12 +284,24 @@ class LegislationIndex:
                              c["section"], c["text"]))
             counts = Counter(tokenize(blob))
             self.doc_len.append(sum(counts.values()))
+            is_en = c.get("lang", "en") != "fr"
+            if is_en:
+                en_docs += 1
+                en_total_len += self.doc_len[-1]
             for term, tf in counts.items():
                 self.postings[term].append((idx, tf))
                 df[term] += 1
+                if is_en:
+                    df_en[term] += 1
         n = len(self.chunks)
         self.avgdl = sum(self.doc_len) / n
         self.idf = {t: math.log(1 + (n - d + 0.5) / (d + 0.5)) for t, d in df.items()}
+        # The English-scoped twins. A term the English corpus never uses is
+        # absent from idf_en rather than zero-weighted: it can only match
+        # chunks the mask already excludes.
+        self.avgdl_en = en_total_len / (en_docs or 1)
+        self.idf_en = {t: math.log(1 + (en_docs - d + 0.5) / (d + 0.5))
+                       for t, d in df_en.items()}
 
     def _build_note_tokens(self):
         """Pre-tokenise each chunk's topical title (see topical_title) for the
@@ -390,28 +421,128 @@ class LegislationIndex:
                 best, best_score = cand, score
         return best
 
-    def _bm25_scores(self, query, allowed=None):
+    @staticmethod
+    def _edit_distance(a, b, cap=3):
+        """Damerau-Levenshtein distance, early-exited past `cap` -- typos
+        are substitutions, insertions, deletions and transpositions, and a
+        transposition ('undeclraed') must cost 1, not 2."""
+        if abs(len(a) - len(b)) > cap:
+            return cap + 1
+        prev2, prev = None, list(range(len(b) + 1))
+        for i, ca in enumerate(a, start=1):
+            cur = [i]
+            best_row = i
+            for j, cb in enumerate(b, start=1):
+                cost = 0 if ca == cb else 1
+                d = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost)
+                if (prev2 is not None and i > 1 and j > 1
+                        and ca == b[j - 2] and a[i - 2] == cb):
+                    d = min(d, prev2[j - 2] + 1)
+                cur.append(d)
+                best_row = min(best_row, d)
+            if best_row > cap:
+                return cap + 1
+            prev2, prev = prev, cur
+        return prev[-1]
+
+    def _correct_word(self, word):
+        """Nearest corpus WORD (not stem) for a misspelling, or None.
+
+        Trigram index generates candidates; Damerau-Levenshtein picks the
+        winner (distance <= 2, shared 2-char prefix, frequency floor of 5
+        so OCR debris in the corpus can never be a 'correction'). The
+        result is real text: it feeds the semantic embedder and the
+        cross-encoder, which score words, not stems."""
+        if self._word_vocab is None:
+            words = Counter()
+            for c in self.chunks:
+                words.update(re.findall(
+                    r"[a-z]{4,}", _fold_accents(c["text"].lower())))
+            tri = defaultdict(set)
+            for w, n in words.items():
+                if n >= 5:                      # junk never enters the index
+                    for i in range(len(w) - 2):
+                        tri[w[i:i + 3]].add(w)
+            self._word_vocab, self._word_tri = words, tri
+        tris = {word[i:i + 3] for i in range(len(word) - 2)}
+        candidates = set()
+        for t in tris:
+            candidates |= self._word_tri.get(t, set())
+        best, best_key = None, (3, 0)           # (distance, -frequency)
+        for cand in candidates:
+            if abs(len(cand) - len(word)) > 2 or cand[:2] != word[:2]:
+                continue
+            dist = self._edit_distance(word, cand, cap=2)
+            if dist > 2:
+                continue
+            key = (dist, -self._word_vocab[cand])
+            if key < best_key:
+                best, best_key = cand, key
+        return best
+
+    def _correct_query(self, query):
+        """The query with misspelled words replaced by their nearest corpus
+        words; the query itself when nothing needed fixing. A word is only
+        'misspelled' when its stem is absent from the whole BM25 vocabulary,
+        so correctly-spelled queries pass through untouched by
+        construction."""
+        out, changed = [], False
+        for tok in query.split():
+            word = _fold_accents(tok.lower()).strip(".,;:!?()'\"")
+            if len(word) >= 4 and word.isalpha():
+                stem = _stem(word)
+                df = len(self.postings.get(stem, ()))
+                if df == 0:
+                    # Unknown everywhere: replace outright.
+                    fix = self._correct_word(word)
+                    if fix and fix != word:
+                        out.append(fix)
+                        changed = True
+                        continue
+                elif df <= 2:
+                    # Nearly unknown -- often a typo that happens to collide
+                    # with OCR debris in the corpus. APPEND a strong
+                    # correction rather than replacing, so a legitimately
+                    # rare term (a chemical in a CDSA schedule, a case name)
+                    # never loses its own recall.
+                    fix = self._correct_word(word)
+                    if (fix and fix != word
+                            and self._word_vocab[fix] >= 100):
+                        out.append(f"{tok} {fix}")
+                        changed = True
+                        continue
+            out.append(tok)
+        return " ".join(out) if changed else query
+
+    def _bm25_scores(self, query, allowed=None, en_only=False):
         """allowed: optional boolean list by chunk index. Filtered searches
         pass it so recall competes only within scope -- filtering after
         recall meant a doc_type='caselaw' search drew from whatever case law
-        happened to survive the open-corpus top-N competition."""
+        happened to survive the open-corpus top-N competition.
+
+        en_only: score with the English-scoped collection statistics (see
+        _build_bm25). Set for every English query, whose mask already bars
+        the French twins from the results they would otherwise be weighting.
+        """
+        idf_map = self.idf_en if en_only else self.idf
+        avgdl = self.avgdl_en if en_only else self.avgdl
         scores = defaultdict(float)
         for term in set(tokenize(query)):
-            idf = self.idf.get(term)
+            idf = idf_map.get(term)
             weight = 1.0
             if idf is None:
                 # Unknown token: try the typo bridge at a discount, so a
                 # misspelled query still reaches the provision while a real
                 # match always outranks a fuzzy one.
                 alt = self._fuzzy_term(term)
-                if alt is None:
+                if alt is None or alt not in idf_map:
                     continue
-                term, idf, weight = alt, self.idf[alt], 0.7
+                term, idf, weight = alt, idf_map[alt], 0.7
             for idx, tf in self.postings[term]:
                 if allowed is not None and not allowed[idx]:
                     continue
                 dl = self.doc_len[idx]
-                denom = tf + K1 * (1 - B + B * dl / self.avgdl)
+                denom = tf + K1 * (1 - B + B * dl / avgdl)
                 scores[idx] += weight * idf * tf * (K1 + 1) / denom
         return scores
 
@@ -472,7 +603,23 @@ class LegislationIndex:
         if doc_type in PRIMARY_DOC_TYPES or doc_type in ("benefits", "commentary"):
             return None
         if doc_type == "memorandum":
-            return ("memorandum", c["section"])   # act_code is a shared constant
+            # The key must be the parent DOCUMENT, and which field carries that
+            # identity differs across the memorandum families. ENF chapters,
+            # PDI pages and the IRB guidelines each have their own act_code and
+            # use `section` to subdivide it, so act_code is the document.
+            # _FLAT_MEMO_CODES are the two families where one act_code covers
+            # many independent items -- ~360 D-memoranda and ~1,000 AMPS
+            # contraventions -- and `section` (memo number / contravention
+            # code) is the unit a reader would cite.
+            #
+            # Keying every memorandum on `section` (as this did) gave each ENF
+            # section and each PDI page its own source key, so neither cap ever
+            # bound at the document level: the ~150-page PDI refugee tree could
+            # fill the whole candidate pool and push IRPA s. 112 out of a PRRA
+            # query entirely.
+            code = c["act_code"]
+            return ("memorandum",
+                    c["section"] if code in _FLAT_MEMO_CODES else code)
         return (doc_type, c["act_code"])          # one decision, keyed by citation
 
     def _capped_top(self, ordered):
@@ -654,9 +801,14 @@ class LegislationIndex:
     def search(self, query, top_k=6, act=None, doc_type=None, court=None,
                decided_after=None):
         """Hybrid candidate fusion (BM25 + semantic), then cross-encoder rerank."""
-        # Expand legal abbreviations (PRRA, H&C, ...) into statutory wording for
-        # the recall stages; the reranker sees the original AND expanded forms.
-        expanded = expand_query(query)
+        # Repair misspellings first (no-op for correctly-spelled queries),
+        # then expand legal abbreviations (PRRA, H&C, ...) into statutory
+        # wording for the recall stages. The corrected text is what feeds
+        # the semantic embedder and the cross-encoder -- the stem-level
+        # fuzzy bridge in _bm25_scores only ever helped BM25, which is why
+        # typo queries used to reach the pool but not the top ranks.
+        corrected = self._correct_query(query)
+        expanded = expand_query(corrected)
         qlang = query_lang(query)
         confidence = None
         fused = defaultdict(float)
@@ -673,7 +825,8 @@ class LegislationIndex:
             mask = self._en_mask
         else:
             mask = None
-        bm25 = self._bm25_scores(expanded, allowed=mask)
+        bm25 = self._bm25_scores(expanded, allowed=mask,
+                                 en_only=(qlang == "en"))
         for rank, idx in enumerate(self._capped_top(
                 sorted(bm25, key=bm25.get, reverse=True))):
             fused[idx] += 1.0 / (RRF_K + rank)
@@ -756,13 +909,18 @@ class LegislationIndex:
             pool = candidates[:RERANK_POOL]
             docs = [self._rerank_doc(i) for i in pool]
             ce_orig = self.reranker.score(query, docs)
-            # Optional (CANLEX_RERANK_DUAL=1): also score the expanded query
-            # and keep each candidate's better score, closing the reranker's
-            # abbreviation blindness ('PRRA' vs 'application for
-            # protection'). Off by default: even promote-only, a second
-            # scoring pass reorders relative ranks, and the 2026-07 eval
-            # could not attribute its effect cleanly on the grown corpus.
-            if expanded != query and os.environ.get("CANLEX_RERANK_DUAL"):
+            # A repaired query is scored alongside the original and each
+            # candidate keeps its better score -- the cross-encoder cannot
+            # ground 'detension' against text saying 'detention'. Applies
+            # only when a correction fired, so legacy queries never take
+            # the second pass.
+            if corrected != query:
+                ce_fix = self.reranker.score(corrected, docs)
+                ce_orig = [max(a, b) for a, b in zip(ce_orig, ce_fix)]
+            # Optional (CANLEX_RERANK_DUAL=1): likewise for the
+            # synonym-expanded form. Off by default: the 2026-07 eval could
+            # not attribute its effect cleanly on the grown corpus.
+            if expanded != corrected and os.environ.get("CANLEX_RERANK_DUAL"):
                 ce_exp = self.reranker.score(expanded, docs)
                 ce_orig = [max(a, b) for a, b in zip(ce_orig, ce_exp)]
             ce = dict(zip(pool, ce_orig))

canlex/pdi.py

@@ -21,19 +21,52 @@ OUT_FILE = PROCESSED_DIR / "pdi.json"
 _OBM = ("https://www.canada.ca/en/immigration-refugees-citizenship/corporate/"
         "publications-manuals/operational-bulletins-manuals/")
 
+# Each set is one PDI category tree, crawled breadth-first from its index to
+# _MAX_DEPTH, capped at _MAX_PAGES pages (dropped pages are LOGGED, never
+# silently skipped). Scope fragments keep the crawl inside the category. The
+# selection is the border-relevant slice of the OBM tree: temporary
+# residents (visitors/eTA/TRV/TRPs), refugee protection (port-of-entry
+# claims), the cross-cutting standard requirements (includes the Tran
+# serious-criminality page), officer designation/delegation, biometrics and
+# identity, and the PR-card section (ENF 27's successor). Deliberately out:
+# citizenship, fees, office service-delivery, and the year-by-year
+# operational-bulletin archives (mostly expired instructions).
+_MAX_DEPTH = 2
+_MAX_PAGES = 160    # 60 capped out: the Temporary Residents tree alone has
+                    # ~150 in-scope pages (2026-07 crawl)
+
 SETS = [
     {"code": "pdi-prcard",
      "short": "PDI PR Card",
      "name": "PDI — Permanent resident card and PR status (successor to ENF 27)",
      "index": _OBM + "permanent-residence/card.html",
-     # sub-pages live under these path fragments relative to the index
      "scope": "/permanent-residence/card"},
-    {"code": "pdi-tran",
-     "short": "PDI Tran",
-     "name": "PDI — Assessing inadmissibility for serious criminality "
-             "following Tran v. Canada",
-     "index": _OBM + "standard-requirements/tran.html",
-     "scope": None},          # single page
+    {"code": "pdi-tr",
+     "short": "PDI Temporary Residents",
+     "name": "PDI — Temporary Resident Program (visitors, eTA, TRV, permits)",
+     "index": _OBM + "temporary-residents.html",
+     "scope": "/temporary-residents"},
+    {"code": "pdi-refugee",
+     "short": "PDI Refugee Protection",
+     "name": "PDI — Refugee Protection (claims, eligibility, port of entry)",
+     "index": _OBM + "refugee-protection.html",
+     "scope": "/refugee-protection"},
+    {"code": "pdi-standard",
+     "short": "PDI Standard Requirements",
+     "name": "PDI — Standard requirements (incl. Tran serious-criminality "
+             "assessment)",
+     "index": _OBM + "standard-requirements.html",
+     "scope": "/standard-requirements"},
+    {"code": "pdi-delegation",
+     "short": "PDI Designation-Delegation",
+     "name": "PDI — Designation of officers and delegation of authority",
+     "index": _OBM + "legislation.html",
+     "scope": "operational-bulletins-manuals/legislation"},
+    {"code": "pdi-identity",
+     "short": "PDI Biometrics-Identity",
+     "name": "PDI — Biometrics and identity management",
+     "index": _OBM + "identity-management.html",
+     "scope": "/identity-management"},
 ]
 
 _MODIFIED = re.compile(r'<time[^>]*>(\d{4}-\d{2}-\d{2})</time>')
@@ -116,17 +149,25 @@ def preserve_failed(chunks, failures, stored):
                             only={url for url, _why in failures})
 
 
-def build(allow_shrink=False):
-    """Rebuild pdi.json. Returns the chunks, or False if the write was refused."""
-    all_chunks, failures = [], []
-    for src in SETS:
-        index_html = fetch_cached(
-            src["index"], RAW / f"{src['code']}-index.html",
-            powershell=True, pause=0.8).decode("utf-8", "replace")
-        pages = [(src["index"], index_html)]
-        if src["scope"]:
-            seen = {src["index"]}
-            for href in _LINK.findall(_main(index_html)):
+def _crawl(src, failures):
+    """Breadth-first crawl of one PDI category: index -> in-scope links ->
+    their in-scope links, to _MAX_DEPTH, capped at _MAX_PAGES pages.
+    Returns [(url, html)]; pages dropped by the cap are LOGGED, and fetch
+    failures land in `failures` so preserve_failed keeps their stored
+    copies."""
+    index_html = fetch_cached(
+        src["index"], RAW / f"{src['code']}-index.html",
+        powershell=True, pause=0.8).decode("utf-8", "replace")
+    pages = [(src["index"], index_html)]
+    if not src["scope"]:
+        return pages
+    seen = {src["index"]}
+    frontier = [(src["index"], index_html)]
+    dropped = 0
+    for _depth in range(_MAX_DEPTH):
+        next_frontier = []
+        for _url, html in frontier:
+            for href in _LINK.findall(_main(html)):
                 if src["scope"] not in href:
                     continue
                 url = href if href.startswith("http") else \
@@ -134,15 +175,38 @@ def build(allow_shrink=False):
                 if url in seen:
                     continue
                 seen.add(url)
+                if len(pages) >= _MAX_PAGES:
+                    dropped += 1
+                    continue
                 name = url.rsplit("/", 1)[1].replace(".html", "")
                 try:
                     page = fetch_cached(
                         url, RAW / f"{src['code']}-{name}.html",
                         powershell=True, pause=0.8).decode("utf-8", "replace")
-                    pages.append((url, page))
                 except Exception as exc:
                     failures.append((url, f"{type(exc).__name__}: {exc}"))
                     print(f"  !! {name}: {type(exc).__name__}: {exc}")
+                    continue
+                pages.append((url, page))
+                next_frontier.append((url, page))
+        frontier = next_frontier
+    if dropped:
+        print(f"  !! {src['short']}: page cap {_MAX_PAGES} hit -- {dropped} "
+              f"in-scope pages NOT ingested (raise _MAX_PAGES to cover them)")
+    return pages
+
+
+def build(allow_shrink=False):
+    """Rebuild pdi.json. Returns the chunks, or False if the write was refused."""
+    all_chunks, failures = [], []
+    for src in SETS:
+        try:
+            pages = _crawl(src, failures)
+        except Exception as exc:
+            failures.append((src["index"], f"{type(exc).__name__}: {exc}"))
+            print(f"  !! {src['short']}: index fetch failed "
+                  f"({type(exc).__name__}: {exc})")
+            continue
         n = 0
         for page_no, (url, html) in enumerate(pages):
             chunks = _page_chunks(src, url, html, page_no)

canlex/refresh.py

@@ -71,7 +71,12 @@ _NON_XML = [
     ("commentary", "commentary.json", "py -m canlex.commentary",
      "Curated commentary chunks"),
     ("union", "ciu.json", "py -m canlex.ciu", "CIU/PSAC union documents"),
-    ("pdi", "pdi.json", "py -m canlex.pdi", "IRCC PDI pages (PR card / Tran)"),
+    ("pdi", "pdi.json", "py -m canlex.pdi",
+     "IRCC PDI trees (TR/refugee/standard/identity/PR-card)"),
+    ("memorandum-fr", "dmemos-fr.json", "py -m canlex.dmemo --fr",
+     "Mémorandums D (français)"),
+    ("directive-fr", "directives-fr.json", "py -m canlex.directive --fr",
+     "Directives du CNM (français)"),
     ("irb-guidelines", "irb_guidelines.json", "py -m canlex.irb_guidelines",
      "IRB Chairperson's Guidelines"),
 ]

data/processed/embeddings.npz

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b6380fda9c44fafe766c7cf83a1b40f8a7ef8a689533069d8181b3a28a79c06f
-size 78503650
+oid sha256:2fa0640d280572773d6e35b23454c4de8e589a5d658a11a79e76175cce4faab3
+size 91712138

tests/test_sprint.py

@@ -142,6 +142,145 @@ class MultiVectorTests(unittest.TestCase):
         self.assertTrue(all(t.startswith("A . ") for t in texts[1:]))
 
 
+class TypoCorrectionTests(unittest.TestCase):
+    def test_edit_distance(self):
+        from canlex.index import LegislationIndex as L
+        self.assertEqual(L._edit_distance("detension", "detention"), 1)
+        self.assertEqual(L._edit_distance("undeclraed", "undeclared"), 1)
+        self.assertEqual(L._edit_distance("serius", "serious"), 1)
+        self.assertEqual(L._edit_distance("same", "same"), 0)
+        self.assertGreater(L._edit_distance("apple", "orange"), 2)
+
+    def test_edit_distance_transposition_costs_one(self):
+        from canlex.index import LegislationIndex as L
+        self.assertEqual(L._edit_distance("sieze", "seize"), 1)
+
+    @staticmethod
+    def _index(text, postings=None):
+        """A bare index carrying just what the correction layer reads."""
+        from canlex.index import LegislationIndex
+        idx = LegislationIndex.__new__(LegislationIndex)
+        idx.chunks = [{"text": text}]
+        idx.postings = postings or {}
+        idx._word_vocab = idx._word_tri = None
+        return idx
+
+    def test_corrects_to_the_nearest_corpus_word(self):
+        idx = self._index("detention review " * 6)
+        self.assertEqual(idx._correct_word("detension"), "detention")
+
+    def test_rare_corpus_word_is_never_a_correction(self):
+        # Frequency floor of 5: OCR debris must not win a correction.
+        idx = self._index("detentlon " * 4 + "detention " * 9)
+        self.assertEqual(idx._correct_word("detension"), "detention")
+
+    def test_known_query_passes_through_untouched(self):
+        query = "detention review timelines"
+        idx = self._index("detention review " * 6,
+                          postings={"detent": [0], "review": [0],
+                                    "timelin": [0]})
+        self.assertIs(idx._correct_query(query), query)
+
+    def test_unknown_word_is_replaced(self):
+        idx = self._index("detention review " * 6, postings={"review": [0]})
+        self.assertEqual(idx._correct_query("detension review"),
+                         "detention review")
+
+    def test_nearly_unknown_word_is_appended_not_replaced(self):
+        # df <= 2 -- a rare legitimate term keeps its own recall, so the
+        # correction is added beside it rather than swapped in.
+        idx = self._index("seizure " * 120 + "seizuer ",
+                          postings={"seizuer": [0]})
+        self.assertEqual(idx._correct_query("seizuer"), "seizuer seizure")
+
+
+class LanguageScopedBM25Tests(unittest.TestCase):
+    """The French twins must not weight English retrieval (see _build_bm25)."""
+
+    @staticmethod
+    def _index(chunks):
+        from canlex.index import LegislationIndex
+        idx = LegislationIndex.__new__(LegislationIndex)
+        idx.chunks = chunks
+        idx._tri_index = None
+        idx._build_bm25()
+        return idx
+
+    @staticmethod
+    def _chunk(text, lang="en"):
+        return {"doc_type": "legislation", "act_code": "I-2.5", "act_name": "A",
+                "act_short": "A", "section": "1", "marginal_note": "",
+                "heading": "", "part": "", "division": "", "lang": lang,
+                "text": text}
+
+    def test_french_chunks_do_not_move_english_idf(self):
+        en = [self._chunk("detention review"), self._chunk("seizure")]
+        idx_en = self._index(list(en))
+        idx_mixed = self._index(en + [self._chunk("detention", lang="fr")] * 8)
+        self.assertAlmostEqual(idx_en.idf["detain"], idx_mixed.idf_en["detain"])
+        # ... while the whole-corpus statistic, which French queries use, does
+        # see them.
+        self.assertNotAlmostEqual(idx_en.idf["detain"], idx_mixed.idf["detain"])
+
+    def test_french_chunks_do_not_move_english_avgdl(self):
+        en = [self._chunk("detention review of a permanent resident")]
+        idx_en = self._index(list(en))
+        idx_mixed = self._index(en + [self._chunk("a b c d e f g h", lang="fr")])
+        self.assertAlmostEqual(idx_en.avgdl, idx_mixed.avgdl_en)
+        self.assertNotAlmostEqual(idx_en.avgdl, idx_mixed.avgdl)
+
+    def test_french_only_term_scores_nothing_in_english_scope(self):
+        idx = self._index([self._chunk("detention review"),
+                           self._chunk("annulé", lang="fr")])
+        self.assertEqual(idx._bm25_scores("annulé", en_only=True), {})
+        self.assertTrue(idx._bm25_scores("annulé"))
+
+
+class MemorandumSourceKeyTests(unittest.TestCase):
+    """The diversity cap must key each memorandum on its parent document."""
+
+    @staticmethod
+    def _keys(chunks):
+        from canlex.index import LegislationIndex
+        idx = LegislationIndex.__new__(LegislationIndex)
+        idx.chunks = [dict(c, doc_type="memorandum") for c in chunks]
+        return [idx._source_key(i) for i in range(len(idx.chunks))]
+
+    def test_enf_sections_share_their_chapter(self):
+        keys = self._keys([{"id": "enf-10-21", "act_code": "ENF-10",
+                            "section": "ENF 10 s. 21"},
+                           {"id": "enf-10-6", "act_code": "ENF-10",
+                            "section": "ENF 10 s. 6"},
+                           {"id": "enf-19-11", "act_code": "ENF-19",
+                            "section": "ENF 19 s. 11"}])
+        self.assertEqual(keys[0], keys[1])
+        self.assertNotEqual(keys[0], keys[2])
+
+    def test_pdi_pages_share_their_tree(self):
+        keys = self._keys([{"id": "pdi-refugee-31-2", "act_code": "PDI-REFUGEE",
+                            "section": "PRRA: Intake"},
+                           {"id": "pdi-refugee-34-0", "act_code": "PDI-REFUGEE",
+                            "section": "PRRA: Applicant"}])
+        self.assertEqual(keys[0], keys[1])
+
+    def test_flat_families_key_on_the_item(self):
+        # One act_code covers every D-memo and every AMPS contravention, so
+        # there the cap must bind per memo / per contravention instead.
+        keys = self._keys([{"id": "dmemo-D19-9-2-1", "act_code": "D-Memo",
+                            "section": "D19-9-2"},
+                           {"id": "dmemo-D19-9-2-2", "act_code": "D-Memo",
+                            "section": "D19-9-2"},
+                           {"id": "dmemo-D2-3-1-1", "act_code": "D-Memo",
+                            "section": "D2-3-1"},
+                           {"id": "amps-C001", "act_code": "AMPS",
+                            "section": "C001"},
+                           {"id": "amps-C004", "act_code": "AMPS",
+                            "section": "C004"}])
+        self.assertEqual(keys[0], keys[1])
+        self.assertNotEqual(keys[1], keys[2])
+        self.assertNotEqual(keys[3], keys[4])
+
+
 class RefreshCuratedTests(unittest.TestCase):
     def test_check_curated_runs_offline(self):
         from canlex.refresh import check_curated