| # CanLex maintainer runbook |
|
|
| Everything a future maintainer needs that the code alone does not say. |
|
|
| ## The refresh choreography |
|
|
| Order matters — chunks feed embeddings feed the eval: |
|
|
| ``` |
| py -m canlex.ingest [--force] [CODES] # Justice Laws XML -> per-Act JSON (incl. Schedules) |
| py -m canlex.ingest --fr # parallel French corpus (same schema; SOR->DORS URLs) |
| py -m canlex.pdi # IRCC PDI trees (TR / refugee / standard / |
| # identity / PR-card; capped BFS, drops logged) |
| py -m canlex.ciu # CIU/PSAC union documents + site VSSAs |
| py -m canlex.dmemo [--fr] # CBSA D-Memoranda (EN; --fr = Mémorandums D) |
| py -m canlex.enf # ENF operational manuals (canada.ca PDFs) |
| py -m canlex.amps # AMPS Master Penalty Document |
| py -m canlex.charter # Charter / Constitution HTML |
| py -m canlex.caselaw # Lexum/Decisia courts + cached CanLII texts |
| py -m canlex.delegation # CBSA/IRCC delegation instruments |
| py -m canlex.directive [--fr] # NJC directives (EN; --fr = éditions françaises) |
| py -m canlex.agreement # FB collective agreement |
| py -m canlex.benefits # benefit-plan booklets |
| py -m canlex.tariff_schedule # Customs Tariff ch. 98/99 (CBSA edition) |
| py -m canlex.commentary # curated datasets -> commentary chunks |
| py -m canlex.embed # re-embed EVERYTHING (run after ANY of the above) |
| py -m unittest discover -s tests # all offline |
| py -m canlex.eval # gate: no metric below the last committed run |
| ``` |
|
|
| `canlex.embed` is mandatory after any chunk change: the index zero-fills |
| vectors for chunks missing from embeddings.npz and quietly degrades them to |
| BM25-only (a stderr warning is the only signal). |
|
|
| ### The corpus-write guard |
|
|
| Every ingester above rebuilds its whole processed file from a fresh scrape, so |
| an upstream page that changes shape turns into data loss at the write. All of |
| them now go through `_common.write_corpus`, which **refuses to write a corpus |
| below 80% of the stored one** and makes the process exit non-zero: |
|
|
| ``` |
| REFUSING to write: 0 chunks would replace 1870 already in dmemos.json. ... |
| ``` |
|
|
| That is not a transient error — it means the scrape broke. Diagnose it, don't |
| retry. Once the drop is understood and intended (a source genuinely shrank), |
| re-run the same module with `--allow-shrink`. Because the runbook chains these |
| as `py -m canlex.X && py -m canlex.embed`, a refusal now correctly stops the |
| embed instead of propagating a gutted corpus. |
|
|
| Most ingesters also **preserve on failure**: an item that errors keeps its |
| last-good chunks (`_common.preserve_dropped`), restricted to items that |
| actually failed, so anything removed on purpose stays gone. Deliberate |
| exception: `enf.py` treats an HTTP 404 as IRCC retiring a chapter — its |
| documented policy — so a 404 drops the chapter while any other error preserves |
| it. Writes are atomic (tmp + `os.replace`): a killed run cannot leave truncated |
| JSON that would read as an empty corpus and disable the guard next time. |
|
|
| ## Deploys |
|
|
| - **MCP Space** (`Beemer0/CanLex`): commit, then `git push space main`. |
| The Space rebuilds automatically. `embeddings.npz` ships in-repo via |
| git-LFS — a fresh clone needs `git lfs install`. |
| - **Webapp Space** (`Beemer0/Canlex-web`): separate repo. Clone it, copy |
| `webapp/app.py` + `webapp/requirements.txt` + `webapp/Dockerfile` + |
| `webapp/README.md` over, commit, push. Do NOT use `hf upload` — HF gates |
| it behind PRO for Docker/Gradio Spaces; git pushes work on the free tier. |
|
|
| ## Secrets and variables (set in each Space's settings) |
|
|
| | Where | Name | Purpose | |
| |---|---|---| |
| | MCP Space | `CANLII_API_KEY` | optional, enables canlex_case | |
| | Webapp | `GEMINI_API_KEY` | required | |
| | Webapp | `CANLEX_WEB_AUTH` | required (`user:pass` per line); the app refuses to start without it | |
| | Webapp | `CANLEX_GEMINI_MODEL` | optional; the deployed Space sets `gemini-2.5-pro` (code default is flash) | |
| |
| Locally, the CanLII key lives in `canlii_key.txt` (gitignored — never |
| commit it). |
|
|
| ## The staleness watch |
|
|
| `py -m canlex.refresh` (alias `--check`) diffs upstream: Justice Laws |
| consolidation dates, CBSA/IRCC delegation instruments, NJC directive dates, |
| the newest per-memo `LAST_UPDATED` in the D-memo datasource, curated-dataset |
| age (>180 days → AGING) and curated-vs-chunk regen drift. Non-zero exit = |
| something drifted. |
| A scheduled task runs it Mondays 03:01 local — if this machine is off, the |
| check silently doesn't happen that week. Note: network fetch errors on the |
| legislation rows DO fail the check (exit 1) so a broken watch can't pass |
| silently. |
|
|
| `_PENDING_BILLS` in canlex/refresh.py names bills heading into force that |
| affect corpus Acts — edit that list when a new one appears (source-code edit, |
| deliberate). A bill stays listed after its in-force date until Justice Laws |
| consolidates it: C-16 came into force 2026-07-18 but the 2026-06-14 |
| consolidation predates it, so the coercive-control provisions are NOT in the |
| corpus yet and the notice correctly keeps printing. |
|
|
| ## The CBSA D-memo index (rebuilt 2026-07-10) |
|
|
| CBSA replaced the static index with a WET DataTable: the page ships **no memo |
| links at all**, the rows come from `dmemoranda-datasource-eng.json` beside it. |
| `dmemo.memo_urls` reads that datasource and falls back to scraping the page |
| only if it disappears. Two consequences worth knowing: |
|
|
| - Cancelled memos (`STATUS: "Cancelled"`, 34 of 398) are skipped — retired |
| guidance must not retrieve as current. D10-14-48 left the corpus this way. |
| - Eight memos are listed Active but their pages 404 (D8-2-2, D8-9-3, D10-2-3, |
| D10-14-3, D10-14-18, D10-14-65, D10-15-16, D10-15-25) — CBSA's own broken |
| links. `dmemo.preserve_failed` re-attaches the last-good chunks of any memo |
| that fails to fetch, so a 404 (or a dropped connection) can no longer delete |
| still-in-force guidance. The two D8s were never ingestable and are absent. |
|
|
| This is the incident the write guard above exists for: the first run after the |
| rebuild scraped zero memos and wrote an empty dmemos.json straight over 1,870 |
| good chunks, recovered from git. Every ingester now carries the guard. |
|
|
| ## Curated-dataset review protocol |
|
|
| `data/curated/us_dispositions.json` and `us_equivalency.json` are the only |
| hand-maintained legal content. To change them: edit → bump `"reviewed"` → |
| `py -m canlex.commentary` → `py -m canlex.embed` → eval → deploy. The refresh |
| check flags any drift between the curated `reviewed` date and the rendered |
| chunks. Provenance and the verbatim-verification log live inside the JSON |
| files themselves; keep them current. |
|
|
| ## Backups |
|
|
| The HF Space remote is also production. A full-history bundle lives at |
| `C:\projects\CanLex-backup.bundle` (recreate: `git bundle create |
| ../CanLex-backup.bundle --all`). For a real offsite mirror, create a private |
| GitHub repo and: |
|
|
| ``` |
| git remote add mirror https://github.com/<you>/CanLex.git |
| git push mirror --all |
| ``` |
|
|
| (GitHub needs git-lfs enabled for embeddings.npz, or add it to a |
| `.lfsconfig`-excluded push.) |
|
|
| ## Known gaps / parked |
|
|
| - ENF coverage (researched and URL-verified 2026-07-22): ENF 20 Detention |
| and the current ENF 3 edition live under non-standard filenames — handled |
| by `_URL_OVERRIDES` in canlex/enf.py (the old enf03-eng.pdf still serves a |
| STALE 2015-era copy; never remove the override). ENF 7 was never publicly |
| released; ENF 13/16/25 never existed; ENF 9 (Judicial Reviews) and ENF 11 |
| (Verifying Departure) are retired with no successor (Wayback copies exist |
| but retired guidance is deliberately not ingested). ENF 27 (PR Card) was |
| replaced by the PDI section at |
| operational-bulletins-manuals/permanent-residence/card.html — ingested by |
| `canlex.pdi` (the `pdi-prcard` set). ENF 2 (Evaluating Inadmissibility) |
| and ENF 14/OP 19 (Criminal Rehabilitation) exist publicly only as |
| ATIP-released copies hosted by a private law firm — NOT ingested pending |
| the maintainer's call on unofficial sources; the Tran PDI page now comes |
| in with the whole `standard-requirements` tree and partially covers |
| ENF 2's ground. |
| - PDI scope: `canlex.pdi` crawls six category trees breadth-first |
| (`_MAX_DEPTH` 2, `_MAX_PAGES` 160 per tree). Deliberately out of scope: |
| citizenship, fees, office service delivery, and the year-by-year |
| operational-bulletin archives (mostly expired instructions). If a tree |
| ever hits the page cap the run PRINTS how many in-scope pages it dropped |
| — raise `_MAX_PAGES` rather than letting the silence read as coverage. |
| - CanLII capture workflow (used for the Hill/Brannson/Li trilogy, |
| 2026-07-22): open the URL a missing-file stub prints in a real browser, |
| copy the page text to the named `data/raw/caselaw/*.txt`, re-run |
| `py -m canlex.caselaw && py -m canlex.embed`. |
| - Dag (FCA) parses to 2 chunks because it genuinely is a 9-paragraph bench |
| judgment — correct, not a bug. |
| - A weekly citator sweep over the ~110 corpus decisions (negative-treatment |
| detection) was considered and deliberately skipped: ~6 minutes of |
| rate-limited CanLII API per run for marginal signal. Run `canlex_case` on |
| a decision before relying on it instead. |
| - French support is now phase 3: the full parallel French legislation corpus |
| is ingested (`--fr`; lang='fr' chunks, '-fr-' id infix, 'art.'/'annexe' |
| citations). Retrieval is language-routed: English queries never see French |
| chunks; French queries (detected by accents or French function words) |
| search both languages. get_section/xref/list_acts stay English-canonical. |
| Phase 2 (a multilingual embedder) was tried and REJECTED 2026-07-22 — |
| regressed every slice and broke confidence calibration; don't retry |
| without re-calibrating HEDGE_THRESHOLD. French guidance (D-memos are |
| published bilingually) is the remaining bilingual frontier. |
| - ENF 2 and ENF 14/OP 19 are both ingested from their ATIP releases |
| (authorized 2026-07-23, bannered UNOFFICIAL on every chunk). ENF 14's |
| image-scan PDF was recovered by OCR (pymupdf rasterization + Windows OCR; |
| cached at data/raw/enf/enf14-ocr.txt — enf.py auto-falls-back to |
| `*-ocr.txt` when a PDF extracts under 500 chars). |
| - Typo robustness, two layers: unknown STEMS get the trigram bridge in |
| `index._fuzzy_term` (BM25 recall only), and misspelled WORDS are repaired |
| by `index._correct_query` (trigram candidates, Damerau-Levenshtein <= 2, |
| shared 2-char prefix, frequency floor 5 so OCR debris can't win) -- the |
| repaired text feeds the semantic embedder and the reranker. Ultra-rare |
| stems (df <= 2) get the correction APPENDED, never substituted, so rare |
| legitimate terms keep their recall. The `typo` eval slice tracks it. |
| - Bilingual guidance: the D-memos and NJC directives now carry their French |
| editions (`--fr` modes; dmemos-fr.json / directives-fr.json) alongside |
| the French legislation, all lang-routed. Remaining English-only: case |
| law, ENF/AMPS/PDI, benefits booklets, and the curated commentary. |
| |
| ## Two retrieval invariants worth not re-breaking (2026-07-29) |
| |
| Ingesting the French guidance and the wider PDI trees exposed both of these. |
| Each was measured question-by-question against the previous corpus, not |
| guessed at — the harness is a per-question rank dump over two corpus |
| variants, which is how a 5-question shuffle gets attributed instead of |
| argued about. |
| |
| - **BM25 collection statistics are language-scoped.** `_build_bm25` keeps |
| `idf`/`avgdl` (whole corpus) AND `idf_en`/`avgdl_en` (English scope); |
| every English query scores with the English pair. An English query can |
| never retrieve the French twins, but a single corpus-wide idf let those |
| unreachable chunks set the rarity of every shared token (numbers, act |
| codes, folded cognates) and drag the average length: adding the French |
| D-memos and directives alone moved legacy Hit@1 0.767 → 0.750 without a |
| single French chunk being retrievable. French queries still use the |
| whole-corpus pair — they search both languages. |
| - **The diversity cap keys memoranda on the parent DOCUMENT.** `_source_key` |
| used `section` for every memorandum, which is the document identity only |
| for the flat families (`_FLAT_MEMO_CODES`: one act_code across ~360 |
| D-memos / ~1,000 AMPS contraventions). ENF chapters, PDI pages and IRB |
| guidelines carry a per-document act_code and use `section` to subdivide |
| it, so each ENF section and each PDI page was its own "source" and |
| neither RECALL_CAP nor SOURCE_CAP ever bound: the ~150-page PDI refugee |
| tree filled the candidate pool and pushed IRPA s. 112 out of the top 20 |
| of "what is a pre-removal risk assessment" entirely. If a new corpus |
| family lands, check which field carries its document identity before |
| trusting the caps. |
|
|
