# CanLex maintainer runbook

Everything a future maintainer needs that the code alone does not say.

## The refresh choreography

Order matters — chunks feed embeddings feed the eval:

```
py -m canlex.ingest [--force] [CODES]   # Justice Laws XML -> per-Act JSON (incl. Schedules)
py -m canlex.ingest --fr                # parallel French corpus (same schema; SOR->DORS URLs)
py -m canlex.pdi                        # IRCC PDI trees (TR / refugee / standard /
                                        #   identity / PR-card; capped BFS, drops logged)
py -m canlex.ciu                        # CIU/PSAC union documents + site VSSAs
py -m canlex.dmemo [--fr]               # CBSA D-Memoranda (EN; --fr = Mémorandums D)
py -m canlex.enf                        # ENF operational manuals (canada.ca PDFs)
py -m canlex.amps                       # AMPS Master Penalty Document
py -m canlex.charter                    # Charter / Constitution HTML
py -m canlex.caselaw                    # Lexum/Decisia courts + cached CanLII texts
py -m canlex.delegation                 # CBSA/IRCC delegation instruments
py -m canlex.directive [--fr]           # NJC directives (EN; --fr = éditions françaises)
py -m canlex.agreement                  # FB collective agreement
py -m canlex.benefits                   # benefit-plan booklets
py -m canlex.tariff_schedule            # Customs Tariff ch. 98/99 (CBSA edition)
py -m canlex.commentary                 # curated datasets -> commentary chunks
py -m canlex.embed                      # re-embed EVERYTHING (run after ANY of the above)
py -m unittest discover -s tests        # all offline
py -m canlex.eval                       # gate: no metric below the last committed run
```

`canlex.embed` is mandatory after any chunk change: the index zero-fills
vectors for chunks missing from embeddings.npz and quietly degrades them to
BM25-only (a stderr warning is the only signal).

### The corpus-write guard

Every ingester above rebuilds its whole processed file from a fresh scrape, so
an upstream page that changes shape turns into data loss at the write. All of
them now go through `_common.write_corpus`, which **refuses to write a corpus
below 80% of the stored one** and makes the process exit non-zero:

```
  REFUSING to write: 0 chunks would replace 1870 already in dmemos.json. ...
```

That is not a transient error — it means the scrape broke. Diagnose it, don't
retry. Once the drop is understood and intended (a source genuinely shrank),
re-run the same module with `--allow-shrink`. Because the runbook chains these
as `py -m canlex.X && py -m canlex.embed`, a refusal now correctly stops the
embed instead of propagating a gutted corpus.

Most ingesters also **preserve on failure**: an item that errors keeps its
last-good chunks (`_common.preserve_dropped`), restricted to items that
actually failed, so anything removed on purpose stays gone. Deliberate
exception: `enf.py` treats an HTTP 404 as IRCC retiring a chapter — its
documented policy — so a 404 drops the chapter while any other error preserves
it. Writes are atomic (tmp + `os.replace`): a killed run cannot leave truncated
JSON that would read as an empty corpus and disable the guard next time.

## Deploys

- **MCP Space** (`Beemer0/CanLex`): commit, then `git push space main`.
  The Space rebuilds automatically. `embeddings.npz` ships in-repo via
  git-LFS — a fresh clone needs `git lfs install`.
- **Webapp Space** (`Beemer0/Canlex-web`): separate repo. Clone it, copy
  `webapp/app.py` + `webapp/requirements.txt` + `webapp/Dockerfile` +
  `webapp/README.md` over, commit, push. Do NOT use `hf upload` — HF gates
  it behind PRO for Docker/Gradio Spaces; git pushes work on the free tier.

## Secrets and variables (set in each Space's settings)

| Where | Name | Purpose |
|---|---|---|
| MCP Space | `CANLII_API_KEY` | optional, enables canlex_case |
| Webapp | `GEMINI_API_KEY` | required |
| Webapp | `CANLEX_WEB_AUTH` | required (`user:pass` per line); the app refuses to start without it |
| Webapp | `CANLEX_GEMINI_MODEL` | optional; the deployed Space sets `gemini-2.5-pro` (code default is flash) |

Locally, the CanLII key lives in `canlii_key.txt` (gitignored — never
commit it).

## The staleness watch

`py -m canlex.refresh` (alias `--check`) diffs upstream: Justice Laws
consolidation dates, CBSA/IRCC delegation instruments, NJC directive dates,
the newest per-memo `LAST_UPDATED` in the D-memo datasource, curated-dataset
age (>180 days → AGING) and curated-vs-chunk regen drift. Non-zero exit =
something drifted.
A scheduled task runs it Mondays 03:01 local — if this machine is off, the
check silently doesn't happen that week. Note: network fetch errors on the
legislation rows DO fail the check (exit 1) so a broken watch can't pass
silently.

`_PENDING_BILLS` in canlex/refresh.py names bills heading into force that
affect corpus Acts — edit that list when a new one appears (source-code edit,
deliberate). A bill stays listed after its in-force date until Justice Laws
consolidates it: C-16 came into force 2026-07-18 but the 2026-06-14
consolidation predates it, so the coercive-control provisions are NOT in the
corpus yet and the notice correctly keeps printing.

## The CBSA D-memo index (rebuilt 2026-07-10)

CBSA replaced the static index with a WET DataTable: the page ships **no memo
links at all**, the rows come from `dmemoranda-datasource-eng.json` beside it.
`dmemo.memo_urls` reads that datasource and falls back to scraping the page
only if it disappears. Two consequences worth knowing:

- Cancelled memos (`STATUS: "Cancelled"`, 34 of 398) are skipped — retired
  guidance must not retrieve as current. D10-14-48 left the corpus this way.
- Eight memos are listed Active but their pages 404 (D8-2-2, D8-9-3, D10-2-3,
  D10-14-3, D10-14-18, D10-14-65, D10-15-16, D10-15-25) — CBSA's own broken
  links. `dmemo.preserve_failed` re-attaches the last-good chunks of any memo
  that fails to fetch, so a 404 (or a dropped connection) can no longer delete
  still-in-force guidance. The two D8s were never ingestable and are absent.

This is the incident the write guard above exists for: the first run after the
rebuild scraped zero memos and wrote an empty dmemos.json straight over 1,870
good chunks, recovered from git. Every ingester now carries the guard.

## Curated-dataset review protocol

`data/curated/us_dispositions.json` and `us_equivalency.json` are the only
hand-maintained legal content. To change them: edit → bump `"reviewed"` →
`py -m canlex.commentary` → `py -m canlex.embed` → eval → deploy. The refresh
check flags any drift between the curated `reviewed` date and the rendered
chunks. Provenance and the verbatim-verification log live inside the JSON
files themselves; keep them current.

## Backups

The HF Space remote is also production. A full-history bundle lives at
`C:\projects\CanLex-backup.bundle` (recreate: `git bundle create
../CanLex-backup.bundle --all`). For a real offsite mirror, create a private
GitHub repo and:

```
git remote add mirror https://github.com/<you>/CanLex.git
git push mirror --all
```

(GitHub needs git-lfs enabled for embeddings.npz, or add it to a
`.lfsconfig`-excluded push.)

## Known gaps / parked

- ENF coverage (researched and URL-verified 2026-07-22): ENF 20 Detention
  and the current ENF 3 edition live under non-standard filenames — handled
  by `_URL_OVERRIDES` in canlex/enf.py (the old enf03-eng.pdf still serves a
  STALE 2015-era copy; never remove the override). ENF 7 was never publicly
  released; ENF 13/16/25 never existed; ENF 9 (Judicial Reviews) and ENF 11
  (Verifying Departure) are retired with no successor (Wayback copies exist
  but retired guidance is deliberately not ingested). ENF 27 (PR Card) was
  replaced by the PDI section at
  operational-bulletins-manuals/permanent-residence/card.html — ingested by
  `canlex.pdi` (the `pdi-prcard` set). ENF 2 (Evaluating Inadmissibility)
  and ENF 14/OP 19 (Criminal Rehabilitation) exist publicly only as
  ATIP-released copies hosted by a private law firm — NOT ingested pending
  the maintainer's call on unofficial sources; the Tran PDI page now comes
  in with the whole `standard-requirements` tree and partially covers
  ENF 2's ground.
- PDI scope: `canlex.pdi` crawls six category trees breadth-first
  (`_MAX_DEPTH` 2, `_MAX_PAGES` 160 per tree). Deliberately out of scope:
  citizenship, fees, office service delivery, and the year-by-year
  operational-bulletin archives (mostly expired instructions). If a tree
  ever hits the page cap the run PRINTS how many in-scope pages it dropped
  — raise `_MAX_PAGES` rather than letting the silence read as coverage.
- CanLII capture workflow (used for the Hill/Brannson/Li trilogy,
  2026-07-22): open the URL a missing-file stub prints in a real browser,
  copy the page text to the named `data/raw/caselaw/*.txt`, re-run
  `py -m canlex.caselaw && py -m canlex.embed`.
- Dag (FCA) parses to 2 chunks because it genuinely is a 9-paragraph bench
  judgment — correct, not a bug.
- A weekly citator sweep over the ~110 corpus decisions (negative-treatment
  detection) was considered and deliberately skipped: ~6 minutes of
  rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
  a decision before relying on it instead.
- French support is now phase 3: the full parallel French legislation corpus
  is ingested (`--fr`; lang='fr' chunks, '-fr-' id infix, 'art.'/'annexe'
  citations). Retrieval is language-routed: English queries never see French
  chunks; French queries (detected by accents or French function words)
  search both languages. get_section/xref/list_acts stay English-canonical.
  Phase 2 (a multilingual embedder) was tried and REJECTED 2026-07-22 —
  regressed every slice and broke confidence calibration; don't retry
  without re-calibrating HEDGE_THRESHOLD. French guidance (D-memos are
  published bilingually) is the remaining bilingual frontier.
- ENF 2 and ENF 14/OP 19 are both ingested from their ATIP releases
  (authorized 2026-07-23, bannered UNOFFICIAL on every chunk). ENF 14's
  image-scan PDF was recovered by OCR (pymupdf rasterization + Windows OCR;
  cached at data/raw/enf/enf14-ocr.txt — enf.py auto-falls-back to
  `*-ocr.txt` when a PDF extracts under 500 chars).
- Typo robustness, two layers: unknown STEMS get the trigram bridge in
  `index._fuzzy_term` (BM25 recall only), and misspelled WORDS are repaired
  by `index._correct_query` (trigram candidates, Damerau-Levenshtein <= 2,
  shared 2-char prefix, frequency floor 5 so OCR debris can't win) -- the
  repaired text feeds the semantic embedder and the reranker. Ultra-rare
  stems (df <= 2) get the correction APPENDED, never substituted, so rare
  legitimate terms keep their recall. The `typo` eval slice tracks it.
- Bilingual guidance: the D-memos and NJC directives now carry their French
  editions (`--fr` modes; dmemos-fr.json / directives-fr.json) alongside
  the French legislation, all lang-routed. Remaining English-only: case
  law, ENF/AMPS/PDI, benefits booklets, and the curated commentary.

## Two retrieval invariants worth not re-breaking (2026-07-29)

Ingesting the French guidance and the wider PDI trees exposed both of these.
Each was measured question-by-question against the previous corpus, not
guessed at — the harness is a per-question rank dump over two corpus
variants, which is how a 5-question shuffle gets attributed instead of
argued about.

- **BM25 collection statistics are language-scoped.** `_build_bm25` keeps
  `idf`/`avgdl` (whole corpus) AND `idf_en`/`avgdl_en` (English scope);
  every English query scores with the English pair. An English query can
  never retrieve the French twins, but a single corpus-wide idf let those
  unreachable chunks set the rarity of every shared token (numbers, act
  codes, folded cognates) and drag the average length: adding the French
  D-memos and directives alone moved legacy Hit@1 0.767 → 0.750 without a
  single French chunk being retrievable. French queries still use the
  whole-corpus pair — they search both languages.
- **The diversity cap keys memoranda on the parent DOCUMENT.** `_source_key`
  used `section` for every memorandum, which is the document identity only
  for the flat families (`_FLAT_MEMO_CODES`: one act_code across ~360
  D-memos / ~1,000 AMPS contraventions). ENF chapters, PDI pages and IRB
  guidelines carry a per-document act_code and use `section` to subdivide
  it, so each ENF section and each PDI page was its own "source" and
  neither RECALL_CAP nor SOURCE_CAP ever bound: the ~150-page PDI refugee
  tree filled the candidate pool and pushed IRPA s. 112 out of the top 20
  of "what is a pre-removal risk assessment" entirely. If a new corpus
  family lands, check which field carries its document identity before
  trusting the caps.