Sprint tests (106 green), rewritten README, maintainer RUNBOOK, findings doc resolved

README.md

@@ -8,18 +8,38 @@ pinned: false
 # CanLex MCP
 
 A remote [Model Context Protocol](https://modelcontextprotocol.io) server for
-Canadian legal research. It exposes four read-only tools over streamable HTTP:
+Canadian legal research, built for front-line border-services work. It exposes
+seven read-only tools over streamable HTTP (or local stdio):
 
 - **canlex_search_legislation** — hybrid (BM25 + semantic) search with a
-  cross-encoder reranker over federal border, customs, criminal, drug,
-  food/health, labour and privacy legislation, CBSA D-Memoranda, Treasury Board
-  collective agreements, and National Joint Council directives.
-- **canlex_get_section** — fetch one provision verbatim by Act and section.
-- **canlex_list_acts** — list the corpus and its currency.
-- **canlex_case** — live CanLII case citator (citation graph for a case URL).
+  cross-encoder reranker over the whole corpus: federal border, customs,
+  criminal, firearms, drug, food/health, labour and privacy legislation with
+  their Schedules; the ENF operational manuals; CBSA D-Memoranda and the AMPS
+  Master Penalty Document; the Charter; Treasury Board collective agreements;
+  NJC directives; ~110 leading decisions; delegation instruments;
+  benefit-plan booklets; and CanLex's curated commentary.
+- **canlex_get_section** — fetch one provision verbatim by Act and section
+  (accepts 's. 34', '34(1)(c)', 'Schedule I').
+- **canlex_us_disposition** — step 1 of the IRPA s. 36 screening chain: is a
+  US disposition a "conviction"? 51-jurisdiction state-by-state dataset.
+- **canlex_us_equivalency** — step 2: the Canadian equivalent offence, its
+  verified maximum penalty, and the inadmissibility branch.
+- **canlex_rehabilitation** — step 3: deemed rehabilitation / eligibility to
+  apply, under IRPA s. 36(3)(c) and IRPR ss. 17–18.1.
+- **canlex_list_acts** — the corpus inventory and its currency.
+- **canlex_case** — live CanLII case citator (citation graph by neutral
+  citation or URL).
+
+The server also publishes an MCP prompt (`us_record_screening`, the guided
+three-step screening flow) and resources (the corpus inventory and the
+conviction/equivalency methodology texts) that any MCP client can use.
 
 Retrieval runs fully locally inside the container and needs no API key. The
 optional case citator reads a CanLII key from the `CANLII_API_KEY` environment
-variable.
+variable. A weekly scheduled staleness check watches upstream sources for
+drift; see RUNBOOK.md for the maintainer's refresh choreography.
 
-Add this Space's `/mcp` URL as a custom connector in any Claude client.
+Add this Space's `/mcp` URL as a custom connector in any Claude client, or
+point any MCP-capable agent (Gemini in Antigravity, etc.) at it. A companion
+Gradio webapp (`webapp/`, deployed as its own Space) gives the same corpus a
+Gemini-powered agentic chat front-end.

RUNBOOK.md

@@ -0,0 +1,107 @@
+# CanLex maintainer runbook
+
+Everything a future maintainer needs that the code alone does not say.
+
+## The refresh choreography
+
+Order matters — chunks feed embeddings feed the eval:
+
+```
+py -m canlex.ingest [--force] [CODES]   # Justice Laws XML -> per-Act JSON (incl. Schedules)
+py -m canlex.dmemo                      # CBSA D-Memoranda
+py -m canlex.enf                        # ENF operational manuals (canada.ca PDFs)
+py -m canlex.amps                       # AMPS Master Penalty Document
+py -m canlex.charter                    # Charter / Constitution HTML
+py -m canlex.caselaw                    # Lexum/Decisia courts + cached CanLII texts
+py -m canlex.delegation                 # CBSA/IRCC delegation instruments
+py -m canlex.directive                  # NJC directives
+py -m canlex.agreement                  # FB collective agreement
+py -m canlex.benefits                   # benefit-plan booklets
+py -m canlex.tariff_schedule            # Customs Tariff ch. 98/99 (CBSA edition)
+py -m canlex.commentary                 # curated datasets -> commentary chunks
+py -m canlex.embed                      # re-embed EVERYTHING (run after ANY of the above)
+py -m unittest discover -s tests        # all offline
+py -m canlex.eval                       # gate: no metric below the last committed run
+```
+
+`canlex.embed` is mandatory after any chunk change: the index zero-fills
+vectors for chunks missing from embeddings.npz and quietly degrades them to
+BM25-only (a stderr warning is the only signal).
+
+## Deploys
+
+- **MCP Space** (`Beemer0/CanLex`): commit, then `git push space main`.
+  The Space rebuilds automatically. `embeddings.npz` ships in-repo via
+  git-LFS — a fresh clone needs `git lfs install`.
+- **Webapp Space** (`Beemer0/Canlex-web`): separate repo. Clone it, copy
+  `webapp/app.py` + `webapp/requirements.txt` + `webapp/Dockerfile` +
+  `webapp/README.md` over, commit, push. Do NOT use `hf upload` — HF gates
+  it behind PRO for Docker/Gradio Spaces; git pushes work on the free tier.
+
+## Secrets and variables (set in each Space's settings)
+
+| Where | Name | Purpose |
+|---|---|---|
+| MCP Space | `CANLII_API_KEY` | optional, enables canlex_case |
+| Webapp | `GEMINI_API_KEY` | required |
+| Webapp | `CANLEX_WEB_AUTH` | required (`user:pass` per line); the app refuses to start without it |
+| Webapp | `CANLEX_GEMINI_MODEL` | optional; the deployed Space sets `gemini-2.5-pro` (code default is flash) |
+
+Locally, the CanLII key lives in `canlii_key.txt` (gitignored — never
+commit it).
+
+## The staleness watch
+
+`py -m canlex.refresh` (alias `--check`) diffs upstream: Justice Laws
+consolidation dates, CBSA/IRCC delegation instruments, NJC directive dates,
+the D-memo index's newest Date modified, curated-dataset age (>180 days →
+AGING) and curated-vs-chunk regen drift. Non-zero exit = something drifted.
+A scheduled task runs it Mondays 03:01 local — if this machine is off, the
+check silently doesn't happen that week. Note: network fetch errors on the
+legislation rows DO fail the check (exit 1) so a broken watch can't pass
+silently.
+
+`_PENDING_BILLS` in canlex/refresh.py names bills heading into force that
+affect corpus Acts — edit that list when a new one appears (source-code edit,
+deliberate).
+
+## Curated-dataset review protocol
+
+`data/curated/us_dispositions.json` and `us_equivalency.json` are the only
+hand-maintained legal content. To change them: edit → bump `"reviewed"` →
+`py -m canlex.commentary` → `py -m canlex.embed` → eval → deploy. The refresh
+check flags any drift between the curated `reviewed` date and the rendered
+chunks. Provenance and the verbatim-verification log live inside the JSON
+files themselves; keep them current.
+
+## Backups
+
+The HF Space remote is also production. A full-history bundle lives at
+`C:\projects\CanLex-backup.bundle` (recreate: `git bundle create
+../CanLex-backup.bundle --all`). For a real offsite mirror, create a private
+GitHub repo and:
+
+```
+git remote add mirror https://github.com/<you>/CanLex.git
+git push mirror --all
+```
+
+(GitHub needs git-lfs enabled for embeddings.npz, or add it to a
+`.lfsconfig`-excluded push.)
+
+## Known gaps / parked
+
+- Hill, Brannson and Li (the FCA equivalency trilogy) have curated
+  `CANLII_CASES` entries but need their page text captured manually from
+  CanLII (DataDome blocks scripts): open the URL each stub prints, save the
+  page text to the named `data/raw/caselaw/*.txt` file, re-run
+  `py -m canlex.caselaw && py -m canlex.embed`.
+- ENF chapters 2, 7, 9, 11, 13, 14, 16, 20 (detention), 25, 27 are not at
+  the legacy PDF URL pattern; the current-policy PDI HTML tree is the
+  eventual replacement source.
+- Dag (FCA) and Kinhnicki (FPSLREB) parse to 2/1 chunks (paragraph-detector
+  fallthrough on those cached pages) — known, low-impact.
+- French support is phase 1 only (accent-folded tokens + FR→EN synonym
+  bridges; the `french` eval slice tracks it). Phase 2 = multilingual
+  embedder via the `CANLEX_EMB_REPO` env hooks; phase 3 = parallel French
+  corpus.

precision-findings.md

@@ -1,5 +1,13 @@
 # CanLex retrieval — precision investigation (2026-05-21)
 
+> **STATUS (2026-07-22): RESOLVED / SUPERSEDED.** The concerns this
+> investigation raised were addressed by the subsequent eval-gated retrieval
+> work — `b2950fb` (reranker pool + doc-cap tuning), `d24e96c` (per-source
+> recall cap), the 2026-06/07 sweeps in `data/eval/sweep.log`, and the
+> 2026-07 filter-aware-recall + repealed-penalty changes. Kept as the
+> diagnostic record; its specific recommendation should NOT be applied on
+> top of the current knobs without a fresh sweep.
+
 Investigation of the persistent eval misses, with a tested, recommended fix.
 **No retrieval-algorithm change has been deployed** — this is for review.
 

tests/test_curated_schema.py

@@ -0,0 +1,90 @@
+"""Schema validation of the REAL curated datasets.
+
+test_commentary.py deliberately exercises the renderer on a minimal in-memory
+dataset; nothing previously validated the actual shipped JSON, so a hand-edit
+typo in the 51-jurisdiction survey surfaced only at runtime in production.
+"""
+import json
+import unittest
+from pathlib import Path
+
+CURATED = Path(__file__).resolve().parent.parent / "data" / "curated"
+
+VERDICTS = {"yes", "likely-yes", "likely-no", "no", "fact-specific", "depends"}
+STATUSES = {"settled", "judicially-considered", "guidance-only", "no-authority"}
+US_STATES = {
+    "Alabama", "Alaska", "Arizona", "Arkansas", "California", "Colorado",
+    "Connecticut", "Delaware", "District of Columbia", "Florida", "Georgia",
+    "Hawaii", "Idaho", "Illinois", "Indiana", "Iowa", "Kansas", "Kentucky",
+    "Louisiana", "Maine", "Maryland", "Massachusetts", "Michigan",
+    "Minnesota", "Mississippi", "Missouri", "Montana", "Nebraska", "Nevada",
+    "New Hampshire", "New Jersey", "New Mexico", "New York",
+    "North Carolina", "North Dakota", "Ohio", "Oklahoma", "Oregon",
+    "Pennsylvania", "Rhode Island", "South Carolina", "South Dakota",
+    "Tennessee", "Texas", "Utah", "Vermont", "Virginia", "Washington",
+    "West Virginia", "Wisconsin", "Wyoming", "general",
+}
+
+
+class DispositionsSchemaTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.data = json.loads(
+            (CURATED / "us_dispositions.json").read_text(encoding="utf-8"))
+
+    def test_reviewed_date_present(self):
+        self.assertRegex(self.data.get("reviewed", ""), r"^\d{4}-\d{2}-\d{2}$")
+
+    def test_entries_have_required_keys(self):
+        for e in self.data["dispositions"]:
+            for key in ("id", "names", "is_conviction", "status", "analysis"):
+                self.assertIn(key, e, f"{e.get('id', '?')} missing {key}")
+            self.assertTrue(e["names"], f"{e['id']}: empty names")
+
+    def test_verdicts_are_legal_values(self):
+        for e in self.data["dispositions"]:
+            self.assertIn(e["is_conviction"], VERDICTS, e["id"])
+            self.assertIn(e["status"], STATUSES, e["id"])
+            for v in e.get("state_variations", []):
+                if v.get("is_conviction"):
+                    self.assertIn(v["is_conviction"], VERDICTS,
+                                  f"{e['id']}/{v.get('state')}")
+
+    def test_state_names_valid(self):
+        for e in self.data["dispositions"]:
+            for v in e.get("state_variations", []):
+                self.assertIn(v["state"], US_STATES,
+                              f"{e['id']}: unknown state {v['state']!r}")
+
+    def test_authorities_carry_citations(self):
+        for e in self.data["dispositions"]:
+            for a in e.get("authorities", []):
+                self.assertTrue(a.get("cite"), f"{e['id']}: authority no cite")
+
+    def test_renderer_accepts_every_entry(self):
+        from canlex.commentary import _entry_text
+        for e in self.data["dispositions"]:
+            text = _entry_text(e)
+            self.assertGreater(len(text), 100, e["id"])
+
+
+class EquivalencySchemaTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.data = json.loads(
+            (CURATED / "us_equivalency.json").read_text(encoding="utf-8"))
+
+    def test_pairings_have_required_keys(self):
+        for p in self.data["pairings"]:
+            for key in ("id", "us_terms", "canadian_offence", "penalty",
+                        "branch", "status", "analysis"):
+                self.assertIn(key, p, f"{p.get('id', '?')} missing {key}")
+            self.assertTrue(p["us_terms"], f"{p['id']}: empty us_terms")
+
+    def test_methodology_framework_present(self):
+        ids = {m["id"] for m in self.data.get("methodology", [])}
+        self.assertIn("equivalency-framework", ids)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_sprint.py

@@ -0,0 +1,118 @@
+"""Offline tests for the sprint's parser, retrieval and server changes."""
+import unittest
+import xml.etree.ElementTree as ET
+
+from canlex.ingest import _schedule_chunks, _table_text
+from canlex.index import _fold_accents, tokenize
+from canlex.server import _normalize_section
+from canlex.citator import canlii_url_from_citation
+
+_SRC = {"short": "TestAct", "name": "Test Act", "web_base": "https://x/"}
+
+_SCHED_XML = """<Statute>
+<Schedule>
+ <ScheduleFormHeading><Label>SCHEDULE I</Label></ScheduleFormHeading>
+ <List><Item><Label>1</Label><Text>Opium (all salts)</Text>
+   <List><Item><Label>(1)</Label><Text>Ketamine</Text></Item></List>
+ </Item></List>
+</Schedule>
+<Schedule>
+ <ScheduleFormHeading><Label>SCHEDULE II</Label></ScheduleFormHeading>
+ <Repealed>[Repealed]</Repealed>
+</Schedule>
+<Schedule id="NifProvs">
+ <ScheduleFormHeading><TitleText>AMENDMENTS NOT IN FORCE</TitleText></ScheduleFormHeading>
+ <List><Item><Text>never rendered here</Text></Item></List>
+</Schedule>
+<Schedule>
+ <ScheduleFormHeading><Label>SCHEDULE III</Label><TitleText>Violations</TitleText></ScheduleFormHeading>
+ <TableGroup><table><tgroup><tbody>
+   <row><entry>Provision</entry><entry>Penalty</entry></row>
+   <row><entry>s. 12</entry><entry>$500</entry></row>
+ </tbody></tgroup></table></TableGroup>
+</Schedule>
+</Statute>"""
+
+
+class ScheduleParsingTests(unittest.TestCase):
+    def setUp(self):
+        self.root = ET.fromstring(_SCHED_XML)
+
+    def chunks(self, src=None):
+        return _schedule_chunks(self.root, "T-1", src or _SRC, "2026-01-01")
+
+    def test_list_schedule_rendered_with_nested_items(self):
+        c = self.chunks()[0]
+        self.assertIn("Opium", c["text"])
+        self.assertIn("Ketamine", c["text"])
+        self.assertEqual(c["section"], "Schedule I")
+        self.assertEqual(c["status"], "in force")
+
+    def test_repealed_schedule_flagged(self):
+        c = next(x for x in self.chunks() if x["section"] == "Schedule Ii")
+        self.assertEqual(c["status"], "repealed")
+
+    def test_nifprovs_excluded(self):
+        self.assertFalse([c for c in self.chunks()
+                          if "never rendered" in c["text"]])
+
+    def test_table_rows_pipe_separated(self):
+        c = next(x for x in self.chunks() if "Violations" in x["marginal_note"])
+        self.assertIn("s. 12 | $500", c["text"])
+
+    def test_skip_schedules_optout(self):
+        self.assertEqual(self.chunks({**_SRC, "skip_schedules": True}), [])
+
+
+class AccentFoldingTests(unittest.TestCase):
+    def test_fold(self):
+        self.assertEqual(_fold_accents("détention"), "detention")
+
+    def test_tokenize_french(self):
+        # Folded French stems to the same normalized token as the English word
+        # ('detention' and 'détention' both -> 'detain').
+        self.assertEqual(tokenize("détention"), tokenize("detention"))
+
+    def test_english_unchanged(self):
+        self.assertEqual(tokenize("seizure of goods"),
+                         tokenize("seizure of goods"))
+
+
+class SectionNormalizationTests(unittest.TestCase):
+    CASES = {
+        "s. 34(1)(c)": "34", "section 20.1": "20.1", "s 34": "34",
+        "ss. 18(2)": "18", "34": "34", "Schedule I": "Schedule I",
+        "schedule 2": "schedule 2", "§34": "34",
+    }
+
+    def test_normalization_table(self):
+        for given, want in self.CASES.items():
+            self.assertEqual(_normalize_section(given), want, given)
+
+
+class CitatorCourtMapTests(unittest.TestCase):
+    def test_fpslreb(self):
+        self.assertIn("/ca/pslreb/doc/2023/2023fpslreb12/",
+                      canlii_url_from_citation("2023 FPSLREB 12"))
+
+    def test_onca_provincial_segment(self):
+        self.assertIn("/on/onca/doc/2024/2024onca608/",
+                      canlii_url_from_citation("2024 ONCA 608"))
+
+    def test_scc_unchanged(self):
+        self.assertIn("/ca/scc/doc/2019/2019scc65/",
+                      canlii_url_from_citation("2019 SCC 65"))
+
+
+class RefreshCuratedTests(unittest.TestCase):
+    def test_check_curated_runs_offline(self):
+        from canlex.refresh import check_curated
+        rows = check_curated("2026-07-22")
+        files = {r["file"] for r in rows}
+        self.assertIn("us_dispositions.json", files)
+        for r in rows:
+            self.assertIn(r["status"], ("ok", "aging", "drift", "error"))
+
+
+if __name__ == "__main__":
+    unittest.main()