Add the public-service benefit-plan member booklets (new doc_type 'benefits')

Ingests the member booklets of the four federal benefit plans as a new
doc_type='benefits', via canlex/benefits.py:

- PSHCP -- Public Service Health Care Plan (health; active + pensioners)
- PSDCP -- Public Service Dental Care Plan (dental; active employees)
- PDSP -- Pensioners' Dental Services Plan (dental; pensioners)
- PSMIP -- Public Service Management Insurance Plan (life/AD&D/disability;
management + excluded employees)

The three Canada Life booklets are PDFs (fetched via urllib, segmented by
locating each table-of-contents heading in the body text); PSMIP is an HTML
page on canada.ca (fetched via PowerShell like agreement.py, since canada.ca
blocks urllib at the TLS layer, segmented at its h2/h3 headings). One chunk per
booklet section, sub-split at ~1800 chars with tiny head/tail remnants folded
in. 310 chunks total (PSHCP 137, PSDCP 71, PDSP 68, PSMIP 34); corpus 16,194 ->
16,504 chunks.

Booklets are guidance, not governing instruments: server.py labels each result
"_Benefit-plan member booklet ... the plan rules govern_" and the corpus
listing gains a "Benefit plans" group. They are NOT in PRIMARY_DOC_TYPES (so
_ensure_primary doesn't treat them as enacted law), but _source_key returns
None for them so a booklet's distinct sections are never collapsed by the
diversity cap (like statute sections). doc_type filter and tool docs updated to
list 'benefits'.

Why PSMIP despite the FB/BSO focus: it covers EX and excluded CBSA staff
(represented FB officers carry the separate DI plan) -- added on request for
completeness.

145-question eval (4 new benefit-plan gold questions, all pass): Hit@1 0.80 /
Hit@3 0.94 / Hit@5 0.97 / Hit@10 0.99 / MRR 0.87. The one-question Hit@5 dip vs
pre-add (D8-2-1 #5->#6) is a benefits "Out-of-Canada Benefit" section lightly
competing on a customs goods-abroad query -- acceptable for the added coverage.
9 new unit tests (TOC parse, segmentation, splitter head/tail fold, emit,
benefits source-key); 40 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

canlex/benefits.py

@@ -0,0 +1,346 @@
+"""Ingest the member booklets of the federal public-service benefit plans.
+
+Four plans cover federal employees (and a Border Services Officer over a
+career), by population:
+
+  - PSHCP -- Public Service Health Care Plan (health; active + pensioners)
+  - PSDCP -- Public Service Dental Care Plan (dental; active employees)
+  - PDSP  -- Pensioners' Dental Services Plan (dental; pensioners)
+  - PSMIP -- Public Service Management Insurance Plan (life / AD&D / disability
+             insurance; management-category and excluded employees -- so EX and
+             excluded CBSA staff rather than the represented FB group, who carry
+             the separate Disability Insurance Plan)
+
+Each plan's *member booklet* is the plain-language guide the plan administrator
+or Treasury Board publishes -- what is covered, who is eligible, how to claim,
+the appeals process. The booklets are guidance: the governing terms live in the
+PSHCP Directive (already ingested as doc_type='directive') and the respective
+plan rules, which prevail in any discrepancy. They are tagged doc_type=
+'benefits' and labelled as such in results.
+
+Sources: the Canada Life PDFs (welcome.canadalife.com / canadalife.com), which
+-- unlike canada.ca -- serve Python's urllib fine; and the PSMIP booklet, an
+HTML page on canada.ca (which blocks urllib at the TLS layer, so it is fetched
+via PowerShell, as agreement.py does). A PDF booklet carries a table of
+contents whose headings are located in the body to segment it; the HTML booklet
+is segmented at its h2/h3 headings. Either way a retrieved chunk is a coherent
+benefit topic.
+
+    py -m canlex.benefits
+"""
+import io
+import json
+import re
+import subprocess
+import time
+import urllib.request
+
+from bs4 import BeautifulSoup
+from pypdf import PdfReader
+
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "benefits"
+OUT = PROCESSED_DIR / "benefits.json"
+
+_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+       "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")
+
+_CHUNK_CHARS = 1800     # target characters per chunk
+
+# Each plan: its booklet, and how to read it. PDF booklets give the page
+# indices of the table of contents and the first body page (everything before
+# it is cover + TOC). The HTML booklet is parsed at its h2/h3 headings.
+# 'edition' is the booklet's own date (for HTML it is read from the page's
+# "Date modified"); bump a PDF edition and re-run when a new booklet is posted.
+SOURCES = {
+    "PSHCP": {
+        "code": "PSHCP", "format": "pdf",
+        "name": "Public Service Health Care Plan",
+        "url": ("https://www.welcome.canadalife.com/content/dam/rfp/"
+                "welcome-sites/pshcp/PSHCP-member-booklet.pdf"),
+        "label": "Member Booklet",
+        "toc_pages": [2, 3, 4],
+        "body_start": 5,
+        "edition": "June 2024",
+    },
+    "PSDCP": {
+        "code": "PSDCP", "format": "pdf",
+        "name": "Public Service Dental Care Plan",
+        "url": ("https://www.welcome.canadalife.com/content/dam/rfp/"
+                "welcome-sites/psdcp/psdcp-member-booklet.pdf"),
+        "label": "Member Booklet",
+        "toc_pages": [2, 3],
+        "body_start": 4,
+        "edition": "May 2026",
+    },
+    "PDSP": {
+        "code": "PDSP", "format": "pdf",
+        "name": "Pensioners' Dental Services Plan",
+        "url": ("https://www.canadalife.com/content/dam/rfp/welcome-sites/"
+                "pdsp/pdsp-member-booklet.pdf"),
+        "label": "Member Booklet",
+        "toc_pages": [2, 3],
+        "body_start": 4,
+        "edition": "November 2024",
+    },
+    "PSMIP": {
+        "code": "PSMIP", "format": "html",
+        "name": "Public Service Management Insurance Plan",
+        "url": ("https://www.canada.ca/en/treasury-board-secretariat/services/"
+                "benefit-plans/management-insurance-plan/main-plan-booklet.html"),
+        "label": "Main Plan Booklet",
+        "edition": "",   # read from the page's "Date modified"
+    },
+}
+
+# canada.ca section labels that are navigation/footer, not plan content.
+_SKIP_HEADINGS = {"on this page", "page details", "from:", "table of contents"}
+
+# A table-of-contents line: optional dot-leaders, the heading text, then the
+# printed page number. The heading must end in a letter or ')' so a bare page
+# number or a dot run alone is not mistaken for a heading.
+_TOC_LINE = re.compile(r"^[\s.]*(.*?[A-Za-z)])\s*\.*\s*(\d{1,3})\s*$")
+
+
+def _fetch(url, dest):
+    if dest.exists():
+        return dest.read_bytes()
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    req = urllib.request.Request(url, headers={"User-Agent": _UA})
+    with urllib.request.urlopen(req, timeout=90) as resp:
+        dest.write_bytes(resp.read())
+    return dest.read_bytes()
+
+
+def _clean(text):
+    """Tidy extracted PDF text: turn bullets into dashes, collapse runs of
+    spaces, and trim excess blank lines. Apostrophes, accents and en-dashes
+    extract as proper Unicode, so no transliteration is needed."""
+    text = text.replace("•", "\n- ").replace("\xa0", " ")
+    text = re.sub(r"[ \t]+", " ", text)
+    text = re.sub(r"\n[ \t]+", "\n", text)
+    return re.sub(r"\n{3,}", "\n\n", text).strip()
+
+
+def _page_texts(pdf_bytes):
+    reader = PdfReader(io.BytesIO(pdf_bytes))
+    return [p.extract_text() or "" for p in reader.pages]
+
+
+def parse_toc(page_texts, toc_pages):
+    """Return the ordered, de-duplicated list of section headings from the
+    table of contents."""
+    seen, headings = set(), []
+    for i in toc_pages:
+        if i >= len(page_texts):
+            continue
+        for line in page_texts[i].split("\n"):
+            line = line.strip()
+            if not line or "Table of Contents" in line:
+                continue
+            m = _TOC_LINE.match(line)
+            if not m:
+                continue
+            title = re.sub(r"\s+", " ", m.group(1)).strip(" .")
+            if len(title) > 2 and title.lower() not in seen:
+                seen.add(title.lower())
+                headings.append(title)
+    return headings
+
+
+def segment(body, headings):
+    """Split the body text at each heading, in order, returning a list of
+    (heading, section_text). A heading the extractor split across lines is
+    matched with flexible whitespace; a heading not found in the body is
+    skipped (its text folds into the preceding section)."""
+    cuts = []
+    pos = 0
+    for h in headings:
+        pattern = re.compile(r"\s*".join(re.escape(w) for w in h.split()), re.I)
+        m = pattern.search(body, pos)
+        if m:
+            cuts.append((h, m.start()))
+            pos = m.end()
+    sections = []
+    for j, (h, start) in enumerate(cuts):
+        end = cuts[j + 1][1] if j + 1 < len(cuts) else len(body)
+        sections.append((h, body[start:end].strip()))
+    return sections
+
+
+def _split(text, target=_CHUNK_CHARS):
+    """Split a long section into ~target-sized pieces at line boundaries,
+    folding a small trailing remnant back into the previous piece so a section
+    never ends in a stub chunk."""
+    if len(text) <= target:
+        return [text]
+    pieces, buf, size = [], [], 0
+    for line in text.split("\n"):
+        if size + len(line) > target and buf:
+            pieces.append("\n".join(buf))
+            buf, size = [], 0
+        buf.append(line)
+        size += len(line) + 1
+    if buf:
+        pieces.append("\n".join(buf))
+    # Fold a small trailing remnant back into the previous piece, and a small
+    # leading remnant (e.g. a heading on its own line ahead of one very long
+    # unbroken paragraph) into the next, so a section never yields a stub chunk.
+    if len(pieces) > 1 and len(pieces[-1]) < 200:
+        pieces[-2] = pieces[-2] + "\n" + pieces[-1]
+        pieces.pop()
+    if len(pieces) > 1 and len(pieces[0]) < 200:
+        pieces[1] = pieces[0] + "\n" + pieces[1]
+        pieces.pop(0)
+    return pieces
+
+
+def _slug(heading, used):
+    """A short, unique, citable section locator from a heading."""
+    base = re.sub(r"[^a-z0-9]+", "-", heading.lower()).strip("-")[:40] or "section"
+    slug, n = base, 2
+    while slug in used:
+        slug = f"{base}-{n}"
+        n += 1
+    used.add(slug)
+    return slug
+
+
+def _emit(sections, src):
+    """Build CanLex chunk dicts from a list of (heading, section_text), shared
+    by the PDF and HTML paths. Skips pure section-divider headings and
+    sub-splits any section over the chunk budget."""
+    chunks = []
+    used_slugs = set()
+    label = src["label"]
+    for heading, text in sections:
+        # Skip pure section dividers -- a heading whose only content is the
+        # heading itself (its substance lives in the child sections below it).
+        # Compare with whitespace collapsed, since a PDF extractor often breaks
+        # a heading across lines ("...Plan \n(PSDCP)").
+        norm_text = re.sub(r"\s+", " ", text).strip()
+        norm_head = re.sub(r"\s+", " ", heading).strip()
+        remainder = (norm_text[len(norm_head):]
+                     if norm_text.lower().startswith(norm_head.lower())
+                     else norm_text)
+        if len(remainder.strip()) < 40:
+            continue
+        slug = _slug(heading, used_slugs)
+        pieces = _split(text)
+        for k, piece in enumerate(pieces, start=1):
+            section = slug if len(pieces) == 1 else f"{slug}-{k}"
+            note = heading if len(pieces) == 1 else f"{heading} (part {k})"
+            chunks.append({
+                "id": f"benefits-{src['code']}-{section}",
+                "doc_type": "benefits",
+                "act_code": src["code"],
+                "act_short": src["code"],
+                "act_name": f"{src['name']} — {label}",
+                "section": section,
+                "marginal_note": note,
+                "part": f"{src['name']} {label}",
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": "",
+                "current_to": src["edition"],
+                "citation": (f"{src['code']} {label}"
+                             + (f" ({src['edition']})" if src["edition"] else "")
+                             + f", “{heading}”"),
+                "source_url": src["url"],
+            })
+    return chunks
+
+
+def parse_pdf_booklet(pdf_bytes, src):
+    """Segment a PDF booklet by its table-of-contents headings."""
+    pages = _page_texts(pdf_bytes)
+    headings = parse_toc(pages, src["toc_pages"])
+    body = _clean("\n".join(pages[src["body_start"]:]))
+    return _emit(segment(body, headings), src)
+
+
+def _html_block_text(heading):
+    """Readable text from an h2/h3 up to the next heading of the same or higher
+    level; lists become dash items."""
+    lines = []
+    for sib in heading.find_next_siblings():
+        if sib.name in ("h2", "h3"):
+            break
+        if sib.name in ("ul", "ol"):
+            for li in sib.find_all("li", recursive=False):
+                item = re.sub(r"\s+", " ", li.get_text(" ", strip=True)).strip()
+                if item:
+                    lines.append(f"- {item}")
+        elif sib.name in ("p", "div", "table", "section", "h4", "h5"):
+            text = re.sub(r"\s+", " ", sib.get_text(" ", strip=True)).strip()
+            if text:
+                lines.append(text)
+    return "\n".join(lines)
+
+
+def parse_html_booklet(html, src):
+    """Segment the canada.ca HTML booklet at its h2/h3 headings."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main") or soup
+    # Read the page's "Date modified" so the booklet's currency is recorded.
+    if not src["edition"]:
+        t = main.find("time", property="dateModified") or soup.find(
+            "time", property="dateModified")
+        if t:
+            src = {**src, "edition": t.get_text(strip=True)}
+    sections = []
+    for h in main.find_all(["h2", "h3"]):
+        title = re.sub(r"\s+", " ", h.get_text(" ", strip=True)).strip()
+        if not title or title.lower() in _SKIP_HEADINGS:
+            continue
+        body = _html_block_text(h)
+        if body:
+            sections.append((title, f"{title}\n{body}"))
+    return _emit(sections, src)
+
+
+def _fetch_html(url, dest):
+    """Fetch a canada.ca page via PowerShell -- the site blocks Python's HTTP
+    client at the TLS layer, but accepts PowerShell's .NET stack (as
+    agreement.py does)."""
+    if dest.exists():
+        return dest.read_bytes()
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    command = (f"Invoke-WebRequest -Uri '{url}' -OutFile '{dest}' "
+               f"-UseBasicParsing -UserAgent '{_UA}'")
+    subprocess.run(["powershell", "-NoProfile", "-NonInteractive",
+                    "-Command", command],
+                   check=True, capture_output=True, timeout=180)
+    time.sleep(0.5)
+    return dest.read_bytes()
+
+
+def build():
+    all_chunks = []
+    for src in SOURCES.values():
+        print(f"Ingesting {src['code']} {src['label'].lower()} ...")
+        try:
+            if src["format"] == "pdf":
+                pdf = _fetch(src["url"], RAW / f"{src['code']}.pdf")
+                chunks = parse_pdf_booklet(pdf, src)
+            else:
+                html = _fetch_html(src["url"], RAW / f"{src['code']}.html"
+                                   ).decode("utf-8", "replace")
+                chunks = parse_html_booklet(html, src)
+        except Exception as exc:
+            print(f"  !! {src['code']}: {type(exc).__name__}: {exc}")
+            continue
+        all_chunks.extend(chunks)
+        print(f"  {len(chunks)} chunks")
+    PROCESSED_DIR.mkdir(parents=True, exist_ok=True)
+    OUT.write_text(json.dumps(all_chunks, ensure_ascii=False, indent=1),
+                   encoding="utf-8")
+    print(f"\n{len(all_chunks)} benefit-plan chunks from {len(SOURCES)} "
+          f"booklet(s) -> {OUT}")
+
+
+if __name__ == "__main__":
+    build()

canlex/index.py

@@ -309,10 +309,18 @@ class LegislationIndex:
         """The parent document a chunk belongs to, for diversity capping. Returns
         None for primary instruments -- legislation, collective agreements and
         directives -- whose sections are distinct provisions and are never
-        capped; case law is keyed by citation, memoranda by memo number."""
+        capped; case law is keyed by citation, memoranda by memo number.
+
+        Benefit-plan booklets also return None: a booklet's sections are
+        distinct topics (eligibility, coverage, claims, appeals...) like
+        statute sections, so capping the whole booklet to SOURCE_CAP chunks
+        would starve a detailed benefits query. They are not in
+        PRIMARY_DOC_TYPES, though -- they are guidance, not governing
+        instruments, so _ensure_primary does not pull them in as if they were
+        enacted law."""
         c = self.chunks[idx]
         doc_type = c.get("doc_type", "legislation")
-        if doc_type in PRIMARY_DOC_TYPES:
+        if doc_type in PRIMARY_DOC_TYPES or doc_type == "benefits":
             return None
         if doc_type == "memorandum":
             return ("memorandum", c["section"])   # act_code is a shared constant

canlex/server.py

@@ -116,6 +116,13 @@ def _format_section(c: dict, related=None) -> str:
                      "designated for functions, under IRPA and the IRPR. "
                      "Administrative; confirm it is still the current version._")
         lines.append(f"(dated {c['current_to'] or 'n/a'})")
+    elif doc_type == "benefits":
+        lines.append("_Benefit-plan member booklet — a plain-language summary "
+                     "published by the plan administrator (Canada Life). The "
+                     "governing terms are in the plan's directive or rules, "
+                     "which prevail in any discrepancy; quote the booklet for "
+                     "what it says but flag that the plan rules control._")
+        lines.append(f"(booklet edition: {c['current_to'] or 'n/a'})")
     else:
         meta = [f"in force; text current to {c['current_to'] or 'n/a'}"]
         if c["last_amended"]:
@@ -183,8 +190,9 @@ class SearchInput(BaseModel):
         description="Optional filter by source type: 'legislation' (Acts and "
         "regulations), 'memorandum' (CBSA D-Memoranda), 'agreement' (collective "
         "agreements), 'directive' (NJC directives), 'caselaw' (court and "
-        "tribunal decisions), or 'delegation' (IRPA/IRPR delegation and "
-        "designation instruments). Omit to search all.",
+        "tribunal decisions), 'delegation' (IRPA/IRPR delegation and "
+        "designation instruments), or 'benefits' (public-service health and "
+        "dental plan member booklets). Omit to search all.",
     )
 
 
@@ -204,7 +212,7 @@ def canlex_search_legislation(params: SearchInput) -> str:
     """Search Canadian federal law, CBSA D-Memoranda, agreements, NJC directives,
     and leading court decisions.
 
-    The CanLex corpus has six kinds of source: 31 federal Acts and regulations
+    The CanLex corpus has seven kinds of source: 31 federal Acts and regulations
     (immigration, customs, criminal, drugs, food/health, labour, privacy and more);
     CBSA D-Memoranda (the Canada Border Services Agency's administrative guidance on
     how it applies customs and border law); Treasury Board collective agreements
@@ -212,8 +220,11 @@ def canlex_search_legislation(params: SearchInput) -> str:
     (travel, relocation, isolated posts and more); leading decisions of the
     courts and federal tribunals: the Supreme Court, Federal Court of Appeal and
     Federal Court, the Immigration and Refugee Board, and the FPSLREB and CIRB
-    labour boards; and instruments of delegation and designation under IRPA and
-    the IRPR (which officials the Minister has authorized to exercise which powers). Use this for ANY question about that material. It ranks results by relevance and returns
+    labour boards; instruments of delegation and designation under IRPA and
+    the IRPR (which officials the Minister has authorized to exercise which powers);
+    and the member booklets of the public-service benefit plans (the PSHCP health
+    plan and the PSDCP and PDSP dental plans). Use this for ANY question about that
+    material. It ranks results by relevance and returns
     their full text so the answer can cite the actual wording; an explicit section
     reference (e.g. "section 34") is always surfaced. Each result is marked with its
     source type.
@@ -224,8 +235,8 @@ def canlex_search_legislation(params: SearchInput) -> str:
             - top_k (int): How many sections to return, 1-20 (default 6).
             - act (Optional[str]): Restrict to one Act by short name/code, or omit for all.
             - doc_type (Optional[str]): 'legislation', 'memorandum', 'agreement',
-              'directive', 'caselaw', or 'delegation' to restrict to one source
-              type; omit for all.
+              'directive', 'caselaw', 'delegation', or 'benefits' to restrict to
+              one source type; omit for all.
 
     Returns:
         str: Markdown with answering instructions followed by the matching sections.
@@ -312,6 +323,7 @@ def canlex_list_acts() -> str:
     directives: dict[str, dict] = {}
     cases: dict[str, dict] = {}
     delegations: dict[str, dict] = {}
+    benefits: dict[str, dict] = {}
     memo_numbers: set[str] = set()
     memo_chunks = 0
     memo_date = ""
@@ -343,6 +355,12 @@ def canlex_list_acts() -> str:
                 "current_to": c["current_to"], "count": 0,
             })
             entry["count"] += 1
+        elif doc_type == "benefits":
+            entry = benefits.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
         else:
             entry = acts.setdefault(c["act_code"], {
                 "short": c["act_short"], "name": c["act_name"],
@@ -379,10 +397,16 @@ def canlex_list_acts() -> str:
         for a in sorted(delegations.values(), key=lambda x: x["short"]):
             lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} items, "
                          f"dated {a['current_to'] or 'n/a'}")
+    if benefits:
+        lines += ["", "## Benefit plans"]
+        for a in sorted(benefits.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} "
+                         f"sections, booklet edition {a['current_to'] or 'n/a'}. "
+                         f"Plain-language member guide; the plan rules govern.")
     lines += ["", "Search with canlex_search_legislation; filter by doc_type "
               "(legislation / memorandum / agreement / directive / caselaw / "
-              "delegation). Fetch a known provision with canlex_get_section, or "
-              "a case's citations with canlex_case."]
+              "delegation / benefits). Fetch a known provision with "
+              "canlex_get_section, or a case's citations with canlex_case."]
     return "\n".join(lines)
 
 

data/eval/questions.json

@@ -132,7 +132,11 @@
   {"query": "What allowances are available to a federal employee posted at an isolated post?", "answers": [["Isolated Posts and Government Housing Directive", ""]]},
   {"query": "What relocation expenses are reimbursed when a federal employee must move for work?", "answers": [["NJC Relocation Directive", ""]]},
   {"query": "What occupational health and safety obligations does the NJC directive place on the employer?", "answers": [["Occupational Health and Safety Directive", ""]]},
-  {"query": "What coverage does the Public Service Health Care Plan provide?", "answers": [["Public Service Health Care Plan Directive", ""]]},
+  {"query": "What coverage does the Public Service Health Care Plan provide?", "answers": [["Public Service Health Care Plan Directive", ""], ["PSHCP", ""]]},
+  {"query": "What does the PSHCP member booklet say about prescription drug coverage?", "answers": [["PSHCP", ""]]},
+  {"query": "What dental services are covered under the Public Service Dental Care Plan for employees?", "answers": [["PSDCP", ""]]},
+  {"query": "What dental coverage do federal pensioners get under the Pensioners' Dental Services Plan?", "answers": [["PDSP", ""]]},
+  {"query": "What life and disability insurance does the Public Service Management Insurance Plan provide for excluded employees?", "answers": [["PSMIP", ""]]},
   {"query": "What support do the Foreign Service Directives give an employee posted outside Canada?", "answers": [["Foreign Service Directives", ""]]},
   {"query": "Who has a right to appeal a decision to the Immigration Appeal Division?", "answers": [["IRPA", "63"]]},
   {"query": "What are the objectives of the Immigration and Refugee Protection Act?", "answers": [["IRPA", "3"]]},

tests/test_benefits.py

@@ -0,0 +1,79 @@
+"""Unit tests for benefit-plan booklet ingestion (canlex/benefits.py)."""
+import unittest
+
+from canlex import benefits
+
+
+SRC = {"code": "PSHCP", "format": "pdf", "name": "Public Service Health Care Plan",
+       "url": "http://example/booklet.pdf", "label": "Member Booklet",
+       "edition": "June 2024"}
+
+
+class TocParseTests(unittest.TestCase):
+    def test_strips_dot_leaders_and_page_numbers(self):
+        pages = ["cover", "",
+                 "Table of Contents\n"
+                 "....................Eligibility 9\n"
+                 "Drug Benefit ........ 21\n"
+                 "Appendix A — Rates 40\n"]
+        headings = benefits.parse_toc(pages, [2])
+        self.assertEqual(headings, ["Eligibility", "Drug Benefit",
+                                    "Appendix A — Rates"])
+
+    def test_dedupes_case_insensitively(self):
+        pages = ["", "", "Eligibility 9\nELIGIBILITY 9\n"]
+        self.assertEqual(benefits.parse_toc(pages, [2]), ["Eligibility"])
+
+
+class SegmentTests(unittest.TestCase):
+    def test_splits_body_at_each_heading_in_order(self):
+        body = ("Eligibility\nYou must be an employee.\n"
+                "Drug Benefit\nDrugs are covered at 80%.")
+        secs = benefits.segment(body, ["Eligibility", "Drug Benefit"])
+        self.assertEqual(secs[0][0], "Eligibility")
+        self.assertIn("employee", secs[0][1])
+        self.assertEqual(secs[1][0], "Drug Benefit")
+        self.assertIn("80%", secs[1][1])
+
+    def test_heading_split_across_lines_is_matched(self):
+        # PDF extraction often breaks a heading across a line; flexible
+        # whitespace matching should still locate it.
+        body = "Welcome to the Public Service Health Care Plan \n(PSHCP) text"
+        secs = benefits.segment(
+            body, ["Welcome to the Public Service Health Care Plan (PSHCP)"])
+        self.assertEqual(len(secs), 1)
+
+
+class SplitTests(unittest.TestCase):
+    def test_short_text_is_one_piece(self):
+        self.assertEqual(benefits._split("short"), ["short"])
+
+    def test_folds_tiny_leading_heading_into_next_piece(self):
+        # A heading on its own line ahead of one very long unbroken paragraph
+        # must not become a stub piece.
+        text = "Basic Life Insurance\n" + ("x" * 2000)
+        pieces = benefits._split(text)
+        self.assertTrue(all(len(p) >= 60 for p in pieces))
+        self.assertTrue(pieces[0].startswith("Basic Life Insurance"))
+
+
+class EmitTests(unittest.TestCase):
+    def test_skips_heading_only_divider_sections(self):
+        sections = [("Introduction", "Introduction"),
+                    ("Eligibility", "Eligibility\n" + "You qualify if employed. " * 4)]
+        chunks = benefits._emit(sections, SRC)
+        codes = {c["marginal_note"] for c in chunks}
+        self.assertNotIn("Introduction", codes)
+        self.assertIn("Eligibility", codes)
+
+    def test_chunk_carries_benefits_doc_type_and_citation(self):
+        sections = [("Drug Benefit", "Drug Benefit\n" + "Covered at 80%. " * 6)]
+        c = benefits._emit(sections, SRC)[0]
+        self.assertEqual(c["doc_type"], "benefits")
+        self.assertEqual(c["act_code"], "PSHCP")
+        self.assertIn("Drug Benefit", c["citation"])
+        self.assertIn("June 2024", c["citation"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_index.py

@@ -90,6 +90,16 @@ class SourceKeyTests(unittest.TestCase):
         self.assertEqual(idx._source_key(0), ("memorandum", "D1-1-1"))
         self.assertEqual(idx._source_key(1), ("caselaw", "2019 SCC 65"))
 
+    def test_benefits_sections_are_not_capped(self):
+        # A benefit-plan booklet's sections are distinct topics, like statute
+        # sections, so the whole booklet is never collapsed to SOURCE_CAP.
+        idx = bare_index([
+            chunk(doc_type="benefits", act_code="PSHCP", section="drug-benefit"),
+            chunk(doc_type="benefits", act_code="PSHCP", section="exclusions"),
+        ])
+        self.assertIsNone(idx._source_key(0))
+        self.assertIsNone(idx._source_key(1))
+
 
 class DiversifyTests(unittest.TestCase):
     def test_caps_caselaw_per_decision(self):