multi-record: anchor-first hybrid resolver (client 0.4.9 + mcp 0.1.8)

Fix the multi-record / linked-record retrieval regression at scale. The old
corpus-fraction selectivity cutoff (round(0.15 * corpus_n)) lost meaning past
~150 records and let cross-cluster records pollute results (~0.36 recall at
50-100 companies). The resolver is now anchor-first with a hybrid gate: a
candidate survives if it is in the query's anchor cluster (matches a rarest-term
anchor) OR clears a high-coverage bar. Scale-invariant; abstention and
terminal/prep gates unchanged.

Also: search() cross-tier rank tiebreaker (content tiers before contentless
journal) and search_entities(category=) anchor filter. mcp pins client>=0.4.9.

Validated: client 110 + mcp 22 tests green; synthetic 480-record A/B (full
recall, 0 cross-cluster pollution vs 1920); LongMemEval retrieval diagnostic
(per-question oracle NEW>=OLD).

sibyl-memory-client/CHANGELOG.md

@@ -4,6 +4,45 @@ All notable changes to `sibyl-memory-client` are recorded here. Format
 follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning
 follows [SemVer](https://semver.org/).
 
+## [0.4.9] - 2026-06-06
+
+### Fixed
+
+- **Multi-record search recall/precision regression at scale (anchor-first hybrid resolver).**
+  `multi_record_search` used a corpus-fraction selectivity cutoff
+  (`round(0.15 * corpus_n)`) calibrated on a 24-record reconstruction. Past ~150
+  records the cutoff lost meaning: almost every term read as "selective," so
+  cross-cluster records cleared the gate and polluted results (tester Sylvain
+  Runs 16/17, ~0.36 recall at 50-100 companies). The resolver is now anchor-first:
+  anchor terms are the rarest tokens, defined RELATIVE to the rarest query term
+  (`df <= ANCHOR_BAND * min_df`, scale-invariant). The gate is a HYBRID: a
+  candidate survives if it is in the anchor's cluster (matches an anchor term) OR
+  clears the high-coverage bar `ANCHOR_HYBRID_HI` (genuinely relevant despite
+  lacking the rare anchor). A pure strict filter killed cross-cluster pollution
+  but over-dropped natural-language evidence; the hybrid keeps both. Abstention
+  (zero-support term) and the terminal/prep gates are unchanged. Validated two
+  ways: (a) synthetic 480-record workflow A/B — full recall, 0 cross-cluster
+  pollution vs the old code's 1,920 polluting hits over 120 queries (matches
+  tester Runs 24-29); (b) real-data LongMemEval retrieval diagnostic — per-question
+  (oracle) retrieval is not regressed (NEW >= OLD, +3.4pts), and in a combined-
+  store contamination stress NEW cuts cross-question pollution ~29% for a small
+  recall trade. Regression guard: `tests/test_anchor_resolver_2026_06_06.py`.
+
+- **Cross-tier rank comparability.** `search()` BM25 ranks are not on a common
+  scale across FTS tables (`journal_events_fts` is contentless). Added a tier
+  tiebreaker so content tiers (entity/state/reference) sort before journal at equal
+  rank, layered on the existing 0.4.7 journal cap. (tester email 19e7eb3096b4dae5)
+
+### Added
+
+- **`search_entities(category=...)`.** Optional exact-match category anchor on
+  entity FTS, removing topical bleed across categories on multi-entity workloads
+  (tester email 19e7e75af0b7780a). Backward compatible (defaults to all categories).
+
+Sourced from Sylvain's beta Runs 24-29 + the bugflow batch dedup; this single
+patch also supersedes ~20 already-fixed entries that had accumulated in the
+bug-batch queue.
+
 ## [0.4.8] - 2026-06-04
 
 ### Fixed

sibyl-memory-client/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sibyl-memory-client"
-version = "0.4.8"
+version = "0.4.9"
 description = "Local-first agentic memory SDK. SQLite-backed five-tier hierarchical schema, FTS5 search, multi-tenant, with self-learning skill detection and local memory linter. Foundation of the Sibyl Memory Plugin family."
 authors = [{ name = "SIBYL, Sibyl Labs LLC", email = "sibyl@sibyllabs.org" }]
 license = { text = "MIT" }

sibyl-memory-client/src/sibyl_memory_client/client.py

@@ -881,7 +881,8 @@ class MemoryClient:
     # ------------------------------------------------------------------
     # FTS5 search
     # ------------------------------------------------------------------
-    def search_entities(self, query: str, *, limit: int = 20, prefix: bool = False) -> list[dict[str, Any]]:
+    def search_entities(self, query: str, *, limit: int = 20, prefix: bool = False,
+                        category: str | None = None) -> list[dict[str, Any]]:
         """Full-text search over entity name + category + body via FTS5.
 
         Returns warm-tier entity rows only. For cross-tier search (entities +
@@ -891,6 +892,11 @@ class MemoryClient:
         (``name:foo``) and unclosed quotes can't escape into the parser.
         Set ``prefix=True`` for prefix matching on the final token.
 
+        Pass ``category="<name>"`` to anchor the search to a single entity
+        category (exact match); this removes topical bleed across categories on
+        multi-entity workloads (tester email 19e7e75af0b7780a). Omit to search
+        all categories.
+
         Returns: list of entity rows. Each row is a dict with keys
             id, tenant_id, category, name, status, body, created_at, updated_at
             (body is JSON-deserialized).
@@ -904,15 +910,18 @@ class MemoryClient:
         # external-content FTS5: join by rowid back to base table.
         # _fts_query handles classification (v0.4.0 KAPPA) + corruption
         # containment (poisoned-index DatabaseError self-heals or returns []).
+        cat_clause = " AND e.category = ?" if category else ""
+        params = ((match_q, self._tenant_id, category, limit) if category
+                  else (match_q, self._tenant_id, limit))
         with self._storage.connection() as conn:
             rows = _fts_query(
                 conn,
                 "SELECT e.id, e.tenant_id, e.category, e.name, e.status, e.body, e.created_at, e.updated_at "
                 "FROM entities_fts f "
                 "JOIN entities e ON e.rowid = f.rowid "
-                "WHERE entities_fts MATCH ? AND f.tenant_id = ? "
+                "WHERE entities_fts MATCH ? AND f.tenant_id = ?" + cat_clause + " "
                 "ORDER BY rank LIMIT ?",
-                (match_q, self._tenant_id, limit),
+                params,
                 "entities_fts",
             )
         return [self._row_to_entity(r) for r in rows]
@@ -1049,8 +1058,12 @@ class MemoryClient:
                         },
                         "snippet": r["snip"], "rank": r["rank"], "ts": r["ts"],
                     })
-        # Sort by rank (lower = better in FTS5) and apply global limit
-        hits.sort(key=lambda h: h["rank"])
+        # Sort by rank (lower = better in FTS5), with a tier tiebreaker: at
+        # comparable rank the content tiers (entity/state/reference) sort before
+        # the contentless journal tier, whose BM25 scores are not on the same
+        # scale (cross-tier rank comparability, tester email 19e7eb3096b4dae5).
+        _tier_rank = {"entity": 0, "state": 0, "reference": 0, "journal": 1}
+        hits.sort(key=lambda h: (h["rank"], _tier_rank.get(h["tier"], 0)))
         return hits[:limit]
 
     # ------------------------------------------------------------------

sibyl-memory-client/src/sibyl_memory_client/multi_record.py

@@ -14,20 +14,37 @@ linked records returns only the single strongest match and misses the rest.
                      (so "rejected" / "denied" / injection queries return []);
                    - on a terminal-state query, drop purely-preparatory records
                      (draft / triage / forecast), negation-aware;
-                   - a candidate must match >= 1 rare/selective term, not just
-                     common ones (kills cross-talk from neighbouring clusters);
-                   - rank by IDF-weighted coverage, keep >= COVERAGE_THRESHOLD.
-
-Bench (reconstructed Run15 oracle, client 0.4.x): baseline single-pass 4/10;
-recall-only multipass 3/10 (REGRESSES — breaks abstention + pulls distractors);
-this 10/10. The verify gates are load-bearing — recall alone regresses.
-
-CAVEAT — the constants below (SELECTIVE_CUTOFF_FRAC, COVERAGE_THRESHOLD, the
-prep/terminal lexicon, the strict zero-support abstention) are tuned on a
-24-record reconstruction, NOT generalized to production-scale corpora. Validate
-against real-scale data or gate behind a flag before relying on it at scale.
-Generalization candidates: corpus-relative IDF percentile, normalized coverage
-threshold, anchor-term / min-coverage abstention, learned state classification.
+                   - ANCHOR-FIRST (hybrid): keep a candidate that is in the
+                     anchor's cluster (matches >= 1 anchor term, the rarest most
+                     discriminating tokens) OR clears the high-coverage bar
+                     ANCHOR_HYBRID_HI. A non-anchor, mid-coverage candidate is
+                     cross-cluster pollution and is dropped. The pure strict
+                     filter killed pollution but over-dropped natural-language
+                     evidence that lacks the rare anchor; the hybrid keeps both;
+                   - rank by IDF-weighted coverage with a tier tiebreaker
+                     (content tiers before contentless journal), keep
+                     >= COVERAGE_THRESHOLD.
+
+Bench: baseline single-pass 4/10; recall-only multipass 3/10 (REGRESSES). The
+prior retrieve-then-verify scored 10/10 at 24 records but only ~0.36 recall at
+50-100 companies (tester Runs 16/17) because its selectivity cutoff was a corpus
+fraction (round(0.15 * corpus_n)) that lost meaning at scale: past ~150 records
+almost every term read as "selective," so cross-cluster records cleared the gate.
+The anchor-first rewrite (this version) defines the anchor RELATIVE to the rarest
+query term, so the precision gate is scale-invariant (tester Runs 24-29:
+100/100 recall, 0 pollution at 100 companies / 1621 writes). Abstention and the
+terminal/prep gates are preserved unchanged.
+
+ANCHOR_HYBRID_HI was tuned on a real-data retrieval diagnostic (LongMemEval text
+combined into one store): the pure anchor-only filter regressed natural-language
+recall (gold evidence that lacks the rare anchor); HI=0.65 restores it while
+keeping synthetic-workflow pollution at 0. Per-question (oracle) retrieval is not
+regressed by this change (NEW >= OLD).
+
+CAVEAT — COVERAGE_THRESHOLD, ANCHOR_BAND, ANCHOR_HYBRID_HI, and the prep/terminal
+lexicon are defaults validated against the synthetic multi-cluster scale test
+(tests/test_anchor_resolver_2026_06_06.py) + the LongMemEval retrieval diagnostic;
+re-validate if corpus structure changes.
 
 Uses only the public MemoryClient surface (search / list_entities), so it adds
 no coupling to client internals.
@@ -51,10 +68,17 @@ _PREP_RE = re.compile(
     r'\b(draft|triage|forecast|planning|proposed|tentative|pending|agenda|'
     r'scheduled|rehearsal|sample|option|wip|follow-?up)\b|work in progress')
 
-# --- bench-tuned constants (see CAVEAT in the module docstring) -------------
-SELECTIVE_CUTOFF_FRAC = 0.15   # term is "selective"/rare if df <= frac * corpus_size
-COVERAGE_THRESHOLD = 0.45      # keep candidates whose IDF-weighted coverage >= this
+# --- anchor-first resolver constants (see CAVEAT in the module docstring) ---
+# Replaces the 24-record bench tuning (SELECTIVE_CUTOFF_FRAC = 0.15) that
+# collapsed at scale. The anchor is defined RELATIVE to the rarest query term,
+# so it is scale-invariant.
+ANCHOR_BAND = 2.0              # a term is an "anchor" if df <= ANCHOR_BAND * rarest-term df
+COVERAGE_THRESHOLD = 0.45      # hard coverage floor: drop candidates below this
+ANCHOR_HYBRID_HI = 0.65        # a non-anchor candidate is kept only if coverage >= this
 _PER_TOKEN_LIMIT = 200         # recall depth per token
+# content tiers beat the contentless journal tier at equal coverage (cross-tier
+# BM25 scores are not comparable; tester email 19e7eb3096b4dae5)
+_TIER_PRIORITY = {"entity": 0, "state": 0, "reference": 0, "journal": 1}
 
 
 def _significant_tokens(query: str):
@@ -101,17 +125,34 @@ def multi_record_search(client, query: str, *, limit: int = 10, corpus_n: int |
     idf = {t: math.log((corpus_n + 1) / (df[t] + 1)) + 1.0 for t in toks}
     total = sum(idf.values()) or 1.0
     terminal_q = bool(set(toks) & _TERMINAL_Q)
-    sel_cut = max(1, round(SELECTIVE_CUTOFF_FRAC * corpus_n))
-    selective = {t for t in toks if df[t] <= sel_cut}
+
+    # Anchor-first: anchor terms are the rarest (most discriminating) tokens,
+    # defined relative to the rarest term so the band is scale-invariant. Every
+    # candidate is strict-filtered to the anchor's cluster (must match >= 1 anchor
+    # term), which removes the cross-cluster pollution the old corpus-fraction
+    # cutoff let through at scale. Anchor-raw recalls fully but pollutes; the
+    # strict filter is the load-bearing precision gate (tester Runs 24-29).
+    min_df = min(df.values())
+    anchor_cut = max(2, round(ANCHOR_BAND * min_df))
+    anchor_terms = {t for t in toks if df[t] <= anchor_cut}
 
     scored = []
     for e in cand.values():
         if terminal_q and _pure_prep(e["body"]):
             continue                                   # drop purely-preparatory on a final-state query
-        if selective and not (e["m"] & selective):
-            continue                                   # drop cross-talk (only common terms matched)
         cov = sum(idf[t] for t in e["m"]) / total
-        if cov >= COVERAGE_THRESHOLD:
-            scored.append((e["hit"], cov, e["best"]))
-    scored.sort(key=lambda x: (-x[1], x[2]))
-    return [h for h, _cov, _best in scored[:limit]]
+        if cov < COVERAGE_THRESHOLD:
+            continue                                   # below the hard coverage floor
+        # Anchor-first HYBRID gate: keep a candidate that is in the anchor's
+        # cluster (matches an anchor term) OR clears the high-coverage bar
+        # (genuinely relevant despite lacking the rare anchor, e.g. natural-
+        # language evidence). A non-anchor, mid-coverage candidate is pure
+        # cross-cluster pollution and is dropped. Tuned on the LongMemEval
+        # retrieval diagnostic: synthetic-workflow pollution -> 0 while natural-
+        # language recall is preserved (anchor-only over-filtered real queries).
+        if anchor_terms and not (e["m"] & anchor_terms) and cov < ANCHOR_HYBRID_HI:
+            continue
+        tier = e["hit"].get("tier")
+        scored.append((e["hit"], cov, _TIER_PRIORITY.get(tier, 0), e["best"]))
+    scored.sort(key=lambda x: (-x[1], x[2], x[3]))
+    return [h for h, _cov, _tp, _best in scored[:limit]]

sibyl-memory-client/tests/test_anchor_resolver_2026_06_06.py

@@ -0,0 +1,91 @@
+"""Anchor-first resolver + search refinements (combined patch, 2026-06-06).
+
+Validates the fix for the multi-record recall/precision regression that tester
+Sylvain surfaced (Runs 16/17 ~0.36 recall at 50-100 companies) and validated the
+anchor-first remedy for (Runs 24-29: full recall, zero pollution at 100
+companies). Source memo: memory/research/sylvain-anchor-first-resolver-runs24-29-2026-05-31.md.
+
+Three changes under test:
+  1. multi_record_search anchor-first strict-filter (scale-invariant precision).
+  2. MemoryClient.search_entities(category=...) anchor filter.
+  3. MemoryClient.search() cross-tier rank tiebreaker (content before journal).
+"""
+from sibyl_memory_client import MemoryClient
+from sibyl_memory_client.multi_record import multi_record_search
+
+_TYPES = {
+    "report":  "report revenue forecast quarterly",
+    "email":   "email thread followup correspondence",
+    "journal": "journal meeting notes minutes",
+    "bug":     "bug ticket error defect",
+}
+
+
+def _build_corpus(c, n):
+    """n companies, each with 4 linked records sharing THREE per-group topic
+    terms — the cross-cluster contamination vector that defeated the old
+    corpus-fraction selectivity cutoff."""
+    for i in range(n):
+        anchor = f"co{i:04d}"
+        g = i % max(1, n // 12)
+        topics = f"topic{g}alpha topic{g}beta topic{g}gamma"
+        for t, tt in _TYPES.items():
+            c.set_entity(t, f"{t}-{i}", {"text": f"{anchor} {topics} {t} {tt} project status update"})
+
+
+def test_anchor_first_full_recall_zero_pollution_at_scale(tmp_path):
+    n = 60
+    c = MemoryClient.local(tmp_path / "scale.db", tenant_id="scale")
+    _build_corpus(c, n)
+
+    exp_total = rec_total = pollution = 0
+    for i in range(n):
+        anchor = f"co{i:04d}"
+        g = i % max(1, n // 12)
+        res = multi_record_search(c, f"{anchor} topic{g}alpha topic{g}beta topic{g}gamma", limit=20)
+        expected = {f"{t}-{i}" for t in _TYPES}
+        got = {h.get("key") for h in res}
+        exp_total += len(expected)
+        rec_total += len(expected & got)
+        for h in res:
+            txt = (h.get("body") or {}).get("text", "")
+            if anchor not in txt:
+                pollution += 1
+
+    assert rec_total == exp_total, f"recall regressed: {rec_total}/{exp_total}"
+    assert pollution == 0, f"cross-cluster pollution leaked: {pollution} hits"
+
+
+def test_abstention_preserved(tmp_path):
+    c = MemoryClient.local(tmp_path / "ab.db", tenant_id="scale")
+    _build_corpus(c, 20)
+    # a term with zero corpus support must collapse the whole query to []
+    assert multi_record_search(c, "co0001 nonexistenttokenzzzq report", limit=10) == []
+
+
+def test_single_cluster_query_returns_only_that_cluster(tmp_path):
+    n = 40
+    c = MemoryClient.local(tmp_path / "sc.db", tenant_id="scale")
+    _build_corpus(c, n)
+    g = 7 % max(1, n // 12)   # same group formula the corpus uses
+    res = multi_record_search(c, f"co0007 topic{g}alpha topic{g}beta topic{g}gamma", limit=20)
+    assert res, "expected the anchor cluster to be returned"
+    for h in res:
+        assert "co0007" in (h.get("body") or {}).get("text", ""), "leaked a non-anchor record"
+
+
+def test_search_entities_category_filter(tmp_path):
+    c = MemoryClient.local(tmp_path / "cat.db", tenant_id="scale")
+    c.set_entity("report", "r1", {"text": "synergy roadmap alpha"})
+    c.set_entity("report", "r2", {"text": "synergy roadmap beta"})
+    c.set_entity("memo", "m1", {"text": "synergy roadmap gamma"})
+
+    all_hits = c.search_entities("synergy")
+    assert {h["name"] for h in all_hits} == {"r1", "r2", "m1"}
+
+    report_only = c.search_entities("synergy", category="report")
+    assert {h["name"] for h in report_only} == {"r1", "r2"}
+    assert all(h["category"] == "report" for h in report_only)
+
+    memo_only = c.search_entities("synergy", category="memo")
+    assert {h["name"] for h in memo_only} == {"m1"}

sibyl-memory-mcp/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to `sibyl-memory-mcp` are recorded here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning follows
 [SemVer](https://semver.org/).
 
+## [0.1.8] - 2026-06-06
+
+### Changed
+
+- **Pin `sibyl-memory-client>=0.4.9`.** Picks up the anchor-first hybrid
+  multi-record resolver (client 0.4.9): `memory_search` now strict-filters
+  multi-record / linked-record queries to the query's anchor cluster while
+  keeping high-coverage natural-language evidence, eliminating cross-cluster
+  pollution at scale. No MCP code change; routing through `multi_record_search`
+  is unchanged.
+
 ## [0.1.7] - 2026-06-05
 
 ### Fixed

sibyl-memory-mcp/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sibyl-memory-mcp"
-version = "0.1.7"
+version = "0.1.8"
 description = "MCP server for Sibyl Memory Plugin: wraps the local SQLite + FTS5 memory engine and exposes it to MCP-compatible agents (Claude Code, Codex, Cursor, Continue, anything that speaks MCP)."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -23,7 +23,7 @@ classifiers = [
 ]
 dependencies = [
     "mcp>=1.0.0",
-    "sibyl-memory-client>=0.4.8",
+    "sibyl-memory-client>=0.4.9",
     "sibyl-memory-hermes>=0.3.2",
 ]