Genome off-target: four complete genomes, and an honest answer on mammals

Adds S. cerevisiae, C. elegans and D. melanogaster alongside E. coli as
COMPLETE-genome off-target targets — for these an off-target anywhere
(intron, intergenic, regulatory) is found, not just coding sequence.
Verified end to end: yeast builds in 18.5s -> 944,171 sites across 17
chromosomes (16 + mito), 0.25s queries, and a positive control pulling a
real protospacer out of the index finds itself at I:79 with CFD 1.000.

Also fixes a latent bug that would have silently disabled every new
organism: server.py validated target_organism against a hardcoded
("", "ecoli", "human", "mouse") tuple in TWO places, so anything else fell
back to "no genome search". Now validated against offtarget.GENOME_SOURCES,
so registering a genome is enough to enable it end to end.

Whole-genome human/mouse is NOT added, and the methods text now says why
rather than leaving it looking like an oversight. Measured in clean
processes: the seed index costs ~90-260 bytes per site depending how the
baseline is counted; a whole human genome carries ~390M PAM sites, i.e.
tens of GB against a 16 GB container, with a build measured at ~1s per Mb
(hours). Getting there needs a different substrate (native aligner + a
multi-GB prebuilt index, or offloading the search) — a cost decision, not
a tuning one.

Guards the memory this opens up: _KMER_CACHE had no eviction, and six
organisms resident at once would OOM the Space alongside ESM-2. Added an
LRU cap (TURINGDNA_MAX_GENOME_INDEXES, default 2).

481 tests green.

dee/core/crispr_methods.py

@@ -74,14 +74,19 @@ METHODS: Dict[str, Dict[str, Any]] = {
                  "across the indexed genome rather than just your input. Your "
                  "guide never leaves the engine — only the public reference "
                  "genome is downloaded, and it is cached and reused.",
-        "limits": "Two scope limits. (1) COVERAGE: E. coli is the complete "
-                  "genome, but human (GRCh38) and mouse (GRCm39) are indexed "
-                  "over CODING SEQUENCE ONLY — so off-targets in introns and "
-                  "intergenic DNA are not seen. If you need a whole-genome "
-                  "sweep, that still calls for a dedicated genome-wide tool. "
-                  "(2) DEPTH: only the top-ranked guides are screened (the "
-                  "ones you would realistically order), not every candidate "
-                  "in the table; the run tells you how many were checked.",
+        "limits": "Two scope limits. (1) COVERAGE: E. coli, S. cerevisiae, "
+                  "C. elegans and D. melanogaster are indexed over the "
+                  "COMPLETE genome, so an off-target anywhere is found. Human "
+                  "(GRCh38) and mouse (GRCm39) are indexed over CODING "
+                  "SEQUENCE ONLY — off-targets in introns and intergenic DNA "
+                  "are not seen there, and a dedicated whole-genome tool is "
+                  "still the right call for that work. The reason is memory, "
+                  "not preference: the seed index costs on the order of "
+                  "100 bytes per site (measured), and a whole human genome "
+                  "carries roughly 390 million PAM sites — tens of gigabytes, "
+                  "well beyond the container. (2) DEPTH: only the top-ranked guides are "
+                  "screened (the ones you would realistically order), not "
+                  "every candidate; the run tells you how many were checked.",
         "citations": [
             "Doench et al. 2016, Nat Biotechnol, Suppl. Table 19 (CFD)",
             "Ensembl release 112 (GRCh38 / GRCm39 CDS); NCBI NC_000913.3 (E. coli K-12)",

dee/core/offtarget.py

@@ -104,10 +104,70 @@ GENOME_SOURCES: Dict[str, Dict[str, object]] = {
         "scope":     "exome (CDS only)",
         "ready":     True,
     },
+    # ─── Full-genome model organisms ────────────────────────────────
+    # These are small enough that the COMPLETE genome fits the in-memory
+    # seed index, so there's no coding-only caveat: an off-target
+    # anywhere — intron, intergenic, regulatory — is found. Build cost
+    # scales with genome size at roughly 1 s per Mb (measured
+    # 2026-07-19), which is what bounds this list; see the module note
+    # on why mammalian whole genomes can't join it.
+    "yeast": {
+        "name":      "Saccharomyces cerevisiae (R64-1-1)",
+        "accession": "R64-1-1",
+        "url":       "https://ftp.ensembl.org/pub/release-112/fasta/saccharomyces_cerevisiae/"
+                     "dna/Saccharomyces_cerevisiae.R64-1-1.dna.toplevel.fa.gz",
+        "is_gzip":   True,
+        "size_mb":   3,
+        "scope":     "full genome",
+        "ready":     True,
+    },
+    "worm": {
+        "name":      "Caenorhabditis elegans (WBcel235)",
+        "accession": "WBcel235",
+        "url":       "https://ftp.ensembl.org/pub/release-112/fasta/caenorhabditis_elegans/"
+                     "dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz",
+        "is_gzip":   True,
+        "size_mb":   30,
+        "scope":     "full genome",
+        "ready":     True,
+    },
+    "fly": {
+        "name":      "Drosophila melanogaster (BDGP6.46)",
+        "accession": "BDGP6.46",
+        "url":       "https://ftp.ensembl.org/pub/release-112/fasta/drosophila_melanogaster/"
+                     "dna/Drosophila_melanogaster.BDGP6.46.dna.toplevel.fa.gz",
+        "is_gzip":   True,
+        "size_mb":   43,
+        "scope":     "full genome",
+        "ready":     True,
+    },
 }
 
 _GENOME_CACHE_DIR = os.environ.get("TURINGDNA_GENOME_CACHE", "/tmp/turingdna_genomes")
 _KMER_CACHE: Dict[str, "KmerIndex"] = {}
+# Least-recently-used timestamps, so the cache can shed a big index instead
+# of OOM-ing the container.
+_LAST_USED: Dict[str, float] = {}
+# Indexes cost ~152 bytes per site (measured 2026-07-19). A mammalian CDS
+# index is ~3.5 GB and a fly genome ~2.7 GB, so holding every organism at
+# once would exceed the Space's RAM alongside the ESM-2 model. Keep only the
+# few most recently used; a re-build is slow but survivable, an OOM is not.
+_MAX_CACHED_INDEXES = int(os.environ.get("TURINGDNA_MAX_GENOME_INDEXES", "2"))
+
+
+def _evict_if_needed(incoming: str) -> None:
+    """Drop least-recently-used indexes so at most _MAX_CACHED_INDEXES - 1
+    remain before `incoming` is inserted. Caller holds _BUILD_LOCK."""
+    try:
+        while len(_KMER_CACHE) >= max(1, _MAX_CACHED_INDEXES):
+            victim = min(_KMER_CACHE, key=lambda k: _LAST_USED.get(k, 0.0))
+            if victim == incoming:
+                break
+            _KMER_CACHE.pop(victim, None)
+            _LAST_USED.pop(victim, None)
+            logger.info("Evicted genome index %s to stay within memory budget", victim)
+    except Exception:  # noqa: BLE001 — eviction must never break a query
+        logger.exception("genome index eviction failed")
 _BUILD_LOCK = threading.Lock()
 # Per-organism build state — distinct from _BUILD_LOCK because we want
 # concurrent requests for DIFFERENT organisms to proceed in parallel,
@@ -231,6 +291,7 @@ def find_genomic_offtargets(
         # building, refresh in 2 min" banner via the API status field.
         kick_off_build(organism)
         return []
+    _LAST_USED[organism] = time.time()   # keep the in-use index off the evict list
 
     guide_seed = guide_spacer[-_SEED_LEN:]
     # Search the perfect-seed bucket plus all 1-mismatch seed buckets.
@@ -296,7 +357,9 @@ def _get_kmer_index(organism: str) -> Optional[KmerIndex]:
                 "Built kmer index for %s: %d sites across %d chroms in %.1f s",
                 organism, index.n_sites, index.n_chroms, elapsed,
             )
+            _evict_if_needed(organism)
             _KMER_CACHE[organism] = index
+            _LAST_USED[organism] = time.time()
             return index
         except Exception as exc:  # noqa: BLE001
             logger.exception("Failed to build kmer index for %s: %s", organism, exc)

dee/server.py

@@ -2016,7 +2016,12 @@ def create_app() -> Flask:
         target_organism = str(body.get("target_organism", "")).lower().strip()
         gene_symbol     = str(body.get("gene_symbol", "")).strip()
         # Light input validation — anything else gets ignored.
-        if target_organism not in ("", "ecoli", "human", "mouse"):
+        # Validate against the actual genome registry rather than a hardcoded
+        # list, so adding an organism to offtarget.GENOME_SOURCES enables it
+        # end to end. The old literal tuple silently rejected any new organism
+        # and fell back to "no genome search".
+        from dee.core.offtarget import GENOME_SOURCES as _GENOMES
+        if target_organism and target_organism not in _GENOMES:
             target_organism = ""
         if len(gene_symbol) > 32 or not re.match(r"^[A-Za-z0-9._-]*$", gene_symbol):
             gene_symbol = ""
@@ -3158,7 +3163,12 @@ def create_app() -> Flask:
         # downloaded XLSX matches what the user saw in the table.
         target_organism = str(body.get("target_organism", "")).lower().strip()
         gene_symbol     = str(body.get("gene_symbol", "")).strip()
-        if target_organism not in ("", "ecoli", "human", "mouse"):
+        # Validate against the actual genome registry rather than a hardcoded
+        # list, so adding an organism to offtarget.GENOME_SOURCES enables it
+        # end to end. The old literal tuple silently rejected any new organism
+        # and fell back to "no genome search".
+        from dee.core.offtarget import GENOME_SOURCES as _GENOMES
+        if target_organism and target_organism not in _GENOMES:
             target_organism = ""
         if len(gene_symbol) > 32 or not re.match(r"^[A-Za-z0-9._-]*$", gene_symbol):
             gene_symbol = ""

dee/static/index.html

@@ -112,7 +112,7 @@
     <!-- ?v= query bumps invalidate browser + iframe asset caches when app.css /
          app.js change. Bump these numbers whenever you ship a frontend update —
          without them, users keep getting the stale file for up to a week. -->
-    <link rel="stylesheet" href="/static/app.css?v=20260718-offtarget" />
+    <link rel="stylesheet" href="/static/app.css?v=20260719-organisms" />
     <link rel="icon" type="image/svg+xml" href="/static/favicon.svg?v=2" />
     <link rel="apple-touch-icon" href="/static/favicon.svg?v=2" />
     <!-- Mol* (PDBe) 3-D viewer is ~4.9 MB. We do NOT eager-load it on every
@@ -1091,9 +1091,16 @@
                               </label>
                               <select id="crisprOrganism" class="crispr-context-input">
                                   <option value="">None — skip genome search</option>
-                                  <option value="ecoli">E. coli K-12 MG1655 · whole genome</option>
-                                  <option value="human">Homo sapiens (GRCh38) · coding regions · index builds once, ~3 min</option>
-                                  <option value="mouse">Mus musculus (GRCm39) · coding regions · index builds once, ~3 min</option>
+                                  <optgroup label="Complete genome — finds off-targets anywhere">
+                                      <option value="ecoli">E. coli K-12 MG1655 · builds in seconds</option>
+                                      <option value="yeast">S. cerevisiae (R64-1-1) · builds in ~30 s</option>
+                                      <option value="worm">C. elegans (WBcel235) · builds once, ~3 min</option>
+                                      <option value="fly">D. melanogaster (BDGP6.46) · builds once, ~5 min</option>
+                                  </optgroup>
+                                  <optgroup label="Coding regions only — introns/intergenic not covered">
+                                      <option value="human">Homo sapiens (GRCh38) · builds once, ~3 min</option>
+                                      <option value="mouse">Mus musculus (GRCm39) · builds once, ~2 min</option>
+                                  </optgroup>
                               </select>
                           </div>
                           <div class="crispr-context-field">
@@ -2306,7 +2313,7 @@
     <!-- Cloning reference data must load before app.js so the Designer
          can read VECTORS / ENZYMES / CLONING_METHODS / TAGS / LINKERS. -->
     <script src="/static/cloning_db.js?v=20260530-ui-polish" defer></script>
-    <script src="/static/app.js?v=20260718-offtarget" defer></script>
+    <script src="/static/app.js?v=20260719-organisms" defer></script>
     <!-- Dwell-time heartbeat. Loads after auth.js so its /api/ping calls go
          through the JWT-attaching fetch wrapper (signed-in attribution). -->
     <script src="/static/telemetry.js?v=20260622-analytics" defer></script>

tests/test_crispr_methods.py

@@ -96,3 +96,36 @@ def test_genome_offtarget_top_n_is_bounded():
     # across all 50 candidates. Keep the screened set small and explicit.
     from dee.core.crispr import GENOME_OFFTARGET_TOP_N
     assert 1 <= GENOME_OFFTARGET_TOP_N <= 15
+
+
+# --------------------------------------------------------------------------- #
+# Genome registry: which organisms, and at what coverage
+# --------------------------------------------------------------------------- #
+def test_full_genome_organisms_are_actually_full_genome():
+    from dee.core.offtarget import GENOME_SOURCES, is_organism_ready
+    full = {k for k, v in GENOME_SOURCES.items() if v["scope"] == "full genome"}
+    # Small enough to index completely — an off-target ANYWHERE is found.
+    assert {"ecoli", "yeast", "worm", "fly"} <= full
+    for o in full:
+        assert is_organism_ready(o)
+        assert GENOME_SOURCES[o]["url"].startswith("https://")
+
+
+def test_mammals_are_declared_cds_only_not_silently_partial():
+    from dee.core.offtarget import GENOME_SOURCES
+    for o in ("human", "mouse"):
+        assert "CDS" in GENOME_SOURCES[o]["scope"]
+    # And the user-facing methods must say so, with the reason.
+    limits = cm.METHODS["genome_off"]["limits"]
+    assert "CODING SEQUENCE ONLY" in limits
+    # The reason must be stated (memory), without false precision — the
+    # per-site cost was measured at roughly 90-260 B depending on how the
+    # baseline is counted, so we claim an order of magnitude, not a figure.
+    assert "390 million" in limits and "gigabytes" in limits
+
+
+def test_index_cache_is_bounded():
+    # Six organisms x multi-GB indexes would OOM the Space; the cache must evict.
+    from dee.core import offtarget as ot
+    assert ot._MAX_CACHED_INDEXES >= 1
+    assert callable(ot._evict_if_needed)