--- license: apache-2.0 tags: - structure-mapping - analogical-retrieval - retrieval - sma - rare-disease - genomics - cybersecurity - legal - finance - cite-or-abstain - novelty-detection - ontology language: - en pipeline_tag: feature-extraction --- # SMA-1 Universal Adapter — adapter-v1 **Structure-Mapping Agentic Memory: a universal structure-mapping retrieval adapter that beats RAG/KG baselines across five unrelated curated-ontology domains — with calibrated abstention and novelty detection that vector RAG structurally cannot provide.** - **Paper:** *"Structure-Mapping Agentic Memory"*, Ayaz Khan (2026, under review, *Nature Machine Intelligence*) - **Source repository:** https://github.com/ayazkhan27/sma-1 - **Tag:** `adapter-v1` (frozen after 5/5 agentic wins; ADR-008) - **License:** Apache-2.0 --- ## What is this adapter? `adapter-v1` is the **universal ontology adapter** for SMA-1. It is not a neural network weight file — it is a frozen configuration that specifies: 1. **Five mounted curated ontologies** (one per evaluation domain) — each parsed from its canonical OBO/OWL/STIX/XML source, mounted onto a predicate lattice, and indexed via the SMA MAC/FAC retrieval engine. 2. **Frozen matcher dials** (see below) — calibrated on held-out validation data before any confirmatory run. 3. **A domain router** — routes a query to the correct ontology by term-id prefix or declared domain. The adapter is "frozen" in the sense that no component (ontology versions, matcher dials, encoder rules) changes after the tag. New domains may be *added* under a new tag (ADR-008); the frozen components do not move. --- ## Mounted curated ontologies | Domain | Ontology | Source format | Approx. active terms | |---|---|---|---| | Medicine | Human Phenotype Ontology (HPO) | OBO | ~17 000 | | Genomics | Gene Ontology (GO) biological process | OBO | ~30 000 | | Cybersecurity | MITRE ATT&CK (STIX 2.1) | JSON/XML | ~700 techniques and sub-techniques | | Legal | USPTO Cooperative Patent Classification (CPC) | OWL/XML | ~254 000 nodes | | Finance | US-GAAP SEC XBRL taxonomy | OWL/XML | ~900 concepts | Each ontology is mounted via `sma.ontology.mount()`: - The is-a hierarchy becomes the **predicate lattice** (ascension depth δ = 2, penalty ρ = 0.95 per hop). - Each annotated entity (disease, protein, threat group, patent, filing) is encoded as a **case**: one `stmt(fid(term_id), subject)` per present term, plus higher-order `stmt(rel, stmt(fid(s), subj), stmt(fid(o), subj))` for typed relations whose subject and object are both present. - The MAC (Memory Analogical Coding) stage indexes cases via a weighted Lemma-2 inverted index (bound-ordered over all candidates for corpora ≤ 20k cases). - The FAC (Full Analogical Coding / SME) stage computes structural alignment kernels; event-type entities are constants (match identically); structure mapping respects parallel connectivity. --- ## Frozen matcher dials (prereg-v1) Calibrated on held-out validation splits (HDFS seed-7, SSB seeds 29/31, Liberty leave-one-out); test seeds never used during calibration. | Parameter | Value | Meaning | |---|---|---| | Scorer | `surprisal` | Surprisal-weighted SES (σ₀ = −log₂ p(functor)) | | Normalisation | `max` | Score / max(score in result set) | | γ (trickle-down weight) | 0.25 | Blueprint §2.5 default | | ρ (ascension penalty) | 0.95 | Per-hop lattice ascension penalty | | δ (ascension depth) | 2 | Maximum ancestor hops for lattice bridging | --- ## Memory-swap evaluation protocol The evaluation uses a **memory-swap** design: one LLM agent, one prompt, one task — the only variable is the retrieval memory. This isolates the contribution of retrieval from language generation. **Agentic suite (5 domains):** the agent is given a query (phenotype set, gene function annotation set, threat-group technique set, patent claims, SEC filing) and must retrieve the top-k matching indexed entities. The gold answer is the correct entity (disease, protein, threat group, CPC code, GAAP concept). **Phase 5 LLM-QA (trustworthy specialist QA):** the agent is given a clinical diagnosis question and must answer it from retrieved evidence, cite its sources, or abstain if the evidence is insufficient. The LLM (DeepSeek, temperature 0) and prompt are fixed; only the retrieval memory varies (none / dense / SMA). The cite-or-abstain threshold is calibrated per memory on a disjoint 60+60 calibration split (Youden's J on retrieval scores only — no LLM spend, no test leakage). --- ## Verified headline metrics All numbers from committed `reports/confirmatory/` CSVs (paired bootstrap 10 000 resamples, Holm-Bonferroni step-down correction). ### 5-domain agentic suite — SMA vs best RAG baseline (tail top-5) "Tail" = rare slice (entity's rarest-term IC > corpus median). Legal arm reported on all-queries (rare slice degenerate for CPC — see limitations). | Domain | SMA tail top-5 | Best RAG | Δ | 95% CI | p_Holm | Cliff's δ | |---|---|---|---|---|---|---| | Medicine (HPO) | 0.949 | 0.606 (hybrid+rerank) | **+0.333** | [0.281, 0.389] | 0.0006 | 0.333 | | Genomics (GO) | 0.849 | 0.682 (dense BGE) | **+0.156** | [0.100, 0.211] | 0.0004 | 0.156 | | Finance (US-GAAP) | 0.418 | 0.231 (hybrid-RRF) | **+0.167** | [0.111, 0.225] | 0.0002 | 0.167 | | Cybersecurity (ATT&CK) | 0.766 | 0.749 (hybrid-RRF) | **+0.073** | [0.008, 0.142] | 0.0346 | 0.073 | | Legal (CPC) | 0.941 (all) | 0.870 (dense BGE, all) | **+0.064** | [0.025, 0.103] | 0.0022 | 0.064 | Four domains survive conservative correction; cyber is directional. RAG/KG baseline gauntlet: BM25, BGE dense, Hybrid-RRF, Hybrid+Rerank (cross-encoder reranker), HippoRAG (phrase-graph + PageRank). **Capability axes (all domains):** SMA achieves lowest AURC (best calibrated selective prediction) and is the only method (along with HippoRAG) with nonzero novelty F1. All pure-RAG baselines: novelty F1 = 0.000. ### Phase 5 LLM-QA — SMA vs dense (medicine, n = 120 answerable + 120 held-out) | Axis | SMA | Dense RAG | Δ | 95% CI | p_Holm | Result | |---|---|---|---|---|---|---| | Accuracy | 0.342 | 0.100 | +0.242 | [+0.167, +0.325] | < 0.001 | WIN | | Grounding AUROC | 0.793 | 0.547 | +0.246 | [+0.159, +0.333] | < 0.001 | WIN | | Novelty F1 | 0.789 | 0.553 | +0.236 | [+0.200, +0.408] (recall) | < 0.001 | WIN | | Selective accuracy | 0.625 | 0.500 | +0.125 | [+0.071, +0.179] | < 0.001 | WIN | | Abstain recall | 0.908 | 0.900 | +0.008 | [−0.058, +0.075] | 0.917 | TIE | 4/5 axes Holm-significant; abstain-recall is an honest tie. **The mechanism (Fig 5b):** SMA's raw structural grounding score separates known (answerable) from unknown (held-out) at AUROC 0.793; dense cosine is near chance (0.547). Dense RAG must refuse 79% of answerable questions to achieve the same abstain-recall as SMA at 45% abstention. ### Structure Synthesis Benchmark (SSB) Zero-lexical-overlap structural retrieval; disjoint per-triple vocabularies bridged only by a declared predicate lattice (seeds 41, 43; n = 100 each): | Method | Forced-choice r@1 | Library r@1 | |---|---|---| | SMA | **1.000** | **0.895** | | BM25 | 0.000 | 0.000 | | TF-IDF Dense | 0.000 | 0.000 | Cliff's δ = 0.895, p_Holm = 0.0004. --- ## Intended use `adapter-v1` is intended for: - **Research** into structure-mapping retrieval and analogical reasoning for LLM agents. - **Evaluation** of SMA-1 claims: reproduce results via `scripts/confirmatory_battery.py` or `scripts/agentic_suite.py`. - **Extension** to new domains: register a new OBO/OWL ontology via `OntologyRegistry`, mount it, run `agentic_suite.py --arm `. A new adapter tag is required for any new frozen ontology (ADR-008). - **The Gradio demo Space** (`release/hf_space/`) which illustrates the medicine arm side-by-side with dense RAG. `adapter-v1` is **not** intended for: - Production clinical decision support (not a medical device; not validated for clinical use). - Domains without a curated ontology (the structural advantage requires a predicate lattice; flat-tabular data yields parity or null — confirmed on UCI Diabetes-130 and IEEE fraud without adapter drafting). - Tasks where surface-retrieval baselines already achieve near-perfect performance (e.g. within-domain log triage with lexically overt labels — BGL in the T2 battery; SMA is statistically tied, not a win). --- ## Limitations and honest nulls 1. **Flat-tabular data.** Where per-record higher-order relational structure is absent or cannot be meaningfully encoded, SMA reaches statistical parity with baselines but does not win (UCI Diabetes-130 before adapter drafting: SMA 0.425 vs BM25 0.537 — not significant; IEEE fraud: SMA below BM25 after adapter drafting — cross-record structure is needed, not handled per-record). 2. **Cross-family transfer.** Structural transfer holds within failure-physics families (supercomputer syslogs, BGL→Spirit, BGL→Thunderbird: SMA +58 F1 pts over dense). It fails across application-vs-infrastructure families (HDFS→OpenStack: all methods collapse to ~0.33). 3. **Legal arm rare slice.** The CPC rare-slice definition (IC > corpus median) degenerates for patent CPC codes (near-uniform IC from closure propagation); legal results are reported on the all-queries slice with an explicit caveat. 4. **Agentic LLM-QA: medicine only.** Phase 5 LLM-QA evaluation is on the medicine (HPO) domain only; the verifiable-specialist result has not been extended to the other four domains under prereg-v2. 5. **ATT&CK cap.** ATT&CK groups with > 30 techniques were capped (SME kernel enumeration timeout without the cap); 41% of AT&CK groups are affected. Results reflect the capped subset. 6. **Novelty F1 threshold.** The SAGE novelty threshold is fixed at 0.5 (not tuned); the absolute novelty F1 values are conservative and would likely improve with threshold calibration. 7. **Phase 4a drift result is INVALID for SMA.** The LongMemEval run (500 instances) is NOT an SMA result — the backend used a broken encoder that collapsed all facts to functor "User"/"The", producing garbage retrieval (SMA accuracy 0.030 = encoder artifact, not reported as an SMA result). --- ## How to use ```python from sma.ontology import OntologyRegistry, DomainRouter # Register a curated ontology reg = OntologyRegistry() reg.register("hpo", "data/hp.obo") # OBO format inferred from extension mounted = reg.get("hpo") # lazily loads, mounts, caches # Build an index over annotated entities from sma.eval.agentic.memories import SmaMemory, IndexItem mem = SmaMemory(mounted) mem.index([ IndexItem(key="OMIM:154700", term_ids=frozenset(["HP:0001166", "HP:0001083", "HP:0002616"]), text="Marfan syndrome arachnodactyly ectopia lentis aortic root dilatation"), # ... more entries ]) # Retrieve from sma.eval.agentic.memories import Query results = mem.retrieve(Query(term_ids=frozenset(["HP:0002616", "HP:0000098"]), text="aortic root dilatation tall stature"), k=5) for r in results: print(r.key, r.score, r.confidence) # Novelty gate nov = mem.novelty(Query(term_ids=frozenset(["HP:0099999"]), text="unknown phenotype")) print(f"Novelty signal: {nov:.3f}") # high = likely out-of-distribution ``` Reproduce the full evaluation: ```bash # Confirmatory battery (single-shot; ~5 h, registered seeds) python3 -u scripts/confirmatory_battery.py --task all # Agentic suite (5 domains) python3 scripts/agentic_suite.py --arm medicine python3 scripts/agentic_suite.py --arm discovery python3 scripts/agentic_suite.py --arm finance python3 scripts/agentic_suite.py --arm cyber python3 scripts/agentic_suite.py --arm legal # Phase 5 LLM-QA (requires SMA_DEEPSEEK_API_KEY) python3 scripts/agentic_qa.py --memory sma --n-index 1500 ``` --- ## Citation ```bibtex @software{khan2026sma, author = {Ayaz Khan}, title = {SMA-1: Structure-Mapping Agentic Memory}, year = {2026}, license = {Apache-2.0}, url = {https://github.com/ayazkhan27/sma-1} } ``` --- ## License Apache-2.0. See `LICENSE` at the repository root. The mounted ontologies are derived from publicly licensed sources: - HPO: hpo.jax.org (CC BY 4.0) - GO: geneontology.org (CC BY 4.0) - MITRE ATT&CK: attack.mitre.org (Apache 2.0) - CPC: USPTO (public domain) - US-GAAP: SEC EDGAR (public domain) The SMA-1 code and adapter configuration are Apache-2.0.