Clean up legacy TS/Node files, fix Dolt backup path sync bug, and update CSP connect-src header

benchmark/COMPARISON.md

@@ -1,158 +0,0 @@
-# AI Agent Memory: Benchmark Comparison
-
-How agentmemory compares against other persistent memory solutions for AI coding agents.
-
-All numbers here come from published benchmarks or public repositories. We link to primary sources wherever possible so you can reproduce.
-
----
-
-## Retrieval Accuracy (LongMemEval)
-
-[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) measures long-term memory retrieval across ~48 sessions per question on the S variant (500 questions, ~115K tokens each).
-
-| System | Benchmark | R@5 | Notes |
-|---|---|---|---|
-| **agentmemory** (BM25 + Vector) | LongMemEval-S | **95.2%** | `all-MiniLM-L6-v2` embeddings, no API key |
-| agentmemory (BM25-only) | LongMemEval-S | 86.2% | Fallback when no embedding provider available |
-| MemPalace | LongMemEval-S | ~96.6% (self-reported) | Vendor-published number we have not independently reproduced. Vector-only with a larger embedding model and no agent-integration surface (no hooks, no MCP, no multi-agent) |
-| Letta / MemGPT | LoCoMo | 83.2% | Different benchmark (LoCoMo, not LongMemEval) |
-| Mem0 | LoCoMo | 68.5% | Different benchmark (LoCoMo, not LongMemEval) |
-
-**⚠️ Apples vs oranges caveat:** only agentmemory's 95.2% is our own measured result, reproducible from the methodology below. Every other number here is the vendor's published claim, on a different benchmark or harness, that we have not independently reproduced: MemPalace reports LongMemEval-S, while Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/). Treat them as ballpark vendor claims, not a head-to-head on identical data. We'd love to run every system on the same dataset; if any maintainer wants to collaborate, open an issue.
-
-Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)
-
----
-
-## Feature Matrix
-
-| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | supermemory | MemPalace | Hippo |
-|---|---|---|---|---|---|---|---|
-| **GitHub stars** | Growing | 58K+ | 23K+ | 35K+ | 26K+ | 54K+ | Trending |
-| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | Memory API + app | Benchmark-focused OSS | Memory system |
-| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ❌ API-side extraction | ❌ Manual | ❌ Manual |
-| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | Vector + RAG | Vector-only (large model) | Decay-weighted |
-| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | ❌ | Multi-agent shared |
-| **Framework lock-in** | None | None | High | Standalone | None (drop-in wrappers) | None | None |
-| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | Managed cloud | Vector store | None |
-| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ❌ Cloud-only | ✅ | ✅ |
-| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ | ❌ |
-| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ✅ Auto-forget | ❌ | ✅ Half-lives |
-| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | ❌ | Episodic + semantic |
-| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ✅ Auto-resolve | ❌ | ❌ |
-| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | Cloud dashboard | ❌ | ❌ |
-| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
-| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ | ❌ |
-| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | MCP + API | Standalone | Multi-agent shared |
-| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ | ❌ |
-| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Python + TS | Python | Node |
-
----
-
-## Token Efficiency
-
-The main reason to use persistent memory at all: token cost. Here's what one year of heavy agent use looks like across approaches.
-
-| Approach | Tokens / year | Cost / year | Notes |
-|---|---|---|---|
-| Paste full history into context | 19.5M+ | Impossible | Exceeds context window after ~200 observations |
-| LLM-summarized memory (extraction-based) | ~650K | ~$500 | Lossy — summarization drops detail |
-| **agentmemory (API embeddings)** | **~170K** | **~$10** | Token-budgeted, only relevant memories injected |
-| **agentmemory (local embeddings)** | **~170K** | **$0** | `all-MiniLM-L6-v2` runs in-process |
-| supermemory | Not published | Cloud pricing | Managed API, no local token budget |
-| Mem0 | Varies by integration | Varies | Extraction-based, no token budget |
-
-**agentmemory ships with a built-in token savings calculator.** Run `npx @agentmemory/agentmemory status` after a few sessions and you'll see exactly how many tokens you've saved vs. pasting the full history.
-
----
-
-## What Each Tool Is Best At
-
-This isn't a "agentmemory wins everything" page. Different tools solve different problems.
-
-**Choose agentmemory if you want:**
-- Automatic capture with zero manual `add()` calls
-- MCP server that works across Claude Code, Cursor, Codex, Gemini CLI, etc.
-- Hybrid BM25 + vector + graph search
-- Real-time viewer to see what your agent is learning
-- Self-hostable with zero external databases
-- Privacy filtering on API keys and secrets
-- Multi-agent coordination (leases, signals, routines)
-
-**Choose Mem0 if you want:**
-- Framework-agnostic API to bolt onto an existing agent
-- Managed cloud option with a dashboard
-- Python + TypeScript SDKs for direct integration
-- Entity/relationship extraction as the primary abstraction
-
-**Choose Letta/MemGPT if you want:**
-- A full agent runtime, not just memory
-- OS-inspired memory tiers (core/archival/recall)
-- Agents that self-edit their memory via function calls
-- Long-running conversational agents (weeks/months)
-
-**Choose Khoj if you want:**
-- A personal AI second brain, not agent infrastructure
-- Document-first search over your files and the web
-- Obsidian/Notion/Emacs integrations
-- Scheduled automations and research tasks
-
-**Choose supermemory if you want:**
-- A managed memory API with server-side auto-extraction and automatic forgetting
-- Drop-in wrappers for major AI frameworks (Vercel AI, LangChain, LangGraph)
-- A hosted dashboard with no infrastructure to run yourself
-- RAG plus memory served from a single query
-
-**Choose MemPalace if you want:**
-- A simple, free, open-source vector memory store
-- To chase its self-reported retrieval benchmark (we have not reproduced it)
-- Pure retrieval over agent workflow features
-- Note: no auto-capture, no MCP, no multi-agent coordination, so you wire all integration yourself
-
-**Choose Hippo if you want:**
-- Biologically-inspired memory model (decay, consolidation, sleep)
-- Multi-agent shared memory as a primary feature
-- "Forget by default, earn persistence through use" philosophy
-
----
-
-## Running Your Own Benchmarks
-
-We encourage you to measure this yourself rather than trust any README. Here's how:
-
-```bash
-# Clone the repo
-git clone https://github.com/rohitg00/agentmemory.git
-cd agentmemory && npm install
-
-# Run LongMemEval-S
-npm run bench:longmemeval
-
-# Run quality benchmark (240 observations, 20 queries)
-npm run bench:quality
-
-# Run scale benchmark
-npm run bench:scale
-
-# Run real embeddings benchmark
-npm run bench:real-embeddings
-```
-
-Results land in `benchmark/results/`. All scripts, datasets, and results are committed for reproducibility.
-
----
-
-## Corrections Welcome
-
-If you maintain one of these tools and we got a number wrong, please open an issue or PR. We'd rather have accurate numbers than convenient ones.
-
-If you want to add your tool to this comparison, open a PR with:
-1. A link to your benchmark methodology
-2. The metric and dataset you're measuring on
-3. A commit hash / version so we can reproduce
-
-**Sources:**
-- Mem0 LoCoMo benchmark: [mem0.ai blog](https://mem0.ai)
-- Letta LoCoMo benchmark: [letta.com/blog/benchmarking-ai-agent-memory](https://letta.com/blog/benchmarking-ai-agent-memory)
-- LongMemEval paper: [arxiv.org/abs/2410.10813](https://arxiv.org/abs/2410.10813)
-- LoCoMo paper: [snap-stanford.github.io/LoCoMo](https://snap-stanford.github.io/LoCoMo/)

benchmark/LONGMEMEVAL.md

@@ -1,79 +0,0 @@
-# LongMemEval-S Benchmark Results
-
-[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) is an academic benchmark for evaluating long-term memory in chat assistants. It tests 5 core abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
-
-## Setup
-
-- **Dataset**: LongMemEval-S (500 questions, ~48 sessions per question, ~115K tokens)
-- **Source**: [xiaowu0162/longmemeval-cleaned](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned)
-- **Metric**: `recall_any@K` — does ANY gold session appear in top-K retrieved results?
-- **Embedding model**: `all-MiniLM-L6-v2` (384 dimensions, local, no API key)
-- **No LLM in the loop**: Pure retrieval evaluation, no answer generation or judge
-
-## Results
-
-| System | R@5 | R@10 | R@20 | NDCG@10 | MRR |
-|---|---|---|---|---|---|
-| **agentmemory BM25+Vector** | **95.2%** | **98.6%** | **99.4%** | **87.9%** | **88.2%** |
-| agentmemory BM25-only | 86.2% | 94.6% | 98.6% | 73.0% | 71.5% |
-| MemPalace raw (vector-only) | 96.6% | ~97.6% | — | — | — |
-
-### By Question Type (BM25+Vector)
-
-| Type | R@5 | R@10 | Count |
-|---|---|---|---|
-| knowledge-update | 98.7% | 100.0% | 78 |
-| multi-session | 97.7% | 100.0% | 133 |
-| single-session-assistant | 96.4% | 98.2% | 56 |
-| temporal-reasoning | 95.5% | 97.7% | 133 |
-| single-session-user | 90.0% | 97.1% | 70 |
-| single-session-preference | 83.3% | 96.7% | 30 |
-
-### By Question Type (BM25-only)
-
-| Type | R@5 | R@10 | Count |
-|---|---|---|---|
-| knowledge-update | 92.3% | 98.7% | 78 |
-| single-session-user | 91.4% | 95.7% | 70 |
-| temporal-reasoning | 88.0% | 94.7% | 133 |
-| multi-session | 86.5% | 96.2% | 133 |
-| single-session-assistant | 80.4% | 91.1% | 56 |
-| single-session-preference | 60.0% | 80.0% | 30 |
-
-## Analysis
-
-1. **BM25+Vector (95.2%) nearly matches pure vector search (96.6%)** with only a 1.4pp gap. Both use the same embedding model (all-MiniLM-L6-v2).
-
-2. **BM25 alone gets 86.2%** — keyword search with Porter stemming and synonym expansion is surprisingly effective on conversational data.
-
-3. **Adding vectors to BM25 gives +9pp** (86.2% → 95.2%), the largest improvement from any single component.
-
-4. **Preferences are the hardest category** for both BM25 (60%) and hybrid (83.3%). These require understanding implicit/indirect statements.
-
-5. **Multi-session and knowledge-update are strongest** (97.7%+ hybrid). The hybrid approach excels when facts are distributed across sessions.
-
-6. **R@10 reaches 98.6%** — nearly all gold sessions are found within the top 10 results.
-
-## Important Notes on Methodology
-
-- These are **retrieval recall** scores, not end-to-end QA accuracy. The official LongMemEval metric is QA accuracy (retrieve + generate answer + GPT-4o judge).
-- Systems on the actual LongMemEval QA leaderboard score 60-95% depending on the LLM reader (Oracle GPT-4o gets ~82.4%).
-- We do NOT claim these as "LongMemEval scores" — they are retrieval-only evaluations on the LongMemEval-S haystack.
-- Each question builds a fresh index from its ~48 sessions, searches with the question text, and checks if gold session IDs appear in results.
-
-## Reproducibility
-
-```bash
-# Download dataset (264 MB)
-pip install huggingface_hub
-python3 -c "
-from huggingface_hub import hf_hub_download
-hf_hub_download(repo_id='xiaowu0162/longmemeval-cleaned', filename='longmemeval_s_cleaned.json', repo_type='dataset', local_dir='benchmark/data')
-"
-
-# Run BM25-only
-npx tsx benchmark/longmemeval-bench.ts bm25
-
-# Run BM25+Vector hybrid (requires @xenova/transformers)
-npx tsx benchmark/longmemeval-bench.ts hybrid
-```

benchmark/QUALITY.md

@@ -1,75 +0,0 @@
-# agentmemory v0.6.0 — Search Quality Evaluation (Internal Dataset)
-
-> For results on the academic LongMemEval-S benchmark (ICLR 2025, 500 questions), see [`LONGMEMEVAL.md`](LONGMEMEVAL.md) — **95.2% R@5, 98.6% R@10**.
-
-**Date:** 2026-03-18T07:44:43.397Z
-**Dataset:** 240 synthetic observations across 30 sessions (internal coding project)
-**Queries:** 20 labeled queries with ground-truth relevance
-**Metric definitions:** Recall@K (fraction of relevant docs in top K), Precision@K (fraction of top K that are relevant), NDCG@10 (ranking quality), MRR (position of first relevant result)
-
-## Head-to-Head Comparison
-
-| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Latency | Tokens/query |
-|--------|----------|-----------|-------------|---------|-----|---------|--------------|
-| Built-in (CLAUDE.md / grep) | 37.0% | 55.8% | 78.0% | 80.3% | 82.5% | 0.50ms | 22,610 |
-| Built-in (200-line MEMORY.md) | 27.4% | 37.8% | 63.0% | 56.4% | 65.5% | 0.16ms | 7,938 |
-| BM25-only | 43.8% | 55.9% | 95.0% | 82.7% | 95.5% | 0.17ms | 3,142 |
-| Dual-stream (BM25+Vector) | 42.4% | 58.6% | 90.0% | 84.7% | 95.4% | 0.71ms | 3,142 |
-| Triple-stream (BM25+Vector+Graph) | 36.8% | 58.0% | 87.0% | 81.7% | 87.9% | 1.02ms | 3,142 |
-
-## Why This Matters
-
-**Recall improvement:** agentmemory triple-stream finds 58.0% of relevant memories at K=10 vs 55.8% for keyword grep (+4%)
-**Token savings:** agentmemory returns only the top 10 results (3,142 tokens) vs loading everything into context (22,610 tokens) — 86% reduction
-**200-line cap:** Claude Code's MEMORY.md is capped at 200 lines. With 240 observations, 37.8% recall at K=10 — memories from later sessions are simply invisible.
-
-## Per-Query Breakdown (Triple-Stream)
-
-| Query | Category | Recall@10 | NDCG@10 | MRR | Relevant | Latency |
-|-------|----------|-----------|---------|-----|----------|---------|
-| How did we set up authentication? | semantic | 50.0% | 100.0% | 100.0% | 20 | 1.7ms |
-| JWT token validation middleware | exact | 50.0% | 64.9% | 100.0% | 10 | 1.2ms |
-| PostgreSQL connection issues | semantic | 33.3% | 100.0% | 100.0% | 30 | 1.0ms |
-| Playwright test configuration | exact | 100.0% | 100.0% | 100.0% | 10 | 1.1ms |
-| Why did the production deployment fail? | cross-session | 33.3% | 100.0% | 100.0% | 30 | 0.8ms |
-| rate limiting implementation | exact | 80.0% | 64.1% | 33.3% | 10 | 0.7ms |
-| What security measures did we add? | semantic | 33.3% | 100.0% | 100.0% | 30 | 0.7ms |
-| database performance optimization | semantic | 0.0% | 0.0% | 7.1% | 25 | 0.8ms |
-| Kubernetes pod crash debugging | entity | 100.0% | 96.7% | 100.0% | 5 | 1.2ms |
-| Docker containerization setup | entity | 100.0% | 100.0% | 100.0% | 10 | 0.9ms |
-| How does caching work in the app? | semantic | 25.0% | 64.9% | 100.0% | 20 | 0.8ms |
-| test infrastructure and factories | exact | 50.0% | 64.9% | 100.0% | 10 | 0.7ms |
-| What happened with the OAuth callback error? | cross-session | 100.0% | 54.1% | 16.7% | 5 | 1.1ms |
-| monitoring and observability setup | semantic | 66.7% | 100.0% | 100.0% | 15 | 0.8ms |
-| Prisma ORM configuration | entity | 25.7% | 93.6% | 100.0% | 35 | 1.8ms |
-| CI/CD pipeline configuration | exact | 20.0% | 64.9% | 100.0% | 25 | 1.0ms |
-| memory leak debugging | cross-session | 100.0% | 100.0% | 100.0% | 5 | 0.7ms |
-| API design decisions | semantic | 25.0% | 64.9% | 100.0% | 20 | 1.4ms |
-| zod validation schemas | entity | 66.7% | 100.0% | 100.0% | 15 | 0.7ms |
-| infrastructure as code Terraform | entity | 100.0% | 100.0% | 100.0% | 5 | 1.5ms |
-
-## By Query Category
-
-| Category | Avg Recall@10 | Avg NDCG@10 | Avg MRR | Queries |
-|----------|---------------|-------------|---------|---------|
-| exact | 60.0% | 71.8% | 86.7% | 5 |
-| semantic | 33.3% | 75.7% | 86.7% | 7 |
-| cross-session | 77.8% | 84.7% | 72.2% | 3 |
-| entity | 78.5% | 98.1% | 100.0% | 5 |
-
-## Context Window Analysis
-
-The fundamental problem with built-in agent memory:
-
-| Observations | MEMORY.md tokens | agentmemory tokens (top 10) | Savings | MEMORY.md reachable |
-|-------------|-----------------|---------------------------|---------|-------------------|
-| 240 | 12,000 | 3,142 | 74% | 83% |
-| 500 | 25,000 | 3,142 | 87% | 40% |
-| 1,000 | 50,000 | 3,142 | 94% | 20% |
-| 5,000 | 250,000 | 3,142 | 99% | 4% |
-
-At 240 observations (our dataset), MEMORY.md already hits its 200-line cap and loses access to the most recent 40 observations. At 1,000 observations, 80% of memories are invisible. agentmemory always searches the full corpus.
-
----
-
-*100 evaluations across 5 systems. Ground-truth labels assigned by concept matching against observation metadata.*

benchmark/README.md

@@ -1,100 +0,0 @@
-# benchmark/
-
-Two kinds of numbers live in this directory:
-
-1. **Quality / retrieval** — `longmemeval-bench.ts`, `quality-eval.ts`,
-   `real-embeddings-eval.ts`, `scale-eval.ts`. Recall, precision, token
-   savings. Documented in `LONGMEMEVAL.md`, `QUALITY.md`,
-   `REAL-EMBEDDINGS.md`, `SCALE.md`.
-
-2. **Load shape** — `load-100k.ts`. p50 / p90 / p99 latency and
-   throughput against a running daemon. This is the file you want when
-   somebody asks "what's p99 at 100k memories under concurrency 100?".
-
-## load-100k.ts
-
-Hand-rolled, dependency-free load harness. Issues real HTTP against a
-local agentmemory daemon at `http://localhost:3111`, records per-request
-latency with `performance.now()`, and writes a JSON report per run.
-
-### What it measures
-
-For each cell in the matrix `(N, concurrency, endpoint)` it records:
-
-- `p50_ms`, `p90_ms`, `p99_ms` — nearest-rank percentiles.
-- `min_ms`, `max_ms`, `ops`, `errors`.
-- `throughput_per_sec` — wall-clock ops / sec for that cell.
-
-Default matrix:
-
-- `N` ∈ {1000, 10000, 100000} — number of memories seeded before the
-  cell runs.
-- `C` ∈ {1, 10, 100} — concurrent in-flight requests during the cell.
-- Endpoints under test:
-  - `POST /agentmemory/remember`
-  - `POST /agentmemory/smart-search`
-  - `GET  /agentmemory/memories?latest=true`
-
-Each cell issues `BENCH_OPS=200` requests by default — enough samples
-for stable p99 without dragging a 100k-seed run past tens of minutes.
-
-### Why p99 is the number that matters
-
-p50 tells you the median request feels fast. p90 tells you the bulk of
-requests feel fast. **p99 tells you the request your tail user hits when
-they really need it feels fast.** Capacity planning lives here — if you
-want to size a fleet, scale your daemon, or set an SLO, p99 is the
-number to plan against. p50 will lie to you.
-
-### Running it
-
-```bash
-# 1. Start the daemon however you normally do (npx, Docker, etc.)
-npx @agentmemory/agentmemory
-
-# 2. From the repo root, in another shell:
-npm run bench:load
-```
-
-To override the matrix:
-
-```bash
-BENCH_N=1000 BENCH_C=1,10 BENCH_OPS=100 npm run bench:load
-```
-
-To have the harness spawn a daemon for the run (after `npm run build`):
-
-```bash
-AGENTMEMORY_BENCH_AUTOSTART=1 npm run bench:load
-```
-
-Other env knobs (see the file header for the canonical list):
-
-- `AGENTMEMORY_URL` — base URL of the daemon (default
-  `http://localhost:3111`).
-- `BENCH_SEED` — seed for the `mulberry32` content RNG. Same seed +
-  same daemon build = byte-identical seed corpus.
-- `BENCH_OUT_DIR` — where the JSON report lands (default
-  `benchmark/results/`).
-
-### Where results land
-
-`benchmark/results/load-100k-<short-git-sha>.json`. The harness
-`mkdir -p`s the directory. The file has a `schema_version: 1` field so
-future format changes don't silently break consumers.
-
-### Content generation is seedable
-
-Synthetic memory content is built from a small noun / verb / concept
-vocabulary fed by a `mulberry32(BENCH_SEED)` PRNG. Same seed + same
-build = same corpus. The point isn't "realistic" content (there isn't
-one realistic content); the point is **reproducibility** — re-running
-the harness against the same git sha should give the same content
-mixture going in, so latency variance comes from the daemon and not
-from JSON payload jitter.
-
-### Publishing numbers per release
-
-The release flow appends a `## Performance` section to `CHANGELOG.md`
-referencing the JSON in `benchmark/results/` for that release's git
-sha. p99 is the headline number; the JSON is the receipt.

benchmark/REAL-EMBEDDINGS.md

@@ -1,67 +0,0 @@
-# agentmemory v0.6.0 — Real Embeddings Quality Evaluation
-
-**Date:** 2026-03-18T07:38:21.450Z
-**Platform:** darwin arm64, Node v20.20.0
-**Dataset:** 240 observations, 30 sessions, 20 labeled queries
-**Embedding model:** Xenova/all-MiniLM-L6-v2 (384d, local, no API key)
-
-## Head-to-Head: Real Embeddings vs Keyword Search
-
-| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Avg Latency | Tokens/query |
-|--------|----------|-----------|-------------|---------|-----|-------------|--------------|
-| Built-in (grep all) | 37.0% | 55.8% | 78.0% | 80.3% | 82.5% | 0.44ms | 19,462 |
-| BM25-only (stemmed+synonyms) | 43.8% | 55.9% | 95.0% | 82.7% | 95.5% | 0.26ms | 1,571 |
-| Dual-stream (BM25+Xenova) | 43.8% | 64.1% | 98.0% | 94.9% | 100.0% | 2.39ms | 1,571 |
-| Triple-stream (BM25+Xenova+Graph) | 43.8% | 64.1% | 98.0% | 94.9% | 100.0% | 2.07ms | 1,571 |
-
-## Improvement from Real Embeddings
-
-Adding real vector embeddings to BM25 improves recall@10 by **8.2 percentage points**.
-Token savings vs loading everything: **92%** (1,571 vs 19,462 tokens).
-
-## Per-Query: Where Real Embeddings Win
-
-Queries where dual-stream (real embeddings) outperforms BM25-only:
-
-| Query | Category | BM25 Recall@10 | +Vector Recall@10 | Delta |
-|-------|----------|---------------|-------------------|-------|
-| How did we set up authentication? | semantic | 25.0% | 45.0% | +20.0pp ** |
-| Playwright test configuration | exact | 50.0% | 90.0% | +40.0pp ** |
-| database performance optimization | semantic | 0.0% | 40.0% | +40.0pp ** |
-| test infrastructure and factories | exact | 50.0% | 80.0% | +30.0pp ** |
-| Prisma ORM configuration | entity | 14.3% | 28.6% | +14.3pp ** |
-| CI/CD pipeline configuration | exact | 20.0% | 40.0% | +20.0pp ** |
-
-## By Category Comparison
-
-| Category | Built-in grep | BM25 (stemmed) | +Real Vectors | +Graph |
-|----------|--------------|----------------|--------------|--------|
-| exact | 48.0% | 54.0% | 72.0% | 72.0% |
-| semantic | 35.5% | 33.3% | 41.9% | 41.9% |
-| cross-session | 77.8% | 77.8% | 77.8% | 77.8% |
-| entity | 79.0% | 76.2% | 79.0% | 79.0% |
-
-## Embedding Performance
-
-| System | Embedding Time | Model | Dimensions |
-|--------|---------------|-------|------------|
-| Dual-stream (BM25+Xenova) | 3.1s | Xenova/all-MiniLM-L6-v2 | 384 |
-| Triple-stream (BM25+Xenova+Graph) | 2.9s | Xenova/all-MiniLM-L6-v2 | 384 |
-
-Embedding is a one-time cost at ingestion. Search is sub-millisecond after indexing.
-
-## Key Findings
-
-1. **Semantic queries improve most**: 8.6pp recall@10 gain from real embeddings
-2. **"database performance optimization"** — the hardest query — goes from BM25 0.0% to vector-augmented 40.0%
-3. **Entity/exact queries** are already well-served by BM25+stemming — vectors add marginal value
-4. **Local embeddings (Xenova)** run without API keys — zero cost, zero latency concerns
-
-## Recommendation
-
-Enable local embeddings by default (`EMBEDDING_PROVIDER=local` or install `@xenova/transformers`).
-This gives agentmemory genuine semantic search that built-in agent memories cannot match —
-understanding that "database performance optimization" relates to "N+1 query fix" and "eager loading".
-
----
-*All measurements use Xenova/all-MiniLM-L6-v2 local embeddings (384 dimensions, no API calls).*

benchmark/SCALE.md

@@ -1,110 +0,0 @@
-# agentmemory v0.6.0 — Scale & Cross-Session Evaluation
-
-**Date:** 2026-03-18T07:45:03.529Z
-**Platform:** darwin arm64, Node v20.20.0
-
-## 1. Scale: agentmemory vs Built-in Memory
-
-Every built-in agent memory (CLAUDE.md, .cursorrules, Cline's memory-bank) loads ALL memory into context every session. agentmemory searches and returns only relevant results.
-
-| Observations | Sessions | Index Build | BM25 Search | Hybrid Search | Heap | Context Tokens (built-in) | Context Tokens (agentmemory) | Savings | Built-in Unreachable |
-|-------------|----------|------------|-------------|---------------|------|--------------------------|-----------------------------|---------|--------------------|
-| 240 | 30 | 177ms | 0.112ms | 0.63ms | 9MB | 10,504 | 1,924 | 82% | 17% |
-| 1,000 | 125 | 155ms | 0.317ms | 1.709ms | 6MB | 43,834 | 1,969 | 96% | 80% |
-| 5,000 | 625 | 810ms | 1.496ms | 8.58ms | 25MB | 220,335 | 1,972 | 99% | 96% |
-| 10,000 | 1250 | 1657ms | 3.195ms | 17.49ms | 1MB | 440,973 | 1,974 | 100% | 98% |
-| 50,000 | 6250 | 9182ms | 22.827ms | 108.722ms | 316MB | 2,216,173 | 1,981 | 100% | 100% |
-
-### What the numbers mean
-
-**Context Tokens (built-in):** How many tokens Claude Code/Cursor/Cline would consume loading ALL memory into the context window. At 5,000 observations, this is ~250K tokens — exceeding most context windows entirely.
-
-**Context Tokens (agentmemory):** How many tokens the top-10 search results consume. Stays constant regardless of corpus size.
-
-**Built-in Unreachable:** Percentage of memories that built-in systems CANNOT access because they exceed the 200-line MEMORY.md cap or context window limits. At 1,000 observations, 80% of your project history is invisible.
-
-### Storage Costs
-
-| Observations | BM25 Index | Vector Index (d=384) | Total Storage |
-|-------------|-----------|---------------------|---------------|
-| 240 | 395 KB | 494 KB | 0.9 MB |
-| 1,000 | 1,599 KB | 2,060 KB | 3.6 MB |
-| 5,000 | 8,006 KB | 10,298 KB | 17.9 MB |
-| 10,000 | 16,005 KB | 20,596 KB | 35.7 MB |
-| 50,000 | 80,126 KB | 102,979 KB | 178.8 MB |
-
-## 2. Cross-Session Retrieval
-
-Can the system find relevant information from past sessions? This is impossible for built-in memory once observations exceed the line/context cap.
-
-| Query | Target Session | Gap | BM25 Found | BM25 Rank | Hybrid Found | Hybrid Rank | Built-in Visible |
-|-------|---------------|-----|-----------|-----------|-------------|-------------|-----------------|
-| How did we set up OAuth providers? | ses_005-009 | 24 | Yes | #1 | Yes | #1 | Yes |
-| What was the N+1 query fix? | ses_010-014 | 18 | Yes | #1 | Yes | #2 | Yes |
-| PostgreSQL full-text search setup | ses_010-014 | 17 | Yes | #1 | Yes | #1 | Yes |
-| bcrypt password hashing configuration | ses_005-009 | 20 | Yes | #1 | Yes | #1 | Yes |
-| Vitest unit testing setup | ses_020-024 | 9 | Yes | #1 | Yes | #1 | Yes |
-| webhook retry exponential backoff | ses_015-019 | 14 | Yes | #1 | Yes | #1 | Yes |
-| ESLint flat config migration | ses_000-004 | 29 | Yes | #1 | Yes | #1 | Yes |
-| Kubernetes HPA autoscaling configuration | ses_025-029 | 4 | Yes | #1 | Yes | #1 | No |
-| Prisma database seed script | ses_010-014 | 16 | Yes | #1 | Yes | #1 | Yes |
-| API cursor-based pagination | ses_015-019 | 14 | Yes | #1 | Yes | #1 | Yes |
-| CSRF protection double-submit cookie | ses_005-009 | 24 | Yes | #1 | Yes | #1 | Yes |
-| blue-green deployment rollback | ses_025-029 | 4 | Yes | #1 | Yes | #1 | No |
-
-**Summary:** agentmemory BM25 found 12/12 cross-session queries. Hybrid found 12/12. Built-in memory (200-line cap) could only reach 10/12.
-
-## 3. The Context Window Problem
-
-```
-Agent context window: ~200K tokens
-System prompt + tools:  ~20K tokens
-User conversation:      ~30K tokens
-Available for memory:  ~150K tokens
-
-At 50 tokens/observation:
-  200 observations  =  10,000 tokens  (fits, but 200-line cap hits first)
-  1,000 observations =  50,000 tokens  (33% of available budget)
-  5,000 observations = 250,000 tokens  (EXCEEDS total context window)
-
-agentmemory top-10 results:
-  Any corpus size     =  ~1,924 tokens  (0.3% of budget)
-```
-
-## 4. What Built-in Memory Cannot Do
-
-| Capability | Built-in (CLAUDE.md) | agentmemory |
-|-----------|---------------------|-------------|
-| Semantic search | No (keyword grep only) | BM25 + vector + graph |
-| Scale beyond 200 lines | No (hard cap) | Unlimited |
-| Cross-session recall | Only if in 200-line window | Full corpus search |
-| Cross-agent sharing | No (per-agent files) | MCP + REST API |
-| Multi-agent coordination | No | Leases, signals, actions |
-| Temporal queries | No | Point-in-time graph |
-| Memory lifecycle | No (manual pruning) | Ebbinghaus decay + eviction |
-| Knowledge graph | No | Entity extraction + traversal |
-| Query expansion | No | LLM-generated reformulations |
-| Retention scoring | No | Time-frequency decay model |
-| Real-time dashboard | No (read files manually) | Viewer on :3113 |
-| Concurrent access | No (file lock) | Keyed mutex + KV store |
-
-## 5. When to Use What
-
-**Use built-in memory (CLAUDE.md) when:**
-- You have < 200 items to remember
-- Single agent, single project
-- Preferences and quick facts only
-- Zero setup is the priority
-
-**Use agentmemory when:**
-- Project history exceeds 200 observations
-- You need to recall specific incidents from weeks ago
-- Multiple agents work on the same codebase
-- You want semantic search ("how does auth work?") not just keyword matching
-- You need to track memory quality, decay, and lifecycle
-- You want a shared memory layer across Claude Code, Cursor, Windsurf, etc.
-
-Built-in memory is your sticky notes. agentmemory is the searchable database behind them.
-
----
-*Scale tests: 5 corpus sizes. Cross-session tests: 12 queries targeting specific past sessions.*

benchmark/dataset.ts

@@ -1,293 +0,0 @@
-import type { CompressedObservation } from "../src/types.js";
-
-export interface LabeledQuery {
-  query: string;
-  relevantObsIds: string[];
-  description: string;
-  category: "exact" | "semantic" | "temporal" | "cross-session" | "entity";
-}
-
-const SESSION_COUNT = 30;
-const OBS_PER_SESSION = 8;
-
-function ts(daysAgo: number): string {
-  return new Date(Date.now() - daysAgo * 86400000).toISOString();
-}
-
-const RAW_SESSIONS: Array<{
-  sessionRange: [number, number];
-  daysAgoRange: [number, number];
-  project: string;
-  observations: Array<Omit<CompressedObservation, "id" | "sessionId" | "timestamp">>;
-}> = [
-  {
-    sessionRange: [0, 4],
-    daysAgoRange: [28, 25],
-    project: "webapp",
-    observations: [
-      { type: "command_run", title: "Initialize Next.js 15 project", subtitle: "create-next-app", facts: ["Created Next.js 15 app with App Router", "TypeScript template selected", "Tailwind CSS v4 configured"], narrative: "Initialized a new Next.js 15 project using create-next-app with TypeScript and Tailwind CSS. Selected the App Router layout.", concepts: ["nextjs", "typescript", "tailwind", "app-router"], files: ["package.json", "tsconfig.json", "tailwind.config.ts"], importance: 6 },
-      { type: "file_edit", title: "Configure ESLint with flat config", subtitle: "eslint.config.mjs", facts: ["Migrated to ESLint flat config format", "Added typescript-eslint plugin", "Configured import sorting rules"], narrative: "Set up ESLint using the new flat config format (eslint.config.mjs). Added typescript-eslint for type-aware linting and configured import sorting with eslint-plugin-import.", concepts: ["eslint", "linting", "code-quality", "typescript"], files: ["eslint.config.mjs", "package.json"], importance: 5 },
-      { type: "file_edit", title: "Set up Prettier with Tailwind plugin", subtitle: "Formatting", facts: ["Installed prettier and prettier-plugin-tailwindcss", "Added .prettierrc with semi: false, singleQuote: true", "Configured format-on-save in VS Code settings"], narrative: "Configured Prettier for automatic code formatting. Added the Tailwind CSS class sorting plugin. Set up VS Code to format on save.", concepts: ["prettier", "formatting", "tailwind", "developer-experience"], files: [".prettierrc", ".vscode/settings.json"], importance: 4 },
-      { type: "file_edit", title: "Create shared UI component library", subtitle: "Components", facts: ["Created Button, Input, Card, Badge components", "Used cva (class-variance-authority) for variant styling", "Added Radix UI primitives for accessibility"], narrative: "Built a shared component library with Button, Input, Card, and Badge components. Used class-variance-authority (cva) for type-safe variant styling and Radix UI primitives for keyboard navigation and screen reader support.", concepts: ["components", "ui-library", "radix-ui", "cva", "accessibility"], files: ["src/components/ui/button.tsx", "src/components/ui/input.tsx", "src/components/ui/card.tsx"], importance: 7 },
-      { type: "file_edit", title: "Add global layout with navigation", subtitle: "Layout", facts: ["Created root layout with metadata", "Added responsive navigation bar", "Implemented mobile hamburger menu"], narrative: "Created the root layout component with SEO metadata, Open Graph tags, and a responsive navigation bar that collapses into a hamburger menu on mobile devices.", concepts: ["layout", "navigation", "responsive-design", "seo"], files: ["src/app/layout.tsx", "src/components/nav.tsx"], importance: 6 },
-      { type: "file_edit", title: "Configure path aliases and absolute imports", subtitle: "tsconfig", facts: ["Added @ alias pointing to src/", "Configured baseUrl for absolute imports"], narrative: "Set up TypeScript path aliases so imports can use @/components instead of relative paths. Configured baseUrl in tsconfig.json.", concepts: ["typescript", "path-aliases", "developer-experience"], files: ["tsconfig.json"], importance: 3 },
-      { type: "command_run", title: "Add Vitest for unit testing", subtitle: "Testing setup", facts: ["Installed vitest and @testing-library/react", "Created vitest.config.ts with jsdom environment", "Added test script to package.json"], narrative: "Set up Vitest as the unit testing framework with React Testing Library for component tests. Configured jsdom environment for DOM testing.", concepts: ["vitest", "testing", "react-testing-library", "configuration"], files: ["vitest.config.ts", "package.json"], importance: 5 },
-      { type: "file_edit", title: "Set up Husky pre-commit hooks", subtitle: "Git hooks", facts: ["Installed husky and lint-staged", "Pre-commit runs ESLint and Prettier", "Added commitlint for conventional commits"], narrative: "Configured Husky git hooks with lint-staged to run ESLint and Prettier on staged files before each commit. Added commitlint to enforce conventional commit message format.", concepts: ["husky", "git-hooks", "lint-staged", "commitlint", "ci"], files: [".husky/pre-commit", ".lintstagedrc", "commitlint.config.js"], importance: 4 },
-    ],
-  },
-  {
-    sessionRange: [5, 9],
-    daysAgoRange: [24, 20],
-    project: "webapp",
-    observations: [
-      { type: "file_edit", title: "Implement NextAuth.js v5 authentication", subtitle: "Auth setup", facts: ["Configured NextAuth.js v5 with Auth.js", "Added GitHub and Google OAuth providers", "Set up JWT session strategy with 30-day expiry"], narrative: "Implemented authentication using NextAuth.js v5 (Auth.js). Configured GitHub and Google as OAuth providers. Using JWT-based sessions with 30-day expiry instead of database sessions for simplicity.", concepts: ["nextauth", "authentication", "oauth", "jwt", "github", "google"], files: ["src/auth.ts", "src/app/api/auth/[...nextauth]/route.ts", ".env.local"], importance: 9 },
-      { type: "file_edit", title: "Create login and signup pages", subtitle: "Auth UI", facts: ["Built login page with OAuth buttons", "Added email/password form with validation", "Implemented error toast notifications"], narrative: "Created the login page with GitHub and Google OAuth sign-in buttons plus an email/password form. Used react-hook-form with zod validation. Added toast notifications for login errors.", concepts: ["login", "signup", "oauth", "form-validation", "react-hook-form", "zod"], files: ["src/app/login/page.tsx", "src/app/signup/page.tsx"], importance: 7 },
-      { type: "file_edit", title: "Add middleware for route protection", subtitle: "Auth middleware", facts: ["Created middleware.ts to protect /dashboard routes", "Redirects unauthenticated users to /login", "Allows public access to /api/webhooks"], narrative: "Added Next.js middleware that checks for valid sessions on protected routes (/dashboard/*). Unauthenticated users are redirected to /login. The /api/webhooks path is excluded from auth checks for third-party integrations.", concepts: ["middleware", "route-protection", "authentication", "security"], files: ["src/middleware.ts"], importance: 8 },
-      { type: "file_edit", title: "Implement role-based access control", subtitle: "RBAC", facts: ["Added user roles: admin, editor, viewer", "Created withAuth HOC for role checking", "Stored roles in JWT custom claims"], narrative: "Implemented role-based access control with three roles: admin, editor, and viewer. Created a withAuth higher-order component that checks user roles before rendering protected components. Roles are stored as custom claims in the JWT token.", concepts: ["rbac", "authorization", "roles", "jwt-claims", "security"], files: ["src/lib/auth/rbac.ts", "src/lib/auth/with-auth.tsx"], importance: 8 },
-      { type: "file_edit", title: "Add password hashing with bcrypt", subtitle: "Security", facts: ["Using bcrypt with cost factor 12", "Added password strength validation (min 8 chars, mixed case, number)", "Implemented rate limiting on login endpoint (5 attempts per 15 min)"], narrative: "Added bcrypt password hashing with cost factor 12 for the email/password authentication flow. Implemented password strength validation requiring minimum 8 characters with mixed case and numbers. Added rate limiting on the login API endpoint: 5 attempts per 15-minute window per IP.", concepts: ["bcrypt", "password-hashing", "rate-limiting", "security", "validation"], files: ["src/lib/auth/password.ts", "src/app/api/auth/login/route.ts"], importance: 9 },
-      { type: "file_edit", title: "Create user profile settings page", subtitle: "User settings", facts: ["Profile page shows avatar, name, email", "Added avatar upload with S3 presigned URLs", "Implemented account deletion flow"], narrative: "Built the user profile settings page showing avatar, name, and email. Added avatar upload using S3 presigned URLs for direct browser-to-S3 uploads. Implemented a full account deletion flow with email confirmation.", concepts: ["user-profile", "settings", "s3", "file-upload", "account-deletion"], files: ["src/app/dashboard/settings/page.tsx", "src/app/api/upload/route.ts"], importance: 6 },
-      { type: "command_run", title: "Debug OAuth callback URL mismatch", subtitle: "Auth debugging", facts: ["GitHub OAuth callback failed with redirect_uri_mismatch", "Fixed: NEXTAUTH_URL was set to http:// but app served on https://", "Lesson: always use HTTPS in production OAuth callback URLs"], narrative: "Spent time debugging why GitHub OAuth login failed in production. The error was redirect_uri_mismatch. Root cause: NEXTAUTH_URL environment variable was set to http://localhost:3000 in production instead of the HTTPS production URL. Fixed by updating the environment variable.", concepts: ["oauth-debugging", "github", "callback-url", "environment-variables", "production"], files: [".env.production"], importance: 7 },
-      { type: "file_edit", title: "Add CSRF protection to API routes", subtitle: "Security", facts: ["Implemented double-submit cookie pattern", "Added CSRF token generation in layout", "Validated CSRF token on all POST/PUT/DELETE requests"], narrative: "Added CSRF protection using the double-submit cookie pattern. A CSRF token is generated on page load and stored in both a cookie and a hidden form field. All mutating API requests (POST, PUT, DELETE) validate the token.", concepts: ["csrf", "security", "cookies", "api-protection"], files: ["src/lib/csrf.ts", "src/middleware.ts"], importance: 8 },
-    ],
-  },
-  {
-    sessionRange: [10, 14],
-    daysAgoRange: [19, 15],
-    project: "webapp",
-    observations: [
-      { type: "file_edit", title: "Set up Prisma ORM with PostgreSQL", subtitle: "Database", facts: ["Initialized Prisma with PostgreSQL provider", "Created User, Post, Comment, Tag models", "Generated migrations with prisma migrate dev"], narrative: "Set up Prisma ORM connecting to a PostgreSQL database. Defined the initial schema with User, Post, Comment, and Tag models including many-to-many relationships between Post and Tag.", concepts: ["prisma", "postgresql", "database", "orm", "schema", "migrations"], files: ["prisma/schema.prisma", "src/lib/db.ts"], importance: 9 },
-      { type: "file_edit", title: "Create database seed script", subtitle: "Seeding", facts: ["Created seed.ts with faker-generated data", "Seeds 10 users, 50 posts, 200 comments", "Runs via prisma db seed command"], narrative: "Built a database seed script using faker.js to generate realistic test data. Creates 10 users with posts, comments, and tags. Configured to run automatically on prisma db seed.", concepts: ["database", "seeding", "faker", "test-data", "prisma"], files: ["prisma/seed.ts", "package.json"], importance: 5 },
-      { type: "file_edit", title: "Implement server actions for CRUD operations", subtitle: "Data layer", facts: ["Created server actions for post CRUD", "Used Prisma transactions for multi-step operations", "Added revalidatePath after mutations"], narrative: "Implemented Next.js server actions for post create, read, update, and delete operations. Used Prisma transactions for operations that modify multiple tables. Called revalidatePath after mutations to refresh cached data.", concepts: ["server-actions", "crud", "prisma", "transactions", "revalidation", "caching"], files: ["src/app/actions/posts.ts"], importance: 8 },
-      { type: "command_run", title: "Fix N+1 query in post listing", subtitle: "Performance", facts: ["Identified N+1 query loading post authors individually", "Fixed with Prisma include for eager loading", "Query count dropped from 52 to 3"], narrative: "Discovered an N+1 query problem on the post listing page — each post was triggering a separate query to load its author. Fixed by using Prisma's include option for eager loading. Total query count dropped from 52 to 3.", concepts: ["n+1", "performance", "prisma", "eager-loading", "query-optimization"], files: ["src/app/actions/posts.ts"], importance: 8 },
-      { type: "file_edit", title: "Add full-text search with PostgreSQL tsvector", subtitle: "Search", facts: ["Created tsvector column on posts table", "Built GIN index for fast text search", "Implemented search API with ts_rank scoring"], narrative: "Added full-text search using PostgreSQL's built-in tsvector functionality. Created a generated tsvector column combining title and body, with a GIN index. The search API uses ts_rank for relevance scoring and supports phrase matching.", concepts: ["full-text-search", "postgresql", "tsvector", "gin-index", "search"], files: ["prisma/migrations/20260301_add_search.sql", "src/app/api/search/route.ts"], importance: 7 },
-      { type: "file_edit", title: "Set up connection pooling with PgBouncer", subtitle: "Database infra", facts: ["Deployed PgBouncer in transaction pooling mode", "Configured max 25 client connections, 10 server connections", "Added DATABASE_URL_DIRECT for migrations (bypasses pooler)"], narrative: "Deployed PgBouncer as a connection pooler for PostgreSQL. Using transaction pooling mode to maximize connection reuse. Configured separate DATABASE_URL for application use (through pooler) and DATABASE_URL_DIRECT for migrations.", concepts: ["pgbouncer", "connection-pooling", "postgresql", "infrastructure"], files: ["docker-compose.yml", ".env"], importance: 7 },
-      { type: "command_run", title: "Debug Prisma migration drift", subtitle: "Database debugging", facts: ["prisma migrate deploy failed with drift detected", "Cause: manual SQL ALTER was run directly on production", "Resolution: ran prisma migrate resolve to mark migration as applied"], narrative: "Production deployment failed because Prisma detected schema drift — someone had run a manual ALTER TABLE directly on the production database. Resolved by using prisma migrate resolve to mark the conflicting migration as already applied.", concepts: ["prisma", "migration-drift", "database", "production", "debugging"], files: ["prisma/schema.prisma"], importance: 7 },
-      { type: "file_edit", title: "Add Redis caching layer for expensive queries", subtitle: "Caching", facts: ["Used ioredis with 60-second TTL for post listings", "Implemented cache-aside pattern", "Added cache invalidation on post mutations"], narrative: "Added a Redis caching layer for expensive database queries. Post listings are cached for 60 seconds using a cache-aside pattern. Cache entries are invalidated when posts are created, updated, or deleted.", concepts: ["redis", "caching", "cache-aside", "ioredis", "performance"], files: ["src/lib/cache.ts", "src/app/actions/posts.ts"], importance: 7 },
-    ],
-  },
-  {
-    sessionRange: [15, 19],
-    daysAgoRange: [14, 10],
-    project: "webapp",
-    observations: [
-      { type: "file_edit", title: "Build REST API with input validation", subtitle: "API", facts: ["Created /api/v1/posts, /api/v1/users endpoints", "Used zod for request body validation", "Added consistent error response format with error codes"], narrative: "Built a versioned REST API under /api/v1/ with endpoints for posts and users. All request bodies are validated with zod schemas. Errors follow a consistent format with error codes, messages, and field-level details.", concepts: ["rest-api", "zod", "validation", "error-handling", "api-design"], files: ["src/app/api/v1/posts/route.ts", "src/app/api/v1/users/route.ts", "src/lib/api/errors.ts"], importance: 8 },
-      { type: "file_edit", title: "Implement cursor-based pagination", subtitle: "API pagination", facts: ["Replaced offset pagination with cursor-based approach", "Uses Prisma cursor with opaque base64-encoded cursors", "Returns hasNextPage and endCursor in response"], narrative: "Switched from offset-based to cursor-based pagination for the post listing API. Cursors are base64-encoded Prisma record IDs. Response includes hasNextPage boolean and endCursor for the client to request the next page.", concepts: ["pagination", "cursor-based", "prisma", "api-design", "performance"], files: ["src/app/api/v1/posts/route.ts", "src/lib/api/pagination.ts"], importance: 7 },
-      { type: "file_edit", title: "Add API rate limiting with Upstash Redis", subtitle: "Rate limiting", facts: ["Used @upstash/ratelimit with sliding window algorithm", "10 requests per 10 seconds per API key", "Returns X-RateLimit-Remaining header"], narrative: "Implemented API rate limiting using Upstash Redis with a sliding window algorithm. Each API key is limited to 10 requests per 10-second window. Rate limit status is communicated via standard X-RateLimit-* headers.", concepts: ["rate-limiting", "upstash", "redis", "api-security", "sliding-window"], files: ["src/middleware.ts", "src/lib/rate-limit.ts"], importance: 8 },
-      { type: "file_edit", title: "Create webhook system for external integrations", subtitle: "Webhooks", facts: ["Built webhook registration and delivery system", "Events: post.created, post.updated, user.signup", "Implemented retry with exponential backoff (max 3 retries)"], narrative: "Created a webhook system allowing external services to subscribe to events. Supports post.created, post.updated, and user.signup events. Webhook deliveries use exponential backoff with up to 3 retries on failure.", concepts: ["webhooks", "events", "integrations", "retry", "exponential-backoff"], files: ["src/lib/webhooks.ts", "src/app/api/v1/webhooks/route.ts"], importance: 7 },
-      { type: "file_edit", title: "Add OpenAPI specification with Swagger UI", subtitle: "API docs", facts: ["Generated OpenAPI 3.1 spec from zod schemas", "Added Swagger UI at /api/docs", "Included request/response examples"], narrative: "Generated an OpenAPI 3.1 specification from the existing zod validation schemas. Added Swagger UI accessible at /api/docs for interactive API documentation with request/response examples.", concepts: ["openapi", "swagger", "api-documentation", "zod"], files: ["src/app/api/docs/route.ts", "src/lib/openapi.ts"], importance: 5 },
-      { type: "command_run", title: "Debug 504 gateway timeout on large queries", subtitle: "Performance debugging", facts: ["Large post queries timing out after 30 seconds on Vercel", "Root cause: missing database index on posts.authorId", "Added composite index (authorId, createdAt DESC), query dropped to 50ms"], narrative: "Investigated 504 Gateway Timeout errors on the post listing endpoint in production (Vercel). Found that large queries filtering by author were doing a full table scan. Added a composite index on (authorId, createdAt DESC) which reduced query time from 30+ seconds to 50ms.", concepts: ["performance", "timeout", "database-index", "postgresql", "vercel", "debugging"], files: ["prisma/migrations/20260310_add_author_index.sql"], importance: 9 },
-      { type: "file_edit", title: "Implement API versioning strategy", subtitle: "API design", facts: ["URL-based versioning: /api/v1/, /api/v2/", "v1 deprecated with Sunset header", "Migration guide in API docs"], narrative: "Established an API versioning strategy using URL-based versioning (/api/v1/, /api/v2/). The v1 API returns a Sunset header indicating its deprecation date. Added a migration guide to the API documentation.", concepts: ["api-versioning", "deprecation", "sunset-header", "backward-compatibility"], files: ["src/app/api/v2/posts/route.ts", "src/lib/api/versioning.ts"], importance: 6 },
-      { type: "file_edit", title: "Add request logging with structured JSON", subtitle: "Observability", facts: ["Used pino for structured JSON logging", "Logs request method, path, status, duration, user ID", "Configured log levels per environment"], narrative: "Added structured JSON request logging using pino. Each request logs method, path, response status, duration in milliseconds, and authenticated user ID. Log levels are configured per environment (debug in dev, info in production).", concepts: ["logging", "pino", "observability", "structured-logging", "monitoring"], files: ["src/lib/logger.ts", "src/middleware.ts"], importance: 6 },
-    ],
-  },
-  {
-    sessionRange: [20, 24],
-    daysAgoRange: [9, 5],
-    project: "webapp",
-    observations: [
-      { type: "file_edit", title: "Write unit tests for auth module", subtitle: "Testing", facts: ["25 test cases covering login, signup, role checking", "Mocked Prisma client with vitest", "Achieved 92% coverage on auth module"], narrative: "Wrote comprehensive unit tests for the authentication module. 25 test cases covering login flow, signup validation, role-based access checks, and password hashing. Mocked the Prisma client using vitest's vi.mock. Achieved 92% code coverage.", concepts: ["unit-testing", "vitest", "mocking", "authentication", "coverage"], files: ["tests/unit/auth.test.ts", "tests/unit/rbac.test.ts"], importance: 7 },
-      { type: "file_edit", title: "Add E2E tests with Playwright", subtitle: "E2E testing", facts: ["Configured Playwright with Chrome and Firefox", "Tests: login flow, post CRUD, search, pagination", "Set up test database with Docker for isolation"], narrative: "Set up Playwright for end-to-end testing with Chrome and Firefox browsers. Created E2E tests for the complete login flow, post CRUD operations, search functionality, and pagination. Each test run gets a fresh database via Docker containers.", concepts: ["playwright", "e2e-testing", "docker", "test-isolation", "browser-testing"], files: ["playwright.config.ts", "tests/e2e/auth.spec.ts", "tests/e2e/posts.spec.ts", "docker-compose.test.yml"], importance: 8 },
-      { type: "command_run", title: "Fix flaky Playwright test on CI", subtitle: "CI debugging", facts: ["Test passed locally but failed in GitHub Actions", "Root cause: missing waitForNavigation after form submit", "Fixed by using page.waitForURL instead of waitForNavigation"], narrative: "Debugged a flaky Playwright test that passed locally but failed intermittently in GitHub Actions CI. The issue was a race condition after form submission — the test was checking the URL before navigation completed. Fixed by replacing the deprecated waitForNavigation with page.waitForURL.", concepts: ["playwright", "flaky-test", "ci", "github-actions", "debugging", "race-condition"], files: ["tests/e2e/auth.spec.ts"], importance: 6 },
-      { type: "file_edit", title: "Add API integration tests with supertest", subtitle: "API testing", facts: ["30 test cases for REST API endpoints", "Tests validation, auth, error responses, pagination", "Uses test database with transaction rollback"], narrative: "Created API integration tests using supertest. 30 test cases covering request validation, authentication requirements, error response formats, and cursor-based pagination. Each test runs in a database transaction that rolls back after completion.", concepts: ["integration-testing", "supertest", "api-testing", "transactions", "test-isolation"], files: ["tests/integration/api.test.ts"], importance: 7 },
-      { type: "file_edit", title: "Set up test coverage reporting with codecov", subtitle: "Coverage", facts: ["Configured vitest coverage with v8 provider", "Minimum coverage thresholds: 80% branches, 85% lines", "Upload to Codecov in CI pipeline"], narrative: "Configured vitest code coverage using the v8 provider. Set minimum coverage thresholds at 80% for branches and 85% for lines. Coverage reports are uploaded to Codecov as part of the GitHub Actions CI pipeline.", concepts: ["code-coverage", "codecov", "vitest", "ci", "quality-gates"], files: ["vitest.config.ts", ".github/workflows/ci.yml"], importance: 5 },
-      { type: "file_edit", title: "Create test fixtures and factories", subtitle: "Test infrastructure", facts: ["Built factory functions for User, Post, Comment, Tag", "Uses faker for realistic data generation", "Supports partial overrides for specific test scenarios"], narrative: "Created test factory functions for all main models (User, Post, Comment, Tag). Factories use faker.js for realistic data and support partial overrides so individual tests can customize specific fields.", concepts: ["test-factories", "faker", "testing-infrastructure", "fixtures"], files: ["tests/fixtures/factories.ts"], importance: 5 },
-      { type: "command_run", title: "Debug memory leak in test suite", subtitle: "Test debugging", facts: ["Tests consuming 2GB+ RAM after 100+ test files", "Root cause: Prisma client not disconnected in afterAll", "Fixed by adding global teardown that calls prisma.$disconnect()"], narrative: "Investigated why the test suite was consuming over 2GB of RAM. The Prisma client was creating new connections in each test file but never disconnecting. Fixed by adding a global teardown hook that calls prisma.$disconnect().", concepts: ["memory-leak", "testing", "prisma", "debugging", "resource-management"], files: ["vitest.config.ts", "tests/setup.ts"], importance: 7 },
-      { type: "file_edit", title: "Add snapshot testing for API responses", subtitle: "Snapshot tests", facts: ["Added toMatchSnapshot for API response shapes", "Snapshot updates require --update flag", "Catches unintended breaking changes in API responses"], narrative: "Added snapshot testing for API response shapes to catch unintended breaking changes. Response bodies are compared against stored snapshots. Snapshots must be explicitly updated with the --update flag when intentional changes are made.", concepts: ["snapshot-testing", "api-testing", "regression-testing", "vitest"], files: ["tests/integration/api.test.ts", "tests/integration/__snapshots__/"], importance: 4 },
-    ],
-  },
-  {
-    sessionRange: [25, 29],
-    daysAgoRange: [4, 0],
-    project: "webapp",
-    observations: [
-      { type: "file_edit", title: "Create multi-stage Dockerfile", subtitle: "Docker", facts: ["Multi-stage build: deps → build → production", "Final image size 180MB (down from 1.2GB)", "Runs as non-root user with UID 1001"], narrative: "Created a multi-stage Dockerfile for the Next.js application. Stage 1 installs dependencies, stage 2 builds the app, stage 3 copies only production artifacts. Final image is 180MB (down from 1.2GB). Application runs as a non-root user for security.", concepts: ["docker", "multi-stage-build", "containerization", "security", "image-optimization"], files: ["Dockerfile", ".dockerignore"], importance: 7 },
-      { type: "file_edit", title: "Set up GitHub Actions CI/CD pipeline", subtitle: "CI/CD", facts: ["Matrix build: Node 18 and 20", "Jobs: lint, test, build, deploy", "Auto-deploy to Vercel on main branch push"], narrative: "Created a comprehensive GitHub Actions CI/CD pipeline with matrix builds for Node 18 and 20. Pipeline runs lint, test (with coverage), build, and deploy jobs. Merges to main automatically trigger Vercel deployment.", concepts: ["github-actions", "ci-cd", "deployment", "vercel", "automation"], files: [".github/workflows/ci.yml", ".github/workflows/deploy.yml"], importance: 8 },
-      { type: "file_edit", title: "Configure Kubernetes deployment manifests", subtitle: "K8s", facts: ["Created Deployment, Service, Ingress, HPA resources", "HPA: min 2, max 10 replicas, CPU target 70%", "Health checks: liveness on /healthz, readiness on /readyz"], narrative: "Created Kubernetes deployment manifests including Deployment, Service, Ingress, and HorizontalPodAutoscaler. HPA scales between 2 and 10 replicas targeting 70% CPU utilization. Added liveness and readiness probes for health monitoring.", concepts: ["kubernetes", "deployment", "hpa", "autoscaling", "health-checks", "ingress"], files: ["k8s/deployment.yaml", "k8s/service.yaml", "k8s/ingress.yaml", "k8s/hpa.yaml"], importance: 8 },
-      { type: "file_edit", title: "Add Terraform for AWS infrastructure", subtitle: "IaC", facts: ["VPC with public/private subnets across 3 AZs", "RDS PostgreSQL with Multi-AZ failover", "ElastiCache Redis cluster with 2 replicas"], narrative: "Created Terraform modules for AWS infrastructure. VPC spans 3 availability zones with public and private subnets. RDS PostgreSQL instance with Multi-AZ failover for high availability. ElastiCache Redis cluster with 2 read replicas.", concepts: ["terraform", "aws", "infrastructure-as-code", "vpc", "rds", "elasticache"], files: ["terraform/main.tf", "terraform/vpc.tf", "terraform/rds.tf", "terraform/redis.tf"], importance: 8 },
-      { type: "command_run", title: "Debug Kubernetes pod crash loop", subtitle: "K8s debugging", facts: ["Pods in CrashLoopBackOff status", "Root cause: DATABASE_URL secret not mounted correctly", "Fixed: Secret key name was 'database-url' but env var expected 'DATABASE_URL'"], narrative: "Debugged pods stuck in CrashLoopBackOff. The application was failing to start because the DATABASE_URL environment variable was empty. Root cause: the Kubernetes secret had the key 'database-url' (kebab-case) but the secretKeyRef expected 'DATABASE_URL' (uppercase).", concepts: ["kubernetes", "debugging", "crashloopbackoff", "secrets", "environment-variables"], files: ["k8s/deployment.yaml", "k8s/secrets.yaml"], importance: 8 },
-      { type: "file_edit", title: "Set up Datadog monitoring and alerting", subtitle: "Monitoring", facts: ["Deployed Datadog agent as DaemonSet", "Custom metrics: request latency, error rate, DB query time", "Alerts: p99 latency > 500ms, error rate > 1%"], narrative: "Deployed the Datadog monitoring agent as a Kubernetes DaemonSet. Created custom metrics for request latency, error rate, and database query time. Set up alerts that trigger when p99 latency exceeds 500ms or error rate exceeds 1%.", concepts: ["datadog", "monitoring", "alerting", "observability", "kubernetes"], files: ["k8s/datadog-agent.yaml", "src/lib/metrics.ts"], importance: 7 },
-      { type: "file_edit", title: "Implement blue-green deployment strategy", subtitle: "Deployment", facts: ["Two identical environments: blue and green", "Health check must pass before traffic switch", "Instant rollback by switching back to previous color"], narrative: "Implemented blue-green deployment strategy. Two identical environments run simultaneously — deploy to the inactive one, run health checks, then switch traffic via Kubernetes service selector update. Rollback is instant by pointing traffic back to the previous color.", concepts: ["blue-green", "deployment-strategy", "zero-downtime", "rollback", "kubernetes"], files: ["k8s/blue-deployment.yaml", "k8s/green-deployment.yaml", "scripts/deploy.sh"], importance: 7 },
-      { type: "file_edit", title: "Add Prometheus metrics and Grafana dashboards", subtitle: "Observability", facts: ["Exported custom metrics via /metrics endpoint", "Metrics: http_request_duration, db_query_duration, cache_hit_ratio", "Created Grafana dashboard with request rate, latency, error panels"], narrative: "Added Prometheus metrics export on a /metrics endpoint. Custom metrics include HTTP request duration histogram, database query duration, and cache hit ratio. Created a Grafana dashboard with panels for request rate, latency percentiles, error rate, and cache performance.", concepts: ["prometheus", "grafana", "metrics", "observability", "dashboards"], files: ["src/lib/metrics.ts", "grafana/dashboard.json"], importance: 6 },
-    ],
-  },
-];
-
-export function generateDataset(): {
-  observations: CompressedObservation[];
-  queries: LabeledQuery[];
-  sessions: Map<string, string[]>;
-} {
-  const observations: CompressedObservation[] = [];
-  const sessions = new Map<string, string[]>();
-
-  for (const group of RAW_SESSIONS) {
-    const [sStart, sEnd] = group.sessionRange;
-    const [dStart, dEnd] = group.daysAgoRange;
-
-    for (let s = sStart; s <= sEnd; s++) {
-      const sessionId = `ses_${s.toString().padStart(3, "0")}`;
-      const daysAgo = dStart - ((s - sStart) / Math.max(1, sEnd - sStart)) * (dStart - dEnd);
-      const obsIds: string[] = [];
-
-      const obsPerSession = Math.min(group.observations.length, OBS_PER_SESSION);
-      for (let o = 0; o < obsPerSession; o++) {
-        const idx = ((s - sStart) * obsPerSession + o) % group.observations.length;
-        const raw = group.observations[idx];
-        const obsId = `obs_${sessionId}_${o.toString().padStart(2, "0")}`;
-        const hourOffset = o * 0.5;
-
-        observations.push({
-          id: obsId,
-          sessionId,
-          timestamp: ts(daysAgo - hourOffset / 24),
-          ...raw,
-        });
-        obsIds.push(obsId);
-      }
-      sessions.set(sessionId, obsIds);
-    }
-  }
-
-  const queries: LabeledQuery[] = [
-    {
-      query: "How did we set up authentication?",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["nextauth", "authentication", "oauth", "jwt", "login", "signup"].includes(c))).map(o => o.id),
-      description: "Should find all auth-related observations across sessions 5-9",
-      category: "semantic",
-    },
-    {
-      query: "JWT token validation middleware",
-      relevantObsIds: observations.filter(o => o.concepts.includes("jwt") || (o.concepts.includes("middleware") && o.concepts.includes("authentication"))).map(o => o.id),
-      description: "Exact match on JWT middleware setup",
-      category: "exact",
-    },
-    {
-      query: "PostgreSQL connection issues",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["postgresql", "pgbouncer", "connection-pooling", "database"].includes(c))).map(o => o.id),
-      description: "Should find database connection and pooling observations",
-      category: "semantic",
-    },
-    {
-      query: "Playwright test configuration",
-      relevantObsIds: observations.filter(o => o.concepts.includes("playwright") || (o.concepts.includes("e2e-testing"))).map(o => o.id),
-      description: "E2E testing setup with Playwright",
-      category: "exact",
-    },
-    {
-      query: "Why did the production deployment fail?",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["debugging", "production", "crashloopbackoff", "timeout", "migration-drift"].includes(c))).map(o => o.id),
-      description: "Cross-session: find all production debugging incidents",
-      category: "cross-session",
-    },
-    {
-      query: "rate limiting implementation",
-      relevantObsIds: observations.filter(o => o.concepts.includes("rate-limiting")).map(o => o.id),
-      description: "Rate limiting across auth and API modules",
-      category: "exact",
-    },
-    {
-      query: "What security measures did we add?",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["security", "csrf", "bcrypt", "rate-limiting", "rbac", "password-hashing"].includes(c))).map(o => o.id),
-      description: "Broad semantic: all security-related work",
-      category: "semantic",
-    },
-    {
-      query: "database performance optimization",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["n+1", "query-optimization", "database-index", "performance", "eager-loading", "caching"].includes(c))).map(o => o.id),
-      description: "Performance optimizations across database and caching",
-      category: "semantic",
-    },
-    {
-      query: "Kubernetes pod crash debugging",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["crashloopbackoff", "kubernetes"].includes(c)) && o.concepts.includes("debugging")).map(o => o.id),
-      description: "Specific K8s debugging incident",
-      category: "entity",
-    },
-    {
-      query: "Docker containerization setup",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["docker", "multi-stage-build", "containerization", "dockerfile"].includes(c))).map(o => o.id),
-      description: "Docker-related observations",
-      category: "entity",
-    },
-    {
-      query: "How does caching work in the app?",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["redis", "caching", "cache-aside", "ioredis", "elasticache"].includes(c))).map(o => o.id),
-      description: "All caching-related observations",
-      category: "semantic",
-    },
-    {
-      query: "test infrastructure and factories",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["test-factories", "testing-infrastructure", "fixtures", "mocking"].includes(c))).map(o => o.id),
-      description: "Test setup infrastructure",
-      category: "exact",
-    },
-    {
-      query: "What happened with the OAuth callback error?",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["oauth-debugging", "callback-url"].includes(c))).map(o => o.id),
-      description: "Specific debugging incident recall",
-      category: "cross-session",
-    },
-    {
-      query: "monitoring and observability setup",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["datadog", "prometheus", "grafana", "monitoring", "observability", "alerting", "metrics", "logging", "pino"].includes(c))).map(o => o.id),
-      description: "All monitoring/observability observations",
-      category: "semantic",
-    },
-    {
-      query: "Prisma ORM configuration",
-      relevantObsIds: observations.filter(o => o.concepts.includes("prisma")).map(o => o.id),
-      description: "All Prisma-related observations",
-      category: "entity",
-    },
-    {
-      query: "CI/CD pipeline configuration",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["ci-cd", "github-actions", "deployment", "ci"].includes(c))).map(o => o.id),
-      description: "CI/CD related observations",
-      category: "exact",
-    },
-    {
-      query: "memory leak debugging",
-      relevantObsIds: observations.filter(o => o.concepts.includes("memory-leak")).map(o => o.id),
-      description: "Memory leak incidents (WebSocket handler, test suite)",
-      category: "cross-session",
-    },
-    {
-      query: "API design decisions",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["rest-api", "api-design", "api-versioning", "pagination", "openapi", "error-handling"].includes(c))).map(o => o.id),
-      description: "API design and architecture decisions",
-      category: "semantic",
-    },
-    {
-      query: "zod validation schemas",
-      relevantObsIds: observations.filter(o => o.concepts.includes("zod")).map(o => o.id),
-      description: "Where zod is used for validation",
-      category: "entity",
-    },
-    {
-      query: "infrastructure as code Terraform",
-      relevantObsIds: observations.filter(o => o.concepts.some(c => ["terraform", "infrastructure-as-code", "aws", "vpc", "rds", "elasticache"].includes(c))).map(o => o.id),
-      description: "Terraform/IaC observations",
-      category: "entity",
-    },
-  ];
-
-  return { observations, queries, sessions };
-}
-
-export function generateScaleDataset(count: number): CompressedObservation[] {
-  const base = generateDataset().observations;
-  const result: CompressedObservation[] = [];
-
-  for (let i = 0; i < count; i++) {
-    const src = base[i % base.length];
-    result.push({
-      ...src,
-      id: `obs_scale_${i.toString().padStart(6, "0")}`,
-      sessionId: `ses_${Math.floor(i / 8).toString().padStart(4, "0")}`,
-      timestamp: ts(Math.random() * 90),
-      title: `${src.title} (iteration ${i})`,
-      narrative: `${src.narrative} [Scale test variant ${i}, session group ${Math.floor(i / 8)}]`,
-    });
-  }
-  return result;
-}

benchmark/lib/percentiles.ts

@@ -1,22 +0,0 @@
-/**
- * Nearest-rank percentile over a pre-sorted ascending array of numbers.
- *
- * No dependencies, no allocation. The caller is responsible for sorting
- * the input ascending (`arr.sort((a, b) => a - b)`) — sorting in here
- * would hide an O(n log n) cost in what looks like a cheap lookup.
- *
- * @param sorted Ascending-sorted samples. Empty array returns `NaN`.
- * @param p Percentile in [0, 100]. Values outside the range are clamped.
- * @returns The sample at the nearest rank, or `NaN` for empty input.
- */
-export function pXX(sorted: number[], p: number): number {
-  const n = sorted.length;
-  if (n === 0) return NaN;
-  const clamped = Math.max(0, Math.min(100, p));
-  if (clamped === 0) return sorted[0]!;
-  if (clamped === 100) return sorted[n - 1]!;
-  // Nearest-rank: rank = ceil(p/100 * n), index = rank - 1.
-  const rank = Math.ceil((clamped / 100) * n);
-  const idx = Math.min(n - 1, Math.max(0, rank - 1));
-  return sorted[idx]!;
-}

benchmark/load-100k.ts

@@ -1,528 +0,0 @@
-/**
- * Load harness — seeds N synthetic memories against a local agentmemory
- * daemon, then drives a matrix of (N, concurrency, endpoint) cells and
- * records p50 / p90 / p99 latency + throughput per cell.
- *
- * Spec: GitHub issue #346.
- *
- * Runs against an already-running daemon at `http://localhost:3111` by
- * default. Set `AGENTMEMORY_BENCH_AUTOSTART=1` to spawn one via
- * `node dist/cli.js start` for the duration of the run.
- *
- * Env knobs:
- *   AGENTMEMORY_BENCH_AUTOSTART   "1" to spawn the daemon (default: assume up)
- *   AGENTMEMORY_URL               base URL of the daemon (default: http://localhost:3111)
- *   BENCH_N                       comma-separated N sizes (default: 1000,10000,100000)
- *   BENCH_C                       comma-separated concurrency levels (default: 1,10,100)
- *   BENCH_OPS                     ops per cell during measurement (default: 200)
- *   BENCH_SEED                    seed for the mulberry32 RNG (default: 0xC0FFEE)
- *   BENCH_OUT_DIR                 results dir (default: benchmark/results)
- *
- * The harness writes one JSON file per run named
- * `load-100k-<short-git-sha>.json`. The git sha is best-effort — falls
- * back to a timestamp when run outside a checkout.
- */
-
-import { spawn, type ChildProcess } from "node:child_process";
-import { execFileSync } from "node:child_process";
-import { mkdirSync, writeFileSync, existsSync } from "node:fs";
-import { join, resolve } from "node:path";
-import { performance } from "node:perf_hooks";
-
-import { pXX } from "./lib/percentiles.js";
-
-/** Seedable PRNG. Mulberry32 — 32-bit state, uniform output in [0, 1). */
-function mulberry32(seed: number): () => number {
-  let s = seed >>> 0;
-  return () => {
-    s = (s + 0x6d2b79f5) >>> 0;
-    let t = s;
-    t = Math.imul(t ^ (t >>> 15), t | 1);
-    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
-    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
-  };
-}
-
-const NOUNS = [
-  "cache", "queue", "router", "stream", "shard", "lock", "buffer", "worker",
-  "engine", "trigger", "function", "memory", "index", "graph", "vector",
-  "session", "observation", "summary", "embedding", "tokenizer", "scheduler",
-  "consumer", "producer", "channel", "actor", "pipeline", "watcher", "pool",
-];
-const VERBS = [
-  "flushes", "rotates", "compacts", "rebalances", "drains", "warms",
-  "expires", "deduplicates", "snapshots", "replays", "promotes", "demotes",
-  "merges", "splits", "indexes", "scans", "compresses", "uploads",
-];
-const CONCEPTS = [
-  "throughput", "latency", "backpressure", "consistency", "isolation",
-  "durability", "idempotency", "fan-out", "cardinality", "skew",
-  "hot-path", "cold-start", "tail-latency", "saturation", "quiescence",
-];
-
-function buildContent(rng: () => number, i: number): string {
-  const n = NOUNS[Math.floor(rng() * NOUNS.length)]!;
-  const v = VERBS[Math.floor(rng() * VERBS.length)]!;
-  const c1 = CONCEPTS[Math.floor(rng() * CONCEPTS.length)]!;
-  const c2 = CONCEPTS[Math.floor(rng() * CONCEPTS.length)]!;
-  const k = Math.floor(rng() * 9999);
-  return `seed-${i} the ${n} ${v} ${c1} under ${c2} pressure (k=${k})`;
-}
-
-interface RunConfig {
-  baseUrl: string;
-  Ns: number[];
-  Cs: number[];
-  opsPerCell: number;
-  seed: number;
-  outDir: string;
-  autoStart: boolean;
-}
-
-function parseIntList(raw: string | undefined, fallback: number[]): number[] {
-  if (!raw) return fallback;
-  const out = raw
-    .split(",")
-    .map((s) => parseInt(s.trim(), 10))
-    .filter((n) => Number.isFinite(n) && n > 0);
-  return out.length > 0 ? out : fallback;
-}
-
-function loadConfig(): RunConfig {
-  return {
-    baseUrl: (process.env["AGENTMEMORY_URL"] || "http://localhost:3111").replace(
-      /\/+$/,
-      "",
-    ),
-    Ns: parseIntList(process.env["BENCH_N"], [1000, 10000, 100000]),
-    Cs: parseIntList(process.env["BENCH_C"], [1, 10, 100]),
-    opsPerCell: parseInt(process.env["BENCH_OPS"] || "200", 10) || 200,
-    seed: parseInt(process.env["BENCH_SEED"] || "12648430", 10) || 12648430,
-    outDir:
-      process.env["BENCH_OUT_DIR"] ||
-      resolve(process.cwd(), "benchmark", "results"),
-    autoStart: process.env["AGENTMEMORY_BENCH_AUTOSTART"] === "1",
-  };
-}
-
-async function waitForLivez(baseUrl: string, timeoutMs: number): Promise<void> {
-  const start = Date.now();
-  let lastErr: unknown = null;
-  while (Date.now() - start < timeoutMs) {
-    try {
-      const res = await fetch(`${baseUrl}/agentmemory/livez`, {
-        signal: AbortSignal.timeout(2000),
-      });
-      if (res.ok) return;
-      lastErr = new Error(`livez HTTP ${res.status}`);
-    } catch (err) {
-      lastErr = err;
-    }
-    await new Promise((r) => setTimeout(r, 500));
-  }
-  const reason =
-    lastErr instanceof Error ? lastErr.message : String(lastErr ?? "unknown");
-  throw new Error(
-    `daemon at ${baseUrl} did not become ready within ${timeoutMs} ms: ${reason}`,
-  );
-}
-
-function maybeStartDaemon(): ChildProcess | null {
-  const candidates = ["dist/cli.mjs", "dist/cli.js"].map((p) =>
-    resolve(process.cwd(), p),
-  );
-  const cliPath = candidates.find((p) => existsSync(p));
-  if (!cliPath) {
-    throw new Error(
-      `AGENTMEMORY_BENCH_AUTOSTART=1 but neither ${candidates.join(" nor ")} exists — run \`npm run build\` first`,
-    );
-  }
-  const child = spawn(process.execPath, [cliPath, "start"], {
-    stdio: ["ignore", "pipe", "pipe"],
-    detached: false,
-  });
-  child.stdout?.on("data", () => {
-    /* drain */
-  });
-  child.stderr?.on("data", () => {
-    /* drain */
-  });
-  return child;
-}
-
-function shortGitSha(): string {
-  try {
-    const sha = execFileSync("git", ["rev-parse", "--short", "HEAD"], {
-      encoding: "utf8",
-      stdio: ["ignore", "pipe", "ignore"],
-    }).trim();
-    if (sha) return sha;
-  } catch {
-    /* no git */
-  }
-  return `nogit-${Date.now().toString(36)}`;
-}
-
-interface CellResult {
-  endpoint: string;
-  N: number;
-  C: number;
-  ops: number;
-  errors: number;
-  wall_ms: number;
-  throughput_per_sec: number;
-  p50_ms: number;
-  p90_ms: number;
-  p99_ms: number;
-  min_ms: number;
-  max_ms: number;
-}
-
-/**
- * Issue `total` requests against `fetcher`, capped at `concurrency`
- * in-flight at any moment. Records per-request latency in ms.
- */
-async function driveLoad(
-  concurrency: number,
-  total: number,
-  fetcher: (i: number) => Promise<void>,
-): Promise<{ latencies: number[]; errors: number; wallMs: number }> {
-  const latencies: number[] = [];
-  let errors = 0;
-  let issued = 0;
-  const wallStart = performance.now();
-
-  async function worker(): Promise<void> {
-    while (true) {
-      const i = issued++;
-      if (i >= total) return;
-      const t0 = performance.now();
-      try {
-        await fetcher(i);
-        latencies.push(performance.now() - t0);
-      } catch {
-        errors++;
-      }
-    }
-  }
-
-  const workers = Array.from({ length: Math.max(1, concurrency) }, () =>
-    worker(),
-  );
-  await Promise.allSettled(workers);
-  const wallMs = performance.now() - wallStart;
-  return { latencies, errors, wallMs };
-}
-
-function summarize(
-  endpoint: string,
-  N: number,
-  C: number,
-  latencies: number[],
-  errors: number,
-  wallMs: number,
-): CellResult {
-  const sorted = latencies.slice().sort((a, b) => a - b);
-  const ops = sorted.length;
-  return {
-    endpoint,
-    N,
-    C,
-    ops,
-    errors,
-    wall_ms: Math.round(wallMs * 1000) / 1000,
-    throughput_per_sec:
-      wallMs > 0 ? Math.round((ops / (wallMs / 1000)) * 100) / 100 : 0,
-    p50_ms: Math.round(pXX(sorted, 50) * 1000) / 1000,
-    p90_ms: Math.round(pXX(sorted, 90) * 1000) / 1000,
-    p99_ms: Math.round(pXX(sorted, 99) * 1000) / 1000,
-    min_ms: ops > 0 ? Math.round(sorted[0]! * 1000) / 1000 : NaN,
-    max_ms: ops > 0 ? Math.round(sorted[ops - 1]! * 1000) / 1000 : NaN,
-  };
-}
-
-async function seedMemories(
-  baseUrl: string,
-  count: number,
-  rng: () => number,
-  seedConcurrency = 32,
-): Promise<{ seeded: number; errors: number; wallMs: number }> {
-  let issued = 0;
-  let seeded = 0;
-  let errors = 0;
-  const t0 = performance.now();
-  async function worker(): Promise<void> {
-    while (true) {
-      const i = issued++;
-      if (i >= count) return;
-      const body = JSON.stringify({
-        content: buildContent(rng, i),
-        type: "observation",
-      });
-      try {
-        const res = await fetch(`${baseUrl}/agentmemory/remember`, {
-          method: "POST",
-          headers: { "content-type": "application/json" },
-          body,
-          signal: AbortSignal.timeout(30_000),
-        });
-        if (res.ok) {
-          seeded++;
-        } else {
-          errors++;
-        }
-        // drain body to free the socket
-        await res.text().catch(() => "");
-      } catch {
-        errors++;
-      }
-    }
-  }
-  await Promise.allSettled(
-    Array.from({ length: seedConcurrency }, () => worker()),
-  );
-  return { seeded, errors, wallMs: performance.now() - t0 };
-}
-
-async function measureRemember(
-  baseUrl: string,
-  rng: () => number,
-  N: number,
-  C: number,
-  ops: number,
-): Promise<CellResult> {
-  const { latencies, errors, wallMs } = await driveLoad(C, ops, async (i) => {
-    const body = JSON.stringify({
-      content: buildContent(rng, N + i),
-      type: "observation",
-    });
-    const res = await fetch(`${baseUrl}/agentmemory/remember`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body,
-      signal: AbortSignal.timeout(30_000),
-    });
-    await res.text().catch(() => "");
-    if (!res.ok) throw new Error(`HTTP ${res.status}`);
-  });
-  return summarize("POST /agentmemory/remember", N, C, latencies, errors, wallMs);
-}
-
-async function measureSmartSearch(
-  baseUrl: string,
-  rng: () => number,
-  N: number,
-  C: number,
-  ops: number,
-): Promise<CellResult> {
-  const queries = Array.from({ length: 32 }, (_, i) => buildContent(rng, i));
-  const { latencies, errors, wallMs } = await driveLoad(C, ops, async (i) => {
-    const body = JSON.stringify({
-      query: queries[i % queries.length],
-      limit: 5,
-    });
-    const res = await fetch(`${baseUrl}/agentmemory/smart-search`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body,
-      signal: AbortSignal.timeout(30_000),
-    });
-    await res.text().catch(() => "");
-    if (!res.ok) throw new Error(`HTTP ${res.status}`);
-  });
-  return summarize(
-    "POST /agentmemory/smart-search",
-    N,
-    C,
-    latencies,
-    errors,
-    wallMs,
-  );
-}
-
-async function measureMemoriesLatest(
-  baseUrl: string,
-  N: number,
-  C: number,
-  ops: number,
-): Promise<CellResult> {
-  const { latencies, errors, wallMs } = await driveLoad(C, ops, async () => {
-    const res = await fetch(`${baseUrl}/agentmemory/memories?latest=true`, {
-      method: "GET",
-      signal: AbortSignal.timeout(30_000),
-    });
-    await res.text().catch(() => "");
-    if (!res.ok) throw new Error(`HTTP ${res.status}`);
-  });
-  return summarize(
-    "GET /agentmemory/memories?latest=true",
-    N,
-    C,
-    latencies,
-    errors,
-    wallMs,
-  );
-}
-
-interface RunReport {
-  schema_version: 1;
-  generated_at: string;
-  git_sha: string;
-  base_url: string;
-  seed: number;
-  matrix: { N: number[]; C: number[] };
-  ops_per_cell: number;
-  cells: CellResult[];
-  notes: string;
-}
-
-async function main(): Promise<void> {
-  const cfg = loadConfig();
-  console.log(
-    `[load-100k] base=${cfg.baseUrl} N=${cfg.Ns.join(",")} C=${cfg.Cs.join(",")} ops/cell=${cfg.opsPerCell} seed=${cfg.seed}`,
-  );
-
-  let daemon: ChildProcess | null = null;
-  if (cfg.autoStart) {
-    console.log("[load-100k] AGENTMEMORY_BENCH_AUTOSTART=1 — spawning daemon");
-    daemon = maybeStartDaemon();
-  }
-
-  try {
-    console.log("[load-100k] waiting for /agentmemory/livez (timeout 30s)");
-    await waitForLivez(cfg.baseUrl, 30_000);
-
-    const cells: CellResult[] = [];
-    // N sorted ascending so each cell builds on the previous seed work.
-    const sortedNs = cfg.Ns.slice().sort((a, b) => a - b);
-    let seededSoFar = 0;
-    for (const N of sortedNs) {
-      const delta = N - seededSoFar;
-      if (delta > 0) {
-        console.log(
-          `[load-100k] seeding ${delta} memories (target N=${N})`,
-        );
-        const rng = mulberry32(cfg.seed + seededSoFar);
-        const seedRes = await seedMemories(cfg.baseUrl, delta, rng);
-        seededSoFar += seedRes.seeded;
-        console.log(
-          `[load-100k]   seeded=${seedRes.seeded} errors=${seedRes.errors} wall=${(
-            seedRes.wallMs / 1000
-          ).toFixed(2)}s`,
-        );
-        if (seedRes.errors > 0 && seedRes.seeded === 0) {
-          throw new Error(
-            `seeding produced 0 successes and ${seedRes.errors} errors — daemon misconfigured`,
-          );
-        }
-      }
-
-      for (const C of cfg.Cs) {
-        const probeRng = mulberry32(cfg.seed ^ (N * 0x9e3779b1) ^ C);
-
-        console.log(`[load-100k] cell N=${N} C=${C} remember`);
-        const remember = await measureRemember(
-          cfg.baseUrl,
-          probeRng,
-          N,
-          C,
-          cfg.opsPerCell,
-        );
-        cells.push(remember);
-
-        console.log(`[load-100k] cell N=${N} C=${C} smart-search`);
-        const search = await measureSmartSearch(
-          cfg.baseUrl,
-          mulberry32(cfg.seed ^ (N * 0x85ebca77) ^ C),
-          N,
-          C,
-          cfg.opsPerCell,
-        );
-        cells.push(search);
-
-        console.log(`[load-100k] cell N=${N} C=${C} memories?latest=true`);
-        const memories = await measureMemoriesLatest(
-          cfg.baseUrl,
-          N,
-          C,
-          cfg.opsPerCell,
-        );
-        cells.push(memories);
-      }
-    }
-
-    const report: RunReport = {
-      schema_version: 1,
-      generated_at: new Date().toISOString(),
-      git_sha: shortGitSha(),
-      base_url: cfg.baseUrl,
-      seed: cfg.seed,
-      matrix: { N: sortedNs, C: cfg.Cs.slice() },
-      ops_per_cell: cfg.opsPerCell,
-      cells,
-      notes:
-        "Single-process load harness. Latency in milliseconds. " +
-        "Throughput is wall-clock ops/sec for the cell (concurrent in-flight = C).",
-    };
-
-    mkdirSync(cfg.outDir, { recursive: true });
-    const outPath = join(cfg.outDir, `load-100k-${report.git_sha}.json`);
-    writeFileSync(outPath, JSON.stringify(report, null, 2) + "\n", "utf8");
-    console.log(`[load-100k] wrote ${outPath} (${cells.length} cells)`);
-
-    // Compact table to stdout for the verification commit.
-    console.log("");
-    console.log(
-      [
-        "endpoint".padEnd(40),
-        "N".padStart(8),
-        "C".padStart(4),
-        "ops".padStart(6),
-        "err".padStart(4),
-        "p50_ms".padStart(8),
-        "p90_ms".padStart(8),
-        "p99_ms".padStart(8),
-        "tp/s".padStart(9),
-      ].join(" "),
-    );
-    for (const c of cells) {
-      console.log(
-        [
-          c.endpoint.padEnd(40),
-          String(c.N).padStart(8),
-          String(c.C).padStart(4),
-          String(c.ops).padStart(6),
-          String(c.errors).padStart(4),
-          c.p50_ms.toFixed(2).padStart(8),
-          c.p90_ms.toFixed(2).padStart(8),
-          c.p99_ms.toFixed(2).padStart(8),
-          c.throughput_per_sec.toFixed(2).padStart(9),
-        ].join(" "),
-      );
-    }
-  } finally {
-    if (daemon) {
-      daemon.kill("SIGTERM");
-      // give it 2s to exit cleanly before SIGKILL
-      await new Promise<void>((resolveFn) => {
-        const t = setTimeout(() => {
-          try {
-            daemon!.kill("SIGKILL");
-          } catch {
-            /* already dead */
-          }
-          resolveFn();
-        }, 2000);
-        daemon!.once("exit", () => {
-          clearTimeout(t);
-          resolveFn();
-        });
-      });
-    }
-  }
-}
-
-main().catch((err) => {
-  console.error("[load-100k] failed:", err instanceof Error ? err.stack : err);
-  process.exit(1);
-});

benchmark/longmemeval-bench.ts

@@ -1,317 +0,0 @@
-import { SearchIndex } from "../src/state/search-index.js";
-import { VectorIndex } from "../src/state/vector-index.js";
-import { HybridSearch } from "../src/state/hybrid-search.js";
-import type {
-  CompressedObservation,
-  EmbeddingProvider,
-} from "../src/types.js";
-import { readFileSync, writeFileSync, existsSync } from "node:fs";
-
-interface LongMemEvalEntry {
-  question_id: string;
-  question_type: string;
-  question: string;
-  question_date: string;
-  answer: string;
-  answer_session_ids: string[];
-  haystack_dates: string[];
-  haystack_session_ids: string[];
-  haystack_sessions: Array<Array<{ role: string; content: string; has_answer?: boolean }>>;
-}
-
-interface SessionChunk {
-  sessionId: string;
-  text: string;
-  turnCount: number;
-}
-
-interface BenchResult {
-  question_id: string;
-  question_type: string;
-  recall_any_at_5: number;
-  recall_any_at_10: number;
-  recall_any_at_20: number;
-  ndcg_at_10: number;
-  mrr: number;
-  retrieved_session_ids: string[];
-  gold_session_ids: string[];
-}
-
-function chunkSessionToText(
-  turns: Array<{ role: string; content: string }>,
-): string {
-  return turns
-    .map((t) => `${t.role}: ${t.content}`)
-    .join("\n");
-}
-
-function recallAny(
-  retrievedSessionIds: string[],
-  goldSessionIds: string[],
-  k: number,
-): number {
-  const topK = new Set(retrievedSessionIds.slice(0, k));
-  return goldSessionIds.some((gid) => topK.has(gid)) ? 1.0 : 0.0;
-}
-
-function dcg(relevances: boolean[], k: number): number {
-  let sum = 0;
-  for (let i = 0; i < Math.min(k, relevances.length); i++) {
-    sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
-  }
-  return sum;
-}
-
-function ndcg(
-  retrievedSessionIds: string[],
-  goldSessionIds: Set<string>,
-  k: number,
-): number {
-  const rels = retrievedSessionIds
-    .slice(0, k)
-    .map((id) => goldSessionIds.has(id));
-  const idealRels = Array.from(
-    { length: Math.min(k, goldSessionIds.size) },
-    () => true,
-  );
-  const idealDCG = dcg(idealRels, k);
-  if (idealDCG === 0) return 0;
-  return dcg(rels, k) / idealDCG;
-}
-
-function mrr(
-  retrievedSessionIds: string[],
-  goldSessionIds: Set<string>,
-): number {
-  for (let i = 0; i < retrievedSessionIds.length; i++) {
-    if (goldSessionIds.has(retrievedSessionIds[i])) return 1 / (i + 1);
-  }
-  return 0;
-}
-
-class MockKV {
-  private store = new Map<string, Map<string, unknown>>();
-  async get<T>(scope: string, key: string): Promise<T> {
-    const m = this.store.get(scope);
-    if (!m || !m.has(key)) throw new Error(`Not found: ${scope}/${key}`);
-    return m.get(key) as T;
-  }
-  async set(scope: string, key: string, value: unknown): Promise<void> {
-    if (!this.store.has(scope)) this.store.set(scope, new Map());
-    this.store.get(scope)!.set(key, value);
-  }
-  async list<T>(scope: string): Promise<T[]> {
-    const m = this.store.get(scope);
-    if (!m) return [];
-    return Array.from(m.values()) as T[];
-  }
-  async delete(scope: string, key: string): Promise<void> {
-    this.store.get(scope)?.delete(key);
-  }
-}
-
-async function runBenchmark(
-  mode: "bm25" | "vector" | "hybrid",
-  embeddingProvider?: EmbeddingProvider,
-) {
-  const dataPath = new URL("./data/longmemeval_s_cleaned.json", import.meta.url).pathname;
-  if (!existsSync(dataPath)) {
-    console.error(`Dataset not found at ${dataPath}`);
-    console.error("Download from: https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned");
-    process.exit(1);
-  }
-
-  console.log(`Loading LongMemEval-S dataset...`);
-  const raw = JSON.parse(readFileSync(dataPath, "utf-8")) as LongMemEvalEntry[];
-
-  const abstentionTypes = new Set([
-    "single-session-user_abs",
-    "multi-session_abs",
-    "knowledge-update_abs",
-    "temporal-reasoning_abs",
-  ]);
-  const entries = raw.filter((e) => !abstentionTypes.has(e.question_type));
-  console.log(
-    `Loaded ${entries.length} questions (${raw.length - entries.length} abstention excluded)`,
-  );
-
-  const results: BenchResult[] = [];
-  let processed = 0;
-
-  for (const entry of entries) {
-    const sessionChunks: SessionChunk[] = [];
-    for (let i = 0; i < entry.haystack_sessions.length; i++) {
-      const sessionId = entry.haystack_session_ids[i];
-      const turns = entry.haystack_sessions[i];
-      const text = chunkSessionToText(turns);
-      sessionChunks.push({ sessionId, text, turnCount: turns.length });
-    }
-
-    const bm25 = new SearchIndex();
-    const vector = mode !== "bm25" ? new VectorIndex() : null;
-    const kv = new MockKV();
-
-    const observations: CompressedObservation[] = [];
-    for (const chunk of sessionChunks) {
-      const obs: CompressedObservation = {
-        id: `obs_${chunk.sessionId}`,
-        sessionId: chunk.sessionId,
-        timestamp: new Date().toISOString(),
-        type: "conversation",
-        title: chunk.text.slice(0, 80),
-        facts: [],
-        narrative: chunk.text,
-        concepts: [],
-        files: [],
-        importance: 5,
-      };
-      observations.push(obs);
-      bm25.add(obs);
-
-      if (vector && embeddingProvider) {
-        try {
-          const embedding = await embeddingProvider.embed(
-            chunk.text.slice(0, 512),
-          );
-          vector.add(obs.id, chunk.sessionId, embedding);
-        } catch {}
-      }
-
-      await kv.set(`mem:obs:${chunk.sessionId}`, obs.id, obs);
-    }
-
-    let retrievedObsIds: string[];
-
-    if (mode === "bm25") {
-      const bm25Results = bm25.search(entry.question, 20);
-      retrievedObsIds = bm25Results.map((r) => r.obsId);
-    } else {
-      const hybridSearch = new HybridSearch(
-        bm25,
-        vector,
-        embeddingProvider || null,
-        kv as any,
-        0.4,
-        0.6,
-        0.0,
-        false,
-      );
-      const hybridResults = await hybridSearch.search(entry.question, 20);
-      retrievedObsIds = hybridResults.map((r) => r.observation.id);
-    }
-
-    const retrievedSessionIds = retrievedObsIds.map((oid) =>
-      oid.replace(/^obs_/, ""),
-    );
-    const goldSet = new Set(entry.answer_session_ids);
-
-    const result: BenchResult = {
-      question_id: entry.question_id,
-      question_type: entry.question_type,
-      recall_any_at_5: recallAny(retrievedSessionIds, entry.answer_session_ids, 5),
-      recall_any_at_10: recallAny(retrievedSessionIds, entry.answer_session_ids, 10),
-      recall_any_at_20: recallAny(retrievedSessionIds, entry.answer_session_ids, 20),
-      ndcg_at_10: ndcg(retrievedSessionIds, goldSet, 10),
-      mrr: mrr(retrievedSessionIds, goldSet),
-      retrieved_session_ids: retrievedSessionIds.slice(0, 10),
-      gold_session_ids: entry.answer_session_ids,
-    };
-    results.push(result);
-    processed++;
-
-    if (processed % 50 === 0) {
-      const avgRecall5 =
-        results.reduce((s, r) => s + r.recall_any_at_5, 0) / results.length;
-      console.log(
-        `  [${processed}/${entries.length}] running recall_any@5: ${(avgRecall5 * 100).toFixed(1)}%`,
-      );
-    }
-  }
-
-  const avgRecallAny5 =
-    results.reduce((s, r) => s + r.recall_any_at_5, 0) / results.length;
-  const avgRecallAny10 =
-    results.reduce((s, r) => s + r.recall_any_at_10, 0) / results.length;
-  const avgRecallAny20 =
-    results.reduce((s, r) => s + r.recall_any_at_20, 0) / results.length;
-  const avgNdcg10 =
-    results.reduce((s, r) => s + r.ndcg_at_10, 0) / results.length;
-  const avgMrr =
-    results.reduce((s, r) => s + r.mrr, 0) / results.length;
-
-  const byType = new Map<string, BenchResult[]>();
-  for (const r of results) {
-    if (!byType.has(r.question_type)) byType.set(r.question_type, []);
-    byType.get(r.question_type)!.push(r);
-  }
-
-  console.log(`\n=== LongMemEval-S Results (${mode}) ===`);
-  console.log(`Questions: ${results.length} (excl. abstention)`);
-  console.log(`recall_any@5:  ${(avgRecallAny5 * 100).toFixed(1)}%`);
-  console.log(`recall_any@10: ${(avgRecallAny10 * 100).toFixed(1)}%`);
-  console.log(`recall_any@20: ${(avgRecallAny20 * 100).toFixed(1)}%`);
-  console.log(`NDCG@10:       ${(avgNdcg10 * 100).toFixed(1)}%`);
-  console.log(`MRR:           ${(avgMrr * 100).toFixed(1)}%`);
-
-  console.log(`\nBy question type:`);
-  for (const [type, typeResults] of byType) {
-    const r5 =
-      typeResults.reduce((s, r) => s + r.recall_any_at_5, 0) /
-      typeResults.length;
-    const r10 =
-      typeResults.reduce((s, r) => s + r.recall_any_at_10, 0) /
-      typeResults.length;
-    console.log(
-      `  ${type.padEnd(30)} R@5: ${(r5 * 100).toFixed(1)}%  R@10: ${(r10 * 100).toFixed(1)}%  (n=${typeResults.length})`,
-    );
-  }
-
-  const outPath = new URL(
-    `./data/longmemeval_results_${mode}.json`,
-    import.meta.url,
-  ).pathname;
-  writeFileSync(
-    outPath,
-    JSON.stringify(
-      {
-        mode,
-        questions: results.length,
-        recall_any_at_5: avgRecallAny5,
-        recall_any_at_10: avgRecallAny10,
-        recall_any_at_20: avgRecallAny20,
-        ndcg_at_10: avgNdcg10,
-        mrr: avgMrr,
-        per_type: Object.fromEntries(
-          Array.from(byType).map(([type, tr]) => [
-            type,
-            {
-              count: tr.length,
-              recall_any_at_5:
-                tr.reduce((s, r) => s + r.recall_any_at_5, 0) / tr.length,
-              recall_any_at_10:
-                tr.reduce((s, r) => s + r.recall_any_at_10, 0) / tr.length,
-            },
-          ]),
-        ),
-        per_question: results,
-      },
-      null,
-      2,
-    ),
-  );
-  console.log(`\nResults saved to ${outPath}`);
-}
-
-const mode = (process.argv[2] || "bm25") as "bm25" | "vector" | "hybrid";
-console.log(`Running LongMemEval-S benchmark in ${mode} mode...`);
-
-if (mode === "bm25") {
-  runBenchmark("bm25").catch(console.error);
-} else {
-  import("../src/providers/embedding/local.js")
-    .then(({ LocalEmbeddingProvider }) => {
-      const provider = new LocalEmbeddingProvider();
-      return runBenchmark(mode, provider);
-    })
-    .catch(console.error);
-}

benchmark/quality-eval.ts

@@ -1,643 +0,0 @@
-import { SearchIndex } from "../src/state/search-index.js";
-import { VectorIndex } from "../src/state/vector-index.js";
-import { HybridSearch } from "../src/state/hybrid-search.js";
-import { GraphRetrieval } from "../src/functions/graph-retrieval.js";
-import { extractEntitiesFromQuery } from "../src/functions/query-expansion.js";
-import type { CompressedObservation, GraphNode, GraphEdge, GraphEdgeType } from "../src/types.js";
-import { generateDataset, type LabeledQuery } from "./dataset.js";
-import { writeFileSync } from "node:fs";
-
-interface QualityMetrics {
-  query: string;
-  category: string;
-  recall_at_5: number;
-  recall_at_10: number;
-  recall_at_20: number;
-  precision_at_5: number;
-  precision_at_10: number;
-  ndcg_at_10: number;
-  mrr: number;
-  relevant_count: number;
-  retrieved_count: number;
-  latency_ms: number;
-}
-
-interface SystemMetrics {
-  system: string;
-  avg_recall_at_5: number;
-  avg_recall_at_10: number;
-  avg_recall_at_20: number;
-  avg_precision_at_5: number;
-  avg_precision_at_10: number;
-  avg_ndcg_at_10: number;
-  avg_mrr: number;
-  avg_latency_ms: number;
-  total_tokens_per_query: number;
-  per_query: QualityMetrics[];
-}
-
-function dcg(relevances: boolean[], k: number): number {
-  let sum = 0;
-  for (let i = 0; i < Math.min(k, relevances.length); i++) {
-    sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
-  }
-  return sum;
-}
-
-function ndcg(retrieved: string[], relevant: Set<string>, k: number): number {
-  const actualRelevances = retrieved.slice(0, k).map(id => relevant.has(id));
-  const idealRelevances = Array.from({ length: Math.min(k, relevant.size) }, () => true);
-  const idealDCG = dcg(idealRelevances, k);
-  if (idealDCG === 0) return 0;
-  return dcg(actualRelevances, k) / idealDCG;
-}
-
-function recall(retrieved: string[], relevant: Set<string>, k: number): number {
-  if (relevant.size === 0) return 1;
-  const topK = new Set(retrieved.slice(0, k));
-  let hits = 0;
-  for (const id of relevant) {
-    if (topK.has(id)) hits++;
-  }
-  return hits / relevant.size;
-}
-
-function precision(retrieved: string[], relevant: Set<string>, k: number): number {
-  const topK = retrieved.slice(0, k);
-  if (topK.length === 0) return 0;
-  let hits = 0;
-  for (const id of topK) {
-    if (relevant.has(id)) hits++;
-  }
-  return hits / topK.length;
-}
-
-function mrr(retrieved: string[], relevant: Set<string>): number {
-  for (let i = 0; i < retrieved.length; i++) {
-    if (relevant.has(retrieved[i])) return 1 / (i + 1);
-  }
-  return 0;
-}
-
-function estimateTokens(text: string): number {
-  return Math.ceil(text.length / 4);
-}
-
-function mockKV() {
-  const store = new Map<string, Map<string, unknown>>();
-  return {
-    get: async <T>(scope: string, key: string): Promise<T | null> => {
-      return (store.get(scope)?.get(key) as T) ?? null;
-    },
-    set: async <T>(scope: string, key: string, data: T): Promise<T> => {
-      if (!store.has(scope)) store.set(scope, new Map());
-      store.get(scope)!.set(key, data);
-      return data;
-    },
-    delete: async (scope: string, key: string): Promise<void> => {
-      store.get(scope)?.delete(key);
-    },
-    list: async <T>(scope: string): Promise<T[]> => {
-      const entries = store.get(scope);
-      return entries ? (Array.from(entries.values()) as T[]) : [];
-    },
-  };
-}
-
-function deterministicEmbedding(text: string, dims = 384): Float32Array {
-  const arr = new Float32Array(dims);
-  const words = text.toLowerCase().split(/\W+/).filter(w => w.length > 2);
-  for (const word of words) {
-    for (let i = 0; i < word.length; i++) {
-      const idx = (word.charCodeAt(i) * 31 + i * 17) % dims;
-      arr[idx] += 1;
-      const idx2 = (word.charCodeAt(i) * 37 + i * 13 + word.length * 7) % dims;
-      arr[idx2] += 0.5;
-    }
-  }
-  const norm = Math.sqrt(arr.reduce((s, v) => s + v * v, 0));
-  if (norm > 0) for (let i = 0; i < dims; i++) arr[i] /= norm;
-  return arr;
-}
-
-async function evalBm25Only(
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-): Promise<SystemMetrics> {
-  const index = new SearchIndex();
-  for (const obs of observations) index.add(obs);
-
-  const perQuery: QualityMetrics[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const start = performance.now();
-    const results = index.search(q.query, 20);
-    const latency = performance.now() - start;
-
-    const retrieved = results.map(r => r.obsId);
-    perQuery.push({
-      query: q.query,
-      category: q.category,
-      recall_at_5: recall(retrieved, relevant, 5),
-      recall_at_10: recall(retrieved, relevant, 10),
-      recall_at_20: recall(retrieved, relevant, 20),
-      precision_at_5: precision(retrieved, relevant, 5),
-      precision_at_10: precision(retrieved, relevant, 10),
-      ndcg_at_10: ndcg(retrieved, relevant, 10),
-      mrr: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      retrieved_count: results.length,
-      latency_ms: latency,
-    });
-  }
-
-  const avgTokens = perQuery.reduce((sum, q) => sum + q.retrieved_count, 0) / perQuery.length;
-  const avgObsTokens = observations.slice(0, 50).reduce((s, o) => s + estimateTokens(JSON.stringify(o)), 0) / 50;
-
-  return {
-    system: "BM25-only",
-    avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
-    avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
-    avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
-    avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
-    avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
-    avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
-    avg_mrr: avg(perQuery.map(q => q.mrr)),
-    avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
-    total_tokens_per_query: Math.round(avgObsTokens * avgTokens),
-    per_query: perQuery,
-  };
-}
-
-async function evalDualStream(
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-): Promise<SystemMetrics> {
-  const kv = mockKV();
-  const bm25 = new SearchIndex();
-  const vector = new VectorIndex();
-  const dims = 384;
-
-  for (const obs of observations) {
-    bm25.add(obs);
-    const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ");
-    vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
-    await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
-  }
-
-  const mockEmbed: any = {
-    name: "deterministic",
-    dimensions: dims,
-    embed: async (text: string) => deterministicEmbedding(text, dims),
-    embedBatch: async (texts: string[]) => texts.map(t => deterministicEmbedding(t, dims)),
-  };
-
-  const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0);
-  const perQuery: QualityMetrics[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const start = performance.now();
-    const results = await hybrid.search(q.query, 20);
-    const latency = performance.now() - start;
-
-    const retrieved = results.map(r => r.observation.id);
-    perQuery.push({
-      query: q.query,
-      category: q.category,
-      recall_at_5: recall(retrieved, relevant, 5),
-      recall_at_10: recall(retrieved, relevant, 10),
-      recall_at_20: recall(retrieved, relevant, 20),
-      precision_at_5: precision(retrieved, relevant, 5),
-      precision_at_10: precision(retrieved, relevant, 10),
-      ndcg_at_10: ndcg(retrieved, relevant, 10),
-      mrr: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      retrieved_count: results.length,
-      latency_ms: latency,
-    });
-  }
-
-  const avgResultTokens = perQuery.reduce((sum, q) => {
-    return sum + q.retrieved_count;
-  }, 0) / perQuery.length;
-  const avgObsTokens2 = observations.slice(0, 50).reduce((s, o) => s + estimateTokens(JSON.stringify(o)), 0) / 50;
-
-  return {
-    system: "Dual-stream (BM25+Vector)",
-    avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
-    avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
-    avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
-    avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
-    avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
-    avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
-    avg_mrr: avg(perQuery.map(q => q.mrr)),
-    avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
-    total_tokens_per_query: Math.round(avgObsTokens2 * avgResultTokens),
-    per_query: perQuery,
-  };
-}
-
-async function evalTripleStream(
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-): Promise<SystemMetrics> {
-  const kv = mockKV();
-  const bm25 = new SearchIndex();
-  const vector = new VectorIndex();
-  const dims = 384;
-
-  for (const obs of observations) {
-    bm25.add(obs);
-    const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ");
-    vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
-    await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
-  }
-
-  const conceptToNodes = new Map<string, string>();
-  const nodeTypes: GraphNode["type"][] = ["concept", "library", "file", "pattern"];
-  const edgeTypes: GraphEdgeType[] = ["uses", "related_to", "depends_on", "modifies"];
-  const now = new Date().toISOString();
-  let nodeId = 0;
-
-  for (const obs of observations) {
-    for (const concept of obs.concepts) {
-      if (!conceptToNodes.has(concept)) {
-        const nid = `gn_${nodeId++}`;
-        conceptToNodes.set(concept, nid);
-        await kv.set("mem:graph:nodes", nid, {
-          id: nid,
-          type: nodeTypes[nodeId % nodeTypes.length],
-          name: concept,
-          properties: {},
-          sourceObservationIds: [],
-          createdAt: now,
-        } as GraphNode);
-      }
-      const nid = conceptToNodes.get(concept)!;
-      const existing = await kv.get<GraphNode>("mem:graph:nodes", nid);
-      if (existing && !existing.sourceObservationIds.includes(obs.id)) {
-        existing.sourceObservationIds.push(obs.id);
-        await kv.set("mem:graph:nodes", nid, existing);
-      }
-    }
-
-    const capped = obs.concepts.slice(0, 10);
-    for (let i = 0; i < capped.length; i++) {
-      for (let j = i + 1; j < capped.length; j++) {
-        const srcNid = conceptToNodes.get(capped[i])!;
-        const tgtNid = conceptToNodes.get(capped[j])!;
-        if (srcNid && tgtNid && srcNid !== tgtNid) {
-          const eid = `ge_${srcNid}_${tgtNid}`;
-          const existing = await kv.get<GraphEdge>("mem:graph:edges", eid);
-          const weight = existing ? Math.min(1.0, existing.weight + 0.1) : 0.5;
-          await kv.set("mem:graph:edges", eid, {
-            id: eid,
-            type: edgeTypes[(i + j) % edgeTypes.length],
-            sourceNodeId: srcNid,
-            targetNodeId: tgtNid,
-            weight,
-            sourceObservationIds: existing
-              ? [...new Set([...existing.sourceObservationIds, obs.id])]
-              : [obs.id],
-            createdAt: now,
-            tcommit: now,
-            version: 1,
-            isLatest: true,
-          } as GraphEdge);
-        }
-      }
-    }
-  }
-
-  const mockEmbed: any = {
-    name: "deterministic",
-    dimensions: dims,
-    embed: async (text: string) => deterministicEmbedding(text, dims),
-    embedBatch: async (texts: string[]) => texts.map(t => deterministicEmbedding(t, dims)),
-  };
-
-  const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0.3);
-  const perQuery: QualityMetrics[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const start = performance.now();
-    const results = await hybrid.search(q.query, 20);
-    const latency = performance.now() - start;
-
-    const retrieved = results.map(r => r.observation.id);
-    perQuery.push({
-      query: q.query,
-      category: q.category,
-      recall_at_5: recall(retrieved, relevant, 5),
-      recall_at_10: recall(retrieved, relevant, 10),
-      recall_at_20: recall(retrieved, relevant, 20),
-      precision_at_5: precision(retrieved, relevant, 5),
-      precision_at_10: precision(retrieved, relevant, 10),
-      ndcg_at_10: ndcg(retrieved, relevant, 10),
-      mrr: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      retrieved_count: results.length,
-      latency_ms: latency,
-    });
-  }
-
-  const avgResultTokens3 = perQuery.reduce((sum, q) => {
-    return sum + q.retrieved_count;
-  }, 0) / perQuery.length;
-  const avgObsTokens3 = observations.slice(0, 50).reduce((s, o) => s + estimateTokens(JSON.stringify(o)), 0) / 50;
-
-  return {
-    system: "Triple-stream (BM25+Vector+Graph)",
-    avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
-    avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
-    avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
-    avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
-    avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
-    avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
-    avg_mrr: avg(perQuery.map(q => q.mrr)),
-    avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
-    total_tokens_per_query: Math.round(avgObsTokens3 * avgResultTokens3),
-    per_query: perQuery,
-  };
-}
-
-async function evalBuiltinMemory(
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-): Promise<SystemMetrics> {
-  const allText = observations.map(o =>
-    `## ${o.title}\n${o.narrative}\nConcepts: ${o.concepts.join(", ")}\nFiles: ${o.files.join(", ")}`
-  ).join("\n\n");
-
-  const totalTokens = estimateTokens(allText);
-
-  const perQuery: QualityMetrics[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const start = performance.now();
-
-    const queryTerms = q.query.toLowerCase().split(/\W+/).filter(w => w.length > 2);
-    const scored: Array<{ id: string; score: number }> = [];
-
-    for (const obs of observations) {
-      const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ").toLowerCase();
-      let score = 0;
-      for (const term of queryTerms) {
-        if (text.includes(term)) score++;
-      }
-      if (score > 0) scored.push({ id: obs.id, score });
-    }
-
-    scored.sort((a, b) => b.score - a.score);
-    const latency = performance.now() - start;
-
-    const retrieved = scored.map(s => s.id).slice(0, 20);
-    perQuery.push({
-      query: q.query,
-      category: q.category,
-      recall_at_5: recall(retrieved, relevant, 5),
-      recall_at_10: recall(retrieved, relevant, 10),
-      recall_at_20: recall(retrieved, relevant, 20),
-      precision_at_5: precision(retrieved, relevant, 5),
-      precision_at_10: precision(retrieved, relevant, 10),
-      ndcg_at_10: ndcg(retrieved, relevant, 10),
-      mrr: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      retrieved_count: Math.min(scored.length, 20),
-      latency_ms: latency,
-    });
-  }
-
-  return {
-    system: "Built-in (CLAUDE.md / grep)",
-    avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
-    avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
-    avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
-    avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
-    avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
-    avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
-    avg_mrr: avg(perQuery.map(q => q.mrr)),
-    avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
-    total_tokens_per_query: totalTokens,
-    per_query: perQuery,
-  };
-}
-
-async function evalBuiltinMemoryTruncated(
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-): Promise<SystemMetrics> {
-  const MAX_LINES = 200;
-  const lines = observations.map(o =>
-    `- ${o.title}: ${o.narrative.slice(0, 80)}... [${o.concepts.slice(0, 3).join(", ")}]`
-  );
-  const truncated = lines.slice(0, MAX_LINES);
-  const truncatedIds = new Set(observations.slice(0, MAX_LINES).map(o => o.id));
-  const totalTokens = estimateTokens(truncated.join("\n"));
-
-  const perQuery: QualityMetrics[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const start = performance.now();
-
-    const queryTerms = q.query.toLowerCase().split(/\W+/).filter(w => w.length > 2);
-    const scored: Array<{ id: string; score: number }> = [];
-
-    for (let i = 0; i < Math.min(MAX_LINES, observations.length); i++) {
-      const obs = observations[i];
-      const line = truncated[i];
-      let score = 0;
-      for (const term of queryTerms) {
-        if (line.toLowerCase().includes(term)) score++;
-      }
-      if (score > 0) scored.push({ id: obs.id, score });
-    }
-
-    scored.sort((a, b) => b.score - a.score);
-    const latency = performance.now() - start;
-
-    const retrieved = scored.map(s => s.id).slice(0, 20);
-
-    const reachableRelevant = new Set(
-      [...relevant].filter(id => truncatedIds.has(id))
-    );
-
-    perQuery.push({
-      query: q.query,
-      category: q.category,
-      recall_at_5: recall(retrieved, relevant, 5),
-      recall_at_10: recall(retrieved, relevant, 10),
-      recall_at_20: recall(retrieved, relevant, 20),
-      precision_at_5: precision(retrieved, relevant, 5),
-      precision_at_10: precision(retrieved, relevant, 10),
-      ndcg_at_10: ndcg(retrieved, relevant, 10),
-      mrr: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      retrieved_count: Math.min(scored.length, 20),
-      latency_ms: latency,
-    });
-  }
-
-  return {
-    system: "Built-in (200-line MEMORY.md)",
-    avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
-    avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
-    avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
-    avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
-    avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
-    avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
-    avg_mrr: avg(perQuery.map(q => q.mrr)),
-    avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
-    total_tokens_per_query: totalTokens,
-    per_query: perQuery,
-  };
-}
-
-function avg(nums: number[]): number {
-  return nums.length ? nums.reduce((a, b) => a + b, 0) / nums.length : 0;
-}
-
-function pct(n: number): string {
-  return (n * 100).toFixed(1) + "%";
-}
-
-function generateReport(systems: SystemMetrics[], obsCount: number, queryCount: number): string {
-  const lines: string[] = [];
-  const w = (s: string) => lines.push(s);
-
-  w("# agentmemory v0.6.0 — Search Quality Evaluation");
-  w("");
-  w(`**Date:** ${new Date().toISOString()}`);
-  w(`**Dataset:** ${obsCount} observations across 30 sessions (realistic coding project)`);
-  w(`**Queries:** ${queryCount} labeled queries with ground-truth relevance`);
-  w(`**Metric definitions:** Recall@K (fraction of relevant docs in top K), Precision@K (fraction of top K that are relevant), NDCG@10 (ranking quality), MRR (position of first relevant result)`);
-  w("");
-
-  w("## Head-to-Head Comparison");
-  w("");
-  w("| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Latency | Tokens/query |");
-  w("|--------|----------|-----------|-------------|---------|-----|---------|--------------|");
-  for (const s of systems) {
-    w(`| ${s.system} | ${pct(s.avg_recall_at_5)} | ${pct(s.avg_recall_at_10)} | ${pct(s.avg_precision_at_5)} | ${pct(s.avg_ndcg_at_10)} | ${pct(s.avg_mrr)} | ${s.avg_latency_ms.toFixed(2)}ms | ${s.total_tokens_per_query.toLocaleString()} |`);
-  }
-
-  w("");
-  w("## Why This Matters");
-  w("");
-
-  const builtin = systems.find(s => s.system.includes("CLAUDE.md / grep"));
-  const truncated = systems.find(s => s.system.includes("200-line"));
-  const triple = systems.find(s => s.system.includes("Triple"));
-  const bm25 = systems.find(s => s.system === "BM25-only");
-
-  if (builtin && triple) {
-    const recallLift = ((triple.avg_recall_at_10 - builtin.avg_recall_at_10) / Math.max(0.001, builtin.avg_recall_at_10) * 100);
-    const tokenSaving = ((1 - triple.total_tokens_per_query / builtin.total_tokens_per_query) * 100);
-    w(`**Recall improvement:** agentmemory triple-stream finds ${pct(triple.avg_recall_at_10)} of relevant memories at K=10 vs ${pct(builtin.avg_recall_at_10)} for keyword grep (${recallLift > 0 ? "+" : ""}${recallLift.toFixed(0)}%)`);
-    w(`**Token savings:** agentmemory returns only the top 10 results (${triple.total_tokens_per_query.toLocaleString()} tokens) vs loading everything into context (${builtin.total_tokens_per_query.toLocaleString()} tokens) — ${tokenSaving.toFixed(0)}% reduction`);
-  }
-
-  if (truncated && triple) {
-    w(`**200-line cap:** Claude Code's MEMORY.md is capped at 200 lines. With ${obsCount} observations, ${pct(truncated.avg_recall_at_10)} recall at K=10 — memories from later sessions are simply invisible.`);
-  }
-
-  w("");
-  w("## Per-Query Breakdown (Triple-Stream)");
-  w("");
-
-  if (triple) {
-    w("| Query | Category | Recall@10 | NDCG@10 | MRR | Relevant | Latency |");
-    w("|-------|----------|-----------|---------|-----|----------|---------|");
-    for (const q of triple.per_query) {
-      w(`| ${q.query.slice(0, 45)}${q.query.length > 45 ? "..." : ""} | ${q.category} | ${pct(q.recall_at_10)} | ${pct(q.ndcg_at_10)} | ${pct(q.mrr)} | ${q.relevant_count} | ${q.latency_ms.toFixed(1)}ms |`);
-    }
-  }
-
-  w("");
-  w("## By Query Category");
-  w("");
-
-  const categories = ["exact", "semantic", "cross-session", "entity"];
-  if (triple) {
-    w("| Category | Avg Recall@10 | Avg NDCG@10 | Avg MRR | Queries |");
-    w("|----------|---------------|-------------|---------|---------|");
-    for (const cat of categories) {
-      const qs = triple.per_query.filter(q => q.category === cat);
-      if (qs.length === 0) continue;
-      w(`| ${cat} | ${pct(avg(qs.map(q => q.recall_at_10)))} | ${pct(avg(qs.map(q => q.ndcg_at_10)))} | ${pct(avg(qs.map(q => q.mrr)))} | ${qs.length} |`);
-    }
-  }
-
-  w("");
-  w("## Context Window Analysis");
-  w("");
-  w("The fundamental problem with built-in agent memory:");
-  w("");
-  w("| Observations | MEMORY.md tokens | agentmemory tokens (top 10) | Savings | MEMORY.md reachable |");
-  w("|-------------|-----------------|---------------------------|---------|-------------------|");
-
-  for (const count of [240, 500, 1000, 5000]) {
-    const memTokens = Math.round(count * 50);
-    const amTokens = triple ? triple.total_tokens_per_query : 500;
-    const saving = ((1 - amTokens / memTokens) * 100);
-    const reachable = count <= 200 ? "100%" : `${((200 / count) * 100).toFixed(0)}%`;
-    w(`| ${count.toLocaleString()} | ${memTokens.toLocaleString()} | ${amTokens.toLocaleString()} | ${saving.toFixed(0)}% | ${reachable} |`);
-  }
-
-  w("");
-  w("At 240 observations (our dataset), MEMORY.md already hits its 200-line cap and loses access to the most recent 40 observations. At 1,000 observations, 80% of memories are invisible. agentmemory always searches the full corpus.");
-
-  w("");
-  w("---");
-  w("");
-  w(`*${systems.reduce((s, sys) => s + sys.per_query.length, 0)} evaluations across ${systems.length} systems. Ground-truth labels assigned by concept matching against observation metadata.*`);
-
-  return lines.join("\n");
-}
-
-async function main() {
-  console.log("Generating labeled dataset...");
-  const { observations, queries, sessions } = generateDataset();
-  console.log(`Dataset: ${observations.length} observations, ${sessions.size} sessions, ${queries.length} queries`);
-  console.log(`Avg relevant docs per query: ${(queries.reduce((s, q) => s + q.relevantObsIds.length, 0) / queries.length).toFixed(1)}`);
-  console.log("");
-
-  console.log("Evaluating: Built-in (CLAUDE.md / grep)...");
-  const builtinResults = await evalBuiltinMemory(observations, queries);
-  console.log(`  Recall@10: ${pct(builtinResults.avg_recall_at_10)}, NDCG@10: ${pct(builtinResults.avg_ndcg_at_10)}`);
-
-  console.log("Evaluating: Built-in (200-line MEMORY.md)...");
-  const truncatedResults = await evalBuiltinMemoryTruncated(observations, queries);
-  console.log(`  Recall@10: ${pct(truncatedResults.avg_recall_at_10)}, NDCG@10: ${pct(truncatedResults.avg_ndcg_at_10)}`);
-
-  console.log("Evaluating: BM25-only...");
-  const bm25Results = await evalBm25Only(observations, queries);
-  console.log(`  Recall@10: ${pct(bm25Results.avg_recall_at_10)}, NDCG@10: ${pct(bm25Results.avg_ndcg_at_10)}`);
-
-  console.log("Evaluating: Dual-stream (BM25+Vector)...");
-  const dualResults = await evalDualStream(observations, queries);
-  console.log(`  Recall@10: ${pct(dualResults.avg_recall_at_10)}, NDCG@10: ${pct(dualResults.avg_ndcg_at_10)}`);
-
-  console.log("Evaluating: Triple-stream (BM25+Vector+Graph)...");
-  const tripleResults = await evalTripleStream(observations, queries);
-  console.log(`  Recall@10: ${pct(tripleResults.avg_recall_at_10)}, NDCG@10: ${pct(tripleResults.avg_ndcg_at_10)}`);
-
-  console.log("");
-
-  const report = generateReport(
-    [builtinResults, truncatedResults, bm25Results, dualResults, tripleResults],
-    observations.length,
-    queries.length,
-  );
-
-  writeFileSync("benchmark/QUALITY.md", report);
-  console.log(report);
-  console.log(`\nReport written to benchmark/QUALITY.md`);
-}
-
-main().catch(console.error);

benchmark/real-embeddings-eval.ts

@@ -1,405 +0,0 @@
-import { SearchIndex } from "../src/state/search-index.js";
-import { VectorIndex } from "../src/state/vector-index.js";
-import { HybridSearch } from "../src/state/hybrid-search.js";
-import { LocalEmbeddingProvider } from "../src/providers/embedding/local.js";
-import type { CompressedObservation, EmbeddingProvider } from "../src/types.js";
-import { generateDataset, type LabeledQuery } from "./dataset.js";
-import { writeFileSync } from "node:fs";
-
-function mockKV() {
-  const store = new Map<string, Map<string, unknown>>();
-  return {
-    get: async <T>(scope: string, key: string): Promise<T | null> =>
-      (store.get(scope)?.get(key) as T) ?? null,
-    set: async <T>(scope: string, key: string, data: T): Promise<T> => {
-      if (!store.has(scope)) store.set(scope, new Map());
-      store.get(scope)!.set(key, data);
-      return data;
-    },
-    delete: async (scope: string, key: string): Promise<void> => {
-      store.get(scope)?.delete(key);
-    },
-    list: async <T>(scope: string): Promise<T[]> => {
-      const entries = store.get(scope);
-      return entries ? (Array.from(entries.values()) as T[]) : [];
-    },
-  };
-}
-
-function estimateTokens(text: string): number {
-  return Math.ceil(text.length / 4);
-}
-
-function obsToText(obs: CompressedObservation): string {
-  return [obs.title, obs.subtitle || "", obs.narrative, ...obs.facts, ...obs.concepts].join(" ");
-}
-
-function recall(retrieved: string[], relevant: Set<string>, k: number): number {
-  if (relevant.size === 0) return 1;
-  const topK = new Set(retrieved.slice(0, k));
-  let hits = 0;
-  for (const id of relevant) if (topK.has(id)) hits++;
-  return hits / relevant.size;
-}
-
-function precision(retrieved: string[], relevant: Set<string>, k: number): number {
-  const topK = retrieved.slice(0, k);
-  if (topK.length === 0) return 0;
-  let hits = 0;
-  for (const id of topK) if (relevant.has(id)) hits++;
-  return hits / topK.length;
-}
-
-function dcg(relevances: boolean[], k: number): number {
-  let sum = 0;
-  for (let i = 0; i < Math.min(k, relevances.length); i++)
-    sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
-  return sum;
-}
-
-function ndcg(retrieved: string[], relevant: Set<string>, k: number): number {
-  const actual = retrieved.slice(0, k).map(id => relevant.has(id));
-  const ideal = Array.from({ length: Math.min(k, relevant.size) }, () => true);
-  const idealDCG = dcg(ideal, k);
-  return idealDCG === 0 ? 0 : dcg(actual, k) / idealDCG;
-}
-
-function mrr(retrieved: string[], relevant: Set<string>): number {
-  for (let i = 0; i < retrieved.length; i++)
-    if (relevant.has(retrieved[i])) return 1 / (i + 1);
-  return 0;
-}
-
-function avg(nums: number[]): number {
-  return nums.length ? nums.reduce((a, b) => a + b, 0) / nums.length : 0;
-}
-
-function pct(n: number): string {
-  return (n * 100).toFixed(1) + "%";
-}
-
-interface QueryResult {
-  query: string;
-  category: string;
-  recall_5: number;
-  recall_10: number;
-  precision_5: number;
-  ndcg_10: number;
-  mrr_val: number;
-  relevant_count: number;
-  latency_ms: number;
-}
-
-interface SystemResult {
-  name: string;
-  results: QueryResult[];
-  embed_time_ms: number;
-  tokens_per_query: number;
-}
-
-async function evalSystem(
-  name: string,
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-  provider: EmbeddingProvider | null,
-  weights: { bm25: number; vector: number; graph: number },
-): Promise<SystemResult> {
-  const kv = mockKV();
-  const bm25 = new SearchIndex();
-  const vector = provider ? new VectorIndex() : null;
-
-  console.log(`  Indexing ${observations.length} observations...`);
-  const embedStart = performance.now();
-
-  for (const obs of observations) {
-    bm25.add(obs);
-    await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
-  }
-
-  if (provider && vector) {
-    const batchSize = 32;
-    for (let i = 0; i < observations.length; i += batchSize) {
-      const batch = observations.slice(i, i + batchSize);
-      const texts = batch.map(o => obsToText(o));
-      const embeddings = await provider.embedBatch(texts);
-      for (let j = 0; j < batch.length; j++) {
-        vector.add(batch[j].id, batch[j].sessionId, embeddings[j]);
-      }
-      if ((i + batchSize) % 100 === 0 || i + batchSize >= observations.length) {
-        process.stdout.write(`\r  Embedded ${Math.min(i + batchSize, observations.length)}/${observations.length}`);
-      }
-    }
-    console.log("");
-  }
-
-  const embedTime = performance.now() - embedStart;
-
-  const hybrid = new HybridSearch(
-    bm25,
-    vector,
-    provider,
-    kv as never,
-    weights.bm25,
-    weights.vector,
-    weights.graph,
-  );
-
-  console.log(`  Running ${queries.length} queries...`);
-  const results: QueryResult[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const start = performance.now();
-    const searchResults = await hybrid.search(q.query, 20);
-    const latency = performance.now() - start;
-
-    const retrieved = searchResults.map(r => r.observation.id);
-    results.push({
-      query: q.query,
-      category: q.category,
-      recall_5: recall(retrieved, relevant, 5),
-      recall_10: recall(retrieved, relevant, 10),
-      precision_5: precision(retrieved, relevant, 5),
-      ndcg_10: ndcg(retrieved, relevant, 10),
-      mrr_val: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      latency_ms: latency,
-    });
-  }
-
-  let totalReturnedTokens = 0;
-  for (const q of queries) {
-    const searchResults = await hybrid.search(q.query, 10);
-    totalReturnedTokens += searchResults.reduce(
-      (sum, r) => sum + estimateTokens(JSON.stringify(r.observation)),
-      0,
-    );
-  }
-  const avgReturnedTokens = Math.round(totalReturnedTokens / queries.length);
-
-  return {
-    name,
-    results,
-    embed_time_ms: embedTime,
-    tokens_per_query: avgReturnedTokens,
-  };
-}
-
-async function evalBuiltinGrep(
-  observations: CompressedObservation[],
-  queries: LabeledQuery[],
-): Promise<SystemResult> {
-  const results: QueryResult[] = [];
-
-  for (const q of queries) {
-    const relevant = new Set(q.relevantObsIds);
-    const queryTerms = q.query.toLowerCase().split(/\W+/).filter(w => w.length > 2);
-    const start = performance.now();
-
-    const scored: Array<{ id: string; score: number }> = [];
-    for (const obs of observations) {
-      const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ").toLowerCase();
-      let score = 0;
-      for (const term of queryTerms) if (text.includes(term)) score++;
-      if (score > 0) scored.push({ id: obs.id, score });
-    }
-    scored.sort((a, b) => b.score - a.score);
-    const latency = performance.now() - start;
-
-    const retrieved = scored.map(s => s.id).slice(0, 20);
-    results.push({
-      query: q.query,
-      category: q.category,
-      recall_5: recall(retrieved, relevant, 5),
-      recall_10: recall(retrieved, relevant, 10),
-      precision_5: precision(retrieved, relevant, 5),
-      ndcg_10: ndcg(retrieved, relevant, 10),
-      mrr_val: mrr(retrieved, relevant),
-      relevant_count: relevant.size,
-      latency_ms: latency,
-    });
-  }
-
-  const allTokens = estimateTokens(observations.map(o =>
-    `## ${o.title}\n${o.narrative}\nConcepts: ${o.concepts.join(", ")}`
-  ).join("\n\n"));
-
-  return { name: "Built-in (grep all)", results, embed_time_ms: 0, tokens_per_query: allTokens };
-}
-
-function generateReport(systems: SystemResult[], obsCount: number): string {
-  const lines: string[] = [];
-  const w = (s: string) => lines.push(s);
-
-  w("# agentmemory v0.6.0 — Real Embeddings Quality Evaluation");
-  w("");
-  w(`**Date:** ${new Date().toISOString()}`);
-  w(`**Platform:** ${process.platform} ${process.arch}, Node ${process.version}`);
-  w(`**Dataset:** ${obsCount} observations, 30 sessions, 20 labeled queries`);
-  w(`**Embedding model:** Xenova/all-MiniLM-L6-v2 (384d, local, no API key)`);
-  w("");
-
-  w("## Head-to-Head: Real Embeddings vs Keyword Search");
-  w("");
-  w("| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Avg Latency | Tokens/query |");
-  w("|--------|----------|-----------|-------------|---------|-----|-------------|--------------|");
-
-  for (const s of systems) {
-    const r = s.results;
-    w(`| ${s.name} | ${pct(avg(r.map(q => q.recall_5)))} | ${pct(avg(r.map(q => q.recall_10)))} | ${pct(avg(r.map(q => q.precision_5)))} | ${pct(avg(r.map(q => q.ndcg_10)))} | ${pct(avg(r.map(q => q.mrr_val)))} | ${avg(r.map(q => q.latency_ms)).toFixed(2)}ms | ${s.tokens_per_query.toLocaleString()} |`);
-  }
-
-  w("");
-  w("## Improvement from Real Embeddings");
-  w("");
-
-  const bm25Only = systems.find(s => s.name === "BM25-only (stemmed+synonyms)");
-  const dual = systems.find(s => s.name.includes("Dual-stream"));
-  const triple = systems.find(s => s.name.includes("Triple-stream"));
-  const builtin = systems.find(s => s.name.includes("grep"));
-
-  if (bm25Only && dual) {
-    const recallDelta = avg(dual.results.map(q => q.recall_10)) - avg(bm25Only.results.map(q => q.recall_10));
-    w(`Adding real vector embeddings to BM25 improves recall@10 by **${(recallDelta * 100).toFixed(1)} percentage points**.`);
-  }
-  if (builtin && dual) {
-    const tokenSaving = (1 - dual.tokens_per_query / builtin.tokens_per_query) * 100;
-    w(`Token savings vs loading everything: **${tokenSaving.toFixed(0)}%** (${dual.tokens_per_query.toLocaleString()} vs ${builtin.tokens_per_query.toLocaleString()} tokens).`);
-  }
-
-  w("");
-  w("## Per-Query: Where Real Embeddings Win");
-  w("");
-
-  if (bm25Only && dual) {
-    w("Queries where dual-stream (real embeddings) outperforms BM25-only:");
-    w("");
-    w("| Query | Category | BM25 Recall@10 | +Vector Recall@10 | Delta |");
-    w("|-------|----------|---------------|-------------------|-------|");
-
-    for (let i = 0; i < bm25Only.results.length; i++) {
-      const bq = bm25Only.results[i];
-      const dq = dual.results[i];
-      const delta = dq.recall_10 - bq.recall_10;
-      const marker = delta > 0 ? " **" : delta < 0 ? " *" : "";
-      if (Math.abs(delta) > 0.001) {
-        w(`| ${bq.query.slice(0, 45)}${bq.query.length > 45 ? "..." : ""} | ${bq.category} | ${pct(bq.recall_10)} | ${pct(dq.recall_10)} | ${delta > 0 ? "+" : ""}${(delta * 100).toFixed(1)}pp${marker} |`);
-      }
-    }
-  }
-
-  w("");
-  w("## By Category Comparison");
-  w("");
-  const categories = ["exact", "semantic", "cross-session", "entity"];
-
-  w("| Category | Built-in grep | BM25 (stemmed) | +Real Vectors | +Graph |");
-  w("|----------|--------------|----------------|--------------|--------|");
-
-  for (const cat of categories) {
-    const vals = systems.map(s => {
-      const qs = s.results.filter(q => q.category === cat);
-      return qs.length ? pct(avg(qs.map(q => q.recall_10))) : "-";
-    });
-    w(`| ${cat} | ${vals.join(" | ")} |`);
-  }
-
-  w("");
-  w("## Embedding Performance");
-  w("");
-  w("| System | Embedding Time | Model | Dimensions |");
-  w("|--------|---------------|-------|------------|");
-  for (const s of systems) {
-    if (s.embed_time_ms > 100) {
-      w(`| ${s.name} | ${(s.embed_time_ms / 1000).toFixed(1)}s | Xenova/all-MiniLM-L6-v2 | 384 |`);
-    }
-  }
-  w("");
-  w("Embedding is a one-time cost at ingestion. Search is sub-millisecond after indexing.");
-
-  w("");
-  w("## Key Findings");
-  w("");
-
-  if (bm25Only && dual) {
-    const semBm25 = bm25Only.results.filter(q => q.category === "semantic");
-    const semDual = dual.results.filter(q => q.category === "semantic");
-    const semImprove = avg(semDual.map(q => q.recall_10)) - avg(semBm25.map(q => q.recall_10));
-
-    w(`1. **Semantic queries improve most**: ${(semImprove * 100).toFixed(1)}pp recall@10 gain from real embeddings`);
-    w(`2. **"database performance optimization"** — the hardest query — goes from BM25 ${pct(bm25Only.results.find(q => q.query.includes("database perf"))?.recall_10 ?? 0)} to vector-augmented ${pct(dual.results.find(q => q.query.includes("database perf"))?.recall_10 ?? 0)}`);
-    w(`3. **Entity/exact queries** are already well-served by BM25+stemming — vectors add marginal value`);
-    w(`4. **Local embeddings (Xenova)** run without API keys — zero cost, zero latency concerns`);
-  }
-
-  w("");
-  w("## Recommendation");
-  w("");
-  w("Enable local embeddings by default (`EMBEDDING_PROVIDER=local` or install `@xenova/transformers`).");
-  w("This gives agentmemory genuine semantic search that built-in agent memories cannot match —");
-  w("understanding that \"database performance optimization\" relates to \"N+1 query fix\" and \"eager loading\".");
-  w("");
-
-  w("---");
-  w(`*All measurements use Xenova/all-MiniLM-L6-v2 local embeddings (384 dimensions, no API calls).*`);
-
-  return lines.join("\n");
-}
-
-async function main() {
-  console.log("=== agentmemory Real Embeddings Benchmark ===\n");
-
-  console.log("Loading Xenova/all-MiniLM-L6-v2 model (first run downloads ~80MB)...");
-  let provider: EmbeddingProvider;
-  try {
-    provider = new LocalEmbeddingProvider();
-    const testEmbed = await provider.embed("test");
-    console.log(`Model loaded. Dimensions: ${testEmbed.length}\n`);
-  } catch (err) {
-    console.error("Failed to load Xenova model:", err);
-    console.error("Install with: npm install @xenova/transformers");
-    process.exit(1);
-  }
-
-  const { observations, queries } = generateDataset();
-  console.log(`Dataset: ${observations.length} observations, ${queries.length} queries\n`);
-
-  console.log("1. Built-in (grep all)...");
-  const builtinResult = await evalBuiltinGrep(observations, queries);
-  console.log(`   Recall@10: ${pct(avg(builtinResult.results.map(q => q.recall_10)))}\n`);
-
-  console.log("2. BM25-only (stemmed+synonyms)...");
-  const bm25Result = await evalSystem(
-    "BM25-only (stemmed+synonyms)",
-    observations, queries, null,
-    { bm25: 1.0, vector: 0, graph: 0 },
-  );
-  console.log(`   Recall@10: ${pct(avg(bm25Result.results.map(q => q.recall_10)))}\n`);
-
-  console.log("3. Dual-stream (BM25 + real Xenova vectors)...");
-  const dualResult = await evalSystem(
-    "Dual-stream (BM25+Xenova)",
-    observations, queries, provider,
-    { bm25: 0.4, vector: 0.6, graph: 0 },
-  );
-  console.log(`   Recall@10: ${pct(avg(dualResult.results.map(q => q.recall_10)))}\n`);
-
-  console.log("4. Triple-stream (BM25 + Xenova + Graph)...");
-  const tripleResult = await evalSystem(
-    "Triple-stream (BM25+Xenova+Graph)",
-    observations, queries, provider,
-    { bm25: 0.4, vector: 0.6, graph: 0.3 },
-  );
-  console.log(`   Recall@10: ${pct(avg(tripleResult.results.map(q => q.recall_10)))}\n`);
-
-  const report = generateReport(
-    [builtinResult, bm25Result, dualResult, tripleResult],
-    observations.length,
-  );
-
-  writeFileSync("benchmark/REAL-EMBEDDINGS.md", report);
-  console.log(report);
-  console.log(`\nReport written to benchmark/REAL-EMBEDDINGS.md`);
-}
-
-main().catch(console.error);

benchmark/results/load-100k-96c0ed0.json

@@ -1,61 +0,0 @@
-{
-  "schema_version": 1,
-  "generated_at": "2026-05-13T19:49:26.116Z",
-  "git_sha": "96c0ed0",
-  "base_url": "http://localhost:3111",
-  "seed": 12648430,
-  "matrix": {
-    "N": [
-      1000
-    ],
-    "C": [
-      10
-    ]
-  },
-  "ops_per_cell": 200,
-  "cells": [
-    {
-      "endpoint": "POST /agentmemory/remember",
-      "N": 1000,
-      "C": 10,
-      "ops": 200,
-      "errors": 0,
-      "wall_ms": 11504.64,
-      "throughput_per_sec": 17.38,
-      "p50_ms": 577.435,
-      "p90_ms": 607.335,
-      "p99_ms": 675.269,
-      "min_ms": 64.46,
-      "max_ms": 683.164
-    },
-    {
-      "endpoint": "POST /agentmemory/smart-search",
-      "N": 1000,
-      "C": 10,
-      "ops": 200,
-      "errors": 0,
-      "wall_ms": 3264.572,
-      "throughput_per_sec": 61.26,
-      "p50_ms": 160.064,
-      "p90_ms": 185.608,
-      "p99_ms": 224.354,
-      "min_ms": 98.498,
-      "max_ms": 251.317
-    },
-    {
-      "endpoint": "GET /agentmemory/memories?latest=true",
-      "N": 1000,
-      "C": 10,
-      "ops": 200,
-      "errors": 0,
-      "wall_ms": 8051.764,
-      "throughput_per_sec": 24.84,
-      "p50_ms": 395.462,
-      "p90_ms": 475.714,
-      "p99_ms": 542.648,
-      "min_ms": 158.79,
-      "max_ms": 635.331
-    }
-  ],
-  "notes": "Single-process load harness. Latency in milliseconds. Throughput is wall-clock ops/sec for the cell (concurrent in-flight = C)."
-}

benchmark/scale-eval.ts

@@ -1,398 +0,0 @@
-import { SearchIndex } from "../src/state/search-index.js";
-import { VectorIndex } from "../src/state/vector-index.js";
-import { HybridSearch } from "../src/state/hybrid-search.js";
-import type { CompressedObservation } from "../src/types.js";
-import { generateScaleDataset, generateDataset } from "./dataset.js";
-import { writeFileSync } from "node:fs";
-
-function mockKV() {
-  const store = new Map<string, Map<string, unknown>>();
-  return {
-    get: async <T>(scope: string, key: string): Promise<T | null> =>
-      (store.get(scope)?.get(key) as T) ?? null,
-    set: async <T>(scope: string, key: string, data: T): Promise<T> => {
-      if (!store.has(scope)) store.set(scope, new Map());
-      store.get(scope)!.set(key, data);
-      return data;
-    },
-    delete: async (scope: string, key: string): Promise<void> => {
-      store.get(scope)?.delete(key);
-    },
-    list: async <T>(scope: string): Promise<T[]> => {
-      const entries = store.get(scope);
-      return entries ? (Array.from(entries.values()) as T[]) : [];
-    },
-  };
-}
-
-function deterministicEmbedding(text: string, dims = 384): Float32Array {
-  const arr = new Float32Array(dims);
-  const words = text.toLowerCase().split(/\W+/).filter(w => w.length > 2);
-  for (const word of words) {
-    for (let i = 0; i < word.length; i++) {
-      const idx = (word.charCodeAt(i) * 31 + i * 17) % dims;
-      arr[idx] += 1;
-      const idx2 = (word.charCodeAt(i) * 37 + i * 13 + word.length * 7) % dims;
-      arr[idx2] += 0.5;
-    }
-  }
-  const norm = Math.sqrt(arr.reduce((s, v) => s + v * v, 0));
-  if (norm > 0) for (let i = 0; i < dims; i++) arr[i] /= norm;
-  return arr;
-}
-
-function estimateTokens(text: string): number {
-  return Math.ceil(text.length / 4);
-}
-
-interface ScaleResult {
-  scale: number;
-  sessions: number;
-  index_build_ms: number;
-  index_build_per_doc_ms: number;
-  bm25_search_ms: number;
-  hybrid_search_ms: number;
-  index_size_kb: number;
-  vector_size_kb: number;
-  heap_mb: number;
-  builtin_tokens: number;
-  builtin_200line_tokens: number;
-  agentmemory_tokens: number;
-  token_savings_pct: number;
-  builtin_unreachable_pct: number;
-}
-
-interface CrossSessionResult {
-  query: string;
-  target_session: string;
-  current_session: string;
-  sessions_apart: number;
-  bm25_found: boolean;
-  bm25_rank: number;
-  hybrid_found: boolean;
-  hybrid_rank: number;
-  builtin_found: boolean;
-  latency_ms: number;
-}
-
-const SEARCH_QUERIES = [
-  "authentication middleware JWT",
-  "PostgreSQL connection pooling",
-  "Kubernetes pod crash",
-  "rate limiting API",
-  "Playwright E2E tests",
-  "Docker multi-stage build",
-  "Redis caching layer",
-  "CI/CD GitHub Actions",
-  "Prisma migration drift",
-  "monitoring Datadog alerts",
-];
-
-async function benchmarkScale(counts: number[]): Promise<ScaleResult[]> {
-  const results: ScaleResult[] = [];
-
-  for (const count of counts) {
-    console.log(`  Scale: ${count.toLocaleString()} observations...`);
-    const observations = generateScaleDataset(count);
-    const sessionCount = new Set(observations.map(o => o.sessionId)).size;
-
-    const heapBefore = process.memoryUsage().heapUsed;
-
-    const buildStart = performance.now();
-    const bm25 = new SearchIndex();
-    const vector = new VectorIndex();
-    const kv = mockKV();
-    const dims = 384;
-
-    for (const obs of observations) {
-      bm25.add(obs);
-      const text = [obs.title, obs.narrative, ...obs.concepts].join(" ");
-      vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
-      await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
-    }
-    const buildMs = performance.now() - buildStart;
-
-    const heapAfter = process.memoryUsage().heapUsed;
-
-    const mockEmbed: any = {
-      name: "deterministic", dimensions: dims,
-      embed: async (t: string) => deterministicEmbedding(t, dims),
-      embedBatch: async (ts: string[]) => ts.map(t => deterministicEmbedding(t, dims)),
-    };
-    const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0);
-
-    let bm25Total = 0;
-    let hybridTotal = 0;
-    const iters = 20;
-
-    for (let i = 0; i < iters; i++) {
-      const q = SEARCH_QUERIES[i % SEARCH_QUERIES.length];
-      const s1 = performance.now();
-      bm25.search(q, 10);
-      bm25Total += performance.now() - s1;
-
-      const s2 = performance.now();
-      await hybrid.search(q, 10);
-      hybridTotal += performance.now() - s2;
-    }
-
-    const bm25Ser = bm25.serialize();
-    const vecSer = vector.serialize();
-
-    const allText = observations.map(o =>
-      `- ${o.title}: ${o.narrative.slice(0, 80)}... [${o.concepts.slice(0, 3).join(", ")}]`
-    ).join("\n");
-    const builtinTokens = estimateTokens(allText);
-
-    const truncatedText = observations.slice(0, 200).map(o =>
-      `- ${o.title}: ${o.narrative.slice(0, 60)}... [${o.concepts.slice(0, 3).join(", ")}]`
-    ).join("\n");
-    const builtin200Tokens = estimateTokens(truncatedText);
-
-    let totalResultTokens = 0;
-    for (let i = 0; i < iters; i++) {
-      const q = SEARCH_QUERIES[i % SEARCH_QUERIES.length];
-      const results = await hybrid.search(q, 10);
-      totalResultTokens += estimateTokens(JSON.stringify(results.map(r => r.observation)));
-    }
-    const agentmemoryTokens = Math.round(totalResultTokens / iters);
-
-    results.push({
-      scale: count,
-      sessions: sessionCount,
-      index_build_ms: Math.round(buildMs),
-      index_build_per_doc_ms: +(buildMs / count).toFixed(3),
-      bm25_search_ms: +(bm25Total / iters).toFixed(3),
-      hybrid_search_ms: +(hybridTotal / iters).toFixed(3),
-      index_size_kb: Math.round(Buffer.byteLength(bm25Ser, "utf-8") / 1024),
-      vector_size_kb: Math.round(Buffer.byteLength(vecSer, "utf-8") / 1024),
-      heap_mb: Math.round((heapAfter - heapBefore) / 1024 / 1024),
-      builtin_tokens: builtinTokens,
-      builtin_200line_tokens: builtin200Tokens,
-      agentmemory_tokens: agentmemoryTokens,
-      token_savings_pct: Math.round((1 - agentmemoryTokens / builtinTokens) * 100),
-      builtin_unreachable_pct: count <= 200 ? 0 : Math.round((1 - 200 / count) * 100),
-    });
-  }
-
-  return results;
-}
-
-async function benchmarkCrossSession(): Promise<CrossSessionResult[]> {
-  const { observations } = generateDataset();
-  const results: CrossSessionResult[] = [];
-
-  const bm25 = new SearchIndex();
-  const kv = mockKV();
-  const vector = new VectorIndex();
-  const dims = 384;
-
-  for (const obs of observations) {
-    bm25.add(obs);
-    const text = [obs.title, obs.narrative, ...obs.concepts].join(" ");
-    vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
-    await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
-  }
-
-  const mockEmbed: any = {
-    name: "deterministic", dimensions: dims,
-    embed: async (t: string) => deterministicEmbedding(t, dims),
-    embedBatch: async (ts: string[]) => ts.map(t => deterministicEmbedding(t, dims)),
-  };
-  const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0);
-
-  const crossQueries: Array<{
-    query: string;
-    targetConcepts: string[];
-    targetSessionRange: [number, number];
-    currentSession: number;
-  }> = [
-    { query: "How did we set up OAuth providers?", targetConcepts: ["oauth", "nextauth"], targetSessionRange: [5, 9], currentSession: 29 },
-    { query: "What was the N+1 query fix?", targetConcepts: ["n+1", "eager-loading"], targetSessionRange: [10, 14], currentSession: 28 },
-    { query: "PostgreSQL full-text search setup", targetConcepts: ["full-text-search", "tsvector"], targetSessionRange: [10, 14], currentSession: 27 },
-    { query: "bcrypt password hashing configuration", targetConcepts: ["bcrypt", "password-hashing"], targetSessionRange: [5, 9], currentSession: 25 },
-    { query: "Vitest unit testing setup", targetConcepts: ["vitest", "unit-testing"], targetSessionRange: [20, 24], currentSession: 29 },
-    { query: "webhook retry exponential backoff", targetConcepts: ["webhooks", "exponential-backoff"], targetSessionRange: [15, 19], currentSession: 29 },
-    { query: "ESLint flat config migration", targetConcepts: ["eslint", "linting"], targetSessionRange: [0, 4], currentSession: 29 },
-    { query: "Kubernetes HPA autoscaling configuration", targetConcepts: ["hpa", "autoscaling", "kubernetes"], targetSessionRange: [25, 29], currentSession: 29 },
-    { query: "Prisma database seed script", targetConcepts: ["seeding", "faker", "prisma"], targetSessionRange: [10, 14], currentSession: 26 },
-    { query: "API cursor-based pagination", targetConcepts: ["cursor-based", "pagination"], targetSessionRange: [15, 19], currentSession: 29 },
-    { query: "CSRF protection double-submit cookie", targetConcepts: ["csrf", "cookies"], targetSessionRange: [5, 9], currentSession: 29 },
-    { query: "blue-green deployment rollback", targetConcepts: ["blue-green", "rollback", "zero-downtime"], targetSessionRange: [25, 29], currentSession: 29 },
-  ];
-
-  for (const cq of crossQueries) {
-    const targetObs = observations.filter(o =>
-      o.concepts.some(c => cq.targetConcepts.includes(c))
-    );
-    const targetIds = new Set(targetObs.map(o => o.id));
-
-    const start = performance.now();
-    const bm25Results = bm25.search(cq.query, 20);
-    const hybridResults = await hybrid.search(cq.query, 20);
-    const latency = performance.now() - start;
-
-    const bm25Rank = bm25Results.findIndex(r => targetIds.has(r.obsId));
-    const hybridRank = hybridResults.findIndex(r => targetIds.has(r.observation.id));
-
-    const builtinLines = 200;
-    const visibleObs = observations.slice(0, builtinLines);
-    const builtinFound = visibleObs.some(o => targetIds.has(o.id));
-
-    const sessionsApart = cq.currentSession - cq.targetSessionRange[0];
-
-    results.push({
-      query: cq.query,
-      target_session: `ses_${cq.targetSessionRange[0].toString().padStart(3, "0")}-${cq.targetSessionRange[1].toString().padStart(3, "0")}`,
-      current_session: `ses_${cq.currentSession.toString().padStart(3, "0")}`,
-      sessions_apart: sessionsApart,
-      bm25_found: bm25Rank >= 0,
-      bm25_rank: bm25Rank >= 0 ? bm25Rank + 1 : -1,
-      hybrid_found: hybridRank >= 0,
-      hybrid_rank: hybridRank >= 0 ? hybridRank + 1 : -1,
-      builtin_found: builtinFound,
-      latency_ms: latency,
-    });
-  }
-
-  return results;
-}
-
-function generateReport(scale: ScaleResult[], cross: CrossSessionResult[]): string {
-  const lines: string[] = [];
-  const w = (s: string) => lines.push(s);
-
-  w("# agentmemory v0.6.0 — Scale & Cross-Session Evaluation");
-  w("");
-  w(`**Date:** ${new Date().toISOString()}`);
-  w(`**Platform:** ${process.platform} ${process.arch}, Node ${process.version}`);
-  w("");
-
-  w("## 1. Scale: agentmemory vs Built-in Memory");
-  w("");
-  w("Every built-in agent memory (CLAUDE.md, .cursorrules, Cline's memory-bank) loads ALL memory into context every session. agentmemory searches and returns only relevant results.");
-  w("");
-  w("| Observations | Sessions | Index Build | BM25 Search | Hybrid Search | Heap | Context Tokens (built-in) | Context Tokens (agentmemory) | Savings | Built-in Unreachable |");
-  w("|-------------|----------|------------|-------------|---------------|------|--------------------------|-----------------------------|---------|--------------------|");
-
-  for (const r of scale) {
-    w(`| ${r.scale.toLocaleString()} | ${r.sessions} | ${r.index_build_ms}ms | ${r.bm25_search_ms}ms | ${r.hybrid_search_ms}ms | ${r.heap_mb}MB | ${r.builtin_tokens.toLocaleString()} | ${r.agentmemory_tokens.toLocaleString()} | ${r.token_savings_pct}% | ${r.builtin_unreachable_pct}% |`);
-  }
-
-  w("");
-  w("### What the numbers mean");
-  w("");
-  w("**Context Tokens (built-in):** How many tokens Claude Code/Cursor/Cline would consume loading ALL memory into the context window. At 5,000 observations, this is ~250K tokens — exceeding most context windows entirely.");
-  w("");
-  w("**Context Tokens (agentmemory):** How many tokens the top-10 search results consume. Stays constant regardless of corpus size.");
-  w("");
-  w("**Built-in Unreachable:** Percentage of memories that built-in systems CANNOT access because they exceed the 200-line MEMORY.md cap or context window limits. At 1,000 observations, 80% of your project history is invisible.");
-  w("");
-
-  w("### Storage Costs");
-  w("");
-  w("| Observations | BM25 Index | Vector Index (d=384) | Total Storage |");
-  w("|-------------|-----------|---------------------|---------------|");
-  for (const r of scale) {
-    const total = r.index_size_kb + r.vector_size_kb;
-    w(`| ${r.scale.toLocaleString()} | ${r.index_size_kb.toLocaleString()} KB | ${r.vector_size_kb.toLocaleString()} KB | ${(total / 1024).toFixed(1)} MB |`);
-  }
-
-  w("");
-  w("## 2. Cross-Session Retrieval");
-  w("");
-  w("Can the system find relevant information from past sessions? This is impossible for built-in memory once observations exceed the line/context cap.");
-  w("");
-  w("| Query | Target Session | Gap | BM25 Found | BM25 Rank | Hybrid Found | Hybrid Rank | Built-in Visible |");
-  w("|-------|---------------|-----|-----------|-----------|-------------|-------------|-----------------|");
-
-  for (const r of cross) {
-    w(`| ${r.query.slice(0, 40)}${r.query.length > 40 ? "..." : ""} | ${r.target_session} | ${r.sessions_apart} | ${r.bm25_found ? "Yes" : "No"} | ${r.bm25_rank > 0 ? `#${r.bm25_rank}` : "-"} | ${r.hybrid_found ? "Yes" : "No"} | ${r.hybrid_rank > 0 ? `#${r.hybrid_rank}` : "-"} | ${r.builtin_found ? "Yes" : "No"} |`);
-  }
-
-  const bm25Found = cross.filter(r => r.bm25_found).length;
-  const hybridFound = cross.filter(r => r.hybrid_found).length;
-  const builtinFound = cross.filter(r => r.builtin_found).length;
-
-  w("");
-  w(`**Summary:** agentmemory BM25 found ${bm25Found}/${cross.length} cross-session queries. Hybrid found ${hybridFound}/${cross.length}. Built-in memory (200-line cap) could only reach ${builtinFound}/${cross.length}.`);
-
-  w("");
-  w("## 3. The Context Window Problem");
-  w("");
-  w("```");
-  w("Agent context window: ~200K tokens");
-  w("System prompt + tools:  ~20K tokens");
-  w("User conversation:      ~30K tokens");
-  w("Available for memory:  ~150K tokens");
-  w("");
-  w("At 50 tokens/observation:");
-  w("  200 observations  =  10,000 tokens  (fits, but 200-line cap hits first)");
-  w("  1,000 observations =  50,000 tokens  (33% of available budget)");
-  w("  5,000 observations = 250,000 tokens  (EXCEEDS total context window)");
-  w("");
-  w("agentmemory top-10 results:");
-  w(`  Any corpus size     =  ~${scale[0]?.agentmemory_tokens.toLocaleString() || "500"} tokens  (0.3% of budget)`);
-  w("```");
-  w("");
-
-  w("## 4. What Built-in Memory Cannot Do");
-  w("");
-  w("| Capability | Built-in (CLAUDE.md) | agentmemory |");
-  w("|-----------|---------------------|-------------|");
-  w("| Semantic search | No (keyword grep only) | BM25 + vector + graph |");
-  w("| Scale beyond 200 lines | No (hard cap) | Unlimited |");
-  w("| Cross-session recall | Only if in 200-line window | Full corpus search |");
-  w("| Cross-agent sharing | No (per-agent files) | MCP + REST API |");
-  w("| Multi-agent coordination | No | Leases, signals, actions |");
-  w("| Temporal queries | No | Point-in-time graph |");
-  w("| Memory lifecycle | No (manual pruning) | Ebbinghaus decay + eviction |");
-  w("| Knowledge graph | No | Entity extraction + traversal |");
-  w("| Query expansion | No | LLM-generated reformulations |");
-  w("| Retention scoring | No | Time-frequency decay model |");
-  w("| Real-time dashboard | No (read files manually) | Viewer on :3113 |");
-  w("| Concurrent access | No (file lock) | Keyed mutex + KV store |");
-  w("");
-
-  w("## 5. When to Use What");
-  w("");
-  w("**Use built-in memory (CLAUDE.md) when:**");
-  w("- You have < 200 items to remember");
-  w("- Single agent, single project");
-  w("- Preferences and quick facts only");
-  w("- Zero setup is the priority");
-  w("");
-  w("**Use agentmemory when:**");
-  w("- Project history exceeds 200 observations");
-  w("- You need to recall specific incidents from weeks ago");
-  w("- Multiple agents work on the same codebase");
-  w("- You want semantic search (\"how does auth work?\") not just keyword matching");
-  w("- You need to track memory quality, decay, and lifecycle");
-  w("- You want a shared memory layer across Claude Code, Cursor, Windsurf, etc.");
-  w("");
-  w("Built-in memory is your sticky notes. agentmemory is the searchable database behind them.");
-  w("");
-
-  w("---");
-  w(`*Scale tests: ${scale.length} corpus sizes. Cross-session tests: ${cross.length} queries targeting specific past sessions.*`);
-
-  return lines.join("\n");
-}
-
-async function main() {
-  console.log("=== agentmemory Scale & Cross-Session Evaluation ===\n");
-
-  console.log("1. Scale benchmarks...");
-  const scaleResults = await benchmarkScale([240, 1_000, 5_000, 10_000, 50_000]);
-
-  console.log("\n2. Cross-session retrieval...");
-  const crossResults = await benchmarkCrossSession();
-
-  console.log("");
-  const report = generateReport(scaleResults, crossResults);
-  writeFileSync("benchmark/SCALE.md", report);
-  console.log(report);
-  console.log(`\nReport written to benchmark/SCALE.md`);
-}
-
-main().catch(console.error);

eval/README.md

@@ -1,111 +0,0 @@
-# agentmemory-evals
-
-Public benchmarks for agentmemory's hybrid memory stack (BM25 + embeddings + consolidation + graph).
-
-Two families, both reproducible:
-
-- **LongMemEval** — public 500-question retrieval benchmark over multi-session chat
-- **coding-agent-life-v1** — in-house corpus of 15 fictional Claude Code sessions for a Rust CLI project (`shipctl`), with 15 hand-graded queries covering bug fixes, refactors, preferences, and multi-session causal reasoning
-
-## Adapters
-
-| Adapter | Backend | API key needed |
-|---|---|---|
-| `grep` | Tokenized substring match | none |
-| `vector` | OpenAI `text-embedding-3-small` + cosine | `OPENAI_API_KEY` |
-| `agentmemory` | Running agentmemory server, smart-search endpoint | none (auth optional via `AGENTMEMORY_SECRET`) |
-
-## Sandbox first
-
-Running the `agentmemory` adapter against your real `~/.agentmemory` directory pollutes the eval with pre-existing memories AND pollutes your real store with eval test data. Always sandbox.
-
-`eval/scripts/sandbox.sh` spins up a clean agentmemory + iii-engine on ports 3411/3412 with state in `/tmp/agentmemory-eval-sandbox/`, exports `AGENTMEMORY_BASE_URL`, and tears down on exit.
-
-```sh
-source eval/scripts/sandbox.sh
-npm run eval:coding-life -- --adapters grep,agentmemory
-```
-
-Requires iii v0.11.2 on PATH (agentmemory pin). If you already have a different version installed, install the pinned build into `~/.local/bin` and make sure that directory comes first on `PATH`:
-
-```sh
-mkdir -p ~/.local/bin
-curl -fsSL https://github.com/iii-hq/iii/releases/download/iii/v0.11.2/iii-aarch64-apple-darwin.tar.gz | tar -xz -C ~/.local/bin
-export PATH="$HOME/.local/bin:$PATH"  # add to ~/.zshrc or ~/.bashrc for persistence
-```
-
-## Quickstart
-
-### coding-agent-life-v1 (in-house, no download)
-
-```sh
-# grep baseline, no sandbox needed
-npm run eval:coding-life -- --adapters grep
-
-# add agentmemory + vector (sandbox + OpenAI key)
-source eval/scripts/sandbox.sh
-OPENAI_API_KEY=sk-... npm run eval:coding-life -- --adapters grep,vector,agentmemory
-```
-
-### LongMemEval `_s` (public, 278MB download)
-
-```sh
-mkdir -p ~/datasets/longmemeval
-curl -Lo ~/datasets/longmemeval/longmemeval_s.json \
-  https://huggingface.co/datasets/xiaowu0162/longmemeval/resolve/main/longmemeval_s
-
-source eval/scripts/sandbox.sh
-
-# Stratified sample of 10 per type (fast iteration, ~$0.20 OpenAI cost)
-OPENAI_API_KEY=sk-... LONGMEMEVAL_PATH=~/datasets/longmemeval/longmemeval_s.json \
-  npm run eval:longmemeval -- --stratify 10
-
-# Full 500 questions × 3 adapters (~$2 OpenAI cost)
-OPENAI_API_KEY=sk-... LONGMEMEVAL_PATH=~/datasets/longmemeval/longmemeval_s.json \
-  npm run eval:longmemeval
-```
-
-## Repo layout
-
-```text
-eval/
-├── README.md
-├── runner/
-│   ├── types.ts                   Adapter, Question, RankedDoc, ScoreRow
-│   ├── score.ts                   P@K, R@K, aggregation
-│   ├── load.ts                    LongMemEval JSON → Question[]
-│   ├── adapters/
-│   │   ├── grep.ts                tokenized substring baseline
-│   │   ├── vector.ts              OpenAI embeddings + cosine
-│   │   └── agentmemory.ts         POST /agentmemory/{remember,smart-search}
-│   ├── longmemeval.ts             public benchmark runner
-│   └── coding-life.ts             in-house benchmark runner
-└── data/
-    └── coding-agent-life-v1/
-        ├── sessions.json          15 fictional sessions (~6KB)
-        └── queries.json           15 queries with gold session IDs
-```
-
-Reports land in `eval/reports/<bench>/` (gitignored): `scores.ndjson` + `summary.json`.
-
-Published scorecards land in `docs/benchmarks/YYYY-MM-DD-<bench>.md`.
-
-## Writing a new adapter
-
-1. Implement `Adapter<State>` from `eval/runner/types.ts`:
-   ```ts
-   import type { Adapter } from "../types.js";
-   export const myAdapter: Adapter<MyState> = {
-     name: "my-adapter",
-     async init(sessions, config) { /* index */ return state; },
-     async query(q, state, k) { /* search */ return ranked; },
-   };
-   ```
-2. Register in `eval/runner/{longmemeval,coding-life}.ts` `ADAPTERS` map.
-3. Run against `coding-agent-life-v1` to sanity-check before committing OpenAI spend on LongMemEval.
-
-## Why a benchmark for agentmemory
-
-agentmemory ships BM25 + embeddings + consolidation + graph retrieval. Numbers from those layers should be measured against grep/vector baselines so the value of each layer is provable.
-
-The in-house corpus is small on purpose (15 sessions) — covers single-session, multi-session, preference, and temporal question types without taking 15 minutes to run. LongMemEval gives the public-comparison axis.

eval/data/coding-agent-life-v1/queries.json

@@ -1,107 +0,0 @@
-[
-  {
-    "id": "q-001",
-    "type": "single-session-bug",
-    "question": "Where did we land the auth env var precedence fix?",
-    "answer": "PR #11 with SHIPCTL_TOKEN > SHIP_TOKEN > SC_TOKEN precedence",
-    "goldSessionIds": ["sess-001"]
-  },
-  {
-    "id": "q-002",
-    "type": "single-session-infra",
-    "question": "What was the multi-arch Docker fix?",
-    "answer": "Added --platform=$BUILDPLATFORM and BUILDX_PLATFORMS for amd64+arm64",
-    "goldSessionIds": ["sess-002"]
-  },
-  {
-    "id": "q-003",
-    "type": "single-session-refactor",
-    "question": "Where did we consolidate the retry logic?",
-    "answer": "src/retry.rs with exponential backoff base=200ms cap=30s full jitter",
-    "goldSessionIds": ["sess-003"]
-  },
-  {
-    "id": "q-004",
-    "type": "single-session-feature",
-    "question": "Which PR introduced helm chart support?",
-    "answer": "PR #14",
-    "goldSessionIds": ["sess-004"]
-  },
-  {
-    "id": "q-005",
-    "type": "single-session-test",
-    "question": "Which test was flaky on macos and how was it fixed?",
-    "answer": "fs-watcher emits_changekind_file_delete; bumped wait to 1500ms + retry: 2",
-    "goldSessionIds": ["sess-005"]
-  },
-  {
-    "id": "q-006",
-    "type": "single-session-perf",
-    "question": "How did we fix the memory leak?",
-    "answer": "Replaced unbounded HashMap with LruCache cap=10k in src/cache.rs (PR #16)",
-    "goldSessionIds": ["sess-006"]
-  },
-  {
-    "id": "q-007",
-    "type": "single-session-api",
-    "question": "How did we handle the github API rate limit?",
-    "answer": "Conditional requests with If-None-Match etag and 304 caching via http-cache",
-    "goldSessionIds": ["sess-007"]
-  },
-  {
-    "id": "q-008",
-    "type": "single-session-db",
-    "question": "What was the schema migration approach for run_history?",
-    "answer": "Three-phase: nullable column + dual-write, backfill + flip reads, drop old column",
-    "goldSessionIds": ["sess-008"]
-  },
-  {
-    "id": "q-009",
-    "type": "single-session-infra",
-    "question": "How is the docs site deployed?",
-    "answer": "GitHub Actions docs.yml workflow + mdbook build + Cloudflare Pages on shipctl.dev",
-    "goldSessionIds": ["sess-009"]
-  },
-  {
-    "id": "q-010",
-    "type": "single-session-release",
-    "question": "Which PR set up the cross-platform release pipeline?",
-    "answer": "PR #19 with cross-rs for linux and native macos/windows builds",
-    "goldSessionIds": ["sess-010"]
-  },
-  {
-    "id": "q-011",
-    "type": "multi-session-causal",
-    "question": "What was the root cause of the staging incident, and where was it fixed?",
-    "answer": "SHIPCTL_TOKEN unset caused fallback to bad SC_TOKEN; fixed in PR #11 (sess-001) with precedence test; documented in post-mortem (sess-014)",
-    "goldSessionIds": ["sess-001", "sess-014"]
-  },
-  {
-    "id": "q-012",
-    "type": "preference",
-    "question": "Which async runtime does the team prefer for new code?",
-    "answer": "async-std (decided in arch review, not tokio)",
-    "goldSessionIds": ["sess-015"]
-  },
-  {
-    "id": "q-013",
-    "type": "preference",
-    "question": "What are the user's formatting preferences?",
-    "answer": "2-space indent in yaml, tab in rust, rust-analyzer over rls, cargo clippy before commit",
-    "goldSessionIds": ["sess-012"]
-  },
-  {
-    "id": "q-014",
-    "type": "multi-session-review",
-    "question": "What was the race condition Aria fixed in the schema migration?",
-    "answer": "Dual-write step had race when two writers hit same row; Aria added per-row advisory lock",
-    "goldSessionIds": ["sess-008", "sess-013"]
-  },
-  {
-    "id": "q-015",
-    "type": "temporal",
-    "question": "What was shipped on April 8th 2026?",
-    "answer": "Auth env-var precedence fix in PR #11, then started docker arm64 work",
-    "goldSessionIds": ["sess-001", "sess-011"]
-  }
-]

eval/data/coding-agent-life-v1/sessions.json

@@ -1,77 +0,0 @@
-[
-  {
-    "id": "sess-001",
-    "timestamp": "2026-04-08T09:14:00Z",
-    "content": "[user] auth check broken on staging when SHIPCTL_TOKEN unset, falling back to wrong env var\n[assistant] looked at src/auth.rs:42. precedence is SHIPCTL_TOKEN > SHIP_TOKEN > legacy SC_TOKEN. legacy path didn't validate. fixing precedence + adding length check.\n[user] yeah ship it\n[assistant] PR #11 opened with fix + regression test. covers all three env names."
-  },
-  {
-    "id": "sess-002",
-    "timestamp": "2026-04-08T14:30:00Z",
-    "content": "[user] docker build failing for arm64 on M2 runner\n[assistant] missing target in Dockerfile FROM line. need --platform=$BUILDPLATFORM on builder stage. updated Dockerfile + Makefile to set BUILDX_PLATFORMS=linux/amd64,linux/arm64. tested locally on apple silicon, builds clean."
-  },
-  {
-    "id": "sess-003",
-    "timestamp": "2026-04-09T10:02:00Z",
-    "content": "[user] retry logic is duplicated across handlers. consolidate.\n[assistant] extracted retry_with_backoff into src/retry.rs. exponential backoff base=200ms cap=30s jitter=full. all 4 callsites now use it. cuts ~80 lines."
-  },
-  {
-    "id": "sess-004",
-    "timestamp": "2026-04-10T11:45:00Z",
-    "content": "[user] add helm chart support\n[assistant] new subcommand `shipctl helm install`. wraps helm CLI, threads through --namespace and --values. PR #14 has the changes + 12 integration tests. Aria reviewed and approved."
-  },
-  {
-    "id": "sess-005",
-    "timestamp": "2026-04-11T16:20:00Z",
-    "content": "[user] test flake on macos-22 runner. emits_changekind_file_delete keeps failing\n[assistant] fsevents propagation slower on apple silicon, the 800ms wait isn't enough. bumped to 1500ms + describe-level retry: 2. all 15 fs-watcher tests pass now."
-  },
-  {
-    "id": "sess-006",
-    "timestamp": "2026-04-12T09:50:00Z",
-    "content": "[user] memory leak in long-running daemon. RSS grows 5MB/min.\n[assistant] traced to unbounded HashMap in src/cache.rs. swapped for LruCache with 10k cap. RSS stable at 180MB after 6h soak test. PR #16."
-  },
-  {
-    "id": "sess-007",
-    "timestamp": "2026-04-13T13:15:00Z",
-    "content": "[user] github API rate-limiting us when listing PRs. 5000/hr isn't enough during ci\n[assistant] added conditional requests with If-None-Match etag. cache 304 responses. drops actual API calls by ~70%. used http-cache crate."
-  },
-  {
-    "id": "sess-008",
-    "timestamp": "2026-04-15T10:30:00Z",
-    "content": "[user] schema migration for run_history table needs zero-downtime\n[assistant] three-phase migration. phase 1: add nullable new column + dual-write. phase 2: backfill + flip reads. phase 3: drop old column. avoids ALTER TABLE locks. ran on staging, took 4s for 2M rows."
-  },
-  {
-    "id": "sess-009",
-    "timestamp": "2026-04-16T15:40:00Z",
-    "content": "[user] docs site is stale. set up auto-deploy from main\n[assistant] added .github/workflows/docs.yml. mdbook build + cloudflare pages deploy. domain shipctl.dev wired up. preview deploys on PRs."
-  },
-  {
-    "id": "sess-010",
-    "timestamp": "2026-04-17T11:00:00Z",
-    "content": "[user] release pipeline. cut v0.3.0 with binaries for linux/macos/windows\n[assistant] new .github/workflows/release.yml. cross-rs for static linux, native for macos + windows. uploads to gh release + crates.io publish. tag-triggered. PR #19 has the workflow."
-  },
-  {
-    "id": "sess-011",
-    "timestamp": "2026-04-08T17:00:00Z",
-    "content": "[user] standup notes: shipped #11 auth fix this morning. starting on docker arm64 next."
-  },
-  {
-    "id": "sess-012",
-    "timestamp": "2026-04-10T18:30:00Z",
-    "content": "[user] preferences: I like rust analyzer over rls. always run cargo clippy before commit. prefer 2-space indent in yaml but tab in rust."
-  },
-  {
-    "id": "sess-013",
-    "timestamp": "2026-04-15T19:00:00Z",
-    "content": "[assistant] reviewed Aria's PR #18 (schema migration). flagged race condition in dual-write step when two writers hit same row. Aria added per-row advisory lock. lgtm now."
-  },
-  {
-    "id": "sess-014",
-    "timestamp": "2026-04-16T20:10:00Z",
-    "content": "[user] post-mortem from prod incident last week: SHIPCTL_TOKEN was unset in staging, fell back to bad SC_TOKEN which had wrong perms. delivery delayed 40min. action items: (1) precedence test (done in #11), (2) startup validation, (3) alert on auth fallback."
-  },
-  {
-    "id": "sess-015",
-    "timestamp": "2026-04-17T16:45:00Z",
-    "content": "[user] preferences: stick to async-std not tokio for new code. team agreed in arch review."
-  }
-]

eval/runner/adapters/agentmemory.ts

@@ -1,93 +0,0 @@
-import type { Adapter, RankedDoc, Session } from "../types.js";
-
-interface AgentMemoryState {
-  baseUrl: string;
-  secret?: string;
-  sessions: Session[];
-  observationToSession: Map<string, string>;
-}
-
-interface RememberResponse {
-  memory?: { id?: string };
-  observationId?: string;
-  id?: string;
-  observation?: { id?: string };
-}
-
-interface SmartSearchResponse {
-  results?: Array<{
-    obsId?: string;
-    id?: string;
-    observationId?: string;
-    sessionId?: string;
-    score?: number;
-    content?: string;
-  }>;
-  observations?: Array<{
-    obsId?: string;
-    id?: string;
-    sessionId?: string;
-    score?: number;
-    content?: string;
-  }>;
-}
-
-function authHeaders(secret?: string): Record<string, string> {
-  const h: Record<string, string> = { "Content-Type": "application/json" };
-  if (secret) h.Authorization = `Bearer ${secret}`;
-  return h;
-}
-
-export const agentmemoryAdapter: Adapter<AgentMemoryState> = {
-  name: "agentmemory-hybrid",
-  async init(sessions, config) {
-    const baseUrl = (config?.baseUrl as string) ?? process.env.AGENTMEMORY_BASE_URL ?? "http://localhost:3111";
-    const secret = (config?.secret as string) ?? process.env.AGENTMEMORY_SECRET;
-    const observationToSession = new Map<string, string>();
-    for (const s of sessions) {
-      const res = await fetch(`${baseUrl}/agentmemory/remember`, {
-        method: "POST",
-        headers: authHeaders(secret),
-        body: JSON.stringify({
-          content: s.content,
-          type: "eval-session",
-          concepts: [s.id],
-        }),
-      });
-      if (!res.ok) {
-        throw new Error(`remember failed for ${s.id}: ${res.status} ${await res.text()}`);
-      }
-      const body = (await res.json()) as RememberResponse;
-      const obsId =
-        body.memory?.id ?? body.observationId ?? body.id ?? body.observation?.id;
-      if (obsId) observationToSession.set(obsId, s.id);
-    }
-    return { baseUrl, secret, sessions, observationToSession };
-  },
-  async query(q, state, k) {
-    const res = await fetch(`${state.baseUrl}/agentmemory/smart-search`, {
-      method: "POST",
-      headers: authHeaders(state.secret),
-      body: JSON.stringify({ query: q, limit: Math.max(k * 10, 50) }),
-    });
-    if (!res.ok) {
-      throw new Error(`smart-search failed: ${res.status} ${await res.text()}`);
-    }
-    const body = (await res.json()) as SmartSearchResponse;
-    const rows = body.results ?? body.observations ?? [];
-    const ranked: RankedDoc[] = [];
-    const seen = new Set<string>();
-    for (const row of rows) {
-      let sessionId = row.sessionId;
-      if (!sessionId) {
-        const memId = row.obsId ?? row.id ?? row.observationId;
-        sessionId = memId ? state.observationToSession.get(memId) : undefined;
-      }
-      if (!sessionId || seen.has(sessionId)) continue;
-      seen.add(sessionId);
-      ranked.push({ sessionId, score: row.score ?? 0 });
-      if (ranked.length >= k) break;
-    }
-    return ranked;
-  },
-};

eval/runner/adapters/grep.ts

@@ -1,36 +0,0 @@
-import type { Adapter, RankedDoc, Session } from "../types.js";
-
-interface GrepState {
-  sessions: Session[];
-}
-
-function tokenize(s: string): string[] {
-  return s
-    .toLowerCase()
-    .replace(/[^a-z0-9_]+/g, " ")
-    .split(/\s+/)
-    .filter((t) => t.length > 2);
-}
-
-export const grepAdapter: Adapter<GrepState> = {
-  name: "grep",
-  async init(sessions) {
-    return { sessions };
-  },
-  async query(q, state, k) {
-    const terms = tokenize(q);
-    const scored: RankedDoc[] = [];
-    for (const s of state.sessions) {
-      const body = s.content.toLowerCase();
-      let hits = 0;
-      for (const t of terms) {
-        if (body.includes(t)) hits += 1;
-      }
-      if (hits > 0) {
-        scored.push({ sessionId: s.id, score: hits });
-      }
-    }
-    scored.sort((a, b) => b.score - a.score);
-    return scored.slice(0, k);
-  },
-};

eval/runner/adapters/vector.ts

@@ -1,108 +0,0 @@
-import type { Adapter, RankedDoc, Session } from "../types.js";
-
-interface VectorState {
-  sessions: Session[];
-  embeddings: Float32Array[];
-}
-
-const OPENAI_URL = "https://api.openai.com/v1/embeddings";
-const MODEL = "text-embedding-3-small";
-const DIM = 1536;
-
-async function embed(text: string, apiKey: string): Promise<Float32Array> {
-  const res = await fetch(OPENAI_URL, {
-    method: "POST",
-    headers: {
-      "Content-Type": "application/json",
-      Authorization: `Bearer ${apiKey}`,
-    },
-    body: JSON.stringify({ input: text, model: MODEL }),
-  });
-  if (!res.ok) {
-    throw new Error(`OpenAI embed failed: ${res.status} ${await res.text()}`);
-  }
-  const data = (await res.json()) as { data: Array<{ embedding: number[] }> };
-  return Float32Array.from(data.data[0].embedding);
-}
-
-async function embedBatch(texts: string[], apiKey: string): Promise<Float32Array[]> {
-  const res = await fetch(OPENAI_URL, {
-    method: "POST",
-    headers: {
-      "Content-Type": "application/json",
-      Authorization: `Bearer ${apiKey}`,
-    },
-    body: JSON.stringify({ input: texts, model: MODEL }),
-  });
-  if (!res.ok) {
-    throw new Error(`OpenAI batch embed failed: ${res.status} ${await res.text()}`);
-  }
-  const data = (await res.json()) as { data: Array<{ embedding: number[]; index: number }> };
-  if (!Array.isArray(data.data) || data.data.length !== texts.length) {
-    throw new Error(
-      `OpenAI batch embed: expected ${texts.length} embeddings, got ${data.data?.length ?? 0}`,
-    );
-  }
-  const out = new Array<Float32Array>(texts.length);
-  for (const row of data.data) {
-    if (
-      !Number.isInteger(row.index) ||
-      row.index < 0 ||
-      row.index >= texts.length ||
-      out[row.index] !== undefined
-    ) {
-      throw new Error(`OpenAI batch embed: invalid or duplicate index ${row.index}`);
-    }
-    if (!Array.isArray(row.embedding) || row.embedding.length === 0) {
-      throw new Error(`OpenAI batch embed: empty embedding at index ${row.index}`);
-    }
-    out[row.index] = Float32Array.from(row.embedding);
-  }
-  return out;
-}
-
-function cosine(a: Float32Array, b: Float32Array): number {
-  let dot = 0;
-  let na = 0;
-  let nb = 0;
-  for (let i = 0; i < a.length; i++) {
-    dot += a[i] * b[i];
-    na += a[i] * a[i];
-    nb += b[i] * b[i];
-  }
-  const denom = Math.sqrt(na) * Math.sqrt(nb);
-  return denom === 0 ? 0 : dot / denom;
-}
-
-export const vectorAdapter: Adapter<VectorState> = {
-  name: "vector",
-  async init(sessions) {
-    const apiKey = process.env.OPENAI_API_KEY;
-    if (!apiKey) throw new Error("OPENAI_API_KEY required for vector adapter");
-    const embeddings: Float32Array[] = new Array(sessions.length);
-    const BATCH = 50;
-    for (let i = 0; i < sessions.length; i += BATCH) {
-      const batch = sessions.slice(i, i + BATCH);
-      const vecs = await embedBatch(
-        batch.map((s) => s.content.slice(0, 8000)),
-        apiKey,
-      );
-      for (let j = 0; j < vecs.length; j++) embeddings[i + j] = vecs[j];
-    }
-    if (embeddings.length > 0 && embeddings[0].length !== DIM) {
-      throw new Error(`unexpected embedding dim: ${embeddings[0].length}`);
-    }
-    return { sessions, embeddings };
-  },
-  async query(q, state, k) {
-    const apiKey = process.env.OPENAI_API_KEY;
-    if (!apiKey) throw new Error("OPENAI_API_KEY required for vector adapter");
-    const qvec = await embed(q, apiKey);
-    const scored: RankedDoc[] = state.sessions.map((s, i) => ({
-      sessionId: s.id,
-      score: cosine(qvec, state.embeddings[i]),
-    }));
-    scored.sort((a, b) => b.score - a.score);
-    return scored.slice(0, k);
-  },
-};

eval/runner/coding-life.ts

@@ -1,101 +0,0 @@
-import { readFileSync, existsSync, mkdirSync, writeFileSync, appendFileSync } from "node:fs";
-import { resolve } from "node:path";
-import { parseArgs } from "node:util";
-import { agentmemoryAdapter } from "./adapters/agentmemory.js";
-import { grepAdapter } from "./adapters/grep.js";
-import { vectorAdapter } from "./adapters/vector.js";
-import { aggregate, scoreQuestion } from "./score.js";
-import type { Adapter, Question, ScoreRow, Session } from "./types.js";
-
-const ADAPTERS: Record<string, Adapter> = {
-  grep: grepAdapter as unknown as Adapter,
-  vector: vectorAdapter as unknown as Adapter,
-  agentmemory: agentmemoryAdapter as unknown as Adapter,
-};
-
-interface CliOptions {
-  data: string;
-  adapters: string;
-  k: string;
-  out: string;
-}
-
-function parse(): CliOptions {
-  const { values } = parseArgs({
-    options: {
-      data: { type: "string", default: "eval/data/coding-agent-life-v1" },
-      adapters: { type: "string", default: "grep,vector,agentmemory" },
-      k: { type: "string", default: "5" },
-      out: { type: "string", default: "eval/reports/coding-life" },
-    },
-  });
-  return values as unknown as CliOptions;
-}
-
-async function main(): Promise<void> {
-  const opts = parse();
-  const k = Number(opts.k);
-  if (!Number.isInteger(k) || k <= 0) {
-    console.error(`--k must be a positive integer, got: ${opts.k}`);
-    process.exit(2);
-  }
-  const sessions = JSON.parse(
-    readFileSync(resolve(opts.data, "sessions.json"), "utf8"),
-  ) as Session[];
-  const queriesRaw = JSON.parse(
-    readFileSync(resolve(opts.data, "queries.json"), "utf8"),
-  ) as Array<Omit<Question, "haystack">>;
-  const questions: Question[] = queriesRaw.map((q) => ({ ...q, haystack: sessions }));
-  const adapterNames = opts.adapters.split(",").map((s) => s.trim()).filter(Boolean);
-  for (const a of adapterNames) {
-    if (!ADAPTERS[a]) {
-      console.error(`unknown adapter: ${a}. options: ${Object.keys(ADAPTERS).join(",")}`);
-      process.exit(2);
-    }
-  }
-  console.log(
-    `loaded ${sessions.length} sessions, ${questions.length} queries, adapters: ${adapterNames.join(",")}, k=${k}`,
-  );
-
-  const outDir = resolve(opts.out);
-  mkdirSync(outDir, { recursive: true });
-  const ndjsonPath = `${outDir}/scores.ndjson`;
-  if (existsSync(ndjsonPath)) writeFileSync(ndjsonPath, "");
-
-  const rows: ScoreRow[] = [];
-  for (const adapterName of adapterNames) {
-    const adapter = ADAPTERS[adapterName];
-    console.log(`\n== ${adapter.name} ==`);
-    const state = await adapter.init(sessions);
-    try {
-      for (const q of questions) {
-        const t0 = performance.now();
-        const ranked = await adapter.query(q.question, state, k);
-        const latencyMs = performance.now() - t0;
-        const row = scoreQuestion(q, ranked, k, adapter.name, latencyMs);
-        rows.push(row);
-        appendFileSync(ndjsonPath, JSON.stringify(row) + "\n");
-        const mark = row.hit ? "+" : "-";
-        console.log(
-          `  ${mark} ${q.id} [${q.type}] R@${k}=${row.recallAtK.toFixed(2)} (${Math.round(latencyMs)}ms)`,
-        );
-      }
-    } finally {
-      if (adapter.teardown) await adapter.teardown(state);
-    }
-  }
-
-  const agg = aggregate(rows);
-  writeFileSync(`${outDir}/summary.json`, JSON.stringify(agg, null, 2));
-  console.log("\n=== Summary ===");
-  for (const [adapter, stats] of Object.entries(agg.byAdapter)) {
-    console.log(
-      `  ${adapter.padEnd(22)} P@${k}=${stats.p.toFixed(3)} R@${k}=${stats.r.toFixed(3)} hit=${stats.hit}/${stats.n} p50=${Math.round(stats.latencyP50)}ms`,
-    );
-  }
-}
-
-main().catch((err) => {
-  console.error(err);
-  process.exit(1);
-});

eval/runner/load.ts

@@ -1,54 +0,0 @@
-import { readFileSync } from "node:fs";
-import type { Question, Session } from "./types.js";
-
-interface LongMemEvalRaw {
-  question_id: string;
-  question_type: string;
-  question: string;
-  answer?: string;
-  answer_session_ids: string[];
-  haystack_session_ids: string[];
-  haystack_sessions: Array<Array<{ role: string; content: string }>>;
-}
-
-function flattenSession(turns: Array<{ role: string; content: string }>): string {
-  return turns.map((t) => `[${t.role}] ${t.content}`).join("\n\n");
-}
-
-export function loadLongMemEval(path: string, limit?: number): Question[] {
-  const raw = JSON.parse(readFileSync(path, "utf8")) as LongMemEvalRaw[];
-  const slice = typeof limit === "number" ? raw.slice(0, limit) : raw;
-  const questions: Question[] = [];
-  for (const r of slice) {
-    if (r.haystack_session_ids.length !== r.haystack_sessions.length) {
-      throw new Error(
-        `LongMemEval row ${r.question_id}: haystack_session_ids (${r.haystack_session_ids.length}) and haystack_sessions (${r.haystack_sessions.length}) length mismatch`,
-      );
-    }
-    const haystack: Session[] = r.haystack_session_ids.map((id, i) => ({
-      id,
-      content: flattenSession(r.haystack_sessions[i]),
-    }));
-    questions.push({
-      id: r.question_id,
-      type: r.question_type,
-      question: r.question,
-      answer: r.answer,
-      goldSessionIds: r.answer_session_ids,
-      haystack,
-    });
-  }
-  return questions;
-}
-
-export function stratifySample(questions: Question[], perType: number): Question[] {
-  const buckets: Record<string, Question[]> = {};
-  for (const q of questions) {
-    (buckets[q.type] ??= []).push(q);
-  }
-  const out: Question[] = [];
-  for (const type of Object.keys(buckets).sort()) {
-    out.push(...buckets[type].slice(0, perType));
-  }
-  return out;
-}

eval/runner/longmemeval.ts

@@ -1,126 +0,0 @@
-import { existsSync, mkdirSync, writeFileSync, appendFileSync } from "node:fs";
-import { dirname, resolve } from "node:path";
-import { parseArgs } from "node:util";
-import { agentmemoryAdapter } from "./adapters/agentmemory.js";
-import { grepAdapter } from "./adapters/grep.js";
-import { vectorAdapter } from "./adapters/vector.js";
-import { loadLongMemEval, stratifySample } from "./load.js";
-import { aggregate, scoreQuestion } from "./score.js";
-import type { Adapter, ScoreRow } from "./types.js";
-
-const ADAPTERS: Record<string, Adapter> = {
-  grep: grepAdapter as unknown as Adapter,
-  vector: vectorAdapter as unknown as Adapter,
-  agentmemory: agentmemoryAdapter as unknown as Adapter,
-};
-
-interface CliOptions {
-  data: string;
-  adapters: string;
-  k: string;
-  limit?: string;
-  stratify?: string;
-  out: string;
-}
-
-function parse(): CliOptions {
-  const { values } = parseArgs({
-    options: {
-      data: { type: "string", default: process.env.LONGMEMEVAL_PATH ?? "" },
-      adapters: { type: "string", default: "grep,vector,agentmemory" },
-      k: { type: "string", default: "5" },
-      limit: { type: "string" },
-      stratify: { type: "string" },
-      out: { type: "string", default: "eval/reports/longmemeval" },
-    },
-  });
-  return values as unknown as CliOptions;
-}
-
-async function main(): Promise<void> {
-  const opts = parse();
-  if (!opts.data) {
-    console.error("--data <path/to/longmemeval_s.json> required (or LONGMEMEVAL_PATH env)");
-    process.exit(2);
-  }
-  const k = Number(opts.k);
-  if (!Number.isInteger(k) || k <= 0) {
-    console.error(`--k must be a positive integer, got: ${opts.k}`);
-    process.exit(2);
-  }
-  let limit: number | undefined;
-  if (opts.limit !== undefined) {
-    limit = Number(opts.limit);
-    if (!Number.isInteger(limit) || limit <= 0) {
-      console.error(`--limit must be a positive integer, got: ${opts.limit}`);
-      process.exit(2);
-    }
-  }
-  let perType: number | undefined;
-  if (opts.stratify !== undefined) {
-    perType = Number(opts.stratify);
-    if (!Number.isInteger(perType) || perType <= 0) {
-      console.error(`--stratify must be a positive integer, got: ${opts.stratify}`);
-      process.exit(2);
-    }
-  }
-  const adapterNames = opts.adapters.split(",").map((s) => s.trim()).filter(Boolean);
-  for (const a of adapterNames) {
-    if (!ADAPTERS[a]) {
-      console.error(`unknown adapter: ${a}. options: ${Object.keys(ADAPTERS).join(",")}`);
-      process.exit(2);
-    }
-  }
-  let questions = loadLongMemEval(resolve(opts.data), limit);
-  if (perType) questions = stratifySample(questions, perType);
-  console.log(
-    `loaded ${questions.length} questions, adapters: ${adapterNames.join(",")}, k=${k}`,
-  );
-
-  const outDir = resolve(opts.out);
-  mkdirSync(outDir, { recursive: true });
-  const ndjsonPath = `${outDir}/scores.ndjson`;
-  if (existsSync(ndjsonPath)) writeFileSync(ndjsonPath, "");
-  mkdirSync(dirname(ndjsonPath), { recursive: true });
-
-  const rows: ScoreRow[] = [];
-  for (const adapterName of adapterNames) {
-    const adapter = ADAPTERS[adapterName];
-    console.log(`\n== ${adapter.name} ==`);
-    for (const q of questions) {
-      const t0 = performance.now();
-      const state = await adapter.init(q.haystack);
-      try {
-        const ranked = await adapter.query(q.question, state, k);
-        const latencyMs = performance.now() - t0;
-        const row = scoreQuestion(q, ranked, k, adapter.name, latencyMs);
-        rows.push(row);
-        appendFileSync(ndjsonPath, JSON.stringify(row) + "\n");
-        const mark = row.hit ? "+" : "-";
-        console.log(
-          `  ${mark} ${q.id} [${q.type}] R@${k}=${row.recallAtK.toFixed(2)} (${Math.round(latencyMs)}ms)`,
-        );
-      } finally {
-        if (adapter.teardown) await adapter.teardown(state);
-      }
-    }
-  }
-
-  const agg = aggregate(rows);
-  const summaryPath = `${outDir}/summary.json`;
-  writeFileSync(summaryPath, JSON.stringify(agg, null, 2));
-
-  console.log("\n=== Summary ===");
-  for (const [adapter, stats] of Object.entries(agg.byAdapter)) {
-    console.log(
-      `  ${adapter.padEnd(22)} P@${k}=${stats.p.toFixed(3)} R@${k}=${stats.r.toFixed(3)} hit=${stats.hit}/${stats.n} p50=${Math.round(stats.latencyP50)}ms`,
-    );
-  }
-  console.log(`\nwrote ${ndjsonPath}`);
-  console.log(`wrote ${summaryPath}`);
-}
-
-main().catch((err) => {
-  console.error(err);
-  process.exit(1);
-});

eval/runner/score.ts

@@ -1,78 +0,0 @@
-import type { Question, RankedDoc, ScoreRow } from "./types.js";
-
-export function scoreQuestion(
-  q: Question,
-  ranked: RankedDoc[],
-  k: number,
-  adapter: string,
-  latencyMs: number,
-): ScoreRow {
-  const topK = ranked.slice(0, k).map((r) => r.sessionId);
-  const gold = new Set(q.goldSessionIds);
-  const hits = topK.filter((id) => gold.has(id)).length;
-  const precisionAtK = k > 0 ? hits / k : 0;
-  const recallAtK = gold.size === 0 ? 0 : hits / gold.size;
-  const hit = hits > 0;
-  let topGoldRank: number | null = null;
-  for (let i = 0; i < ranked.length; i++) {
-    if (gold.has(ranked[i].sessionId)) {
-      topGoldRank = i + 1;
-      break;
-    }
-  }
-  return {
-    questionId: q.id,
-    questionType: q.type,
-    adapter,
-    k,
-    precisionAtK,
-    recallAtK,
-    hit,
-    topGoldRank,
-    latencyMs,
-  };
-}
-
-export function aggregate(rows: ScoreRow[]): {
-  byAdapter: Record<string, { p: number; r: number; hit: number; n: number; latencyP50: number }>;
-  byType: Record<string, Record<string, { p: number; r: number; hit: number; n: number }>>;
-} {
-  const byAdapter: Record<
-    string,
-    { p: number; r: number; hit: number; n: number; latencyP50: number }
-  > = {};
-  const latencies: Record<string, number[]> = {};
-  for (const r of rows) {
-    const a = (byAdapter[r.adapter] ??= { p: 0, r: 0, hit: 0, n: 0, latencyP50: 0 });
-    a.p += r.precisionAtK;
-    a.r += r.recallAtK;
-    a.hit += r.hit ? 1 : 0;
-    a.n += 1;
-    (latencies[r.adapter] ??= []).push(r.latencyMs);
-  }
-  for (const adapter of Object.keys(byAdapter)) {
-    const a = byAdapter[adapter];
-    a.p = a.p / a.n;
-    a.r = a.r / a.n;
-    const sorted = latencies[adapter].slice().sort((x, y) => x - y);
-    a.latencyP50 = sorted[Math.floor(sorted.length / 2)] ?? 0;
-  }
-  const byType: Record<string, Record<string, { p: number; r: number; hit: number; n: number }>> =
-    {};
-  for (const r of rows) {
-    const t = (byType[r.questionType] ??= {});
-    const a = (t[r.adapter] ??= { p: 0, r: 0, hit: 0, n: 0 });
-    a.p += r.precisionAtK;
-    a.r += r.recallAtK;
-    a.hit += r.hit ? 1 : 0;
-    a.n += 1;
-  }
-  for (const t of Object.keys(byType)) {
-    for (const adapter of Object.keys(byType[t])) {
-      const a = byType[t][adapter];
-      a.p = a.p / a.n;
-      a.r = a.r / a.n;
-    }
-  }
-  return { byAdapter, byType };
-}

eval/runner/types.ts

@@ -1,38 +0,0 @@
-export interface Session {
-  id: string;
-  timestamp?: string;
-  content: string;
-}
-
-export interface Question {
-  id: string;
-  type: string;
-  question: string;
-  answer?: string;
-  goldSessionIds: string[];
-  haystack: Session[];
-}
-
-export interface RankedDoc {
-  sessionId: string;
-  score: number;
-}
-
-export interface Adapter<State = unknown> {
-  name: string;
-  init(sessions: Session[], config?: Record<string, unknown>): Promise<State>;
-  query(q: string, state: State, k: number): Promise<RankedDoc[]>;
-  teardown?(state: State): Promise<void>;
-}
-
-export interface ScoreRow {
-  questionId: string;
-  questionType: string;
-  adapter: string;
-  k: number;
-  precisionAtK: number;
-  recallAtK: number;
-  hit: boolean;
-  topGoldRank: number | null;
-  latencyMs: number;
-}

eval/scripts/sandbox.sh

@@ -1,117 +0,0 @@
-#!/usr/bin/env bash
-# Boot a sandboxed agentmemory + iii-engine on alt ports with a clean data dir,
-# so eval runs aren't polluted by (and don't pollute) your real ~/.agentmemory.
-# Source it: `source eval/scripts/sandbox.sh` then run eval scripts;
-# the sandbox is torn down on EXIT.
-
-set -euo pipefail
-
-SANDBOX_ROOT="${SANDBOX_ROOT:-/tmp/agentmemory-eval-sandbox}"
-SANDBOX_PORT="${SANDBOX_PORT:-3411}"
-SANDBOX_STREAM_PORT="${SANDBOX_STREAM_PORT:-3412}"
-
-REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-
-if ! command -v iii >/dev/null 2>&1; then
-  echo "iii binary not on PATH. Install pinned version:"
-  echo "  curl -fsSL https://github.com/iii-hq/iii/releases/download/iii/v0.11.2/iii-aarch64-apple-darwin.tar.gz | tar -xz -C ~/.local/bin"
-  exit 1
-fi
-
-iii_ver=$(iii --version 2>&1 | head -1)
-if [[ "$iii_ver" != "0.11.2" ]]; then
-  echo "warning: iii version on PATH is $iii_ver; agentmemory pins 0.11.2"
-fi
-
-if [[ ! -f "$REPO_ROOT/dist/index.mjs" ]]; then
-  echo "dist/ missing. Run: npm run build" >&2
-  exit 1
-fi
-
-if [[ -z "${SANDBOX_ROOT:-}" || "$SANDBOX_ROOT" == "/" || "$SANDBOX_ROOT" != /tmp/* ]]; then
-  echo "refusing to wipe SANDBOX_ROOT='$SANDBOX_ROOT' — must be non-empty and under /tmp/" >&2
-  exit 1
-fi
-rm -rf "$SANDBOX_ROOT"
-mkdir -p "$SANDBOX_ROOT/data" "$SANDBOX_ROOT/.agentmemory"
-
-cat > "$SANDBOX_ROOT/iii-config.yaml" <<EOF
-workers:
-  - name: iii-http
-    config:
-      port: $SANDBOX_PORT
-      host: 127.0.0.1
-      default_timeout: 180000
-      cors:
-        allowed_origins: ["http://localhost:$SANDBOX_PORT", "http://127.0.0.1:$SANDBOX_PORT"]
-        allowed_methods: [GET, POST, PUT, DELETE, OPTIONS]
-  - name: iii-state
-    config:
-      adapter:
-        name: kv
-        config:
-          store_method: file_based
-          file_path: $SANDBOX_ROOT/data/state_store.db
-  - name: iii-queue
-    config:
-      adapter:
-        name: builtin
-  - name: iii-pubsub
-    config:
-      adapter:
-        name: local
-  - name: iii-cron
-    config:
-      adapter:
-        name: kv
-  - name: iii-stream
-    config:
-      port: $SANDBOX_STREAM_PORT
-      host: 127.0.0.1
-      adapter:
-        name: kv
-        config:
-          store_method: file_based
-          file_path: $SANDBOX_ROOT/data/stream_store
-  - name: iii-observability
-    config:
-      enabled: true
-      service_name: agentmemory-eval
-      exporter: memory
-      sampling_ratio: 1.0
-      metrics_enabled: true
-      logs_enabled: false
-      logs_console_output: false
-  - name: iii-exec
-    config:
-      exec:
-        - node $REPO_ROOT/dist/index.mjs
-EOF
-
-cd "$SANDBOX_ROOT"
-HOME="$SANDBOX_ROOT" iii --config "$SANDBOX_ROOT/iii-config.yaml" > "$SANDBOX_ROOT/iii.log" 2>&1 &
-SANDBOX_PID=$!
-
-cleanup() {
-  echo "tearing down sandbox (pid $SANDBOX_PID)"
-  kill "$SANDBOX_PID" 2>/dev/null || true
-  sleep 1
-  kill -9 "$SANDBOX_PID" 2>/dev/null || true
-}
-trap cleanup EXIT
-
-# wait for livez
-for i in $(seq 1 30); do
-  if curl -sS --max-time 1 "http://localhost:$SANDBOX_PORT/agentmemory/livez" 2>/dev/null | grep -q '"status":"ok"'; then
-    export AGENTMEMORY_BASE_URL="http://localhost:$SANDBOX_PORT"
-    echo "sandbox ready: $AGENTMEMORY_BASE_URL"
-    echo "  state: $SANDBOX_ROOT/data/"
-    echo "  logs:  $SANDBOX_ROOT/iii.log"
-    return 0 2>/dev/null || exit 0
-  fi
-  sleep 1
-done
-
-echo "sandbox failed to come up within 30s. last log lines:" >&2
-tail -10 "$SANDBOX_ROOT/iii.log" >&2
-exit 1

integrations/filesystem-watcher/README.md

@@ -1,63 +0,0 @@
-# @agentmemory/fs-watcher
-
-Filesystem connector for agentmemory. Watches one or more directories and emits an observation to the running agentmemory server every time a file changes.
-
-Part of the data-source-connectors effort tracked in issue #62.
-
-## Install
-
-```bash
-npm install -g @agentmemory/fs-watcher
-```
-
-Or run without installing:
-
-```bash
-npx @agentmemory/fs-watcher ~/work/my-repo
-```
-
-## Usage
-
-```bash
-# CLI args win over env.
-agentmemory-fs-watcher ~/work/my-repo ~/notes
-
-# Or set env once in your shell.
-export AGENTMEMORY_FS_WATCH_DIRS=~/work/my-repo,~/notes
-export AGENTMEMORY_URL=http://localhost:3111
-export AGENTMEMORY_SECRET=...   # only if the server requires auth
-agentmemory-fs-watcher
-```
-
-Every file change inside the watched roots becomes a `post_tool_use` observation whose `data.changeKind` is `file_change` or `file_delete`. The first 4 KB of each text file is included as `data.content` so retrieval can match by substring; larger files are truncated with `data.truncated: true`. Binary files are not read (set `AGENTMEMORY_FS_WATCH_ALLOW_BINARY=1` to override).
-
-Session id and project are required by the observe endpoint — set them via env, or the watcher generates a per-process `fs-watcher-<ts>-<rand>` session id and uses the first root's directory name as the project.
-
-Requires Node.js **>=20 LTS**. Recursive `fs.watch` needs Node 19.1.0+ on Linux; Node 20 is the minimum supported LTS line.
-
-## Configuration
-
-| Variable | Default | Meaning |
-|---|---|---|
-| `AGENTMEMORY_FS_WATCH_DIRS` | — | Comma-separated list of directories to watch |
-| `AGENTMEMORY_FS_WATCH_IGNORE` | — | Comma-separated regex patterns to ignore (applied to relative paths) |
-| `AGENTMEMORY_FS_WATCH_ALLOW_BINARY` | `0` | `1` to include binary files in the preview read |
-| `AGENTMEMORY_URL` | `http://localhost:3111` | agentmemory server URL |
-| `AGENTMEMORY_SECRET` | — | Bearer token, required if the server has `AGENTMEMORY_SECRET` set |
-| `AGENTMEMORY_PROJECT` | — | Optional project label attached to each observation |
-| `AGENTMEMORY_SESSION_ID` | — | Optional session id to attribute observations to |
-
-## Defaults
-
-Ignored out of the box: `.git/`, `node_modules/`, `dist/`, `build/`, `.next/`, `.turbo/`, `coverage/`, `.DS_Store`, `*.log`, `*.lock`. Extend with `AGENTMEMORY_FS_WATCH_IGNORE`.
-
-Text extensions read for preview: common source, config, and docs (`.ts/.js/.py/.go/.rs/.md/.yaml/...`). Unknown extensions are recorded as a path-only observation without content.
-
-Writes are debounced 500 ms per path so a stream of saves from your editor becomes a single observation.
-
-## Notes
-
-- Uses Node's built-in `fs.watch` with `{ recursive: true }`. Works natively on macOS, Linux, and Windows 10+. No native deps.
-- If `fs.watch` errors on a specific root (permission, platform quirk), the watcher logs and continues on the others.
-- The process must keep running. Use a process manager (`launchd`, `systemd`, `pm2`) to supervise it.
-- This connector is intentionally one-way: it writes observations and never reads the agentmemory store.

integrations/filesystem-watcher/bin.mjs

@@ -1,28 +0,0 @@
-#!/usr/bin/env node
-import { FilesystemWatcher, configFromEnv } from "./watcher.mjs";
-
-const cliArgs = process.argv.slice(2);
-const envCfg = configFromEnv(process.env);
-
-const roots = cliArgs.length > 0 ? cliArgs : envCfg.roots;
-if (!roots || roots.length === 0) {
-  process.stderr.write(
-    "agentmemory-fs-watcher: no directories to watch.\n" +
-      "Usage: agentmemory-fs-watcher <dir> [<dir>...]\n" +
-      "Or set AGENTMEMORY_FS_WATCH_DIRS=path1,path2\n",
-  );
-  process.exit(2);
-}
-
-const watcher = new FilesystemWatcher({ ...envCfg, roots });
-watcher.start();
-process.stderr.write(
-  `[fs-watcher] emitting to ${envCfg.baseUrl || "http://localhost:3111"}\n`,
-);
-
-const shutdown = () => {
-  watcher.stop();
-  process.exit(0);
-};
-process.on("SIGINT", shutdown);
-process.on("SIGTERM", shutdown);

integrations/filesystem-watcher/package.json

@@ -1,22 +0,0 @@
-{
-  "name": "@agentmemory/fs-watcher",
-  "version": "0.1.0",
-  "description": "Filesystem connector for agentmemory — emits observations on file changes.",
-  "type": "module",
-  "bin": {
-    "agentmemory-fs-watcher": "./bin.mjs"
-  },
-  "main": "./watcher.mjs",
-  "exports": {
-    ".": "./watcher.mjs"
-  },
-  "files": ["watcher.mjs", "bin.mjs", "README.md"],
-  "engines": { "node": ">=20" },
-  "license": "Apache-2.0",
-  "homepage": "https://github.com/rohitg00/agentmemory/tree/main/integrations/filesystem-watcher",
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/rohitg00/agentmemory.git",
-    "directory": "integrations/filesystem-watcher"
-  }
-}

integrations/filesystem-watcher/watcher.mjs

@@ -1,317 +0,0 @@
-import { watch, promises as fsp, statSync } from "node:fs";
-import { resolve, relative, join, extname, sep, basename } from "node:path";
-import { randomBytes } from "node:crypto";
-
-const TEXT_EXTENSIONS = new Set([
-  ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs",
-  ".py", ".rb", ".go", ".rs", ".java", ".kt", ".swift",
-  ".c", ".cc", ".cpp", ".h", ".hpp",
-  ".md", ".mdx", ".txt", ".rst",
-  ".json", ".yaml", ".yml", ".toml", ".ini", ".env",
-  ".html", ".css", ".scss", ".vue", ".svelte",
-  ".sh", ".bash", ".zsh", ".fish",
-  ".sql", ".graphql", ".proto",
-]);
-
-const DEFAULT_IGNORE = [
-  /(?:^|\/)\.git(?:\/|$)/,
-  /(?:^|\/)node_modules(?:\/|$)/,
-  /(?:^|\/)dist(?:\/|$)/,
-  /(?:^|\/)build(?:\/|$)/,
-  /(?:^|\/)\.next(?:\/|$)/,
-  /(?:^|\/)\.turbo(?:\/|$)/,
-  /(?:^|\/)coverage(?:\/|$)/,
-  /(?:^|\/)\.DS_Store$/,
-  /\.log$/,
-  /\.lock$/,
-];
-
-const MAX_PREVIEW_BYTES = 4096;
-const DEBOUNCE_MS = 500;
-const REDACTED = "[REDACTED]";
-const PEM_BEGIN_RE = /-----BEGIN [A-Z ]*PRIVATE KEY-----/;
-const PEM_END_RE = /-----END [A-Z ]*PRIVATE KEY-----/;
-const JWT_RE = /\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\b/g;
-const JWT_MIN_LEN = 100;
-
-function isDotEnvPath(path) {
-  const name = basename(path).toLowerCase();
-  return name === ".env" || name.startsWith(".env.");
-}
-
-function isSensitiveKey(key) {
-  const normalized = key.replace(/[^a-z0-9]/gi, "").toLowerCase();
-  return [
-    "apikey",
-    "accesstoken",
-    "accesskey",
-    "authorization",
-    "bearer",
-    "clientsecret",
-    "password",
-    "passwd",
-    "privatekey",
-    "pwd",
-    "secret",
-    "token",
-  ].some((needle) => normalized.includes(needle));
-}
-
-function redactJwtTokens(line) {
-  return line.replace(JWT_RE, (match) => (match.length >= JWT_MIN_LEN ? REDACTED : match));
-}
-
-function redactSensitiveLine(line) {
-  if (PEM_BEGIN_RE.test(line) || PEM_END_RE.test(line)) {
-    return line;
-  }
-  const assignment = line.match(
-    /^(\s*(?:export\s+)?["']?([A-Za-z_][A-Za-z0-9_.-]*)["']?\s*([=:])\s*)(.*)$/,
-  );
-  if (assignment && isSensitiveKey(assignment[2])) {
-    const bearer = assignment[3] === ":" ? assignment[4].match(/^(Bearer\s+).+/i) : null;
-    return `${assignment[1]}${bearer ? bearer[1] : ""}${REDACTED}`;
-  }
-  const bearerRedacted = line.replace(
-    /\b(Bearer\s+)[A-Za-z0-9._~+/=-]{8,}\b/gi,
-    `$1${REDACTED}`,
-  );
-  return redactJwtTokens(bearerRedacted);
-}
-
-function redactPemBlocks(preview) {
-  const lines = preview.split("\n");
-  const out = [];
-  let inBlock = false;
-  for (const line of lines) {
-    if (!inBlock) {
-      const beginMatch = line.match(PEM_BEGIN_RE);
-      if (!beginMatch) {
-        out.push(line);
-        continue;
-      }
-      const beginIdx = beginMatch.index;
-      const endMatch = line.match(PEM_END_RE);
-      if (endMatch && endMatch.index > beginIdx) {
-        const before = line.slice(0, beginIdx);
-        const after = line.slice(endMatch.index + endMatch[0].length);
-        out.push(`${before}${beginMatch[0]}${REDACTED}${endMatch[0]}${after}`);
-      } else {
-        out.push(`${line.slice(0, beginIdx)}${beginMatch[0]}`);
-        out.push(REDACTED);
-        inBlock = true;
-      }
-    } else {
-      const endMatch = line.match(PEM_END_RE);
-      if (endMatch) {
-        out.push(`${endMatch[0]}${line.slice(endMatch.index + endMatch[0].length)}`);
-        inBlock = false;
-      }
-    }
-  }
-  return out.join("\n");
-}
-
-function redactSensitivePreview(preview) {
-  return redactPemBlocks(preview).split("\n").map(redactSensitiveLine).join("\n");
-}
-
-export class FilesystemWatcher {
-  constructor(config = {}) {
-    this.roots = (config.roots || []).map((r) => resolve(r));
-    this.baseUrl = (config.baseUrl || "http://localhost:3111").replace(/\/+$/, "");
-    this.secret = config.secret;
-    this.project =
-      config.project ||
-      (this.roots[0] ? basename(this.roots[0]) : "filesystem-watcher");
-    this.sessionId =
-      config.sessionId ||
-      `fs-watcher-${Date.now().toString(36)}-${randomBytes(3).toString("hex")}`;
-    this.ignore = [...DEFAULT_IGNORE, ...(config.ignorePatterns || [])];
-    this.allowBinary = Boolean(config.allowBinary);
-    this.logger = config.logger || console;
-    this.watchers = [];
-    this.pendingByPath = new Map();
-  }
-
-  isIgnored(path) {
-    return this.ignore.some((re) => re.test(path));
-  }
-
-  isTextFile(path) {
-    if (this.allowBinary) return true;
-    const ext = extname(path).toLowerCase();
-    return TEXT_EXTENSIONS.has(ext) || isDotEnvPath(path);
-  }
-
-  async readPreview(path) {
-    try {
-      const fh = await fsp.open(path, "r");
-      try {
-        const buf = Buffer.alloc(MAX_PREVIEW_BYTES);
-        const { bytesRead } = await fh.read(buf, 0, MAX_PREVIEW_BYTES, 0);
-        return buf.slice(0, bytesRead).toString("utf-8");
-      } finally {
-        await fh.close();
-      }
-    } catch {
-      return null;
-    }
-  }
-
-  async emit(event) {
-    const headers = { "content-type": "application/json" };
-    if (this.secret) headers.authorization = `Bearer ${this.secret}`;
-    try {
-      const res = await fetch(`${this.baseUrl}/agentmemory/observe`, {
-        method: "POST",
-        headers,
-        body: JSON.stringify(event),
-        signal: AbortSignal.timeout(5000),
-      });
-      if (!res.ok) {
-        this.logger.warn?.(
-          `[fs-watcher] observe ${res.status}: ${await res.text().catch(() => "")}`,
-        );
-      }
-    } catch (err) {
-      this.logger.warn?.(`[fs-watcher] observe failed: ${err?.message || err}`);
-    }
-  }
-
-  schedule(rootDir, relPath) {
-    const key = join(rootDir, relPath);
-    const existing = this.pendingByPath.get(key);
-    if (existing) clearTimeout(existing.timer);
-    const timer = setTimeout(() => {
-      this.pendingByPath.delete(key);
-      this.flush(rootDir, relPath).catch((err) =>
-        this.logger.warn?.(`[fs-watcher] flush failed: ${err?.message || err}`),
-      );
-    }, DEBOUNCE_MS);
-    this.pendingByPath.set(key, { timer });
-  }
-
-  async flush(rootDir, relPath) {
-    const absPath = join(rootDir, relPath);
-    if (this.isIgnored(relPath)) return;
-    let exists = true;
-    let size = 0;
-    try {
-      const st = statSync(absPath);
-      if (!st.isFile()) return;
-      size = st.size;
-    } catch {
-      exists = false;
-    }
-    const changeKind = exists ? "file_change" : "file_delete";
-    let preview = null;
-    if (exists && this.isTextFile(absPath)) {
-      preview = await this.readPreview(absPath);
-      if (preview !== null) preview = redactSensitivePreview(preview);
-    }
-    const truncated = exists && size > MAX_PREVIEW_BYTES;
-    const payload = {
-      hookType: "post_tool_use",
-      sessionId: this.sessionId,
-      project: this.project,
-      cwd: rootDir,
-      timestamp: new Date().toISOString(),
-      data: {
-        source: "filesystem-watcher",
-        changeKind,
-        files: [relPath],
-        content: this.formatContent(relPath, changeKind, preview, {
-          size,
-          truncated,
-        }),
-        rootDir,
-        absPath,
-        size,
-        truncated,
-      },
-    };
-    await this.emit(payload);
-  }
-
-  formatContent(relPath, changeKind, preview, { size, truncated }) {
-    if (changeKind === "file_delete") return `deleted: ${relPath}`;
-    const head = `${relPath} (${size} bytes${truncated ? ", truncated" : ""})`;
-    if (preview === null) return head;
-    return `${head}\n\n${preview}`;
-  }
-
-  start() {
-    if (this.roots.length === 0) {
-      throw new Error("filesystem-watcher: at least one root directory is required");
-    }
-    const failures = [];
-    for (const root of this.roots) {
-      try {
-        const handle = watch(
-          root,
-          { recursive: true, persistent: true },
-          (_eventType, filename) => {
-            if (!filename) return;
-            const rel = filename.split(sep).join("/");
-            if (this.isIgnored(rel)) return;
-            this.schedule(root, rel);
-          },
-        );
-        handle.on("error", (err) => {
-          this.logger.warn?.(`[fs-watcher] watch error on ${root}: ${err?.message || err}`);
-        });
-        this.watchers.push(handle);
-        this.logger.info?.(`[fs-watcher] watching ${root}`);
-      } catch (err) {
-        const msg = err?.message || String(err);
-        failures.push(`${root}: ${msg}`);
-        this.logger.error?.(`[fs-watcher] failed to watch ${root}: ${msg}`);
-      }
-    }
-    if (this.watchers.length === 0) {
-      throw new Error(
-        `filesystem-watcher: could not watch any of the configured roots. ` +
-          `If you are on Node 18 + Linux, recursive fs.watch requires Node >=19.1.0; upgrade to Node 20 LTS or newer. ` +
-          `Failures: ${failures.join("; ")}`,
-      );
-    }
-  }
-
-  stop() {
-    for (const w of this.watchers) {
-      try {
-        w.close();
-      } catch {}
-    }
-    this.watchers = [];
-    for (const { timer } of this.pendingByPath.values()) {
-      clearTimeout(timer);
-    }
-    this.pendingByPath.clear();
-  }
-}
-
-// Small helper used by tests and bin.mjs to parse env.
-export function configFromEnv(env = process.env) {
-  const roots = (env.AGENTMEMORY_FS_WATCH_DIRS || "")
-    .split(",")
-    .map((s) => s.trim())
-    .filter(Boolean);
-  const extraIgnore = (env.AGENTMEMORY_FS_WATCH_IGNORE || "")
-    .split(",")
-    .map((s) => s.trim())
-    .filter(Boolean)
-    .map((s) => new RegExp(s));
-  return {
-    roots,
-    baseUrl: env.AGENTMEMORY_URL,
-    secret: env.AGENTMEMORY_SECRET,
-    project: env.AGENTMEMORY_PROJECT || null,
-    sessionId: env.AGENTMEMORY_SESSION_ID || null,
-    ignorePatterns: extraIgnore,
-    allowBinary: env.AGENTMEMORY_FS_WATCH_ALLOW_BINARY === "1",
-  };
-}
-
-export { relative as _relativeForTests };

integrations/openclaw/README.md

@@ -1,155 +0,0 @@
-<p align="center">
-  <img src="../../assets/banner.png" alt="agentmemory" width="640" />
-</p>
-
-<h1 align="center">
-  <img src="https://github.com/openclaw.png?size=80" alt="OpenClaw" width="28" height="28" align="center" />
-  &nbsp;agentmemory for OpenClaw
-</h1>
-
-<p align="center">
-  <strong>Your OpenClaw agents remember everything. No more re-explaining.</strong><br/>
-  <sub>Persistent cross-session memory via <a href="https://github.com/rohitg00/agentmemory">agentmemory</a> — 95.2% retrieval accuracy on <a href="https://arxiv.org/abs/2410.10813">LongMemEval-S</a>.</sub>
-</p>
-
-<p align="center">
-  <img src="https://img.shields.io/badge/MCP-43_tools-1f6feb?style=flat-square" alt="43 MCP tools" />
-  <img src="https://img.shields.io/badge/Plugin-memory_slot-1f6feb?style=flat-square" alt="OpenClaw memory plugin" />
-  <img src="https://img.shields.io/badge/R@5-95.2%25-00875f?style=flat-square" alt="95.2% R@5" />
-  <img src="https://img.shields.io/badge/Self--hosted-yes-00875f?style=flat-square" alt="Self-hosted" />
-  <img src="https://img.shields.io/badge/License-Apache_2.0-blue?style=flat-square" alt="Apache 2.0" />
-</p>
-
----
-
-## Install it in 30 seconds
-
-**Paste this prompt into OpenClaw** and it does the whole setup for you:
-
-```text
-Install agentmemory for OpenClaw. Run `npx @agentmemory/agentmemory` in a separate terminal to start the memory server on localhost:3111.
-
-For zero-code setup, add this MCP server so OpenClaw gets all 43 memory tools:
-
-{
-  "mcpServers": {
-    "agentmemory": {
-      "command": "npx",
-      "args": ["-y", "@agentmemory/mcp"]
-    }
-  }
-}
-
-For deeper memory integration, copy `integrations/openclaw` from the agentmemory repo to `~/.openclaw/extensions/agentmemory`, then enable it in `~/.openclaw/openclaw.json`:
-
-{
-  "plugins": {
-    "slots": {
-      "memory": "agentmemory"
-    },
-    "entries": {
-      "agentmemory": {
-        "enabled": true,
-        "config": {
-          "base_url": "http://localhost:3111",
-          "token_budget": 2000,
-          "min_confidence": 0.5,
-          "fallback_on_error": true,
-          "timeout_ms": 5000
-        }
-      }
-    }
-  }
-}
-
-Restart OpenClaw. Verify with `curl http://localhost:3111/agentmemory/health`. Open http://localhost:3113 for the real-time viewer.
-```
-
-That's it. OpenClaw handles the rest.
-
-## Option 1: MCP server (zero code)
-
-Start the agentmemory server in a separate terminal:
-
-```bash
-npx @agentmemory/agentmemory
-```
-
-Then add to your OpenClaw MCP config:
-
-```json
-{
-  "mcpServers": {
-    "agentmemory": {
-      "command": "npx",
-      "args": ["-y", "@agentmemory/mcp"]
-    }
-  }
-}
-```
-
-OpenClaw now has access to all 43 MCP tools including `memory_recall`, `memory_save`, `memory_smart_search`, `memory_timeline`, `memory_profile`, and more.
-
-## Option 2: OpenClaw memory plugin (deeper integration)
-
-Copy this folder into OpenClaw's extension directory:
-
-```bash
-mkdir -p ~/.openclaw/extensions
-cp -r integrations/openclaw ~/.openclaw/extensions/agentmemory
-```
-
-Then enable it in `~/.openclaw/openclaw.json`:
-
-```json
-{
-  "plugins": {
-    "slots": {
-      "memory": "agentmemory"
-    },
-    "entries": {
-      "agentmemory": {
-        "enabled": true,
-        "config": {
-          "base_url": "http://localhost:3111",
-          "token_budget": 2000,
-          "min_confidence": 0.5,
-          "fallback_on_error": true,
-          "timeout_ms": 5000
-        }
-      }
-    }
-  }
-}
-```
-
-What the plugin does:
-
-- claims the `plugins.slots.memory = "agentmemory"` slot via `api.registerMemoryCapability({ promptBuilder })` so OpenClaw recognises it as the active memory plugin
-- recalls relevant long-term memory before the agent starts (via the `before_agent_start` hook)
-- captures completed conversation turns after the agent finishes (via the `agent_end` hook)
-- shares the same backend with Claude Code, Codex CLI, Gemini CLI, Hermes, pi, and other agents
-
-### Memory runtime (current scope)
-
-The plugin currently registers a `promptBuilder` only — not a full `MemoryPluginRuntime` adapter. OpenClaw's `MemoryRuntimeBackendConfig` type today is `{ backend: "builtin" }` or `{ backend: "qmd" }`; both are openclaw-internal backends that don't fit agentmemory's external REST shape. The hook-driven recall + capture flow above is the working integration path. If you need OpenClaw's in-process memory-runtime APIs (e.g. `getMemorySearchManager`) backed by agentmemory, file an upstream request against `openclaw` for an `"external"` backend type and we'll wire `runtime` here once the contract supports it.
-
-## Troubleshooting
-
-**Plugin validates but does not load** — make sure the folder contains `package.json`, `openclaw.plugin.json`, and `plugin.mjs`, and that `plugins.slots.memory` is set to `agentmemory`.
-
-**`plugins.slots.memory = "agentmemory"` reports `unavailable`** — upgrade to v0.9.11+. Older versions of this plugin registered hooks but never called `api.registerMemoryCapability(...)`, so the memory-slot machinery did not consider the slot claimed. The current plugin registers a memory capability (prompt builder) at startup, which is the documented OpenClaw API for occupying the slot.
-
-**Connection refused on port 3111** — the agentmemory server is not running. Start it with `npx @agentmemory/agentmemory`.
-
-**No memories returned** — open `http://localhost:3113` and verify observations are being captured.
-
-## See also
-
-- [agentmemory main README](../../README.md)
-- [Hermes integration](../hermes/README.md)
-- [pi integration](../pi/README.md)
-
-## License
-
-Apache-2.0 (same as agentmemory)

integrations/openclaw/openclaw.plugin.json

@@ -1,27 +0,0 @@
-{
-  "id": "agentmemory",
-  "kind": "memory",
-  "name": "agentmemory",
-  "description": "Persistent cross-session memory for OpenClaw via agentmemory.",
-  "version": "0.9.4",
-  "configSchema": {
-    "type": "object",
-    "additionalProperties": false,
-    "properties": {
-      "enabled": { "type": "boolean" },
-      "base_url": { "type": "string" },
-      "token_budget": { "type": "number" },
-      "min_confidence": { "type": "number" },
-      "fallback_on_error": { "type": "boolean" },
-      "timeout_ms": { "type": "number" }
-    }
-  },
-  "uiHints": {
-    "enabled": { "label": "Enabled" },
-    "base_url": { "label": "Base URL", "help": "agentmemory REST server base URL" },
-    "token_budget": { "label": "Token Budget", "help": "Approximate context budget to inject before the agent starts" },
-    "min_confidence": { "label": "Min Confidence" },
-    "fallback_on_error": { "label": "Fallback On Error" },
-    "timeout_ms": { "label": "Timeout (ms)" }
-  }
-}

integrations/openclaw/package.json

@@ -1,10 +0,0 @@
-{
-  "name": "agentmemory",
-  "version": "0.9.4",
-  "type": "module",
-  "openclaw": {
-    "extensions": [
-      "./plugin.mjs"
-    ]
-  }
-}

integrations/openclaw/plugin.mjs

@@ -1,217 +0,0 @@
-/**
- * agentmemory plugin for OpenClaw
- *
- * Deeper integration than raw MCP:
- * - claims the plugins.slots.memory slot via api.registerMemoryCapability({ promptBuilder })
- * - recalls relevant memories before the agent starts (before_agent_start hook)
- * - captures completed conversation turns after the agent finishes (agent_end hook)
- *
- * Requires the agentmemory server on localhost:3111.
- * Start it with: npx @agentmemory/agentmemory
- */
-
-const DEFAULT_BASE_URL = "http://localhost:3111";
-const DEFAULT_TIMEOUT_MS = 5000;
-const LOOPBACK_HOSTS = new Set(["localhost", "127.0.0.1", "::1"]);
-
-const configSchema = {
-  type: "object",
-  additionalProperties: false,
-  properties: {
-    enabled: { type: "boolean" },
-    base_url: { type: "string" },
-    token_budget: { type: "number" },
-    min_confidence: { type: "number" },
-    fallback_on_error: { type: "boolean" },
-    timeout_ms: { type: "number" },
-  },
-};
-
-function extractText(content) {
-  if (typeof content === "string") return content;
-  if (!Array.isArray(content)) return "";
-  return content
-    .flatMap((block) => {
-      if (!block || typeof block !== "object") return [];
-      if (block.type === "text" && typeof block.text === "string") return [block.text];
-      return [];
-    })
-    .join("\n")
-    .trim();
-}
-
-function lastAssistantText(messages) {
-  for (const message of [...messages].reverse()) {
-    if (!message || typeof message !== "object") continue;
-    if (message.role !== "assistant") continue;
-    const text = extractText(message.content);
-    if (text) return text;
-  }
-  return "";
-}
-
-function latestUserText(messages) {
-  for (const message of [...messages].reverse()) {
-    if (!message || typeof message !== "object") continue;
-    if (message.role !== "user") continue;
-    const text = extractText(message.content);
-    if (text) return text;
-  }
-  return "";
-}
-
-function formatResults(results) {
-  if (!Array.isArray(results) || results.length === 0) return "";
-  return results
-    .slice(0, 5)
-    .map((result, index) => {
-      const obs = result?.observation ?? result ?? {};
-      const title = (obs.title || `Memory ${index + 1}`).trim();
-      const narrative = (obs.narrative || "").trim();
-      const type = (obs.type || "memory").trim();
-      return `- ${title} (${type})${narrative ? `: ${narrative}` : ""}`;
-    })
-    .join("\n");
-}
-
-function normalizedHostname(hostname) {
-  return hostname.replace(/^\[|\]$/g, "").toLowerCase();
-}
-
-function usesPlaintextBearerAuth(baseUrl, secret) {
-  if (!secret) return false;
-  try {
-    const parsed = new URL(baseUrl);
-    return parsed.protocol === "http:" && !LOOPBACK_HOSTS.has(normalizedHostname(parsed.hostname));
-  } catch {
-    return false;
-  }
-}
-
-function plaintextBearerAuthMessage(baseUrl) {
-  return `agentmemory: AGENTMEMORY_SECRET is configured for plaintext HTTP to ${baseUrl}. Bearer tokens and memory payloads can be observed on the network; use HTTPS or an SSH tunnel.`;
-}
-
-export function createPlaintextBearerAuthGuard(warn, env) {
-  let warned = false;
-  return function guardPlaintextBearerAuth(baseUrl, secret) {
-    if (!usesPlaintextBearerAuth(baseUrl, secret)) return;
-    const message = plaintextBearerAuthMessage(baseUrl);
-    if ((env || process.env).AGENTMEMORY_REQUIRE_HTTPS === "1") throw new Error(message);
-    if (!warned) {
-      warned = true;
-      warn(message);
-    }
-  };
-}
-
-function createClient(cfg, api) {
-  const baseUrl = String(cfg.base_url || DEFAULT_BASE_URL).replace(/\/+$/, "");
-  const timeoutMs = Number(cfg.timeout_ms || DEFAULT_TIMEOUT_MS);
-  const fallbackOnError = cfg.fallback_on_error !== false;
-  const secret = process.env.AGENTMEMORY_SECRET;
-  const guardPlaintextBearerAuth = createPlaintextBearerAuthGuard(
-    (message) => api.logger.warn?.(message),
-  );
-  if (process.env.AGENTMEMORY_REQUIRE_HTTPS === "1") {
-    guardPlaintextBearerAuth(baseUrl, secret);
-  }
-
-  async function postJson(path, payload) {
-    guardPlaintextBearerAuth(baseUrl, secret);
-    const headers = { "Content-Type": "application/json" };
-    if (secret) headers.Authorization = `Bearer ${secret}`;
-    try {
-      const res = await fetch(`${baseUrl}${path}`, {
-        method: "POST",
-        headers,
-        body: JSON.stringify(payload),
-        signal: AbortSignal.timeout(timeoutMs),
-      });
-      if (!res.ok) {
-        if (fallbackOnError) return null;
-        const body = await res.text().catch(() => "");
-        throw new Error(`agentmemory ${path} failed: ${res.status} ${body}`);
-      }
-      return await res.json();
-    } catch (error) {
-      if (!fallbackOnError) throw error;
-      api.logger.warn?.(`agentmemory: ${String(error)}`);
-      return null;
-    }
-  }
-
-  return { postJson, baseUrl };
-}
-
-const plugin = {
-  id: "agentmemory",
-  name: "agentmemory",
-  description: "Shared cross-session memory via the local agentmemory server.",
-  configSchema,
-  register(api) {
-    const cfg = {
-      enabled: api.pluginConfig?.enabled !== false,
-      base_url: api.pluginConfig?.base_url || DEFAULT_BASE_URL,
-      token_budget: api.pluginConfig?.token_budget || 2000,
-      min_confidence: api.pluginConfig?.min_confidence || 0.5,
-      fallback_on_error: api.pluginConfig?.fallback_on_error !== false,
-      timeout_ms: api.pluginConfig?.timeout_ms || DEFAULT_TIMEOUT_MS,
-    };
-    const client = createClient(cfg, api);
-
-    if (typeof api.registerMemoryCapability === "function") {
-      api.registerMemoryCapability({
-        // OpenClaw passes { availableTools: Set<string>, citationsMode? }. We
-        // don't currently branch on tool availability, but accept the params
-        // object so the signature matches MemoryPromptSectionBuilder exactly.
-        promptBuilder: (_params) => [
-          "Long-term memory provider: agentmemory (external REST service on " +
-            client.baseUrl +
-            ").",
-          "agentmemory recalls relevant prior observations before each turn via the before_agent_start hook and captures completed turns via agent_end.",
-          "Treat recalled context as background, not authoritative — prefer current workspace state and explicit user instructions when they conflict.",
-        ],
-      });
-    }
-
-    api.on("before_agent_start", async (event) => {
-      if (!cfg.enabled) return;
-      const prompt = typeof event?.prompt === "string" ? event.prompt.trim() : "";
-      if (!prompt) return;
-      const result = await client.postJson("/agentmemory/smart-search", {
-        query: prompt,
-        limit: 5,
-      });
-      const block = formatResults(result?.results || []);
-      if (!block) return;
-      return {
-        prependContext: `Relevant long-term memory from agentmemory:\n${block}`,
-      };
-    });
-
-    api.on("agent_end", async (event) => {
-      if (!cfg.enabled || !event?.success || !Array.isArray(event.messages)) return;
-      const userText = latestUserText(event.messages);
-      const assistantText = lastAssistantText(event.messages);
-      if (!userText || !assistantText) return;
-      const sessionId =
-        event.sessionId ||
-        event.sessionKey ||
-        event.runId ||
-        `openclaw-${Date.now()}`;
-      await client.postJson("/agentmemory/observe", {
-        hookType: "post_tool_use",
-        sessionId,
-        timestamp: new Date().toISOString(),
-        data: {
-          tool_name: "conversation",
-          tool_input: userText.slice(0, 1000),
-          tool_output: assistantText.slice(0, 4000),
-        },
-      });
-    });
-  },
-};
-
-export default plugin;

integrations/openclaw/plugin.yaml

@@ -1,27 +0,0 @@
-name: agentmemory
-version: 0.8.1
-description: "Persistent cross-session memory for OpenClaw via agentmemory. 95.2% retrieval accuracy on LongMemEval-S."
-author: "Rohit Ghumare"
-homepage: "https://github.com/rohitg00/agentmemory"
-license: Apache-2.0
-
-category: memory
-tags:
-  - memory
-  - persistence
-  - mcp
-  - context
-
-hooks:
-  - on_session_start
-  - on_pre_llm_call
-  - on_post_tool_use
-  - on_session_end
-
-config:
-  enabled: true
-  base_url: http://localhost:3111
-  token_budget: 2000
-  min_confidence: 0.5
-  fallback_on_error: true
-  timeout_ms: 5000

integrations/pi/README.md

@@ -1,77 +0,0 @@
-<p align="center">
-  <img src="../../assets/banner.png" alt="agentmemory" width="640" />
-</p>
-
-<h1 align="center">
-  &nbsp;agentmemory for pi
-</h1>
-
-<p align="center">
-  <strong>Your pi sessions remember everything. No more re-explaining.</strong><br/>
-  <sub>Persistent cross-session memory via <a href="https://github.com/rohitg00/agentmemory">agentmemory</a> — shared with Claude Code, Codex CLI, Gemini CLI, Hermes, OpenClaw, and more.</sub>
-</p>
-
----
-
-## Quick setup
-
-Start the agentmemory server in a separate terminal:
-
-```bash
-npx @agentmemory/agentmemory
-```
-
-Copy this folder into pi's global extensions directory:
-
-```bash
-mkdir -p ~/.pi/agent/extensions/agentmemory
-cp integrations/pi/index.ts ~/.pi/agent/extensions/agentmemory/index.ts
-```
-
-Then enable it in `~/.pi/agent/settings.json` if you prefer explicit loading:
-
-```json
-{
-  "extensions": ["~/.pi/agent/extensions/agentmemory"]
-}
-```
-
-If you place it under `~/.pi/agent/extensions/agentmemory/`, pi will also auto-discover it and `/reload` can hot-reload it.
-
-## What it adds
-
-- `memory_health` — confirm the shared memory server is reachable
-- `memory_search` — search prior decisions, bugs, workflows, and preferences
-- `memory_save` — write durable facts back to long-term memory
-- `/agentmemory-status` — check health from inside pi
-- `before_agent_start` recall — injects relevant memories into the prompt
-- `agent_end` capture — saves completed conversation turns back to agentmemory
-
-## Environment variables
-
-| Variable | Default | Description |
-|---|---|---|
-| `AGENTMEMORY_URL` | `http://localhost:3111` | agentmemory server URL |
-| `AGENTMEMORY_SECRET` | (none) | Bearer token for protected instances |
-| `AGENTMEMORY_REQUIRE_HTTPS` | (off) | When set to `1`, refuse to send a bearer token over plaintext HTTP to a non-loopback host. Sends the token only when `AGENTMEMORY_URL` is `https://...` or points at `localhost`/`127.0.0.1`/`::1`. With this off, the plugin warns once but still sends. |
-
-## Smoke test
-
-Run pi and ask it to use the `memory_health` tool, or call the command directly:
-
-```text
-/agentmemory-status
-```
-
-You should see `agentmemory healthy` and a footer status like `🧠 agentmemory`.
-
-## Notes
-
-- This extension uses pi's extension API, not MCP, so it can hook directly into the agent lifecycle.
-- One local agentmemory server can be shared across pi, pi2, Hermes, OpenClaw, Claude Code, Codex CLI, and Gemini CLI.
-
-## See also
-
-- [agentmemory main README](../../README.md)
-- [Hermes integration](../hermes/README.md)
-- [OpenClaw integration](../openclaw/README.md)

integrations/pi/index.ts

@@ -1,275 +0,0 @@
-import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
-import { Type } from "typebox";
-import path from "node:path";
-import crypto from "node:crypto";
-import { createPlaintextBearerAuthGuard } from "./security.js";
-
-type TextBlock = { type?: string; text?: string };
-type AssistantMessage = { role?: string; content?: unknown };
-type SmartSearchResult = {
-  title?: string;
-  narrative?: string;
-  type?: string;
-  combinedScore?: number;
-  score?: number;
-  observation?: {
-    title?: string;
-    narrative?: string;
-    type?: string;
-  };
-};
-
-type HealthResponse = {
-  status?: string;
-  service?: string;
-  version?: string;
-  health?: {
-    status?: string;
-    notes?: string[];
-  };
-};
-
-const DEFAULT_URL = process.env.AGENTMEMORY_URL || "http://localhost:3111";
-const guardPlaintextBearerAuth = createPlaintextBearerAuthGuard();
-const TOOL_GUIDANCE = [
-  "agentmemory is available for cross-session memory.",
-  "Use memory_search to recall prior decisions, preferences, bugs, and workflows.",
-  "Use memory_save when you discover durable facts worth remembering beyond this session.",
-].join(" ");
-
-function normalizeBaseUrl(url: string): string {
-  return url.replace(/\/+$/, "");
-}
-
-function getText(content: unknown): string {
-  if (typeof content === "string") return content;
-  if (!Array.isArray(content)) return "";
-  return content
-    .flatMap((part) => {
-      if (!part || typeof part !== "object") return [] as string[];
-      const block = part as TextBlock;
-      if (block.type === "text" && typeof block.text === "string") return [block.text];
-      return [] as string[];
-    })
-    .join("\n")
-    .trim();
-}
-
-function getLastAssistantText(messages: unknown[]): string {
-  for (const msg of [...messages].reverse()) {
-    if (!msg || typeof msg !== "object") continue;
-    const assistant = msg as AssistantMessage;
-    if (assistant.role !== "assistant") continue;
-    const text = getText(assistant.content);
-    if (text) return text;
-  }
-  return "";
-}
-
-function formatSearchResults(results: SmartSearchResult[]): string {
-  if (!results.length) return "No relevant memories found.";
-  return results
-    .slice(0, 5)
-    .map((result, index) => {
-      const obs = result.observation ?? result;
-      const title = obs.title?.trim() || `Memory ${index + 1}`;
-      const narrative = obs.narrative?.trim() || "";
-      const type = obs.type?.trim() || "memory";
-      const score = result.combinedScore ?? result.score;
-      const scoreText = typeof score === "number" ? ` [score=${score.toFixed(3)}]` : "";
-      return `- ${title} (${type})${scoreText}${narrative ? `: ${narrative}` : ""}`;
-    })
-    .join("\n");
-}
-
-async function callAgentMemory<T>(
-  pathname: string,
-  options?: {
-    method?: "GET" | "POST";
-    body?: unknown;
-    baseUrl?: string;
-  },
-): Promise<T | null> {
-  const baseUrl = normalizeBaseUrl(options?.baseUrl || process.env.AGENTMEMORY_URL || DEFAULT_URL);
-  const method = options?.method || "POST";
-  const url = `${baseUrl}/agentmemory/${pathname.replace(/^\/+/, "")}`;
-  const headers: Record<string, string> = {};
-  const secret = process.env.AGENTMEMORY_SECRET;
-  guardPlaintextBearerAuth(baseUrl, secret);
-  if (options?.body !== undefined) headers["Content-Type"] = "application/json";
-  if (secret) headers.Authorization = `Bearer ${secret}`;
-
-  try {
-    const response = await fetch(url, {
-      method,
-      headers,
-      body: options?.body !== undefined ? JSON.stringify(options.body) : undefined,
-    });
-    if (!response.ok) return null;
-    return (await response.json()) as T;
-  } catch {
-    return null;
-  }
-}
-
-export default function agentmemoryExtension(pi: ExtensionAPI) {
-  if (process.env.AGENTMEMORY_REQUIRE_HTTPS === "1") {
-    guardPlaintextBearerAuth(
-      normalizeBaseUrl(process.env.AGENTMEMORY_URL || DEFAULT_URL),
-      process.env.AGENTMEMORY_SECRET,
-    );
-  }
-  let sessionId = `ephemeral-${crypto.randomUUID().slice(0, 8)}`;
-  let currentProject = process.cwd();
-  let lastPrompt = "";
-  let lastHealthOk = false;
-
-  async function getHealth() {
-    return await callAgentMemory<HealthResponse>("health", { method: "GET" });
-  }
-
-  async function refreshStatus(ctx: { ui: { setStatus: (key: string, text: string) => void } }) {
-    const health = await getHealth();
-    lastHealthOk = !!health && (health.status === "healthy" || health.health?.status === "healthy");
-    ctx.ui.setStatus("agentmemory", lastHealthOk ? "🧠 agentmemory" : "🧠 agentmemory off");
-  }
-
-  pi.registerCommand("agentmemory-status", {
-    description: "Check local agentmemory server health",
-    handler: async (_args, ctx) => {
-      const health = await getHealth();
-      if (!health) {
-        ctx.ui.notify("agentmemory is unreachable at http://localhost:3111", "warning");
-        return;
-      }
-      ctx.ui.notify(
-        `agentmemory ${health.status || health.health?.status || "unknown"}${health.version ? ` v${health.version}` : ""}`,
-        "info",
-      );
-    },
-  });
-
-  pi.registerTool({
-    name: "memory_health",
-    label: "Memory Health",
-    description: "Check whether the local agentmemory server is reachable and healthy",
-    parameters: Type.Object({}),
-    async execute() {
-      const health = await getHealth();
-      if (!health) {
-        return {
-          content: [{ type: "text", text: "agentmemory is unreachable at http://localhost:3111" }],
-          details: { ok: false },
-        };
-      }
-      return {
-        content: [
-          {
-            type: "text",
-            text: `agentmemory status: ${health.status || health.health?.status || "unknown"}${health.version ? ` (v${health.version})` : ""}`,
-          },
-        ],
-        details: health,
-      };
-    },
-  });
-
-  pi.registerTool({
-    name: "memory_search",
-    label: "Memory Search",
-    description: "Search agentmemory for cross-session project memory, prior decisions, bugs, and user preferences",
-    parameters: Type.Object({
-      query: Type.String({ description: "What to search for in memory" }),
-      limit: Type.Optional(Type.Integer({ minimum: 1, maximum: 10, default: 5, description: "Maximum results" })),
-    }),
-    async execute(_toolCallId, params) {
-      const result = await callAgentMemory<{ results?: SmartSearchResult[] }>("smart-search", {
-        body: { query: params.query, limit: params.limit ?? 5 },
-      });
-      const results = result?.results || [];
-      return {
-        content: [{ type: "text", text: formatSearchResults(results) }],
-        details: { query: params.query, results },
-      };
-    },
-  });
-
-  pi.registerTool({
-    name: "memory_save",
-    label: "Memory Save",
-    description: "Save a durable fact, convention, workflow, preference, or bug fix into agentmemory",
-    parameters: Type.Object({
-      content: Type.String({ description: "What should be remembered" }),
-      type: Type.Optional(
-        Type.String({
-          description: "Memory type",
-          default: "fact",
-        }),
-      ),
-    }),
-    async execute(_toolCallId, params) {
-      const result = await callAgentMemory<Record<string, unknown>>("remember", {
-        body: { content: params.content, type: params.type || "fact" },
-      });
-      if (!result) {
-        return {
-          content: [{ type: "text", text: "Failed to save memory to agentmemory." }],
-          details: { ok: false },
-        };
-      }
-      return {
-        content: [{ type: "text", text: `Saved memory (${params.type || "fact"}): ${params.content}` }],
-        details: result,
-      };
-    },
-  });
-
-  pi.on("session_start", async (_event, ctx) => {
-    const sessionFile = ctx.sessionManager.getSessionFile();
-    sessionId = sessionFile ? path.basename(sessionFile).replace(/\.[^.]+$/, "") : `ephemeral-${crypto.randomUUID().slice(0, 8)}`;
-    currentProject = process.cwd();
-    await refreshStatus(ctx);
-  });
-
-  pi.on("before_agent_start", async (event, ctx) => {
-    currentProject = event.systemPromptOptions.cwd || process.cwd();
-    lastPrompt = event.prompt?.trim() || "";
-    if (!lastPrompt) return;
-
-    const result = await callAgentMemory<{ results?: SmartSearchResult[] }>("smart-search", {
-      body: { query: lastPrompt, limit: 5 },
-    });
-    const results = result?.results || [];
-    const recallBlock = results.length
-      ? [
-          "Relevant long-term memory from agentmemory:",
-          formatSearchResults(results),
-        ].join("\n")
-      : "";
-
-    await refreshStatus(ctx);
-    return {
-      systemPrompt: [event.systemPrompt, TOOL_GUIDANCE, recallBlock].filter(Boolean).join("\n\n"),
-    };
-  });
-
-  pi.on("agent_end", async (event) => {
-    if (!lastHealthOk || !lastPrompt) return;
-    const assistantText = getLastAssistantText(event.messages as unknown[]);
-    if (!assistantText) return;
-    void callAgentMemory("observe", {
-      body: {
-        hookType: "post_tool_use",
-        sessionId,
-        project: currentProject,
-        cwd: currentProject,
-        timestamp: new Date().toISOString(),
-        data: {
-          tool_name: "conversation",
-          tool_input: lastPrompt.slice(0, 500),
-          tool_output: assistantText.slice(0, 4000),
-        },
-      },
-    });
-  });
-}

integrations/pi/package.json

@@ -1,5 +0,0 @@
-{
-  "name": "agentmemory-pi-extension",
-  "private": true,
-  "type": "module"
-}

integrations/pi/security.ts

@@ -1,35 +0,0 @@
-const LOOPBACK_HOSTS = new Set(["localhost", "127.0.0.1", "::1"]);
-
-function normalizedHostname(hostname: string): string {
-  return hostname.replace(/^\[|\]$/g, "").toLowerCase();
-}
-
-export function usesPlaintextBearerAuth(baseUrl: string, secret?: string): boolean {
-  if (!secret) return false;
-  try {
-    const parsed = new URL(baseUrl);
-    return parsed.protocol === "http:" && !LOOPBACK_HOSTS.has(normalizedHostname(parsed.hostname));
-  } catch {
-    return false;
-  }
-}
-
-export function plaintextBearerAuthMessage(baseUrl: string): string {
-  return `agentmemory: AGENTMEMORY_SECRET is configured for plaintext HTTP to ${baseUrl}. Bearer tokens and memory payloads can be observed on the network; use HTTPS or an SSH tunnel.`;
-}
-
-export function createPlaintextBearerAuthGuard(
-  warn: (message: string) => void = (message) => console.warn(message),
-  env?: { AGENTMEMORY_REQUIRE_HTTPS?: string },
-): (baseUrl: string, secret?: string) => void {
-  let warned = false;
-  return (baseUrl, secret) => {
-    if (!usesPlaintextBearerAuth(baseUrl, secret)) return;
-    const message = plaintextBearerAuthMessage(baseUrl);
-    if ((env || process.env).AGENTMEMORY_REQUIRE_HTTPS === "1") throw new Error(message);
-    if (!warned) {
-      warned = true;
-      warn(message);
-    }
-  };
-}

plugin/.claude-plugin/plugin.json

@@ -1,13 +0,0 @@
-{
-  "name": "agentmemory",
-  "version": "0.9.27",
-  "description": "Persistent memory for AI coding agents -- captures tool usage, compresses via LLM, injects context into future sessions. 12 hooks, 53 MCP tools, 8 skills, real-time viewer.",
-  "author": {
-    "name": "Rohit Ghumare",
-    "url": "https://github.com/rohitg00"
-  },
-  "license": "Apache-2.0",
-  "homepage": "https://github.com/rohitg00/agentmemory",
-  "repository": "https://github.com/rohitg00/agentmemory",
-  "skills": ["./skills/"]
-}

plugin/.codex-plugin/plugin.json

@@ -1,15 +0,0 @@
-{
-  "name": "agentmemory",
-  "version": "0.9.27",
-  "description": "Persistent memory for AI coding agents -- captures tool usage, compresses via LLM, injects context into future sessions. 6 hooks, 53 MCP tools, 8 skills, real-time viewer.",
-  "author": {
-    "name": "Rohit Ghumare",
-    "url": "https://github.com/rohitg00"
-  },
-  "license": "Apache-2.0",
-  "homepage": "https://github.com/rohitg00/agentmemory",
-  "repository": "https://github.com/rohitg00/agentmemory",
-  "skills": "./skills/",
-  "mcpServers": "./.mcp.json",
-  "hooks": "./hooks/hooks.codex.json"
-}

plugin/.mcp.copilot.json

@@ -1,15 +0,0 @@
-{
-  "mcpServers": {
-    "agentmemory": {
-      "type": "local",
-      "command": "npx",
-      "args": ["-y", "@agentmemory/mcp"],
-      "env": {
-        "AGENTMEMORY_URL": "${AGENTMEMORY_URL:-http://localhost:3111}",
-        "AGENTMEMORY_SECRET": "${AGENTMEMORY_SECRET:-}",
-        "AGENTMEMORY_TOOLS": "${AGENTMEMORY_TOOLS:-all}"
-      },
-      "tools": ["*"]
-    }
-  }
-}

plugin/.mcp.json

@@ -1,13 +0,0 @@
-{
-  "mcpServers": {
-    "agentmemory": {
-      "command": "npx",
-      "args": ["-y", "@agentmemory/mcp"],
-      "env": {
-        "AGENTMEMORY_URL": "${AGENTMEMORY_URL:-http://localhost:3111}",
-        "AGENTMEMORY_SECRET": "${AGENTMEMORY_SECRET:-}",
-        "AGENTMEMORY_TOOLS": "${AGENTMEMORY_TOOLS:-all}"
-      }
-    }
-  }
-}

plugin/hooks/hooks.codex.json

@@ -1,67 +0,0 @@
-{
-  "hooks": {
-    "SessionStart": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/session-start.mjs\"",
-            "statusMessage": "agentmemory: loading session context"
-          }
-        ]
-      }
-    ],
-    "UserPromptSubmit": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/prompt-submit.mjs\"",
-            "statusMessage": "agentmemory: recalling relevant memories"
-          }
-        ]
-      }
-    ],
-    "PreToolUse": [
-      {
-        "matcher": "Edit|Write|Read|Glob|Grep",
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/pre-tool-use.mjs\""
-          }
-        ]
-      }
-    ],
-    "PostToolUse": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/post-tool-use.mjs\""
-          }
-        ]
-      }
-    ],
-    "PreCompact": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/pre-compact.mjs\""
-          }
-        ]
-      }
-    ],
-    "Stop": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/stop.mjs\""
-          }
-        ]
-      }
-    ]
-  }
-}

plugin/hooks/hooks.copilot.json

@@ -1,72 +0,0 @@
-{
-  "version": 1,
-  "hooks": {
-    "sessionStart": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/session-start.mjs"
-      }
-    ],
-    "userPromptSubmitted": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/prompt-submit.mjs"
-      }
-    ],
-    "preToolUse": [
-      {
-        "type": "command",
-        "matcher": "edit|write|create|read|view|glob|grep",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/pre-tool-use.mjs"
-      }
-    ],
-    "postToolUse": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/post-tool-use.mjs"
-      }
-    ],
-    "postToolUseFailure": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/post-tool-failure.mjs"
-      }
-    ],
-    "preCompact": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/pre-compact.mjs"
-      }
-    ],
-    "agentStop": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/stop.mjs"
-      }
-    ],
-    "sessionEnd": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/session-end.mjs"
-      }
-    ],
-    "subagentStart": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/subagent-start.mjs"
-      }
-    ],
-    "subagentStop": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/subagent-stop.mjs"
-      }
-    ],
-    "notification": [
-      {
-        "type": "command",
-        "command": "node ${COPILOT_PLUGIN_ROOT}/scripts/notification.mjs"
-      }
-    ]
-  }
-}

plugin/hooks/hooks.json

@@ -1,125 +0,0 @@
-{
-  "hooks": {
-    "SessionStart": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/session-start.mjs\""
-          }
-        ]
-      }
-    ],
-    "UserPromptSubmit": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/prompt-submit.mjs\""
-          }
-        ]
-      }
-    ],
-    "PreToolUse": [
-      {
-        "matcher": "Edit|Write|Read|Glob|Grep",
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/pre-tool-use.mjs\""
-          }
-        ]
-      }
-    ],
-    "PostToolUse": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/post-tool-use.mjs\""
-          }
-        ]
-      }
-    ],
-    "PostToolUseFailure": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/post-tool-failure.mjs\""
-          }
-        ]
-      }
-    ],
-    "PreCompact": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/pre-compact.mjs\""
-          }
-        ]
-      }
-    ],
-    "SubagentStart": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/subagent-start.mjs\""
-          }
-        ]
-      }
-    ],
-    "SubagentStop": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/subagent-stop.mjs\""
-          }
-        ]
-      }
-    ],
-    "Notification": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/notification.mjs\""
-          }
-        ]
-      }
-    ],
-    "TaskCompleted": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/task-completed.mjs\""
-          }
-        ]
-      }
-    ],
-    "Stop": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/stop.mjs\""
-          }
-        ]
-      }
-    ],
-    "SessionEnd": [
-      {
-        "hooks": [
-          {
-            "type": "command",
-            "command": "node \"${CLAUDE_PLUGIN_ROOT}/scripts/session-end.mjs\""
-          }
-        ]
-      }
-    ]
-  }
-}

plugin/opencode/README.md

@@ -1,229 +0,0 @@
-<h1 align="center">
-  <img src="https://github.com/opencode-ai.png?size=80" alt="OpenCode" width="28" height="28" align="center" />
-  &nbsp;agentmemory for OpenCode
-</h1>
-
-<p align="center">
-  <strong>Your OpenCode agents remember everything. No more re-explaining.</strong><br/>
-  <sub>Persistent cross-session memory via <a href="https://github.com/rohitg00/agentmemory">agentmemory</a> — 95.2% retrieval accuracy on <a href="https://arxiv.org/abs/2410.10813">LongMemEval-S</a>.</sub>
-</p>
-
-<p align="center">
-  <img src="https://img.shields.io/badge/MCP-53_tools-1f6feb?style=flat-square" alt="53 MCP tools" />
-  <img src="https://img.shields.io/badge/Plugin-22_hooks-1f6feb?style=flat-square" alt="22 hooks" />
-  <img src="https://img.shields.io/badge/Commands-2_slash-1f6feb?style=flat-square" alt="2 slash commands" />
-  <img src="https://img.shields.io/badge/R@5-95.2%25-00875f?style=flat-square" alt="95.2% R@5" />
-</p>
-
----
-
-## Quick start
-
-### 1. Start the agentmemory server
-
-```bash
-npx @agentmemory/agentmemory
-```
-
-The server starts on `http://localhost:3111`.
-
-### 2. Configure the MCP server
-
-Add to `~/.config/opencode/opencode.json` or your project's `.opencode/opencode.json`:
-
-```json
-{
-  "mcp": {
-    "agentmemory": {
-      "type": "local",
-      "command": ["npx", "-y", "@agentmemory/mcp"],
-      "enabled": true
-    }
-  }
-}
-```
-
-### 3. Install the plugin
-
-Add to `~/.config/opencode/opencode.json`:
-
-```json
-{
-  "plugin": ["./plugins/agentmemory-capture.ts"]
-}
-```
-
-Copy the plugin file from this repo:
-
-```bash
-mkdir -p ~/.config/opencode/plugins
-cp plugin/opencode/agentmemory-capture.ts ~/.config/opencode/plugins/
-```
-
-### 4. Add the slash commands
-
-Copy the commands into your project or global `.opencode/commands/` directory:
-
-```bash
-mkdir -p ~/.config/opencode/commands
-cp plugin/opencode/commands/recall.md ~/.config/opencode/commands/
-cp plugin/opencode/commands/remember.md ~/.config/opencode/commands/
-```
-
-Restart OpenCode or open a new session. The plugin auto-captures everything.
-
-## What gets captured
-
-### Session lifecycle
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| Session start | `session.created` | POST /session/start |
-| Idle → summarize | `session.idle` + `session.status` (idle) | POST /summarize |
-| Status transitions | `session.status` (idle/busy/retry) | POST /observe |
-| Compaction | `session.compacted` | POST /summarize + POST /observe |
-| Metadata updates | `session.updated` | POST /observe |
-| Code change tracking | `session.diff` | POST /observe |
-| Session delete | `session.deleted` | POST /session/end |
-| Session error | `session.error` | POST /observe |
-
-### Messages & prompts
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| User prompt (rich) | `chat.message` | POST /observe |
-| User prompt metadata | `message.updated` (user) | POST /observe |
-| Assistant response | `message.updated` (assistant) | POST /observe |
-| Message removed (undo) | `message.removed` | POST /observe |
-
-### Parts & steps
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| Subagent start | `message.part.updated` (subtask) | POST /observe |
-| Tool completed | `message.part.updated` (tool completed) | POST /observe |
-| Tool error | `message.part.updated` (tool error) | POST /observe |
-| Step finish (cost/tokens) | `message.part.updated` (step-finish) | POST /observe |
-| Reasoning trace | `message.part.updated` (reasoning) | POST /observe |
-| Patch applied | `message.part.updated` (patch) | POST /observe |
-| Auto/manual compaction | `message.part.updated` (compaction) | POST /observe |
-| Agent selection | `message.part.updated` (agent) | POST /observe |
-| API retry | `message.part.updated` (retry) | POST /observe |
-
-### File enrichment pipeline
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| File tool params | `tool.execute.before` → stash paths | — |
-| File edited | `file.edited` → stash paths | — |
-| File part attached | `message.part.updated` (file) → stash paths | — |
-| Enrichment inject | `experimental.chat.system.transform` | POST /enrich → `output.system[]` |
-| Memory context inject | `experimental.chat.system.transform` | POST /context → `output.system[]` |
-
-### Permissions
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| Permission prompt | `permission.updated` | POST /observe |
-| Permission reply | `permission.replied` | POST /observe |
-
-### Tasks & commands
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| Task tracking (w/ priority) | `todo.updated` | POST /observe |
-| Command executed | `command.executed` | POST /observe |
-
-### Model & config
-
-| Event | Hook | agentmemory API |
-|---|---|---|
-| LLM parameters | `chat.params` | POST /observe |
-| Config loaded | `config` | POST /observe |
-| Compaction (WIP) | `experimental.session.compacting` | POST /context → `output.context[]` |
-
-### File enrichment + memory injection (two-layer pipeline)
-
-`experimental.chat.system.transform` fires before every LLM call and injects two layers of context:
-
-1. **Memory context** (once per session): calls `/agentmemory/context` and injects project profile, recent session summaries, and important past observations into the system prompt. This is the OpenCode equivalent of Claude's MEMORY.md bridge — instead of syncing to a markdown file, context is injected directly into the system prompt.
-
-2. **File enrichment** (every turn with stashed files): calls `/agentmemory/enrich` with files stashed by `tool.execute.before`, `file.edited`, and `message.part.updated` (file parts). File-specific context (past observations, related bugs, semantic search) is injected into the system prompt.
-
-```text
-System prompt = [OpenCode instructions] + [memory context] + [file enrichment] + [user message]
-                                        ^                 ^
-                               first turn only         every file-touching turn
-```
-
-**Differences from Claude's PreToolUse:**
-
-| Dimension | Claude (PreToolUse) | OpenCode (two-hop pipeline) |
-|---|---|---|
-| Injection mechanism | stdout → context window | `output.system[]` → system prompt |
-| Timing | Same turn (parallel with tool) | Next turn (before next LLM call) |
-| File set | Per-tool (immediate) | Batched (all files since last enrichment) |
-| Coverage | Edit/Write/Read/Glob/Grep only | Edit/Write/Read/Glob/Grep only |
-| What gets injected | `<agentmemory-file-context>` + bug memories | Identical `/enrich` response |
-
-## MEMORY.md vs AGENTS.md: how context flows
-
-Claude Code and OpenCode take fundamentally different approaches to injecting memory context into the agent's system prompt.
-
-### Claude Code: file-backed bridge (two-hop)
-
-```
-agentmemory  ──write──▶  MEMORY.md  ──read──▶  Claude system prompt
-```
-
-- The `claude-bridge/sync` endpoint serializes agentmemory observations into a `MEMORY.md` file in the project root
-- Claude Code reads `MEMORY.md` on session start and prepends it to the system prompt
-- **Sync is periodic** — sessions only get fresh context when the bridge last ran (session end, pre-compact)
-- **Coupling**: memory data lives in a git-trackable file, visible to CI, team members, and other tools
-
-### OpenCode: direct injection (one-hop)
-
-```
-agentmemory  ──push──▶  OpenCode system prompt
-```
-
-- `experimental.chat.system.transform` calls `/context` at runtime and pushes the response directly into `output.system[]`
-- **Always current** — context is fetched at session start (once) and before file-touching turns (per-batch)
-- **No file intermediary** — no stale copies, no merge conflicts, no disk I/O
-- `AGENTS.md` is a static instruction file for project conventions, coding standards, and tool guidance — agentmemory does not read or write it
-
-### Tradeoffs
-
-| Dimension | Claude (MEMORY.md bridge) | OpenCode (direct injection) |
-|---|---|---|
-| Freshness | Stale between syncs | Always current (fetched at call time) |
-| Visibility | Human-readable file in repo | In-memory injection only |
-| Simplicity | Two moving parts (bridge + file) | One step (API → system prompt) |
-| Team sharing | File is git-trackable, CI-friendly | Memory shared via agentmemory server API |
-| Integration | Any tool can read MEMORY.md | Requires OpenCode plugin SDK |
-
-### Why OpenCode goes direct
-
-agentmemory already persists everything in SQLite (`data/state_store.db`). Adding an intermediate MEMORY.md file would duplicate data, introduce sync lag, and require the model to re-parse structured context from markdown. Direct injection delivers the same data with lower latency and zero staleness — the agent always sees what agentmemory knows right now.
-
-## Slash commands
-
-- `/recall <query>` — Search past observations and lessons
-- `/remember <text>` — Save an insight to long-term memory
-
-## Session instruction injection
-
-Agentmemory usage instructions are injected into the system prompt on the first turn of every session via `experimental.chat.system.transform` (alongside memory context from `/context`). This is functionally equivalent to Claude Code's skills mechanism — the agent learns which `agentmemory_memory_*` tools to use and when, without needing separate skill invocations.
-
-## What's not covered (vs Claude Code plugin)
-
-| Claude feature | Reason |
-|---|---|
-| SubagentStop | OpenCode's `SubtaskPart` type has no completion/result fields; subtask lifecycle ends are not exposed as distinct events in the OpenCode SDK |
-| TaskCompleted | No team/teammate concept in OpenCode; `todo.updated` captures task state changes as a partial equivalent |
-| Stop | `session.compacted` event handler exists; `experimental.session.compacting` injection hook defined in SDK but Go binary (v1.14.41) doesn't wire it — will auto-activate when upstream implements it |
-| Skills (remember/recall/forget/session-history) | Covered by injected system instructions via `experimental.chat.system.transform` — agent receives usage guidance on first turn |
-| Consolidation pipeline (crystals/auto + consolidate-pipeline) | Now called on `session.deleted` — mirrors Claude's `CONSOLIDATION_ENABLED=true` behavior |
-| Claude MEMORY.md bridge | OpenCode-specific; OpenCode uses its own AGENTS.md mechanism, not Claude's MEMORY.md |
-
-All other Claude Code hooks have direct or pipeline equivalents in this plugin. 12 of 12 Claude hook types covered.

plugin/opencode/agentmemory-capture.ts

@@ -1,687 +0,0 @@
-import type { Plugin } from "@opencode-ai/plugin";
-
-const API = process.env.AGENTMEMORY_URL || "http://localhost:3111";
-const FILE_TOOLS = new Set(["Read", "Write", "Edit", "Glob", "Grep"]);
-const FILE_KEYS = ["filePath", "file_path", "path", "file", "pattern"];
-const MAX_STASHED_FILES = 20;
-
-const DEBUG = process.env.OPENCODE_AGENTMEMORY_DEBUG === "1";
-const SECRET = process.env.AGENTMEMORY_SECRET || "";
-
-function authHeaders(): Record<string, string> {
-  const headers: Record<string, string> = { "Content-Type": "application/json" };
-  if (SECRET) headers["Authorization"] = `Bearer ${SECRET}`;
-  return headers;
-}
-
-async function post(path: string, body: Record<string, unknown>, timeoutMs = 5000): Promise<void> {
-  try {
-    await fetch(`${API}/agentmemory${path}`, {
-      method: "POST",
-      headers: authHeaders(),
-      body: JSON.stringify(body),
-      signal: AbortSignal.timeout(timeoutMs),
-    });
-  } catch (e) {
-    if (DEBUG) console.error(`[agentmemory] POST ${path} failed:`, (e as Error).message);
-  }
-}
-
-async function postJson(path: string, body: Record<string, unknown>): Promise<unknown | null> {
-  try {
-    const res = await fetch(`${API}/agentmemory${path}`, {
-      method: "POST",
-      headers: authHeaders(),
-      body: JSON.stringify(body),
-      signal: AbortSignal.timeout(5000),
-    });
-    return res.ok ? await res.json() : null;
-  } catch (e) {
-    if (DEBUG) console.error(`[agentmemory] POST ${path} failed:`, (e as Error).message);
-    return null;
-  }
-}
-
-async function observe(
-  sessionId: string,
-  hookType: string,
-  data: Record<string, unknown>,
-): Promise<void> {
-  await post("/observe", {
-    hookType,
-    sessionId,
-    project: projectPath,
-    cwd: projectPath,
-    timestamp: new Date().toISOString(),
-    data,
-  });
-}
-
-let activeSessionId: string | null = null;
-let pendingConfig: Record<string, unknown> | null = null;
-let projectPath: string | null = null;
-const stashedFiles = new Map<string, Set<string>>();
-const seenSubtaskIds = new Map<string, Set<string>>();
-const seenToolCallIds = new Map<string, Set<string>>();
-const contextInjectedSessions = new Set<string>();
-// cache the context returned by POST /session/start so the chat
-// system-transform hook can inject it without a second /context fetch.
-// Auto-injection now happens at session.created (immediately) AND at
-// the first prompt_submit (fallback for older OpenCode builds that
-// don't implement experimental.chat.system.transform).
-const startContextCache = new Map<string, string>();
-
-function stashFor(sid: string): Set<string> {
-  let s = stashedFiles.get(sid);
-  if (!s) { s = new Set<string>(); stashedFiles.set(sid, s); }
-  return s;
-}
-
-function subtaskSetFor(sid: string): Set<string> {
-  let s = seenSubtaskIds.get(sid);
-  if (!s) { s = new Set<string>(); seenSubtaskIds.set(sid, s); }
-  return s;
-}
-
-function toolCallSetFor(sid: string): Set<string> {
-  let s = seenToolCallIds.get(sid);
-  if (!s) { s = new Set<string>(); seenToolCallIds.set(sid, s); }
-  return s;
-}
-
-function pruneSessionMaps(sid: string): void {
-  stashedFiles.delete(sid);
-  seenSubtaskIds.delete(sid);
-  seenToolCallIds.delete(sid);
-}
-
-function safeSlice(v: unknown, max: number): string {
-  if (typeof v === "string") return v.slice(0, max);
-  if (v == null) return "";
-  try { return JSON.stringify(v).slice(0, max); } catch { return ""; }
-}
-
-const AGENTMEMORY_INSTRUCTIONS = `<agentmemory-instructions>
-You have access to agentmemory for persistent cross-session memory. Use these tools proactively.
-
-CORE TOOLS:
-
-memory_save — Save an insight, decision, or fact to long-term memory.
-  Required: content (text), concepts (2-5 comma-separated keywords), type (pattern/preference/architecture/bug/workflow/fact)
-  Optional: files (comma-separated paths)
-  Use when: user says "remember this", after discovering a bug, after making an architectural decision, after learning a project convention.
-
-memory_recall — Search past observations by keywords.
-  Use when: user says "recall", "what did we do", "do you remember", or needs context from past sessions.
-
-memory_smart_search — Hybrid semantic+keyword search with progressive disclosure.
-  Use when: you need the most relevant past context, fuzzy/conceptual searches, or recall doesn't find what you need.
-
-memory_sessions — List recent sessions with status and observation counts.
-  Use when: user asks about session/past history, "what did we work on".
-
-memory_file_history — Get past observations about specific files (across all sessions).
-  Use when: you're about to edit a file and want to know its history, common pitfalls, or past edits.
-
-memory_lesson_save — Save a lesson learned (what worked, what to avoid).
-  Use when: you discover a pattern that could help future sessions avoid mistakes.
-
-memory_lesson_recall — Search lessons by query. Returns lessons sorted by confidence.
-  Use when: before making a decision, check if past lessons apply.
-
-memory_governance_delete ��� Delete specific memories. Requires explicit user confirmation.
-  Use when: user says "forget this", "delete that memory".
-
-memory_patterns — Detect recurring patterns across sessions.
-  Use when: you want to understand project-level trends over time.
-
-memory_consolidate — Run the 4-tier memory consolidation pipeline.
-  Use when: you want to compress and organize accumulated session observations.
-
-All memory tools start with \`agentmemory_memory_\`. Use the exact names as they appear in your tool list. Tool results are JSON. Always check what was returned before presenting to the user.
-</agentmemory-instructions>`;
-
-function extractFilePaths(args: Record<string, unknown>): string[] {
-  const files: string[] = [];
-  for (const key of FILE_KEYS) {
-    const val = args[key];
-    if (typeof val === "string" && val.length > 0) {
-      files.push(val);
-    }
-  }
-  return files;
-}
-
-function extractErrorMessage(err: unknown): string {
-  if (typeof err === "string") return err;
-  if (err && typeof err === "object") {
-    const e = err as Record<string, unknown>;
-    if (typeof e.message === "string") return e.message;
-    if (e.data && typeof e.data === "object") {
-      const d = e.data as Record<string, unknown>;
-      if (typeof d.message === "string") return d.message;
-    }
-    if (typeof e.name === "string") return e.name;
-    try { return JSON.stringify(err); } catch { return ""; }
-  }
-  return String(err ?? "");
-}
-
-export const AgentmemoryCapturePlugin: Plugin = async (ctx) => {
-  projectPath = ctx.worktree || ctx.project?.id || process.cwd();
-
-  return {
-    event: async ({ event }) => {
-      const type = event.type;
-      const props = (event as any).properties || {};
-
-      // ── session.created ──
-      if (type === "session.created") {
-        const info = props.info as Record<string, unknown> | undefined;
-        activeSessionId = (info?.id as string) || props.sessionID || null;
-        if (!activeSessionId) return;
-        stashedFiles.set(activeSessionId, new Set());
-        seenSubtaskIds.delete(activeSessionId);
-        seenToolCallIds.delete(activeSessionId);
-        contextInjectedSessions.delete(activeSessionId);
-        // Snapshot the session id locally — `activeSessionId` is mutable
-        // and another `session.created` event during the await could
-        // rebind it, causing context to be cached against the wrong key.
-        const sessionId = activeSessionId;
-        const startResult = await postJson("/session/start", {
-          sessionId,
-          title: info?.title ?? null,
-          parentID: info?.parentID ?? null,
-          version: info?.version ?? null,
-          project: projectPath,
-          cwd: projectPath,
-        });
-        // cache the context returned at session/start so the
-        // chat.system.transform hook injects it without a second fetch.
-        const startCtx = (startResult as any)?.context;
-        if (typeof startCtx === "string" && startCtx.length > 0) {
-          startContextCache.set(sessionId, startCtx);
-        }
-        if (pendingConfig) {
-          await observe(sessionId, "config_loaded", pendingConfig);
-          pendingConfig = null;
-        }
-      }
-
-      // ── session.idle ── (summarize handled in session.status idle branch)
-
-      // ── session.status ──
-      if (type === "session.status") {
-        const status = props.status as Record<string, unknown> | undefined;
-        const sid = props.sessionID || activeSessionId;
-        if (!sid || !status) return;
-        if (status.type === "idle") {
-          await post("/summarize", { sessionId: sid });
-        }
-        await observe(sid, "session_status", {
-          status_type: status.type,
-          attempt: status.attempt ?? null,
-          message: safeSlice(status.message, 2000),
-        });
-      }
-
-      // ── session.compacted ──
-      if (type === "session.compacted") {
-        const sid = props.sessionID || activeSessionId;
-        if (sid) {
-          await post("/summarize", { sessionId: sid });
-          await observe(sid, "session_compacted", {});
-        }
-      }
-
-      // ── session.updated ──
-      if (type === "session.updated") {
-        const info = props.info as Record<string, unknown> | undefined;
-        const sid = (info?.id as string) || props.sessionID || activeSessionId;
-        if (!sid) return;
-        await observe(sid, "session_updated", {
-          title: info?.title ?? null,
-          parentID: info?.parentID ?? null,
-          additions: (info?.summary as any)?.additions ?? null,
-          deletions: (info?.summary as any)?.deletions ?? null,
-          files: (info?.summary as any)?.files ?? null,
-        });
-      }
-
-      // ── session.diff ──
-      if (type === "session.diff") {
-        const sid = props.sessionID || activeSessionId;
-        if (!sid || !Array.isArray(props.diff)) return;
-        const diffs = props.diff as Array<Record<string, unknown>>;
-        await observe(sid, "session_diff", {
-          files: diffs.map(d => d.file),
-          additions: diffs.reduce((s, d) => s + ((d.additions as number) || 0), 0),
-          deletions: diffs.reduce((s, d) => s + ((d.deletions as number) || 0), 0),
-          diffs: diffs.slice(0, 50),
-        });
-      }
-
-      // ── session.deleted ──
-      if (type === "session.deleted") {
-        const sid = props.info?.id || props.sessionID || activeSessionId;
-        if (!sid) {
-          if (DEBUG) console.error("[agentmemory] session.deleted with no session ID");
-          return;
-        }
-        await post("/session/end", { sessionId: sid });
-        post("/crystals/auto", { olderThanDays: 7 }, 30000);
-        post("/consolidate-pipeline", { tier: "all", force: true }, 30000);
-        if (sid === activeSessionId) activeSessionId = null;
-        stashedFiles.delete(sid);
-        startContextCache.delete(sid);
-        seenSubtaskIds.delete(sid);
-        seenToolCallIds.delete(sid);
-        contextInjectedSessions.delete(sid);
-      }
-
-      // ── session.error ──
-      if (type === "session.error") {
-        const sid = props.sessionID || activeSessionId;
-        if (sid) {
-          await observe(sid, "post_tool_failure", {
-            tool_name: "session.error",
-            tool_input: "",
-            tool_output: safeSlice(props.error, 8000),
-          });
-        }
-      }
-
-      // ── message.updated ──
-      if (type === "message.updated") {
-        const info = props.info as Record<string, unknown> | undefined;
-        if (!info) return;
-
-        if (info.role === "assistant") {
-          const sid = props.sessionID || (info.sessionID as string) || activeSessionId;
-          if (!sid) return;
-          const tokens = info.tokens as Record<string, unknown> | undefined;
-          const error = info.error ? extractErrorMessage(info.error) : null;
-          await observe(sid, "assistant_message", {
-            messageID: info.id,
-            parentID: info.parentID,
-            modelID: info.modelID,
-            providerID: info.providerID,
-            mode: info.mode,
-            cost: info.cost ?? 0,
-            tokens: {
-              input: tokens?.input ?? 0,
-              output: tokens?.output ?? 0,
-              reasoning: tokens?.reasoning ?? 0,
-              cache_read: (tokens?.cache as any)?.read ?? 0,
-              cache_write: (tokens?.cache as any)?.write ?? 0,
-            },
-            finish: info.finish ?? null,
-            error,
-            duration_ms: (info.time && typeof (info.time as any).completed === "number")
-              ? (info.time as any).completed - ((info.time as any).created || 0)
-              : null,
-          });
-        }
-      }
-
-      // ── message.removed ──
-      if (type === "message.removed") {
-        const sid = props.sessionID || activeSessionId;
-        if (sid) {
-          await observe(sid, "message_removed", {
-            messageID: props.messageID,
-          });
-        }
-      }
-
-      // ── message.part.updated ──
-      if (type === "message.part.updated") {
-        const part = props.part as Record<string, unknown> | undefined;
-        if (!part) return;
-        const sid = (part.sessionID as string) || props.sessionID || activeSessionId;
-        if (!sid) return;
-
-        if (part.type === "subtask") {
-          const subtaskId = part.id as string;
-          if (!subtaskId) return;
-          const subtaskSet = subtaskSetFor(sid);
-          if (subtaskSet.has(subtaskId)) return;
-          subtaskSet.add(subtaskId);
-          await observe(sid, "subagent_start", {
-            subtask_id: part.id,
-            agent: part.agent,
-            prompt: safeSlice(part.prompt, 4000),
-            description: safeSlice(part.description, 2000),
-          });
-          return;
-        }
-
-        if (part.type === "tool") {
-          const state = part.state as Record<string, unknown> | undefined;
-          if (!state) return;
-          const callId = part.callID as string;
-          if (!callId) return;
-          const toolName = part.tool as string;
-
-          if (state.status === "completed") {
-            const callSet = toolCallSetFor(sid);
-            if (callSet.has(callId)) return;
-            callSet.add(callId);
-            const st = state as Record<string, unknown>;
-            const rawTime = (st.time as any) || {};
-            const startTime = typeof rawTime.start === "number" ? rawTime.start : null;
-            const endTime = typeof rawTime.end === "number" ? rawTime.end : null;
-            await observe(sid, "post_tool_use", {
-              tool_name: toolName,
-              call_id: callId,
-              tool_input: safeSlice(st.input, 4000),
-              tool_output: safeSlice(st.output, 8000),
-              title: st.title ?? null,
-              metadata: st.metadata || {},
-              duration_ms: (startTime != null && endTime != null) ? endTime - startTime : null,
-              attachments: Array.isArray(st.attachments)
-                ? (st.attachments as Array<Record<string, unknown>>).map(a => a.filename || a.url)
-                : [],
-            });
-          } else if (state.status === "error") {
-            const callSet = toolCallSetFor(sid);
-            if (callSet.has(callId)) return;
-            callSet.add(callId);
-            const st = state as Record<string, unknown>;
-            const rawTime = (st.time as any) || {};
-            const startTime = typeof rawTime.start === "number" ? rawTime.start : null;
-            const endTime = typeof rawTime.end === "number" ? rawTime.end : null;
-            await observe(sid, "post_tool_failure", {
-              tool_name: toolName,
-              call_id: callId,
-              tool_input: safeSlice(st.input, 4000),
-              tool_output: safeSlice(st.error, 8000),
-              duration_ms: (startTime != null && endTime != null) ? endTime - startTime : null,
-            });
-          }
-          return;
-        }
-
-        if (part.type === "step-finish") {
-          await observe(sid, "step_finish", {
-            messageID: part.messageID,
-            reason: part.reason ?? null,
-            cost: (part as any).cost ?? 0,
-            input_tokens: ((part as any).tokens?.input as number) ?? 0,
-            output_tokens: ((part as any).tokens?.output as number) ?? 0,
-            reasoning_tokens: ((part as any).tokens?.reasoning as number) ?? 0,
-          });
-          return;
-        }
-
-        if (part.type === "reasoning") {
-          await observe(sid, "reasoning", {
-            messageID: part.messageID,
-            text: safeSlice((part as any).text, 4000),
-          });
-          return;
-        }
-
-        if (part.type === "file") {
-          const filename = (part as any).filename || (part as any).url || null;
-          if (filename) stashFor(sid).add(filename);
-          return;
-        }
-
-        if (part.type === "patch") {
-          await observe(sid, "patch_applied", {
-            messageID: part.messageID,
-            hash: (part as any).hash,
-            files: (part as any).files || [],
-          });
-          return;
-        }
-
-        if (part.type === "compaction") {
-          await observe(sid, "compaction_event", {
-            messageID: part.messageID,
-            auto: (part as any).auto ?? false,
-          });
-          return;
-        }
-
-        if (part.type === "agent") {
-          await observe(sid, "agent_selected", {
-            messageID: part.messageID,
-            name: (part as any).name,
-          });
-          return;
-        }
-
-        if (part.type === "retry") {
-          await observe(sid, "retry_attempt", {
-            messageID: part.messageID,
-            attempt: (part as any).attempt,
-            error: safeSlice((part as any).error, 2000),
-          });
-          return;
-        }
-      }
-
-      // ── file.edited ──
-      if (type === "file.edited") {
-        const sid = props.sessionID || activeSessionId;
-        if (sid && typeof props.file === "string" && props.file.length > 0) {
-          const stash = stashFor(sid);
-          stash.add(props.file);
-          if (stash.size > MAX_STASHED_FILES) {
-            const keep = [...stash].slice(-MAX_STASHED_FILES);
-            stash.clear();
-            for (const f of keep) stash.add(f);
-          }
-        }
-      }
-
-      // ── permission.updated ──
-      if (type === "permission.updated") {
-        const sid = props.sessionID || activeSessionId;
-        if (!sid) return;
-        await observe(sid, "notification", {
-          notification_type: "permission_prompt",
-          permission: props.type || "unknown",
-          pattern: Array.isArray(props.pattern)
-            ? props.pattern.join(", ")
-            : (props.pattern || ""),
-          tool_call_id: props.callID || null,
-          title: props.title || props.type || "",
-          metadata: props.metadata || {},
-        });
-      }
-
-      // ── permission.replied ──
-      if (type === "permission.replied") {
-        const sid = props.sessionID || activeSessionId;
-        if (!sid) return;
-        await observe(sid, "permission_replied", {
-          permission_id: props.permissionID || props.requestID || "",
-          response: props.response || props.reply || "",
-        });
-      }
-
-      // ── todo.updated ──
-      if (type === "todo.updated") {
-        const sid = props.sessionID || activeSessionId;
-        const todos = Array.isArray(props.todos) ? props.todos.slice(0, 100) : [];
-        if (!sid || todos.length === 0) return;
-        const completed = todos.filter((t: any) => t.status === "completed");
-        const active = todos.filter((t: any) => t.status !== "completed");
-        await observe(sid, "task_completed", {
-          completed: completed.map((t: any) => ({ content: t.content, priority: t.priority })),
-          in_progress: active.map((t: any) => ({ content: t.content, priority: t.priority })),
-          total: todos.length,
-        });
-      }
-
-      // ── command.executed ──
-      if (type === "command.executed") {
-        const sid = props.sessionID || activeSessionId;
-        if (sid) {
-          await observe(sid, "command_executed", {
-            name: props.name,
-            arguments: props.arguments || "",
-          });
-        }
-      }
-    },
-
-    // ── chat.message ──
-    "chat.message": async (input, output) => {
-      const sid = input.sessionID || activeSessionId;
-      if (!sid) return;
-      const parts = output.parts || [];
-      const files = parts
-        .filter((p: any) => p.type === "file")
-        .map((p: any) => p.filename || p.url)
-        .filter(Boolean);
-      for (const f of files) {
-        const stash = stashFor(sid);
-        stash.add(f);
-        if (stash.size > MAX_STASHED_FILES) {
-          const keep = [...stash].slice(-MAX_STASHED_FILES);
-          stash.clear();
-          for (const k of keep) stash.add(k);
-        }
-      }
-
-      const textParts = parts.filter((p: any) => p.type === "text" && !p.synthetic && !p.ignored);
-      const userText = textParts.map((p: any) => p.text || "").join("\n");
-
-      await observe(sid, "prompt_submit", {
-        agent: input.agent ?? null,
-        model: input.model ?? null,
-        variant: input.variant ?? null,
-        prompt: userText.slice(0, 8000),
-        files: files.slice(0, 20),
-        parts_summary: parts.map((p: any) => p.type).filter(Boolean),
-      });
-    },
-
-    // ── chat.params ──
-    "chat.params": async (input, output) => {
-      if (!input.model || !output) return;
-      const sid = input.sessionID || activeSessionId;
-      if (!sid) return;
-      await observe(sid, "llm_params", {
-        agent: input.agent,
-        model: `${input.model.providerID}/${input.model.id}`,
-        provider_url: input.model.api?.url ?? null,
-        temperature: output.temperature,
-        topP: output.topP,
-        max_output_tokens: input.model.limit?.output ?? null,
-        context_limit: input.model.limit?.context ?? null,
-        cost_1k_input: input.model.cost?.input ?? 0,
-        cost_1k_output: input.model.cost?.output ?? 0,
-      });
-    },
-
-    // ── tool.execute.before ──
-    "tool.execute.before": async (input, output) => {
-      if (!FILE_TOOLS.has(input.tool)) return;
-      const sid = input.sessionID || activeSessionId;
-      if (!sid) return;
-      const args = output.args as Record<string, unknown> | undefined;
-      if (!args) return;
-      const stash = stashFor(sid);
-      for (const fp of extractFilePaths(args)) {
-        stash.add(fp);
-      }
-      if (stash.size > MAX_STASHED_FILES) {
-        const keep = [...stash].slice(-MAX_STASHED_FILES);
-        stash.clear();
-        for (const f of keep) stash.add(f);
-      }
-    },
-
-    // ── experimental.chat.system.transform ──
-    "experimental.chat.system.transform": async (input, output) => {
-      const sid = input.sessionID || activeSessionId;
-      if (!sid) return;
-
-      if (!contextInjectedSessions.has(sid)) {
-        if (!Array.isArray(output.system)) return;
-        output.system.push(AGENTMEMORY_INSTRUCTIONS);
-        // prefer the context already fetched at session.created;
-        // fall back to a fresh /context call if the cache missed (e.g.
-        // session resumed across plugin reloads).
-        let ctx = startContextCache.get(sid);
-        if (typeof ctx !== "string" || ctx.length === 0) {
-          const result = await postJson("/context", {
-            sessionId: sid,
-            project: projectPath,
-          });
-          ctx = (result as any)?.context;
-        } else {
-          startContextCache.delete(sid);
-        }
-        if (typeof ctx === "string" && ctx.length > 0) {
-          output.system.push(ctx);
-        }
-        contextInjectedSessions.add(sid);
-      }
-
-      const stash = stashFor(sid);
-      if (stash.size === 0) return;
-      const files = [...stash].slice(0, 10);
-
-      const enrichResult = await postJson("/enrich", {
-        sessionId: sid,
-        files,
-        toolName: "enrich_inject",
-      });
-
-      const enrichCtx = (enrichResult as any)?.context;
-      if (typeof enrichCtx === "string" && enrichCtx.length > 0) {
-        if (Array.isArray(output.system)) {
-          output.system.push(enrichCtx);
-        }
-        for (const f of files) stash.delete(f);
-      }
-    },
-
-    // ── experimental.session.compacting (WIP) ──
-    "experimental.session.compacting": async (input, output) => {
-      const sid = input.sessionID || activeSessionId;
-      if (!sid) return;
-
-      const result = await postJson("/context", {
-        sessionId: sid,
-        project: projectPath,
-      });
-      const ctx = (result as any)?.context;
-      if (typeof ctx === "string" && ctx.length > 0) {
-        if (Array.isArray(output.context)) {
-          output.context.push(ctx);
-        }
-      }
-    },
-
-    // ── config ──
-    config: async (input) => {
-      const payload: Record<string, unknown> = {
-        theme: input.theme ?? null,
-        model: input.model ?? null,
-        autoupdate: input.autoupdate ?? null,
-        agents: typeof input.agent === "object" && input.agent !== null && !Array.isArray(input.agent)
-          ? Object.keys(input.agent as Record<string, unknown>)
-          : Array.isArray(input.agent) ? input.agent : [],
-        mcp_servers: typeof input.mcp === "object" && input.mcp !== null && !Array.isArray(input.mcp)
-          ? Object.keys(input.mcp as Record<string, unknown>)
-          : Array.isArray(input.mcp) ? input.mcp : [],
-        providers: typeof input.provider === "object" && input.provider !== null && !Array.isArray(input.provider)
-          ? Object.keys(input.provider as Record<string, unknown>)
-          : Array.isArray(input.provider) ? input.provider : [],
-        permission: input.permission ?? null,
-      };
-      if (activeSessionId) {
-        await observe(activeSessionId, "config_loaded", payload);
-      } else {
-        pendingConfig = payload;
-      }
-    },
-  };
-};

plugin/opencode/commands/recall.md

@@ -1,19 +0,0 @@
-Search past session observations and lessons for relevant context. Wrap the `memory_smart_search` and `memory_lesson_recall` MCP tools.
-
-## Usage
-
-```
-/recall [query]
-```
-
-## Instructions
-
-1. Call `memory_smart_search` with the query and `limit: 10` (hybrid BM25 + vector + graph search).
-2. Call `memory_lesson_recall` with the same query and `limit: 5` (lesson search).
-3. Combine results and present to the user:
-   - Group by session
-   - Show type, title, and narrative for each observation
-   - Highlight high-importance (>= 7) observations
-   - Show lessons separately with confidence scores
-4. If no results, suggest 2-3 alternative search terms.
-5. **Never hallucinate results.** Only present what the MCP tools actually return.

plugin/opencode/commands/remember.md

@@ -1,19 +0,0 @@
-Explicitly save an insight, decision, or learning to agentmemory for future sessions. Wraps the `memory_save` MCP tool.
-
-## Usage
-
-```
-/remember [what to remember]
-```
-
-## Instructions
-
-1. Analyze what needs to be remembered — extract the core insight, decision, or fact.
-2. Extract 2-5 searchable concepts (lowercased keyword phrases). Prefer specific terms ("jwt-refresh-rotation" over "auth").
-3. Extract relevant file paths the memory references.
-4. Call `memory_save` with:
-   - `content` — full text to remember (preserve user's phrasing)
-   - `concepts` — extracted concept list
-   - `files` — extracted file list (empty array if none)
-   - `type` — choose from: pattern, preference, architecture, bug, workflow, fact
-5. Confirm the save and show the concepts tagged so the user knows retrieval terms.