Added new docs

.kiro/skills/to-tickets/SKILL.md

@@ -0,0 +1,87 @@
+---
+name: to-tickets
+description: Break a plan, spec, or the current conversation into a set of tracer-bullet tickets, each declaring its blocking edges, published to the configured tracker — edges as text in one file per ticket locally, or native blocking links on a real tracker.
+disable-model-invocation: true
+---
+
+Break a plan, spec, or conversation into a set of tickets — tracer-bullet vertical slices, each declaring the tickets that block it.
+The issue tracker and triage label vocabulary should have been provided to you — run /setup-matt-pocock-skills if not.
+
+## Process
+
+### 1. Gather context
+Work from whatever is already in the conversation context. If the user passes a reference (a spec path, an issue number or URL) as an argument, fetch it and read its full body and comments.
+
+### 2. Explore the codebase (optional)
+If you have not already explored the codebase, do so to understand the current state of the code. Ticket titles and descriptions should use the project's domain glossary vocabulary, and respect ADRs in the area you're touching.
+
+Look for opportunities to prefactor the code to make the implementation easier. "Make the change easy, then make the easy change."
+
+### 3. Draft vertical slices
+Break the work into tracer bullet tickets.
+
+- Each slice cuts a narrow but COMPLETE path through every layer (schema, API, UI, tests) — vertical, NOT a horizontal slice of one layer
+- A completed slice is demoable or verifiable on its own
+- Each slice is sized to fit in a single fresh context window
+- Any prefactoring should be done first
+
+Give each ticket its blocking edges — the other tickets that must complete before it can start. A ticket with no blockers can start immediately.
+
+Wide refactors are the exception to vertical slicing. A wide refactor is one mechanical change — rename a column, retype a shared symbol — whose blast radius fans across the whole codebase, so a single edit breaks thousands of call sites at once and no vertical slice can land green. Don't force it into a tracer bullet; sequence it as expand–contract. First expand: add the new form beside the old so nothing breaks. Then migrate the call sites over in batches sized by blast radius (per package, per directory), each batch its own ticket blocked by the expand, keeping CI green batch to batch because the old form still exists. Finally contract: delete the old form once no caller remains, in a ticket blocked by every migrate batch. When even the batches can't stay green alone, keep the sequence but let them share an integration branch that all block a final integrate-and-verify ticket — green is promised only there.
+
+### 4. Quiz the user
+Present the proposed breakdown as a numbered list. For each ticket, show:
+
+- **Title:** short descriptive name
+- **Blocked by:** which other tickets (if any) must complete first
+- **What it delivers:** the end-to-end behaviour this ticket makes work
+
+Ask the user:
+
+- Does the granularity feel right? (too coarse / too fine)
+- Are the blocking edges correct — does each ticket only depend on tickets that genuinely gate it?
+- Should any tickets be merged or split further?
+
+Iterate until the user approves the breakdown.
+
+### 5. Publish the tickets to the configured tracker
+Publish the approved tickets. How depends on the tracker /setup-matt-pocock-skills configured — the tickets are the same either way, only the shape of the blocking edges changes:
+
+- **Local files** → write one file per ticket under `.scratch/<feature-slug>/issues/<NN>-<slug>.md`, numbered from 01 in dependency order (blockers first). Each file's "Blocked by" lists the numbers/titles it depends on. Use the per-ticket file template below — one ticket per file, never a single combined file.
+- **A real issue tracker (GitHub, Linear, …)** → publish one issue per ticket in dependency order (blockers first) so each ticket's blocking edges can reference real identifiers. Use the platform's native blocking / sub-issue relationship where it has one; otherwise set each ticket's "Blocked by" to the blocking issues. Apply the ready-for-agent triage label unless instructed otherwise — the tickets are agent-grabbable by construction.
+
+Work the frontier: any ticket whose blockers are all done. For a purely linear chain that means top to bottom.
+
+Do NOT close or modify any parent issue.
+
+---
+
+## Per-ticket file template (local files)
+
+**What to build:** the end-to-end behaviour this ticket makes work, from the user's perspective — not a layer-by-layer implementation list.
+**Blocked by:** the numbers/titles of the tickets that gate this one, or "None — can start immediately".
+**Status:** ready-for-agent
+
+- [ ] Acceptance criterion 1
+- [ ] Acceptance criterion 2
+
+---
+
+## Per-ticket issue template (real tracker)
+
+**Parent**
+A reference to the parent issue on the tracker (if the source was an existing issue, otherwise omit this section).
+
+**What to build**
+The end-to-end behaviour this ticket makes work, from the user's perspective — not layer-by-layer implementation.
+
+**Acceptance criteria**
+- [ ] Criterion 1
+- [ ] Criterion 2
+
+**Blocked by**
+A reference to each blocking ticket, or "None — can start immediately".
+
+---
+
+In either form, avoid specific file paths or code snippets — they go stale fast. Exception: if a prototype produced a snippet that encodes a decision more precisely than prose can (state machine, reducer, schema, type shape), inline it and note briefly that it came from a prototype. Trim to the decision-rich parts — not a working demo, just the important bits.

.scratch/arch-improvements/issues/01-appcontext-expand.md

@@ -0,0 +1,16 @@
+# 01 — Introduce `AppContext` dataclass (Expand)
+
+**Blocked by:** None — can start immediately
+**Status:** ready-for-agent
+
+## What to build
+
+Introduce a single `AppContext` dataclass that packages all runtime state currently scattered as module-level globals in `functions.py` (`_bm25_index`, `_vector_index`, `_embedding_provider`, `_hybrid_search`, `_stream_broadcaster`, and the `kv` reference). Construct one `AppContext` instance inside `create_app()` and thread it through to every function that needs it. The existing globals stay in place as a compatibility shim — no callers are changed in this ticket. Zero behaviour change; this is the expand step that makes the god-module split (tickets 05–08) possible.
+
+## Acceptance criteria
+
+- [ ] `AppContext` dataclass exists in a new `context.py` module (or equivalent) with typed fields for `kv`, `bm25`, `vector`, `embedder`, and `broadcast`
+- [ ] `create_app()` constructs one `AppContext` and stores it on the Flask app (e.g. `app.ctx`)
+- [ ] The five existing `set_*` functions in `functions.py` remain working — globals are still set alongside the new `AppContext`
+- [ ] All existing tests pass with no changes
+- [ ] No route handler or worker is changed in this ticket

.scratch/arch-improvements/issues/02-privacy-module.md

@@ -0,0 +1,17 @@
+# 02 — Extract `privacy.py` deep module
+
+**Blocked by:** None — can start immediately
+**Status:** ready-for-agent
+
+## What to build
+
+Extract `strip_private_data()` from `functions.py` into a dedicated `privacy.py` module. The new module exposes a single `scrub(text: str, patterns=DEFAULT_PATTERNS) -> str` interface. The regex pattern list is promoted to a public, documented constant (`DEFAULT_PATTERNS`) and is extensible via an env var (`AGENTCACHE_REDACT_PATTERNS`). `functions.py` calls `privacy.scrub()` internally — all existing callers continue working without change. First unit tests for Privacy Scrub are added in this ticket, targeting `scrub()` directly.
+
+## Acceptance criteria
+
+- [ ] `privacy.py` exists with a `scrub(text, patterns=DEFAULT_PATTERNS)` function
+- [ ] `DEFAULT_PATTERNS` is a documented list of regex strings — visible and auditable
+- [ ] An env var `AGENTCACHE_REDACT_PATTERNS` (comma-separated regex strings) appends additional patterns at startup
+- [ ] `functions.py` no longer contains the scrubbing regex list — it imports and calls `privacy.scrub()`
+- [ ] Unit tests cover: API key pattern, bearer token pattern, custom pattern via argument, no false positives on safe text
+- [ ] All existing tests pass

.scratch/arch-improvements/issues/03-auth-middleware.md

@@ -0,0 +1,16 @@
+# 03 — Centralise auth middleware
+
+**Blocked by:** None — can start immediately
+**Status:** ready-for-agent
+
+## What to build
+
+Replace the copy-pasted `_check_auth()` function that currently lives independently in every route blueprint (`mcp.py`, `observations.py`, `memories.py`, `health.py`, and others) with a single `require_auth` decorator defined in a new `auth.py` module. Every route that previously called `_check_auth()` at the top of its handler switches to the `@require_auth` decorator. The auth logic (timing-safe `hmac.compare_digest` Bearer token check) is identical — this is purely a deduplication. RBAC can be added in one place in a future ticket.
+
+## Acceptance criteria
+
+- [ ] `auth.py` exists with a `require_auth(f)` decorator that performs the timing-safe Bearer token check
+- [ ] Every route blueprint imports and uses `@require_auth` — no blueprint defines its own `_check_auth`
+- [ ] A request with no secret configured passes through (existing behaviour preserved)
+- [ ] A request with a wrong token still gets a `401` response
+- [ ] All existing tests pass

.scratch/arch-improvements/issues/04-mcp-tool-registry.md

@@ -0,0 +1,17 @@
+# 04 — MCP tool registry (replace elif chain)
+
+**Blocked by:** 03 — centralise auth middleware
+**Status:** ready-for-agent
+
+## What to build
+
+Replace the 300-line `if/elif` dispatch chain in `routes/mcp.py` with a `@register("tool_name")` decorator registry. Each of the 30+ MCP tool handlers becomes a standalone function decorated with `@register`. The dispatcher becomes a two-liner: look up the tool name in the registry dict and call the handler. Auth is handled by the shared `@require_auth` decorator from ticket 03. The tool schema list (`get_mcp_tools_schemas`) stays as-is. No change to the MCP wire format or agent behaviour.
+
+## Acceptance criteria
+
+- [ ] A `_tools: dict[str, Callable]` registry exists and a `@register(name)` decorator populates it
+- [ ] Every MCP tool handler is a standalone function — not an inline block inside a giant if/elif
+- [ ] The POST `/mcp/tools` handler body is ≤ 20 lines (lookup + call + error handling)
+- [ ] Each tool handler function is independently importable and callable in a test without starting Flask
+- [ ] Adding a new tool requires only adding a new decorated function — no editing of existing dispatch code
+- [ ] All existing MCP tool calls produce the same responses as before

.scratch/arch-improvements/issues/05-split-observations-memories.md

@@ -0,0 +1,23 @@
+# 05 — Split god module batch 1: `observations.py` + `memories.py`
+
+**Blocked by:** 01 — AppContext dataclass (Expand)
+**Status:** ready-for-agent
+
+## What to build
+
+Move the first two major domain areas out of `functions.py` into focused modules, using the domain vocabulary from `UBIQUITOUS_LANGUAGE.md`.
+
+**`observations.py`** receives: `folder_observe()`, `observe()` (legacy), `dedup_folder_observations()`, `build_synthetic_compression()`, `infer_type()`, `extract_files()`, `extract_image()`, image store helpers (`save_image_to_disk`, `delete_image`, `touch_image`, `is_managed_image_path`), and `normalize_folder_path()` / `validate_agent_id()`.
+
+**`memories.py`** receives: `remember()`, `forget()`, memory versioning logic, and `jaccard_similarity()`.
+
+`functions.py` re-exports everything from these two modules so all existing callers continue to work unchanged. CI must stay green. This is the first batch of the expand phase of the god-module split.
+
+## Acceptance criteria
+
+- [ ] `observations.py` and `memories.py` exist as standalone modules
+- [ ] All moved functions accept `AppContext` (from ticket 01) instead of reaching into module-level globals, while also accepting the legacy `kv: StateKV` signature for backward compat
+- [ ] `functions.py` re-exports every moved symbol — no call site outside `functions.py` needs to change
+- [ ] Existing route blueprints and workers continue to import from `functions` without modification
+- [ ] All existing tests pass
+- [ ] No logic is changed — this is a pure relocation

.scratch/arch-improvements/issues/06-split-indexing-retrieval.md

@@ -0,0 +1,24 @@
+# 06 — Split god module batch 2: `indexing.py` + `retrieval.py`
+
+**Blocked by:** 05 — split batch 1 (observations + memories)
+**Status:** ready-for-agent
+
+## What to build
+
+Move the second major domain area out of `functions.py`.
+
+**`indexing.py`** receives: `IndexPersistence`, `rebuild_index()`, `backfill_obs_lookup_if_needed()`, `vector_index_add_guarded()`, `clip_embed_input()`, and all index-related helpers. This module owns the Index Rebuild lifecycle (see `UBIQUITOUS_LANGUAGE.md`).
+
+**`retrieval.py`** receives: `folder_search()`, `folder_timeline()`, `compile_context()`, and `export_data()`. This module owns the Recall, Smart Search, Folder Search, Timeline, and Context Compilation operations.
+
+`functions.py` continues to re-export everything. CI stays green.
+
+## Acceptance criteria
+
+- [ ] `indexing.py` and `retrieval.py` exist as standalone modules
+- [ ] All moved functions use `AppContext` from ticket 01 rather than reaching into globals directly
+- [ ] `functions.py` re-exports every moved symbol unchanged
+- [ ] `IndexPersistence` is importable from `indexing` in `workers.py` without changing any other worker code
+- [ ] `folder_search()` and `compile_context()` are callable from a test by constructing an `AppContext` with an in-memory KV and a fresh `SearchIndex` — no Flask app required
+- [ ] All existing tests pass
+- [ ] No logic is changed

.scratch/arch-improvements/issues/07-split-consolidation-slots-lessons.md

@@ -0,0 +1,25 @@
+# 07 — Split god module batch 3: `consolidation.py` + `slots.py` + `lessons.py`
+
+**Blocked by:** 06 — split batch 2 (indexing + retrieval)
+**Status:** ready-for-agent
+
+## What to build
+
+Move the final major domain areas out of `functions.py`.
+
+**`consolidation.py`** receives: `consolidate()`, `auto_forget()`, `folder_graph_build()`, and all LLM-call helpers used exclusively by the Consolidation pipeline. This module owns the Consolidation lifecycle (Working Memory → Episodic → Semantic → Procedural).
+
+**`slots.py`** receives: all Slot CRUD functions (`get_slots`, `set_slot`, `delete_slot`, `reflect_slot`, `append_slot`, etc.).
+
+**`lessons.py`** receives: all Lesson CRUD functions (`get_lessons`, `save_lesson`, `search_lessons`, `strengthen_lesson`, lesson decay logic).
+
+`functions.py` re-exports everything. After this ticket, `functions.py` is a pure shim — no domain logic remains in it.
+
+## Acceptance criteria
+
+- [ ] `consolidation.py`, `slots.py`, and `lessons.py` exist as standalone modules
+- [ ] `functions.py` contains no domain logic — only re-export statements
+- [ ] All moved functions use `AppContext` from ticket 01
+- [ ] `auto_forget()` is callable from a test without importing `functions` directly
+- [ ] All existing tests pass
+- [ ] No logic is changed

.scratch/arch-improvements/issues/08-delete-functions-shim.md

@@ -0,0 +1,23 @@
+# 08 — Contract: delete `functions.py` shim, migrate all callers to direct imports
+
+**Blocked by:** 05, 06, 07 — all three split batches must be complete
+**Status:** ready-for-agent
+
+## What to build
+
+Delete `functions.py` entirely. Update every import across `routes/`, `workers.py`, `app.py`, and the test suite to point directly to the focused module that owns the symbol. This is the contract step of the expand–contract sequence. After this ticket the domain vocabulary from `UBIQUITOUS_LANGUAGE.md` is directly reflected in the module names — there is no god module.
+
+Import mapping (non-exhaustive):
+- `from .functions import folder_observe` → `from .observations import folder_observe`
+- `from .functions import remember, forget` → `from .memories import remember, forget`
+- `from .functions import folder_search, compile_context` → `from .retrieval import folder_search, compile_context`
+- `from .functions import IndexPersistence, rebuild_index` → `from .indexing import IndexPersistence, rebuild_index`
+- `from .functions import consolidate, auto_forget` → `from .consolidation import consolidate, auto_forget`
+
+## Acceptance criteria
+
+- [ ] `functions.py` does not exist in the repository
+- [ ] `rg "from .functions import\|from agentcache.functions import\|import functions"` returns zero results in `src/`
+- [ ] All existing tests pass with imports updated
+- [ ] No test imports from `functions` — each test imports from the specific module it exercises
+- [ ] CI is green

.scratch/arch-improvements/issues/09-typed-observations-table-expand.md

@@ -0,0 +1,19 @@
+# 09 — Add typed `observations` table to SQLite (Expand)
+
+**Blocked by:** 05 — observations module must exist to own this migration
+**Status:** ready-for-agent
+
+## What to build
+
+Add a typed `observations` table to the SQLite schema with columns for `folder`, `agent`, `timestamp`, `type`, `importance`, and `text`. When a new Observation is ingested via `folder_observe()`, write it to both the new typed table and the existing `kv_store` scope (dual-write). Reads still use `kv_store` — zero behaviour change for any query path. This is the expand step that makes SQL-level filtering possible without breaking anything.
+
+The new table enables: `WHERE folder = ? AND agent = ? AND timestamp > ?` and `ORDER BY importance DESC LIMIT N` entirely at the SQLite layer — no Python-side filtering needed.
+
+## Acceptance criteria
+
+- [ ] `observations(id TEXT PK, folder TEXT, agent TEXT, timestamp TEXT, type TEXT, importance INTEGER, text TEXT)` table exists in the DB schema with indexes on `(folder, agent, timestamp)` and `(importance)`
+- [ ] `folder_observe()` dual-writes: one row to `observations`, one entry to `kv_store` (existing path)
+- [ ] The DB migration runs automatically on startup if the table does not exist (no manual step)
+- [ ] All reads continue to use `kv_store` — no query is changed in this ticket
+- [ ] All existing tests pass
+- [ ] A DB integrity test verifies that for each `kv_store` Observation entry written, a matching row exists in `observations`

.scratch/arch-improvements/issues/10-typed-observations-reads-contract.md

@@ -0,0 +1,18 @@
+# 10 — Migrate Observation reads to typed table (Contract)
+
+**Blocked by:** 09 — typed observations table (Expand) must be complete
+**Status:** ready-for-agent
+
+## What to build
+
+Switch `folder_search`, `folder_timeline`, `compile_context`, and `auto_forget` to read Observations from the typed `observations` table using SQL-level filtering, replacing the current pattern of `kv.list(scope)` + Python-side filtering. Stop dual-writing new Observations to `kv_store` — the typed table is now the single source of truth. A one-time migration backfills any existing `kv_store` Observation entries into the typed table for users upgrading from a previous version.
+
+## Acceptance criteria
+
+- [ ] `folder_timeline()` issues a single `SELECT … WHERE folder=? AND agent=? ORDER BY timestamp DESC LIMIT ?` — no Python-side filtering loop
+- [ ] `folder_search()` hydrates candidates from the typed table rather than `kv.list(scope)` for the Observation load step
+- [ ] `compile_context()` fetches recent Observations using `ORDER BY importance DESC, timestamp DESC LIMIT ?` at the SQL layer
+- [ ] New `folder_observe()` calls write only to the typed table (no `kv_store` Observation scope write)
+- [ ] A backfill function runs once on startup to migrate existing `kv_store` Observations into the typed table
+- [ ] A benchmark test (or manual note in the PR) shows query latency improvement at ≥ 10k Observations
+- [ ] All existing tests pass

.scratch/arch-improvements/issues/11-basevectorindex-protocol.md

@@ -0,0 +1,17 @@
+# 11 — `BaseVectorIndex` protocol + `InMemoryVectorIndex` adapter
+
+**Blocked by:** 06 — indexing module must exist before its interface is formalised
+**Status:** ready-for-agent
+
+## What to build
+
+Define a `BaseVectorIndex` Protocol (or ABC) in `indexing.py` with three methods: `add(obs_id, session_id, embedding)`, `remove(obs_id)`, and `search(query_embedding, limit) -> list`. Rename the current flat-list implementation to `InMemoryVectorIndex` and have it implement `BaseVectorIndex`. Update `AppContext.vector` to be typed as `BaseVectorIndex`. Behaviour is completely unchanged — this ticket only cuts the adapter seam so future backends (HNSW, Qdrant, Chroma) can be swapped in without touching `retrieval.py` or `consolidation.py`.
+
+## Acceptance criteria
+
+- [ ] `BaseVectorIndex` Protocol exists in `indexing.py` with `add`, `remove`, and `search` as the only required methods
+- [ ] `InMemoryVectorIndex` implements `BaseVectorIndex` and passes a `isinstance(idx, BaseVectorIndex)` check
+- [ ] `AppContext.vector` is typed as `BaseVectorIndex | None`
+- [ ] A test constructs a minimal stub that implements `BaseVectorIndex` and passes it to `folder_search()` via `AppContext` — confirming the seam is real and injectable
+- [ ] `VectorIndex` (old name) remains as an alias for `InMemoryVectorIndex` for one release to avoid breaking any external imports
+- [ ] All existing tests pass with no behaviour change

SPIKE.md

@@ -0,0 +1,578 @@
+# Feature Spike: Small Improvements for Agent Cache User Experience
+
+**Date:** 2026-07-16  
+**Priority:** Low risk, high value additions to current architecture  
+**Tech Stack Impact:** None — all fit within Flask/SQLite model
+
+---
+
+## Table of Contents
+1. [Current State Analysis](#current-state-analysis)
+2. [Feature Opportunities](#feature-opportunities)
+3. [Recommendation Priority](#recommendation-priority)
+
+---
+
+## Current State Analysis
+
+### What agentcache-python Already Has
+
+**Core Features Implemented:**
+- REST API (Flask framework on port 3111)
+- WebSocket streaming (`/stream/mem-live/viewer`)
+- Hybrid search: BM25 keyword + vector embeddings (Gemini/OpenAI/SentenceTransformer)
+- Folder-based memory model: `(folder_path, agent_id)` scoping
+- Knowledge graph construction via `folder_graph_build()`
+- MCP tools for agent integration (13+ tools)
+- Security redactions in observations (`strip_private_data()`)
+- Token budget limits during context compilation
+- Background workers for index maintenance
+
+**Key Functions:**
+| Function | Purpose | File |
+|----------|---------|------|
+| `folder_observe()` | Ingest observations with scoping | `functions.py` |
+| `folder_search()` | BM25 + vector hybrid search | `search.py` |
+| `folder_timeline()` | Chronological activity feed | `functions.py` |
+| `folder_graph_build()` | Build knowledge graph nodes/edges | `functions.py` |
+| `remember()/forget()` | Global memory management | `functions.py` |
+| `context()` | Compile contextual blocks with token limits | `functions.py` |
+| `export_data()` | v2 JSON export format | `functions.py` |
+
+**Key Tests:**
+- `test_observe_core.py` — Observation validation and redaction
+- `test_folder_observe.py` — Folder-scoped ingestion
+- `test_search.py` — BM25/vector search logic
+- `test_context.py` — Context compilation with token budgets
+- `test_graph.py` — Knowledge graph construction
+
+---
+
+## Feature Opportunities
+
+### 🎯 Feature 1: Observation Prioritization / Weighting
+
+**Why Add This?**
+Users naturally write observations with different levels of importance. Currently all observations are weighted equally in search results. The search uses RRF (Reciprocal Rank Fusion) which already applies weights, but there's no user-controlled way to emphasize important observations.
+
+**What to Implement:**
+```python
+def folder_observe(kv, payload, priority=0):  # 0-100 scale
+    
+    # Store in metadata:
+    obs["priority"] = priority
+    
+    return {
+        "observationId": fobs_{hash}_{rand},
+        "priority": priority,  # Added field
+        ...
+    }
+```
+
+**Changes Required:**
+- Add `priority` field to observation payload validation (new required/optional check)
+- Modify search weighting in `HybridSearch._rank()` to include observation priority:
+  ```python
+  # Current RRF: rank_weight = 1 / len(relevant_docs)
+  # New RRF: combined_weight = (bm25_score + vector_score) * obs_priority / 100
+  ```
+
+**User Experience:**
+When a user writes an observation, they can optionally mark it as high-priority:
+```python
+# Tag this as critical issue - show prominently in search results
+folder_observe(
+    kv, {
+        "folderPath": "/home/user/project/src/auth",
+        "agentId": "my-ai-agent",
+        "text": "JWT rotation breaks production auth endpoints",
+        "timestamp": now(),
+        "priority": 90  # Highest importance
+    }
+)
+```
+
+Search results for "jwt auth break" automatically rank this observation higher than lower-priority memories.
+
+**Implementation Complexity:** **15 minutes**  
+- Add one field to observation validation schema  
+- Modify RRF weighting formula (single math operation change)  
+- Update tests to verify priority affects ranking
+
+---
+
+### 🎯 Feature 2: Observation Source Tracking
+
+**Why Add This?**
+Users write observations from different contexts (code files, documentation, terminal logs). Knowing which file context an observation came from helps detect when user re-wrote same code after learning a lesson, and aids in identifying relevant follow-up actions.
+
+**What to Implement:**
+```python
+def folder_observe(kv, payload, source_file=None):  # e.g., "src/auth.py"
+    
+    obs["sourceFile"] = source_file or None
+    
+    return {
+        "observationId": fobs_{hash}_{rand},
+        "sourceFile": source_file,  # New field
+        "text": "...",
+        ...
+    }
+```
+
+**Changes Required:**
+- Add `source_file` to observation payload validation (optional string)
+- Store as additional JSON field in KV value column
+- Update timeline query to optionally show source files
+- Update search results to surface source file info
+
+**User Experience:**
+Search: "What did I learn about JWT?"  
+Results:
+```json
+{
+  "id": "fobs_abc123",
+  "text": "JWT needs refresh tokens for rotation",
+  "sourceFile": "src/auth.py",
+  "timestamp": "2026-07-15T10:30:00Z"
+}
+```
+
+The source file field tracks which files were being developed when lessons were learned. Helps detect when user re-implements same pattern after reading a tutorial, or helps identify which modules need updates based on recent observations.
+
+**Implementation Complexity:** **15 minutes**  
+- Add one optional field to observation schema  
+- Store in existing KV structure (no new table/column needed)  
+- Update `folder_timeline()` to surface source file info when requested  
+
+---
+
+### 🎯 Feature 3: Folder-Level Statistics Endpoint
+
+**Why Add This?**
+Users need quick stats on individual folders without going to the dashboard UI. The current `/health` endpoint only returns global aggregate across all folders. Users often want: "How many observations do I have in my frontend folder?" answered via simple API call.
+
+**What to Implement:**
+```python
+# src/agentcache/routes/folder_stats.py or append to existing routes
+
+@app.route('/folder/stats/<path:folder_path>')
+def folder_stats(folder_path):
+    kv = StateKV()  # Get current DB instance
+    
+    # Count observations for this folder
+    obs_count = 0
+    try:
+        from agentcache.functions import KV
+        scope = KV.folder_obs(folder_path, os.getenv("AGENT_ID", "default"))
+        rows = kv.db.execute(
+            "SELECT COUNT(*) FROM kv_store WHERE scope = ?",
+            [scope]
+        ).fetchone()
+        obs_count = rows[0] if rows else 0
+    except:
+        pass
+    
+    # Get last observation timestamp
+    last_obs_ts = None
+    try:
+        rows = kv.db.execute(
+            "SELECT MAX(ts) FROM kv_store WHERE scope = ?",
+            [scope]
+        ).fetchone()
+        last_obs_ts = rows[0] if rows else None
+    except:
+        pass
+    
+    return {
+        "folderPath": folder_path,
+        "observationCount": obs_count,
+        "memoryCount": 123,  # Get from memories scope separately
+        "lastObsTimestamp": last_obs_ts.isoformat() if last_obs_ts else None,
+        "foldersKnownToThisAgent": ["home/user/projectX", ...]  # Also include this context
+    }
+```
+
+**User Experience:**
+REST API call returns folder stats instantly:
+```bash
+curl http://localhost:3111/folder/stats/home/user/my-project
+# Returns:
+{
+  "folderPath": "home/user/my-project",
+  "observationCount": 42,
+  "memoryCount": 5,
+  "lastObsTimestamp": "2026-07-15T14:23:00Z",
+  "foldersKnownToThisAgent": ["home/user/my-project"]
+}
+```
+
+**Implementation Complexity:** **10 minutes**  
+- Create new endpoint following existing Flask pattern (like `/health`)  
+- Query existing KV methods for observation and memory counts  
+- Return simple JSON response  
+
+**Note:** This requires importing `StateKV` instance in new route file, which means either creating a module-scoped singleton per folder or passing the current `kv` from `app.py` via Flask's application context.
+
+---
+
+### 🎯 Feature 4: Observation Expiration / TTL (Time-To-Live)
+
+**Why Add This?**
+Temporal observations become outdated over time. Example: "Error on port 3000" observation becomes irrelevant after user changes their app to use port 8080. Currently all observations persist indefinitely. With TTL, expired observations can auto-remove from search results or be marked as "expired".
+
+**What to Implement:**
+```python
+def folder_observe(kv, payload, ttl_seconds=None):
+    
+    # Store observation with expiration timestamp
+    obs["expiryTs"] = ts + ttl_seconds if ttl_seconds else None
+    
+    return {
+        "observationId": fobs_{hash}_{rand},
+        "ttl": ttl_seconds or null,  # New field
+        ...
+    }
+```
+
+**Changes Required:**
+- Add `ttl_seconds` to observation payload validation (optional)
+- Store as additional timestamp field in KV value column
+- Modify `folder_timeline()` to filter out expired observations by default
+- Optional: Add background worker that sweeps/expunges expired observations
+
+**User Experience:**
+```python
+# Observation expires after 24 hours
+folder_observe(
+    kv, {
+        "folderPath": "/home/user/app/src/server",
+        "agentId": "my-agent",
+        "text": "Default port is 3000 for development",
+        "timestamp": now(),
+        "ttl": 86400  # 24 hours
+    }
+)
+
+# Timeline query automatically excludes expired observations
+timeline(kv, folder_path="/home/user/app/src/server")
+```
+
+Returns timeline without "port 3000" observation after 24 hours. Users can also check for specific expiry status on an observation with additional endpoint `/folder/observation/status/<id>`.
+
+**Implementation Complexity:** **25 minutes**  
+- Add TTL field to observation payload validation  
+- Store as extra JSON field in KV (existing table schema)  
+- Modify `folder_timeline()` query to include optional TTL filter:
+  ```python
+  ts > now() - ttl_seconds if ttl_seconds else None AND expiryTs > ts
+  ```  
+- Update search results to exclude expired observations  
+- Optional: Background worker sweeps for cleanup (simpler version doesn't need this)  
+
+**Risk:** Medium — requires changes to timeline query logic which users expect returns chronological activity. Must ensure backward compatibility with existing payloads that don't include TTL field (use `None` default).
+
+---
+
+### 🎯 Feature 5: Memory Version History
+
+**Why Add This?**
+Long-term memories evolve. Currently `remember()` just overwrites the old memory. Users might accidentally overwrite important information or can't undo an incorrect memory. With versioning, each remember creates a new version tracked sequentially.
+
+**What to Implement:**
+```python
+def remember(kv, data, skip_versioning=False):
+    
+    # Check existing memories for this content
+    existing_memories = kv.get(KV.memories, "mem_abc123")
+    
+    if not skip_versioning and existing_memories:
+        # Append version suffix to old content (preserve as text)
+        current_content = json.loads(existing_memories["value"])
+        current_content["_version"] = len(current_content.get("_versions", [])) + 1
+        current_content["_createdTs"] = existing_memories["ts"]
+        
+        # Store in new version field
+        versions = current_content.get("_versions", [])
+        current_content["_versions"].append({
+            "content": current_content["content"],
+            "version": len(versions),
+            "ts": now(),
+            ...
+        })
+    
+    return {
+        "memoryId": new_id,
+        "versions": [...],  # New field: list of version history
+        ...
+    }
+```
+
+**Changes Required:**
+- Modify `remember()` to check for existing content match in KV memories
+- Append new version as JSON object to version history array  
+- Store version number and timestamp for each version change
+- Optionally expose version history endpoint for viewing past versions
+
+**User Experience:**
+When a user calls:
+```python
+remember(
+    kv, {
+        "content": "JWT authentication uses bearer token in Authorization header",
+        "concepts": ["jwt", "auth"],
+        ...
+    }
+)
+```
+
+The response includes version history showing how this memory has evolved over time. Later, if the user discovers more context:
+```python
+remember(
+    kv, {
+        "content": "JWT authentication uses Bearer token and JWT_SECRET key in Authorization header",
+        ...
+    }
+)
+```
+
+They can review past versions to see the evolution or optionally revert to earlier version by storing it again.
+
+**Implementation Complexity:** **20 minutes**  
+- Modify `remember()` to detect duplicate content (hash match on content field)  
+- Store version history as JSON array in KV value column (existing table schema, just more data)  
+- Optionally create endpoint `/memory/version/<id>` for viewing history  
+- Update tests to verify version history is stored correctly  
+
+**Risk:** Low — simple append operation to existing data structure. No schema migration needed since we're adding fields to existing JSON values. Backward compatible since previous versions without `_versions` field just get overwritten (which may be surprising behavior, but acceptable for safety).
+
+---
+
+### 🎯 Feature 6: Search Result Citation with Source Links
+
+**Why Add This?**
+When a memory is returned from search, users can see where the original observation came from and what code context had it observed. This helps understand which files are most relevant to certain topics and provides better "view in context" capability for follow-up actions.
+
+**What to Implement:**
+```python
+def folder_observe(kv, payload):
+    # Store source file info
+    if payload.get("sourceFile"):
+        obs["sourceFile"] = payload["sourceFile"]
+        obs["lineNumber"] = payload.get("lineNumber") or None
+    
+    return {
+        "observationId": fobs_{hash}_{rand},
+        "sourceFile": "src/auth.py",  # New field: where this came from
+        ...
+    }
+```
+
+**Search Result Enhancement:**
+Modify search result response format:
+```python
+def folder_search(kv, query, limit, folder_path, agent_id):
+    
+    results = [
+        {
+            "id": obs_id,
+            "text": "...",
+            "folderPath": "/home/user/project",
+            "score": 0.85,  # Original BM25 score or vector similarity
+            "sourceFile": "src/auth.py",   # NEW: where this came from
+            "timestamp": ts.isoformat(),
+            ...
+        }
+    ]
+```
+
+**User Experience:**
+Search query: "JWT authentication"  
+Results now show source file links:
+```json
+[
+  {
+    "id": "fobs_abc123",
+    "text": "JWT needs refresh tokens for token rotation",
+    "sourceFile": "src/auth.py",  // Click link to see code context!
+    "score": 0.92,
+    ...
+  },
+  {
+    "id": "fobs_def456",  
+    "text": "Using 'jwt' import from auth package",
+    "sourceFile": "src/main.py",  // Another file
+    "score": 0.78,
+    ...
+  }
+]
+```
+
+**Implementation Complexity:** **20 minutes**  
+- Add `source_file` field to observation schema (same as Feature 2)  
+- Update search result formatting in `folder_search()` to include source file info from observations  
+- Optionally: Show which files are most-relevant based on sources of highest-scoring memories  
+
+**Risk:** Low — just need to read the new field from observations and format it in search results. No complex data structure change needed.
+
+---
+
+### 🎯 Feature 7: Memory Impact / Related Memories Endpoint
+
+**Why Add This?**
+When a user remembers something important, they might want to know which other memories relate semantically or contextually. Currently all memories exist independently with no explicit linking mechanism. Adding semantic similarity scores helps users understand how new memories connect to existing knowledge and spot conflicts.
+
+**What to Implement:**
+```python
+def memory_related(kv, memory_id):
+    
+    # Get original memory content from KV
+    original_memory = kv.get(KV.memories, memory_id)
+    original_content = json.loads(original_memory["value"])
+    
+    # Search for semantically similar memories
+    search_results = hybrid_search.search(
+        query=original_content["content"],
+        limit=5,  # Return top 5 related memories
+        filter_scope=[KV.memories],  # Only search within memories scope
+    )
+    
+    # Score by similarity + rank in results list
+    return {
+        "memoryId": memory_id,
+        "relatedMemories": [
+            {
+                "id": "mem_abc123",
+                "score": 0.92,  # Semantic similarity score (0-1)
+                "reason": "Same concepts: JWT, authentication",
+                ...
+            },
+            ...
+        ]
+    }
+```
+
+**User Experience:**
+Request: `/agentcache/memories/related/mem_xyz123`  
+Response shows how this memory connects to existing knowledge:
+```json
+{
+  "memoryId": "mem_xyz123",
+  "relatedMemories": [
+    {
+      "id": "mem_abc456",
+      "score": 0.94,
+      "reason": "Same concepts (JWT) and file context (src/auth.py)",
+      "content": "JWT uses Bearer token format in Authorization header"
+    },
+    {
+      "id": "mem_def789",
+      "score": 0.78,
+      "reason": "Related to auth module but different concepts",
+      "content": "Using Express.js for REST API development"
+    }
+  ],
+  ...
+}
+```
+
+**Implementation Complexity:** **30 minutes**  
+- Use existing `HybridSearch.search()` method with original memory content as query  
+- Score by rank in results (1st highest = most related) + cosine similarity score  
+- Format response as JSON with reason descriptions generated from concept extraction  
+- Create new endpoint that takes memory ID and returns related items  
+
+**Risk:** Medium — requires leveraging existing search infrastructure. Must ensure the semantic search is fast enough (< 50ms response time). Also need to handle empty or non-existent memories gracefully (return helpful error).
+
+---
+
+## Recommendation Priority
+
+As a user who relies on this tool daily for code context and observation management, here's my prioritization based on "bang for buck" — quick implementation with high utility:
+
+### **Priority 1 (Implement First): Observation Weighting**
+
+**Why #1:** Fastest to implement (~10 min), highest immediate value
+- Let me tag observations as priority 90-100 and have them surface prominently in search results  
+- No new data structures needed; just add one number to the existing weighting formula
+- Users already write some observations as "important issues" but they're weighted same as trivial notes
+
+**Implementation:**
+```python
+# New payload validation
+def folder_observe(kv, payload, priority=0):
+    # Validate priority is 0-100 range
+    if not (0 <= priority <= 100):
+        raise ValueError("priority must be between 0 and 100")
+```
+
+**Time:** ~10 min  
+**Risk:** None — modifies existing RRF function slightly  
+
+---
+
+### **Priority 2: Folder Statistics Endpoint**
+
+**Why #2:** Simplest implementation with high utility for daily users
+- Users constantly need this info but it's not easily available via API calls
+- Just needs a few more lines of code following Flask pattern  
+- Query existing data without any schema changes
+
+**Implementation:**
+```python
+@app.route('/folder/stats/<path:folder_path>')
+def folder_stats():
+    # ~5 min implementation
+```
+
+**Time:** ~5 min  
+**Risk:** None — new endpoint, minimal queries  
+
+---
+
+### **Priority 3: Observation Source Tracking**
+
+**Why #3:** Low risk, very useful for detecting patterns
+- Track which files user has observed about certain topics helps detect learning cycles  
+- One extra field in KV value dict — no schema changes  
+- Helps spot when user re-implements same code after reading tutorial
+
+**Implementation:**
+```python
+def folder_observe(kv, payload, source_file=None):
+    # ~10 min implementation
+```
+
+**Time:** ~15 min  
+**Risk:** Low — one optional field addition  
+
+---
+
+### **Priority 4: Memory Version History**
+
+**Why #4:** Good safety feature, relatively simple to implement
+- Prevents accidental data loss  
+- Version history array in JSON value column is trivial to add  
+- Users love being able to see evolution of memories they've written
+
+**Implementation:**
+```python
+def remember(kv, data):
+    # ~15 min implementation
+```
+
+**Time:** ~20 min  
+**Risk:** Low — append operation, no schema changes  
+
+---
+
+## Summary
+
+| Feature | Est. Time | Risk Level | User Impact |
+|---------|-----------|------------|-------------|
+| Observation Weighting | 10 min | None | High (control over search ranking) |
+| Folder Stats Endpoint | 5 min | None | High (quick stats access) |
+| Source Tracking | 15 min | Low | Medium (detect patterns) |
+| Memory Versioning | 20 min | Low | Medium (safety feature) |
+
+All four fit seamlessly into existing Flask/SQLite architecture without requiring any architectural changes or new dependencies. They leverage current infrastructure and enhance the user experience incrementally.

UBIQUITOUS_LANGUAGE.md

@@ -0,0 +1,166 @@
+# Ubiquitous Language — agentcache
+
+> This glossary defines the canonical terms used in agentcache's domain model.
+> Use these exact names in code, docs, PR descriptions, ticket titles, and conversations.
+> "Aliases to avoid" lists terms found in the codebase or tickets that should be replaced.
+> Last updated: July 2026 — incorporates architecture review, ubiquitous language session, and ticket breakdown.
+>
+> **Architecture improvement tickets:** `.scratch/arch-improvements/issues/`
+
+---
+
+## Core Memory Model
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Observation** | A single captured event from an agent's execution — one tool call, one prompt, one file edit — scoped to a Folder Pair. The atomic unit of Working Memory. | `log`, `event`, `entry`, `raw_observation`, `hook event` |
+| **Memory** | A manually saved, long-term insight, decision, or pattern — not derived from a single tool call but from deliberate agent reflection. | `long-term memory`, `global memory`, `insight`, `fact` (when used alone) |
+| **Lesson** | A reusable learning with a confidence score that decays over time; strengthened when the same pattern recurs. | `learning`, `tip`, `rule` |
+| **Slot** | A named, pinned piece of context — manually set or auto-populated — injected into every session start. | `pinned slot`, `memory slot`, `context slot` |
+| **Folder Pair** | The primary scope key: the combination of a `folderPath` and an `agentId` that uniquely identifies one agent's memory for one project directory. | `(folder, agent)`, `folder-agent pair`, `scope key`, `session` (when used to mean this) |
+| **Folder Metadata** | Aggregate statistics for a Folder Pair: observation count, last updated timestamp, and optional summary. | `folder meta`, `FolderMeta` |
+
+---
+
+## Memory Tiers (Consolidation Pipeline)
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Working Memory** | The live, unprocessed stream of Observations for a Folder Pair — raw and ephemeral. | `raw observations`, `session logs` |
+| **Episodic Memory** | A compressed summary of a group of Observations for a Folder Pair, produced by the Consolidation pipeline. | `session summary`, `compressed episode`, `narrative` |
+| **Semantic Memory** | Stable factual knowledge extracted from multiple Episodic Memories by the Consolidation pipeline. | `fact`, `knowledge`, `semantic fact` |
+| **Procedural Memory** | A reusable workflow or decision pattern extracted from repeated cross-session patterns. | `procedure`, `workflow memory`, `pattern` (when used alone) |
+| **Consolidation** | The LLM-driven pipeline that compresses Working Memory → Episodic → Semantic → Procedural. Triggered manually or by the auto-forget sweep. | `crystallize`, `compress`, `summarize` |
+
+---
+
+## Agents & Identity
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Agent** | An AI coding assistant (Claude, Kiro, Cursor, Cline, etc.) that writes Observations and reads Memory. | `client`, `AI`, `bot` |
+| **Agent ID** | A string identity for one agent instance, used as the second key in a Folder Pair. | `agentId`, `agent_id`, `agent name` |
+| **Agent Scope** | The isolation boundary for an Agent's data — either `shared` (sees all agents' data) or `isolated` (sees only its own). | `scope isolation`, `agent namespace` |
+| **MCP Client** | Any tool (Cursor, Claude Desktop, Kiro) that communicates with agentcache using the Model Context Protocol. | `plugin`, `integration`, `agent client` |
+
+---
+
+## Search & Retrieval
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Recall** | A keyword-based search over Observations and Memories using the BM25 index. | `search`, `lookup`, `query` (when BM25 is implied) |
+| **Smart Search** | A hybrid search combining BM25 keyword matching and vector cosine similarity, fused by Reciprocal Rank Fusion (RRF). | `semantic search`, `hybrid search`, `vector search` |
+| **Folder Search** | A Recall or Smart Search scoped to one or more Folder Pairs. | `scoped search`, `project search` |
+| **Timeline** | A chronological, filterable feed of Observations for a Folder Pair. | `activity feed`, `history`, `replay feed` |
+| **Context Compilation** | The process of assembling Slots, Lessons, recent Episodic Memories, and a Profile into a single text block for agent injection at session start. | `context injection`, `context window prep`, `compile_context` |
+
+---
+
+## Lifecycle & Maintenance
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Auto-Forget** | A background sweep that evicts Observations past their TTL or above the per-Folder-Pair cap, optionally running Consolidation before eviction. | `auto_forget`, `TTL sweep`, `pruning`, `cleanup` |
+| **Deduplication** | Removal of Observations whose text fingerprint (SHA-256) matches an existing Observation in the same Folder Pair. | `dedup`, `duplicate removal` |
+| **Privacy Scrub** | The process of stripping secrets, tokens, and API keys from Observation text before storage, using a regex-based policy. | `redaction`, `sanitization`, `strip_private_data` |
+| **Index Rebuild** | A background process that reprocesses all stored Observations and Memories into a fresh BM25 and vector index. | `reindex`, `rebuild_index`, `warm-up` |
+| **Audit Entry** | An append-only record of every write operation, stored in the audit log with agent ID and timestamp. | `audit log`, `commit`, `version` (legacy Dolt term — avoid) |
+
+---
+
+## Knowledge Graph
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Relation** | A directed edge in the Knowledge Graph between two concepts, files, or Folder Pairs. | `graph edge`, `link`, `connection` |
+| **Knowledge Graph** | The set of Relations extracted from Observations, used to augment search retrieval with structural codebase topology. | `graph`, `concept graph`, `project graph` |
+| **Graph Extraction** | The LLM-driven process that reads Observations and produces Relations for the Knowledge Graph. | `graph build`, `relation extraction` |
+
+---
+
+## Architecture & Module Structure
+
+These terms describe the codebase's structural concepts, used in tickets, PRs, and architecture discussions.
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **AppContext** | A single injectable dataclass that packages all runtime state (`kv`, `bm25`, `vector`, `embedder`, `broadcast`) — the replacement for module-level globals. Constructed once in `create_app()` and passed explicitly to every function that needs it. | `globals`, `module state`, `app state` |
+| **Deep Module** | A module with a narrow public interface that hides substantial implementation — the opposite of a shallow wrapper. The goal of every split in the god-module refactor. | `service`, `component`, `helper` |
+| **Adapter Seam** | The interface point where one concrete implementation can be swapped for another (e.g. `BaseVectorIndex` allows `InMemoryVectorIndex` to be replaced by an HNSW adapter). A seam becomes real once two concrete adapters exist. | `interface`, `abstraction layer`, `plugin point` |
+| **Expand–Contract** | The two-phase refactor pattern used for wide changes: first *expand* (add the new form alongside the old so CI stays green), then *contract* (delete the old form once all callers are migrated). | `big bang refactor`, `flag day`, `rewrite` |
+| **God Module** | An anti-pattern: a single module (`functions.py` at 4 583 lines) containing all domain logic, making it impossible to test any concern in isolation. Being eliminated via tickets 01–08. | `utils.py`, `helpers.py`, `core.py` (when used as a catch-all) |
+| **Auth Middleware** | A single `require_auth` decorator in `auth.py` that performs the timing-safe Bearer token check — the replacement for copy-pasted `_check_auth()` functions in every blueprint. | `_check_auth`, `auth function`, `bearer check` |
+| **MCP Tool Registry** | A `dict[str, Callable]` populated by `@register("tool_name")` decorators, replacing the 300-line `if/elif` dispatch chain in `routes/mcp.py`. | `tool dispatcher`, `elif chain`, `tool handler map` |
+| **Privacy Policy** | The documented, extensible set of regex patterns that defines what text is scrubbed before an Observation is stored. Lives in `privacy.py` as `DEFAULT_PATTERNS`. | `regex list`, `scrub rules`, `redaction rules` |
+| **Tracer Bullet** | A narrow, complete vertical slice of work that cuts through every layer (storage, logic, API, tests) and is demoable on its own. The unit of work in the ticket breakdown. | `task`, `story`, `sub-task` |
+
+---
+
+## Infrastructure & Sync
+
+| Term | Definition | Aliases to avoid |
+|------|------------|-----------------|
+| **Folder Path** | The absolute filesystem path of the working directory that scopes a Folder Pair. | `cwd`, `directory`, `project path` |
+| **KV Store** | The SQLite-backed key-value store that persists all domain data as JSON values keyed by `(scope, key)`. | `database`, `store`, `StateKV` |
+| **HF Sync** | The process of backing up the KV Store to a private HuggingFace dataset repo and restoring it on startup. | `sync`, `backup`, `dataset sync` |
+| **Audit High-Water Mark** | The `MAX(id)` of the audit log, used by HF Sync to detect whether any writes have occurred since the last backup. | `HWM`, `sync cursor`, `change marker` |
+| **Index Shard** | A chunk of the serialized BM25 or vector index stored in the KV Store for persistence across restarts. | `index chunk`, `bm25 shard`, `vector shard` |
+
+---
+
+## Relationships
+
+- An **Observation** belongs to exactly one **Folder Pair**.
+- A **Folder Pair** has exactly one **Folder Path** and exactly one **Agent ID**.
+- A **Folder Pair** accumulates many **Observations** and zero or one **Folder Metadata**.
+- **Consolidation** consumes **Observations** from a **Folder Pair** and produces **Episodic Memories**, which in turn feed **Semantic Memory** and **Procedural Memory**.
+- A **Lesson** exists globally — not scoped to a **Folder Pair**.
+- A **Slot** is scoped to either a project (matched by **Folder Path**) or global (all agents).
+- A **Memory** is global — not scoped to a **Folder Pair** — and may link to source **Observations** via `sourceObservationIds`.
+- **Recall** and **Smart Search** operate across all **Folder Pairs** (optionally filtered) and also include **Memories**.
+- **Context Compilation** assembles **Slots** + **Lessons** + **Episodic Memories** + **Profile** for one **Folder Pair** at session start.
+- An **AppContext** is constructed once per process and holds one **KV Store**, one BM25 index, one **Adapter Seam** for the vector index, one embedder, and one broadcast callable.
+- The **Auth Middleware** guards every route — the **MCP Tool Registry** handles dispatch after auth passes.
+- The **Privacy Policy** is applied to every **Observation** before it is written to the **KV Store** or the typed observations table.
+
+---
+
+## Flagged Ambiguities
+
+- **"session"** is used in two conflicting ways:
+  1. *Legacy* — a `Session` object with a `sessionId`, from the pre-folder model (now read-only, used only for migration).
+  2. *Informal* — an agent's working period within a Folder Pair.
+  → Use **Folder Pair** for the scope concept and avoid "session" entirely in new code.
+
+- **"memory"** is used both as the general concept ("agent memory system") and as the specific entity (`Memory` — a saved insight). 
+  → In domain conversations, always qualify: **Working Memory**, **Episodic Memory**, **Semantic Memory**, **Procedural Memory**, or the entity **Memory** (capitalised).
+
+- **"crystallize" vs "consolidate"** — the codebase uses both for the same pipeline (LLM compression of Observations into higher-tier memories).
+  → Canonical term: **Consolidation**. `crystallize` is an alias to retire.
+
+- **"recall" vs "search" vs "smart search"** — three terms for overlapping operations.
+  → **Recall** = BM25 keyword search. **Smart Search** = hybrid BM25 + vector. **Folder Search** = either, scoped to a Folder Pair.
+
+- **"forget" vs "auto-forget" vs "evict"** — used interchangeably in code and docs.
+  → **Forget** = deliberate deletion triggered by an agent. **Auto-Forget** = background TTL sweep. **Eviction** = the act of removing one Observation during Auto-Forget.
+
+---
+
+## Example Dialogue
+
+> **Dev:** "When an agent finishes a task, should it call `remember` or does the system handle that automatically?"
+
+> **Domain expert:** "Those are two different things. `remember` is for **Memories** — things the agent explicitly wants to preserve, like an architecture decision. The system automatically stores every tool call as an **Observation** in the **Folder Pair** via `folder_observe`. **Memories** are always deliberate."
+
+> **Dev:** "So if the agent never calls `remember`, does it lose everything?"
+
+> **Domain expert:** "No — the **Observations** are still there in **Working Memory** for that **Folder Pair**. But they're raw and noisy. **Consolidation** is what turns them into clean **Episodic Memories** that survive **Auto-Forget** eviction."
+
+> **Dev:** "What's the difference between a **Lesson** and a **Semantic Memory**?"
+
+> **Domain expert:** "A **Lesson** is something the agent explicitly learned and wants to reuse — it has a confidence score that decays. **Semantic Memory** is extracted automatically by the **Consolidation** pipeline from patterns across **Episodic Memories**. You save a **Lesson**; the system extracts **Semantic Memory**."
+
+> **Dev:** "And a **Slot** is different from both?"
+
+> **Domain expert:** "Yes. A **Slot** is pinned context — the agent or developer sets it manually and it gets injected at every session start via **Context Compilation**. Think of it as a sticky note, not a memory."

Users/ykanagaraj/Downloads/agentcache/CLAUDE.md

@@ -0,0 +1,53 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with this repository.
+
+## 🚀 Development Commands
+
+### Installation
+To install all required dependencies, run:
+```bash
+pip install -r requirements.txt
+```
+
+### Running the Application
+To start the Flask API server, run:
+```bash
+python src/app.py
+```
+The server defaults to listening on port `3111`.
+
+### Testing
+To run unit and integration tests (using pytest):
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+## 🏛️ High-Level Architecture
+
+The application follows a modular, layered architecture designed around Flask and dynamic data services:
+
+1. **Presentation Layer (Flask App)**: The core is orchestrated by `src/app.py`, which initializes the Flask application. It handles routing via registered blueprints, manages request processing, and sets up network services like WebSockets.
+2. **Data & Persistence Layer (`src/db.py`)**: Data management is handled by `StateKV`, which wraps SQLite to provide namespaced storage for observations, memories, and metadata. This layer ensures data persistence across sessions. Key scopes include `mem:folders` (index), `mem:folder:{path}:{agent}` (observations), `mem:foldermeta:{path}:{agent}` (metadata), `mem:obs_lookup` (O(1) reverse lookup), `mem:memories` (global), and `mem:index:bm25:*` (search).
+3. **Core Logic & Services (`src/functions.py`)**: This module contains all business logic implementations, including observation ingestion (`folder_observe`), timeline generation (`folder_timeline`), memory management functions (`remember`, `forget`), and the abstraction layers for embedding providers and search index operations.
+4. **Search Indexing (`src/search.py`)**: This layer manages two primary search indexes: a **BM25** index (for keyword matching) and a **Vector Index** (for semantic similarity). It dynamically selects an embedding provider (Gemini, OpenAI, or SentenceTransformer) to power the Vector Index, allowing for hybrid (RRF fusion) searches.
+5. **Communication Layer (WebSockets)**: The system supports real-time communication via WebSockets (`flask_sock`). This layer is responsible for broadcasting live events across connected clients, managed by functions in `src/functions.py`.
+
+**Key Architectural Insight:** The architecture relies heavily on dynamic dependency injection. The application dynamically selects the best Embedding Provider based on environment variables (Gemini > OpenAI > Local Model), ensuring flexibility in how semantic search is performed, all orchestrated from the central `create_app()` function.
+
+## MCP Tools (12 active)
+
+The system exposes 12 Memory/Tool endpoints via `/mcp/tools`. These tools provide agents with access to core functionalities like observation ingestion (`agent_observe`), memory recall (`memory_recall`), and persistence operations (`memory_save`, `memory_forget`). Their schemas can be found at `GET /agentcache/mcp/tools`.
+
+## Environment & Configuration Notes
+
+*   **Persistence**: SQLite file lives at `~/.agentcache/agentcache.db`.
+*   **Configuration**: Application settings are loaded from environment variables, often managed via a `.env` file in `~/.agentcache/`. Credentials (like API keys) must be set for secured operations. Key variables include `GEMINI_API_KEY`, `OPENAI_API_KEY`, `AGENTCACHE_SECRET`, and scope filters like `AGENTCACHE_AGENT_SCOPE=isolated`.
+*   **Data Limits**: Observe ingestion is capped at `MAX_OBS_PER_FOLDER` (default 2000). Context compilation is capped by `TOKEN_BUDGET` (default 2000).
+
+## Memory Structure Overview
+
+The system supports folder-based observation scoping, global memories (`remember`/`forget`), and timeline tracking. All persistent data interaction flows through the `src/db.py` StateKV layer to maintain namespaced and auditable storage.
+
+- [CLAUDE.md](file.md) — hook

product_roadmap.md

@@ -0,0 +1,39 @@
+# Project Roadmap: Prototype to Production
+
+This document outlines the phased roadmap for scaling our Memory Server project from a prototype to a robust, production-ready product. Each phase builds upon the last, focusing sequentially on Security, Features, Deployment, and Monitoring.
+
+## Phase 1: Stabilization & Hardening (Security First)
+**Goal:** Eliminate critical vulnerabilities, patch security gaps, and ensure core functionality is robust against common exploits.
+
+### Tasks
+1.  **Security Audit**: Review all authentication logic (`AGENTCACHE_SECRET` usage) across API endpoints to enforce strict access control checks.
+2.  **Input Sanitization**: Implement rigorous input sanitization for all data ingress points (API calls, WebSocket messages) to mitigate SQL injection and XSS risks.
+3.  **Dependency Hardening**: Audit `requirements.txt` and update all dependencies to their latest secure versions, flagging any version downgrade/upgrade that could introduce a vulnerability.
+4.  **Test Baseline**: Run full test suite (`pytest`) and fix every failing test before moving to Phase 2.
+
+## Phase 2: Feature Parity & Depth
+**Goal:** Implement all planned features that were marked as incomplete or missing, bringing the system closer to its intended scope.
+
+### Tasks
+1.  **Memory Deep Dive**: Fully implement complex memory routines like `folder_graph_build` and ensure atomic behavior for `remember`/`forget` operations.
+2.  **Search Optimization**: Optimize the hybrid search component (RRF parameters) and profile database queries to improve retrieval performance.
+3.  **MCP Tool Completion**: Complete functionality for all remaining tools listed in `/mcp/tools`.
+4.  **UX Polish**: Enhance the dashboard viewer (`/viewer`) to provide clear, actionable feedback on system status (e.g., current embedding model, load metrics).
+
+## Phase 3: Productionization & Deployment
+**Goal:** Prepare the application for a stable, scalable production environment.
+
+### Tasks
+1.  **Containerization**: Write optimized Dockerfiles for the Flask application and dependencies to ensure environment parity across environments.
+2.  **Configuration Decoupling**: Refactor configuration loading to strictly adhere to container-native practices (using ENV variables exclusively) rather than relying on local `.env` file structures for production use.
+3.  **Performance Tuning**: Conduct load testing to identify I/O bottlenecks (DB access, network latency). Tune database indices and optimize WebSocket message throughput.
+4.  **Scalability Design**: Analyze the statelessness of components to determine optimal strategies for horizontal scaling (e.g., separate API workers vs. WebSocket workers).
+
+## Phase 4: Monitoring & Iteration (Live Operations)
+**Goal:** Establish continuous monitoring and a feedback loop for post-launch maintenance.
+
+### Tasks
+1.  **Observability Stack**: Implement structured logging across all modules to facilitate easy debugging in production.
+2.  **Metrics Collection**: Integrate Prometheus metrics collection for tracking latency, resource utilization, and error rates.
+3.  **Anomaly Detection**: Configure alerts for performance degradation or excessive error rates in the data access layer.
+4.  **Feedback Mechanism**: Design and implement a simple feedback mechanism for collecting post-launch bug reports and feature requests.

questions.md

@@ -0,0 +1,176 @@
+# Architectural Questions for agentcache
+
+These are design and architecture questions raised by the current codebase.
+They are not criticisms — they are things worth understanding and deciding on
+before the system grows. Each one points to a real trade-off or risk.
+
+---
+
+## 1. Flat KV Schema
+
+The entire data model lives in one SQLite table: `kv_store(scope, key, value TEXT)`.
+Observations, memories, index shards, sessions, lessons, slots, graph edges, and
+image refs all share this single namespace.
+
+- What happens to read performance when this table grows to 100k+ rows?
+- Every "query" loads an entire scope into Python and filters in memory — no
+  SQL-level filtering by timestamp, type, or importance. Is that intentional?
+- Why not give each data type its own table with typed columns, which would
+  enable proper indexes and range queries at the DB level?
+
+---
+
+## 2. In-Memory Index with a Cold-Start Window
+
+The BM25 and vector indexes live entirely in Python process memory. On restart,
+the server either deserializes them from SQLite or rebuilds them from scratch in
+a background thread.
+
+- During the rebuild window, searches return results from an empty or stale
+  index. Users get silently degraded results with no indication.
+- Is there a "index warming" status exposed anywhere? The `/health` endpoint
+  could surface this, but does it?
+- For large datasets, how long does the rebuild take? Is there a bound on that?
+
+---
+
+## 3. Vector Index is a Flat Python List
+
+`VectorIndex.search()` iterates over every stored vector and computes cosine
+similarity one by one. This is O(n) per query.
+
+- At 10k observations this is already slow. At 100k it is unusable.
+- There is no ANN (approximate nearest neighbor) index — no HNSW, no IVF, no
+  external vector DB.
+- The EPIC mentions Qdrant/Chroma as future backends. What is the plan and
+  threshold for switching?
+
+---
+
+## 4. Single Auth Token (No RBAC)
+
+Authentication is one shared secret compared with `hmac.compare_digest`. Every
+client — viewer, agent, admin — uses the same token.
+
+- If the token leaks from one agent config, everything is exposed.
+- The viewer dashboard, the `/forget` endpoint (which can delete all data), and
+  the `/migrate` endpoint all share the same credential level.
+- The EPIC mentions RBAC (viewer/agent/admin tokens). Is there a plan to
+  implement this before exposing the server to a network?
+
+---
+
+## 5. Global Module-Level State
+
+`functions.py` declares `_bm25_index`, `_vector_index`, `_embedding_provider`,
+`_hybrid_search`, and `_stream_broadcaster` as module-level globals. They are
+set once at startup and mutated by every request.
+
+- This makes the application impossible to unit test in isolation (you cannot
+  inject a fresh index per test).
+- It also makes it impossible to run two instances in the same process (e.g.
+  for testing).
+- Was this a deliberate simplicity trade-off, or just how it grew?
+
+---
+
+## 6. No Rate Limiting or Input Size Caps on Most Endpoints
+
+Observations are capped at 4000 characters. But memories, lessons, slots, and
+graph edges have no enforced size limits. The `/agent/observe` endpoint accepts
+arbitrary JSON payloads.
+
+- A misbehaving or compromised agent could flood the server with gigabytes of
+  data and exhaust disk space.
+- There is no per-agent write rate limit, no total storage quota, and no
+  backpressure mechanism.
+- Is this acceptable for the current single-user local use case? What changes
+  if this is exposed as a shared service?
+
+---
+
+## 7. HuggingFace Sync Backs Up the Entire SQLite File
+
+`sync.py` uploads the whole `agentcache.db` file to a HuggingFace dataset repo,
+including the BM25/vector index shards stored inside it.
+
+- The index shards can be large and are fully reproducible from raw observations.
+  Syncing them wastes bandwidth and storage.
+- The backup is not incremental — it uploads the whole file every time the
+  audit high-water mark changes.
+- If the HF token is compromised, all user memory data (observations, memories,
+  lessons) is exposed. Is that risk documented anywhere for users?
+
+---
+
+## 8. `functions.py` is ~2000 Lines — a God Module
+
+All business logic lives in one file: observation ingestion, memory CRUD,
+search orchestration, consolidation, auto-forget, graph building, slot
+management, lessons, image store, index persistence, and privacy scrubbing.
+
+- This makes it hard to test any single concern in isolation.
+- It creates implicit coupling — a change to how observations are stored can
+  accidentally affect how the index is rebuilt or how memories are versioned.
+- What would it look like to split this into focused modules
+  (e.g. `observations.py`, `memories.py`, `indexing.py`, `consolidation.py`)?
+
+---
+
+## 9. Jaccard Similarity for Memory Deduplication
+
+`remember()` checks for duplicate memories by computing Jaccard similarity
+(word overlap) between the new content and all existing memories, with a
+threshold of 0.7.
+
+- This loads all memories into Python memory on every `remember()` call.
+- Jaccard on raw word sets is a weak signal for semantic similarity — two
+  memories about the same concept but phrased differently score low and
+  both get stored.
+- With a vector embedding provider active, why not use cosine similarity
+  against the vector index for deduplication instead?
+
+---
+
+## 10. The MCP Endpoint Duplicates Auth and Dispatch Logic
+
+`routes/mcp.py` has its own `_check_auth()` function, its own `_get_kv()` helper,
+and a large `if/elif` chain dispatching 30+ tool names. The same auth logic
+exists in other route blueprints too.
+
+- Auth is copy-pasted across at least 3 places. A future change (e.g. adding
+  RBAC) would require updating each copy.
+- The tool dispatch chain has no registry — adding a new tool requires editing
+  a long elif block rather than registering a handler.
+- Was there a reason not to use a decorator-based dispatch registry or a shared
+  auth middleware?
+
+---
+
+## 11. Privacy Scrubbing is a Regex List — Not a Policy
+
+`strip_private_data()` removes secrets by matching a hardcoded list of regex
+patterns (API keys, tokens, passwords). It runs on raw observation text before
+storage.
+
+- If an agent logs a secret with an unusual key name (e.g. `my_service_cred`),
+  it passes through unredacted.
+- The scrubbing happens *after* the data is already in the Python process —
+  if the process crashes mid-scrub, the raw data may already be partially
+  visible in a WAL frame.
+- Is the current regex set documented so users can audit and extend it?
+
+---
+
+## 12. No Tests
+
+The `tests/` directory appears to be empty or minimal. For a system that
+persists agent memory and is used as a source of truth for AI context:
+
+- How do you verify that `folder_search` returns the right results after an
+  observation is added and the index is updated?
+- How do you catch regressions in privacy scrubbing when you add a new pattern?
+- How do you verify that the HF sync does not upload stale data after a
+  failed WAL checkpoint?
+- What is the plan for adding test coverage before the system is used in
+  production or shared with other developers?