Initial open source release

.dockerignore

@@ -0,0 +1,24 @@
+.venv/
+__pycache__/
+*.pyc
+*.pyo
+data_raw/
+data_processed/
+embeddings/
+.git/
+.qdrant/
+*.db
+*.index
+*.db-shm
+*.db-wal
+faraday-server/
+scripts/
+00 - Inbox/
+01 - Raw Sources/
+02 - Wiki/
+03 - Data Lake/
+.obsidian/
+*.md
+!README.md
+test_*.py
+results.txt

.gitignore

@@ -0,0 +1,13 @@
+# Python environments
+.venv/
+__pycache__/
+*.pyc
+
+# Local data / DBs
+data_raw/
+data_processed/
+embeddings/
+
+# Keys / Config
+.env
+secret

DATA_INGESTION.md

@@ -0,0 +1,64 @@
+# Faraday Memory: Data Ingestion Guide
+
+Your AI memory system has built-in smart parsers designed to ingest various file formats. You do not need to convert your documents manually — the ingestion pipeline handles extraction, chunking, and vector embedding automatically based on file extensions.
+
+## 📥 Where to Drop Files
+
+You have two primary designated "Drop Zones". The `sync.py push` command recursively scans these directories and all their subfolders:
+
+1. **Your Obsidian Vault's Inbox or Reference Folders**
+   - `00 - Inbox/`
+   - `01 - Raw Sources/`
+2. **The Dedicated Raw Data Directory**
+   - `Faraday/ai-memory-mcp/data_raw/`
+
+## 📄 Supported Formats & Naming Rules
+
+### 1. ChatGPT Exports (JSON)
+*   **Format:** `.json`
+*   **Rule:** The filename **must** contain the word `conversations`.
+*   **Example:** `chatgpt_conversations.json` or `conversations_2025.json`
+*   *Behavior:* Explodes the dump and parses it specifically tracking Human vs AI messages.
+
+### 2. Gemini Exports (HTML / Takeout)
+*   **Format:** `.html`
+*   **Rule:** The filename **must** contain the word `activity` or `gemini`.
+*   **Example:** `My_Gemini_Activity.html`
+*   *Behavior:* Strips out Google tracking headers and parses out prompts and responses cleanly.
+
+### 3. PDF Documents (Coursework, Papers, E-books)
+*   **Format:** `.pdf`
+*   **Rule:** Standard PDF documents (text-based). 
+*   **Example:** `Linear_Algebra_Notes.pdf` or `Attention_Is_All_You_Need.pdf`
+*   *Behavior:* Extracts multi-page text using PDFMiner/PyMuPDF into logical reading chunks.
+
+### 4. Images & Scans (Diagrams, Receipts, Screenshots)
+*   **Format:** `.png`, `.jpg`, `.jpeg`, `.webp`, `.bmp`, `.tiff`
+*   **Rule:** Can be named anything.
+*   **Requirements:** You must have [Tesseract-OCR](https://github.com/UB-Mannheim/tesseract/wiki) installed on your system. 
+*   *Behavior:* Automatically performs Optical Character Recognition (OCR) to rip the text out of the image and inject it into your vector database.
+
+### 5. Plain Text, Emails, Code, CSVs, and Markdown
+*   **Format:** `.md`, `.txt`, `.csv`, `.log`, `.rst`
+*   **Rule:** Can be named anything.
+*   **Example:** `Meeting_Notes.md` or `email_invoice.txt`
+*   *Behavior:* The default fallback. Reads the file as plain structured text and intelligently slices it up. Large CSVs are skipped if they exceed 15MB to prevent clogging the embedding memory.
+
+---
+
+## 🚀 How to Add It to the Cloud
+
+Once your files are dropped into the folders:
+1. Open your terminal in the `ai-memory-mcp` folder.
+2. Run the ingestion and cloud synchronization command:
+   ```bash
+   python sync.py push
+   ```
+3. The script will:
+   - Identify *only* the new files.
+   - Run the appropriate parser (OCR, HTML extractor, etc.).
+   - Break them down into semantic chunks and generate embeddings.
+   - Compress the new database.
+   - Securely upload the compressed database to Supabase.
+   
+Your Claude mobile app (and Antigravity!) will automatically have access to this new data upon their next query!

Dockerfile

@@ -0,0 +1,49 @@
+# ─────────────────────────────────────────────────────
+# Faraday AI Memory — Cloud Run Container
+# ─────────────────────────────────────────────────────
+# Multi-stage build:
+#   1. Pre-download the embedding model at build time
+#   2. Install all Python dependencies
+#   3. Runs cloud_server.py on port 8080
+
+FROM python:3.11-slim AS base
+
+# System dependencies for faiss-cpu
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set working directory
+WORKDIR /app
+
+# Copy requirements first (Docker layer caching)
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Pre-download the embedding model at build time
+# This avoids a ~100MB download on every cold start
+RUN python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"
+
+# Copy application code
+COPY config.py .
+COPY database/ database/
+COPY processing/ processing/
+COPY ingestion/ ingestion/
+COPY mcp_server/ mcp_server/
+
+# Create data directories
+RUN mkdir -p /tmp/faraday-data data_raw data_processed embeddings
+
+# Hugging Face Spaces uses PORT env var (default 7860)
+ENV PORT=7860
+ENV CLOUD_DATA_DIR=/tmp/faraday-data
+
+# Expose the port
+EXPOSE 7860
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
+    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:7860/sse')" || exit 1
+
+# Run the cloud MCP server
+CMD ["python", "mcp_server/cloud_server.py"]

README.md

@@ -0,0 +1,76 @@
+---
+title: Faraday Memory
+emoji: 🧠
+colorFrom: blue
+colorTo: purple
+sdk: docker
+app_port: 7860
+---
+
+# Faraday AI Memory
+
+**Author:** Saurab Mishra
+
+Look, I got tired of switching between five different "AI" coding tools and repeating the same context a million times like a broken record. Claude Code, Cursor, Antigravity, Copilot—they all suffer from the exact same severe, chronic amnesia every time you close the tab. You tell them your tech stack on Monday, and by Tuesday they're confidently rewriting your backend in PHP. 
+
+So I built **Faraday**. 
+
+Faraday is a brutally fast, brutally simple Model Context Protocol (MCP) server that slams your local documents, chat histories, emails, and PDFs directly into a vector database, compresses it, and force-feeds it to whatever vibe-coding AI toy you are currently obsessing over. 
+
+You drop files in a folder, run a script, and suddenly your AI actually remembers who you are and what you're working on. Groundbreaking concept, I know.
+
+---
+
+## What It Actually Does
+
+1. **Ingests literally everything:** Drop `.md`, `.json` (ChatGPT exports), `.html` (Gemini exports), `.pdf`, or even `.png` (images with text) into the `data_raw` folder.
+2. **Chunking & Vectorization:** It chunks the files, runs them through an extremely lightweight open-source embedding model (`all-MiniLM-L6-v2`), and indexes them in FAISS.
+3. **Cloud Synchronization:** Compresses the gigantic, bloated vector indexes down to a few megabytes and shoves them into a private Supabase bucket.
+4. **SSE Server:** Boots up an HTTP/SSE server (perfect for Hugging Face Spaces or Cloud Run) that safely pulls your index from Supabase into memory, hiding it behind a simple API key firewall. 
+
+Your data stays entirely under your control (no relying on bloated SaaS corporate subscriptions).
+
+## The Local Dev Setup (If you want to run it on your own laptop)
+
+If you're deploying this to the cloud, the `Dockerfile` takes care of the heavy lifting. But if you want to run the ingestion and test locally:
+
+### 1. Requirements
+
+Install the stuff. You probably already have half of this globally installed anyway.
+```bash
+pip install -r requirements.txt
+```
+*(Pro-tip: If you want it to actually read the text out of your images, you need to suffer through installing `Tesseract-OCR` on your OS level. Don't blame me, complain to Google).*
+
+### 2. Configuration
+
+Open `config.py`. It's stupidly simple.
+- Put your raw messy files in the `data_raw/` directory.
+- For the Supabase cloud connection, set these environment variables (or hardcode them if you like playing with fire):
+  - `SUPABASE_URL`
+  - `SUPABASE_KEY` (Needs to be the service key to bypass Row Level Security, since we use private storage buckets)
+
+### 3. Update the Brain
+
+Whenever you hoard more documents, just run:
+```bash
+python sync.py push
+```
+It will automatically find the new files, ignore the ones it already did, run the embeddings, compress the database, and shoot it to the cloud.
+
+### 4. Connect to your AI
+
+Connect any MCP-compatible agent directly to the cloud server, or run it locally:
+```bash
+# Local standard I/O (For Cursor / Desktop apps)
+python mcp_server/main.py
+
+# Or connect to the Cloud SSE endpoint
+URL: https://<your-huggingface-space>.hf.space/sse
+Header: X-API-Key: <your_secret_password>
+```
+
+Enjoy not having to constantly remind your AI what programming language you're using.
+
+---
+*No personal data, keys, or vector blobs are included in this repo. I `.gitignore`'d all of it. If you manage to leak your API keys, that's entirely on you.*

config.py

@@ -0,0 +1,110 @@
+"""
+config.py — Central configuration for the AI Memory MCP system.
+
+All paths, model settings, and tuning constants live here.
+Every module imports from this file; nothing is hardcoded elsewhere.
+"""
+
+import os
+from pathlib import Path
+
+# ─────────────────────────────────────────────────────
+# Directory Layout
+# ─────────────────────────────────────────────────────
+
+BASE_DIR = Path(__file__).parent.resolve()
+
+# Where raw exports / documents are dropped for ingestion
+DATA_RAW = BASE_DIR / "data_raw"
+
+# Processed artefacts (SQLite DB, FAISS index)
+DATA_PROCESSED = BASE_DIR / "data_processed"
+
+# FAISS index file
+EMBEDDINGS_DIR = BASE_DIR / "embeddings"
+
+# Obsidian vault root (parent of ai-memory-mcp/)
+OBSIDIAN_VAULT = BASE_DIR.parent
+
+# Obsidian source directories to scan during update
+OBSIDIAN_SCAN_DIRS = [
+    OBSIDIAN_VAULT / "00 - Inbox",
+    OBSIDIAN_VAULT / "01 - Raw Sources",
+    OBSIDIAN_VAULT / "02 - Wiki",
+    OBSIDIAN_VAULT / "03 - Data Lake",
+]
+
+# Directories / patterns to skip during recursive scanning
+SKIP_PATTERNS = [
+    ".qdrant", ".chroma", ".venv", "__pycache__",
+    "faraday-server", "ai-memory-mcp", "node_modules",
+    ".git", ".obsidian",
+]
+
+# Ensure critical dirs exist
+for _d in [DATA_RAW, DATA_PROCESSED, EMBEDDINGS_DIR]:
+    _d.mkdir(parents=True, exist_ok=True)
+
+# ─────────────────────────────────────────────────────
+# Database Paths
+# ─────────────────────────────────────────────────────
+
+SQLITE_DB_PATH = DATA_PROCESSED / "memory.db"
+FAISS_INDEX_PATH = EMBEDDINGS_DIR / "memory.index"
+
+# ─────────────────────────────────────────────────────
+# Embedding Model
+# ─────────────────────────────────────────────────────
+
+EMBEDDING_MODEL = "all-MiniLM-L6-v2"
+EMBEDDING_DIM = 384       # Output dimension for all-MiniLM-L6-v2
+BATCH_SIZE = 64            # SentenceTransformer encode batch size
+
+# ─────────────────────────────────────────────────────
+# Chunking
+# ─────────────────────────────────────────────────────
+
+CHUNK_MAX_WORDS = 300      # Target words per chunk
+CHUNK_OVERLAP_WORDS = 50   # Overlap between adjacent chunks
+CHUNK_MIN_LENGTH = 30      # Minimum characters — skip trivially short chunks
+
+# ─────────────────────────────────────────────────────
+# Search / Scoring
+# ─────────────────────────────────────────────────────
+
+SEMANTIC_WEIGHT = 0.7      # Weight for vector similarity in hybrid score
+RECENCY_WEIGHT = 0.3       # Weight for timestamp recency in hybrid score
+DEFAULT_TOP_K = 5          # Default number of results
+
+# ─────────────────────────────────────────────────────
+# FAISS Tuning
+# ─────────────────────────────────────────────────────
+
+# When total vectors exceed this threshold, rebuild as IVFFlat
+# Below this, IndexFlatIP is fine (brute-force is fast enough)
+FAISS_IVF_THRESHOLD = 10_000
+FAISS_NLIST = 100          # Number of Voronoi cells for IVFFlat
+FAISS_NPROBE = 10          # Cells to search at query time (speed/accuracy tradeoff)
+
+# ─────────────────────────────────────────────────────
+# OCR (optional)
+# ─────────────────────────────────────────────────────
+
+OCR_ENABLED = True         # Set False to skip image ingestion entirely
+# If tesseract is not in PATH, set the full path here:
+# e.g. r"C:\Program Files\Tesseract-OCR\tesseract.exe"
+TESSERACT_CMD = None
+
+# ─────────────────────────────────────────────────────
+# Cloud Sync (optional — Supabase)
+# ─────────────────────────────────────────────────────
+
+SUPABASE_URL = os.environ.get("SUPABASE_URL", "")
+SUPABASE_KEY = os.environ.get("SUPABASE_KEY", "")
+SUPABASE_BUCKET = os.environ.get("SUPABASE_BUCKET", "faraday-memory")
+
+# ─────────────────────────────────────────────────────
+# Logging
+# ─────────────────────────────────────────────────────
+
+LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO")

database/__init__.py

@@ -0,0 +1,3 @@
+"""
+database — Storage layer with SQLite (metadata + FTS5) and FAISS (vectors).
+"""

database/faiss_db.py

@@ -0,0 +1,203 @@
+"""
+database.faiss_db — Production FAISS vector index with adaptive indexing.
+
+Key design decisions:
+  - Uses IndexFlatIP (inner product on L2-normalized vectors) for small datasets.
+  - Automatically trains and rebuilds as IVFFlat when vectors exceed threshold
+    (default 10K), giving O(√N) search instead of O(N).
+  - Deferred disk writes: caller must explicitly call save() — eliminates
+    the O(N²) disk I/O from writing after every batch.
+  - IndexIDMap wrapper maps FAISS positions to SQLite row IDs.
+"""
+
+import os
+import sys
+from pathlib import Path
+from typing import List, Optional, Tuple
+
+import faiss
+import numpy as np
+
+# Import from root config
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from config import (
+    EMBEDDING_DIM,
+    FAISS_INDEX_PATH,
+    FAISS_IVF_THRESHOLD,
+    FAISS_NLIST,
+    FAISS_NPROBE,
+)
+
+
+class VectorDB:
+    """
+    FAISS-backed vector index with adaptive index type selection.
+
+    For < FAISS_IVF_THRESHOLD vectors: brute-force IndexFlatIP.
+    For >= threshold: IVFFlat with configurable nlist/nprobe.
+    """
+
+    def __init__(
+        self,
+        dim: int = EMBEDDING_DIM,
+        index_path: Optional[str] = None,
+    ):
+        self.dim = dim
+        self.index_path = index_path or str(FAISS_INDEX_PATH)
+        self.index: Optional[faiss.Index] = None
+        self._load_or_create()
+
+    def _load_or_create(self):
+        """Load existing index from disk or create a fresh one."""
+        if os.path.exists(self.index_path):
+            self.index = faiss.read_index(self.index_path)
+            print(
+                f"[VectorDB] Loaded FAISS index: {self.index.ntotal} vectors",
+                file=sys.stderr,
+            )
+        else:
+            print("[VectorDB] Creating new FAISS IndexFlatIP.", file=sys.stderr)
+            base = faiss.IndexFlatIP(self.dim)
+            self.index = faiss.IndexIDMap(base)
+
+    def add_embeddings(self, embeddings: np.ndarray, ids: np.ndarray):
+        """
+        Add a batch of embeddings with their corresponding SQLite IDs.
+
+        Args:
+            embeddings: (N, dim) float32 array — MUST be L2-normalized.
+            ids:        (N,) int64 array of SQLite row IDs.
+
+        NOTE: Does NOT write to disk. Call save() explicitly when done.
+        """
+        if embeddings.shape[0] == 0:
+            return
+
+        assert embeddings.shape[1] == self.dim, (
+            f"Dimension mismatch: got {embeddings.shape[1]}, expected {self.dim}"
+        )
+
+        embeddings = embeddings.astype(np.float32)
+        ids = ids.astype(np.int64)
+
+        # L2-normalize for cosine similarity via inner product
+        faiss.normalize_L2(embeddings)
+
+        self.index.add_with_ids(embeddings, ids)
+
+    def search(
+        self, query_embedding: np.ndarray, top_k: int = 5
+    ) -> List[Tuple[int, float]]:
+        """
+        Find the top_k most similar vectors.
+
+        Args:
+            query_embedding: (1, dim) or (dim,) float32 array.
+            top_k: Number of results.
+
+        Returns:
+            List of (sqlite_id, similarity_score) tuples.
+            Score is cosine similarity (higher = better, range 0-1).
+        """
+        if self.index is None or self.index.ntotal == 0:
+            return []
+
+        query_embedding = query_embedding.astype(np.float32)
+        if query_embedding.ndim == 1:
+            query_embedding = query_embedding.reshape(1, -1)
+
+        # Normalize query for cosine similarity
+        faiss.normalize_L2(query_embedding)
+
+        # Set nprobe for IVF indices (no-op for flat indices)
+        try:
+            # Access the underlying IVF quantizer if it exists
+            ivf = faiss.extract_index_ivf(self.index)
+            if ivf is not None:
+                ivf.nprobe = FAISS_NPROBE
+        except Exception:
+            pass
+
+        distances, indices = self.index.search(query_embedding, top_k)
+
+        results = []
+        for i in range(len(indices[0])):
+            idx = int(indices[0][i])
+            score = float(distances[0][i])
+            if idx != -1:  # FAISS returns -1 for missing matches
+                results.append((idx, score))
+
+        return results
+
+    def maybe_rebuild_ivf(self):
+        """
+        If total vectors exceed the IVF threshold, rebuild as IVFFlat
+        for O(√N) search performance. Called after a full update cycle.
+
+        This is an expensive operation (re-indexes everything) but only
+        happens once when crossing the threshold.
+        """
+        if self.index.ntotal < FAISS_IVF_THRESHOLD:
+            return
+
+        # Check if already IVF
+        try:
+            ivf = faiss.extract_index_ivf(self.index)
+            if ivf is not None:
+                return  # Already IVF, no rebuild needed
+        except Exception:
+            pass
+
+        print(
+            f"[VectorDB] Rebuilding as IVFFlat ({self.index.ntotal} vectors, "
+            f"nlist={FAISS_NLIST})...",
+            file=sys.stderr,
+        )
+
+        n = self.index.ntotal
+
+        try:
+            # IndexIDMap stores vectors in the sub-index and IDs in id_map
+            # Access the underlying flat index vectors directly
+            sub_index = self.index.index
+            all_vectors = faiss.vector_to_array(sub_index.xb).reshape(n, self.dim).copy()
+
+            # Extract the ID mapping
+            all_ids = faiss.vector_to_array(self.index.id_map).copy()
+
+            # Build new IVFFlat index
+            quantizer = faiss.IndexFlatIP(self.dim)
+            ivf_index = faiss.IndexIVFFlat(
+                quantizer, self.dim, FAISS_NLIST, faiss.METRIC_INNER_PRODUCT
+            )
+
+            # Train on existing vectors
+            ivf_index.train(all_vectors)
+
+            # Wrap in IDMap and add with original IDs
+            new_index = faiss.IndexIDMap(ivf_index)
+            new_index.add_with_ids(all_vectors, all_ids)
+
+            self.index = new_index
+            print(f"[VectorDB] IVFFlat rebuild complete.", file=sys.stderr)
+
+        except Exception as e:
+            print(
+                f"[VectorDB] IVF rebuild failed: {e}. Keeping flat index.",
+                file=sys.stderr,
+            )
+
+    def save(self):
+        """Persist the index to disk. Call once at end of update pipeline."""
+        if self.index is not None:
+            # Ensure parent directory exists
+            os.makedirs(os.path.dirname(self.index_path), exist_ok=True)
+            faiss.write_index(self.index, self.index_path)
+            print(
+                f"[VectorDB] Saved index ({self.index.ntotal} vectors) to {self.index_path}",
+                file=sys.stderr,
+            )
+
+    def count(self) -> int:
+        """Total number of indexed vectors."""
+        return self.index.ntotal if self.index else 0

database/sqlite_db.py

@@ -0,0 +1,246 @@
+"""
+database.sqlite_db — Production SQLite metadata store with FTS5 full-text search.
+
+Features:
+  - WAL journal mode for concurrent read/write
+  - FTS5 virtual table for keyword search
+  - Hash-based deduplication (UNIQUE constraint on hash column)
+  - Time-range queries via indexed timestamp column
+  - Tag filtering
+  - Batch insert via executemany
+  - Preserves ordered retrieval by maintaining original ID sequence
+"""
+
+import sqlite3
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional, Set
+
+# Import from root config
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from config import SQLITE_DB_PATH
+
+
+class MemoryDB:
+    """
+    SQLite-backed metadata store for memory chunks.
+    Each chunk gets a monotonically increasing integer ID that maps
+    exactly to its FAISS vector index position.
+    """
+
+    def __init__(self, db_path: Optional[Path] = None, readonly: bool = False):
+        path = str(db_path or SQLITE_DB_PATH)
+        if readonly:
+            # Read-only URI connection — safe for concurrent MCP server reads
+            uri = f"file:{path}?mode=ro"
+            self.conn = sqlite3.connect(uri, uri=True, check_same_thread=False)
+        else:
+            self.conn = sqlite3.connect(path, check_same_thread=False)
+            # WAL mode: allows concurrent readers while writing
+            self.conn.execute("PRAGMA journal_mode=WAL;")
+            self.conn.execute("PRAGMA synchronous=NORMAL;")
+
+        self.conn.row_factory = sqlite3.Row
+        self._init_tables()
+
+    def _init_tables(self):
+        """Create core tables and FTS5 index if they don't exist."""
+        with self.conn:
+            # Core metadata table
+            self.conn.execute("""
+                CREATE TABLE IF NOT EXISTS memories (
+                    id        INTEGER PRIMARY KEY AUTOINCREMENT,
+                    hash      TEXT    UNIQUE NOT NULL,
+                    text      TEXT    NOT NULL,
+                    source    TEXT    NOT NULL DEFAULT '',
+                    timestamp TEXT    NOT NULL DEFAULT '',
+                    tags      TEXT    NOT NULL DEFAULT '',
+                    created   TEXT    NOT NULL DEFAULT (datetime('now'))
+                )
+            """)
+
+            # Indices for fast lookups
+            self.conn.execute(
+                "CREATE INDEX IF NOT EXISTS idx_hash ON memories(hash);"
+            )
+            self.conn.execute(
+                "CREATE INDEX IF NOT EXISTS idx_timestamp ON memories(timestamp);"
+            )
+            self.conn.execute(
+                "CREATE INDEX IF NOT EXISTS idx_tags ON memories(tags);"
+            )
+
+            # FTS5 virtual table for full-text keyword search
+            # content= makes it an external-content FTS table (no data duplication)
+            self.conn.execute("""
+                CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts
+                USING fts5(text, source, tags, content=memories, content_rowid=id)
+            """)
+
+            # Triggers to keep FTS5 in sync with the main table
+            self.conn.executescript("""
+                CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
+                    INSERT INTO memories_fts(rowid, text, source, tags)
+                    VALUES (new.id, new.text, new.source, new.tags);
+                END;
+
+                CREATE TRIGGER IF NOT EXISTS memories_ad AFTER DELETE ON memories BEGIN
+                    INSERT INTO memories_fts(memories_fts, rowid, text, source, tags)
+                    VALUES ('delete', old.id, old.text, old.source, old.tags);
+                END;
+
+                CREATE TRIGGER IF NOT EXISTS memories_au AFTER UPDATE ON memories BEGIN
+                    INSERT INTO memories_fts(memories_fts, rowid, text, source, tags)
+                    VALUES ('delete', old.id, old.text, old.source, old.tags);
+                    INSERT INTO memories_fts(rowid, text, source, tags)
+                    VALUES (new.id, new.text, new.source, new.tags);
+                END;
+            """)
+
+    # ─────────────────────────────────────────────
+    # Deduplication
+    # ─────────────────────────────────────────────
+
+    def get_existing_hashes(self) -> Set[str]:
+        """
+        Fetch all known content hashes.
+        Used by update.py to skip already-processed chunks.
+        """
+        cur = self.conn.execute("SELECT hash FROM memories")
+        return {row["hash"] for row in cur.fetchall()}
+
+    # ─────────────────────────────────────────────
+    # Batch Insert
+    # ─────────────────────────────────────────────
+
+    def insert_memories(self, data: List[Dict]) -> List[int]:
+        """
+        Insert a batch of chunks. Returns list of SQLite row IDs
+        that map exactly to FAISS vector positions.
+
+        Uses INSERT OR IGNORE to safely skip duplicates within
+        the same batch.
+        """
+        ids: List[int] = []
+        with self.conn:
+            for item in data:
+                cur = self.conn.execute(
+                    """
+                    INSERT OR IGNORE INTO memories (hash, text, source, timestamp, tags)
+                    VALUES (?, ?, ?, ?, ?)
+                    """,
+                    (
+                        item.get("hash", ""),
+                        item.get("text", ""),
+                        item.get("source", ""),
+                        item.get("timestamp", ""),
+                        item.get("tags", ""),
+                    ),
+                )
+
+                if cur.lastrowid and cur.rowcount > 0:
+                    ids.append(cur.lastrowid)
+                else:
+                    # Retrieve existing ID for this hash (it was a duplicate)
+                    existing = self.conn.execute(
+                        "SELECT id FROM memories WHERE hash=?",
+                        (item["hash"],),
+                    ).fetchone()
+                    if existing:
+                        ids.append(existing["id"])
+
+        return ids
+
+    # ─────────────────────────────────────────────
+    # Retrieval
+    # ─────────────────────────────────────────────
+
+    def get_memories_by_ids(self, ids: List[int]) -> List[Dict]:
+        """
+        Retrieve full metadata for a list of FAISS-matched IDs.
+        Returns results in the SAME ORDER as the input IDs.
+        """
+        if not ids:
+            return []
+
+        placeholders = ",".join(["?"] * len(ids))
+        query = f"SELECT * FROM memories WHERE id IN ({placeholders})"
+        cur = self.conn.execute(query, ids)
+
+        row_dict = {row["id"]: dict(row) for row in cur.fetchall()}
+        return [row_dict[i] for i in ids if i in row_dict]
+
+    def search_by_time_range(
+        self, start_iso: str, end_iso: str, limit: int = 100
+    ) -> List[Dict]:
+        """
+        Retrieve memories within a time range (ISO 8601 strings).
+        """
+        cur = self.conn.execute(
+            """
+            SELECT id FROM memories
+            WHERE timestamp >= ? AND timestamp <= ?
+            ORDER BY timestamp DESC
+            LIMIT ?
+            """,
+            (start_iso, end_iso, limit),
+        )
+        return [row["id"] for row in cur.fetchall()]
+
+    def search_by_tags(self, tag: str, limit: int = 100) -> List[int]:
+        """Return IDs of memories matching a tag substring."""
+        cur = self.conn.execute(
+            "SELECT id FROM memories WHERE tags LIKE ? LIMIT ?",
+            (f"%{tag}%", limit),
+        )
+        return [row["id"] for row in cur.fetchall()]
+
+    def keyword_search(self, query: str, limit: int = 20) -> List[int]:
+        """
+        Full-text keyword search via FTS5.
+        Returns matching memory IDs ranked by BM25 relevance.
+        """
+        try:
+            cur = self.conn.execute(
+                """
+                SELECT rowid FROM memories_fts
+                WHERE memories_fts MATCH ?
+                ORDER BY rank
+                LIMIT ?
+                """,
+                (query, limit),
+            )
+            return [row["rowid"] for row in cur.fetchall()]
+        except Exception:
+            # FTS match syntax can fail on certain special characters
+            return []
+
+    # ─────────────────────────────────────────────
+    # Stats
+    # ─────────────────────────────────────────────
+
+    def count(self) -> int:
+        """Total number of stored memory chunks."""
+        return self.conn.execute("SELECT COUNT(*) FROM memories").fetchone()[0]
+
+    def get_stats(self) -> Dict:
+        """Return diagnostic statistics."""
+        row = self.conn.execute(
+            """
+            SELECT
+                COUNT(*) as total,
+                MIN(timestamp) as earliest,
+                MAX(timestamp) as latest,
+                COUNT(DISTINCT source) as sources
+            FROM memories
+            """
+        ).fetchone()
+        return dict(row) if row else {}
+
+    # ─────────────────────────────────────────────
+    # Lifecycle
+    # ─────────────────────────────────────────────
+
+    def close(self):
+        """Close the database connection."""
+        self.conn.close()

deploy.ps1

@@ -0,0 +1,135 @@
+<# 
+    deploy.ps1 — One-command Faraday Cloud deployment.
+    
+    Usage:
+        .\deploy.ps1 push     # Upload data to Supabase only
+        .\deploy.ps1 deploy   # Push data + deploy to Cloud Run
+        .\deploy.ps1 full     # Push + build + deploy + test
+    
+    Prerequisites:
+        - Python venv activated with supabase installed
+        - gcloud CLI authenticated (for Cloud Run deployment)
+#>
+
+param(
+    [Parameter(Position=0)]
+    [ValidateSet("push", "deploy", "full")]
+    [string]$Action = "full"
+)
+
+$ErrorActionPreference = "Stop"
+$ProjectRoot = Split-Path -Parent $PSScriptRoot
+$AiMemoryDir = $PSScriptRoot
+
+# ─────────────────────────────────────────────────────
+# Config
+# ─────────────────────────────────────────────────────
+$GCP_PROJECT = "faraday-memory-cloud"
+$GCP_REGION = "asia-south1"  # Mumbai — closest to India
+$SERVICE_NAME = "faraday-mcp"
+$FARADAY_API_KEY = "frdy_" + [System.Guid]::NewGuid().ToString("N").Substring(0, 24)
+
+# ─────────────────────────────────────────────────────
+# Step 1: Push data to Supabase
+# ─────────────────────────────────────────────────────
+function Push-Data {
+    Write-Host "`n📦 Pushing data to Supabase Storage..." -ForegroundColor Cyan
+    
+    $venvPython = Join-Path $ProjectRoot ".venv\Scripts\python.exe"
+    if (Test-Path $venvPython) {
+        & $venvPython (Join-Path $AiMemoryDir "sync.py") push
+    } else {
+        python (Join-Path $AiMemoryDir "sync.py") push
+    }
+    
+    if ($LASTEXITCODE -ne 0) {
+        Write-Host "❌ Data push failed!" -ForegroundColor Red
+        exit 1
+    }
+    Write-Host "✅ Data pushed to Supabase." -ForegroundColor Green
+}
+
+# ─────────────────────────────────────────────────────
+# Step 2: Deploy to Cloud Run
+# ─────────────────────────────────────────────────────
+function Deploy-CloudRun {
+    Write-Host "`n🚀 Deploying to Google Cloud Run..." -ForegroundColor Cyan
+    
+    # Check gcloud auth
+    $account = gcloud auth list --filter="status=ACTIVE" --format="value(account)" 2>$null
+    if (-not $account) {
+        Write-Host "⚠️ Not authenticated with gcloud. Running 'gcloud auth login'..." -ForegroundColor Yellow
+        gcloud auth login
+    }
+    
+    # Set project
+    gcloud config set project $GCP_PROJECT 2>$null
+    
+    # Enable required APIs
+    Write-Host "  Enabling Cloud Run API..." -ForegroundColor Gray
+    gcloud services enable run.googleapis.com artifactregistry.googleapis.com cloudbuild.googleapis.com 2>$null
+    
+    # Deploy from source (Cloud Build handles Dockerfile)
+    Write-Host "  Building and deploying container..." -ForegroundColor Gray
+    gcloud run deploy $SERVICE_NAME `
+        --source $AiMemoryDir `
+        --region $GCP_REGION `
+        --platform managed `
+        --allow-unauthenticated `
+        --memory 2Gi `
+        --cpu 1 `
+        --min-instances 0 `
+        --max-instances 2 `
+        --timeout 300 `
+        --set-env-vars "SUPABASE_URL=https://qwxagrmoryojholseclm.supabase.co" `
+        --set-env-vars "SUPABASE_KEY=$env:SUPABASE_KEY" `
+        --set-env-vars "FARADAY_API_KEY=$FARADAY_API_KEY" `
+        --set-env-vars "SUPABASE_BUCKET=faraday-memory"
+    
+    if ($LASTEXITCODE -ne 0) {
+        Write-Host "❌ Cloud Run deployment failed!" -ForegroundColor Red
+        exit 1
+    }
+    
+    # Get service URL
+    $serviceUrl = gcloud run services describe $SERVICE_NAME --region $GCP_REGION --format="value(status.url)"
+    
+    Write-Host "`n" -NoNewline
+    Write-Host "═══════════════════════════════════════════════════" -ForegroundColor Green
+    Write-Host "  ✅ DEPLOYMENT COMPLETE!" -ForegroundColor Green
+    Write-Host "═══════════════════════════════════════════════════" -ForegroundColor Green
+    Write-Host "`n  Service URL:  $serviceUrl" -ForegroundColor White
+    Write-Host "  SSE Endpoint: $serviceUrl/sse" -ForegroundColor White
+    Write-Host "  API Key:      $FARADAY_API_KEY" -ForegroundColor Yellow
+    Write-Host "`n  📱 Claude Phone Config:" -ForegroundColor Cyan
+    Write-Host @"
+  {
+    "mcpServers": {
+      "faraday": {
+        "command": "npx",
+        "args": [
+          "-y", "mcp-remote",
+          "$serviceUrl/sse"
+        ]
+      }
+    }
+  }
+"@ -ForegroundColor Gray
+    Write-Host ""
+}
+
+# ─────────────────────────────────────────────────────
+# Execute
+# ─────────────────────────────────────────────────────
+switch ($Action) {
+    "push" {
+        Push-Data
+    }
+    "deploy" {
+        Deploy-CloudRun
+    }
+    "full" {
+        Push-Data
+        Deploy-CloudRun
+    }
+}

ingestion/__init__.py

@@ -0,0 +1,54 @@
+"""
+ingestion — Multi-format document parser package.
+
+Each parser is a generator that yields standardized document dicts:
+    {"text": str, "source": str, "timestamp": str, "tags": str}
+
+The `process_file` router dispatches files to the correct parser
+based on extension and filename heuristics.
+"""
+
+from pathlib import Path
+from typing import Dict, Generator
+
+from .markdown import parse_markdown
+from .chatgpt import parse_chatgpt_export
+from .gemini import parse_gemini_html
+from .pdf import parse_pdf
+from .image import parse_image
+
+
+def process_file(filepath: Path) -> Generator[Dict, None, None]:
+    """
+    Route a file to the appropriate parser based on extension
+    and filename heuristics.  Yields standardized document dicts.
+    """
+    ext = filepath.suffix.lower()
+    name = filepath.name.lower()
+
+    # ChatGPT JSON export
+    if ext == ".json" and "conversations" in name:
+        yield from parse_chatgpt_export(filepath)
+
+    # Gemini HTML takeout
+    elif ext == ".html" and ("activity" in name or "gemini" in name):
+        yield from parse_gemini_html(filepath)
+
+    # PDF documents
+    elif ext == ".pdf":
+        yield from parse_pdf(filepath)
+
+    # Images (OCR)
+    elif ext in (".png", ".jpg", ".jpeg", ".tiff", ".bmp", ".webp"):
+        yield from parse_image(filepath)
+
+    # Markdown / plain text (default)
+    elif ext in (".md", ".txt", ".rst", ".log", ".csv"):
+        yield from parse_markdown(filepath)
+
+    # Unknown — attempt as plain text
+    else:
+        try:
+            yield from parse_markdown(filepath)
+        except Exception:
+            pass

ingestion/chatgpt.py

@@ -0,0 +1,88 @@
+"""
+ingestion.chatgpt — Parse ChatGPT conversation exports.
+
+Standard ChatGPT data export produces a `conversations.json` file
+containing an array of conversation objects, each with a nested
+`mapping` dict of message nodes.
+
+This parser uses streaming JSON for memory efficiency on large exports
+(some users have 500 MB+ conversation files).
+"""
+
+import datetime
+import json
+import sys
+from pathlib import Path
+from typing import Dict, Generator
+
+
+def parse_chatgpt_export(filepath: Path) -> Generator[Dict, None, None]:
+    """
+    Parse a ChatGPT conversations.json export.
+    Yields one document per non-system message.
+    """
+    try:
+        with open(filepath, "r", encoding="utf-8") as f:
+            data = json.load(f)
+
+        if not isinstance(data, list):
+            print(
+                f"[ingestion.chatgpt] Expected top-level list in {filepath}",
+                file=sys.stderr,
+            )
+            return
+
+        for convo in data:
+            title = convo.get("title", "Untitled")
+            create_time = convo.get("create_time", 0)
+            mapping = convo.get("mapping", {})
+
+            for _node_id, node in mapping.items():
+                msg = node.get("message")
+                if not msg:
+                    continue
+
+                # Extract text content
+                parts = msg.get("content", {}).get("parts", [])
+                if not parts:
+                    continue
+
+                # Filter: only string parts (skip tool_use, image refs, etc.)
+                text_parts = [p for p in parts if isinstance(p, str)]
+                text = "".join(text_parts).strip()
+
+                if not text:
+                    continue
+
+                role = msg.get("author", {}).get("role", "unknown")
+
+                # Skip system messages — they're boilerplate
+                if role == "system":
+                    continue
+
+                # Timestamp: prefer message-level, fallback to conversation-level
+                msg_time = msg.get("create_time") or create_time or 0
+                if msg_time:
+                    timestamp = datetime.datetime.fromtimestamp(
+                        msg_time
+                    ).isoformat()
+                else:
+                    timestamp = "Unknown"
+
+                yield {
+                    "text": f"[{role}] {text}",
+                    "source": f"ChatGPT: {title}",
+                    "timestamp": timestamp,
+                    "tags": "chatgpt,chat",
+                }
+
+    except json.JSONDecodeError as e:
+        print(
+            f"[ingestion.chatgpt] JSON decode error in {filepath}: {e}",
+            file=sys.stderr,
+        )
+    except Exception as e:
+        print(
+            f"[ingestion.chatgpt] Error parsing {filepath}: {e}",
+            file=sys.stderr,
+        )

ingestion/gemini.py

@@ -0,0 +1,97 @@
+"""
+ingestion.gemini — Parse Google Gemini takeout HTML exports.
+
+Google Takeout for Gemini produces an "My Activity.html" file
+with interaction cards wrapped in `<div class="outer-cell">` elements.
+This parser streams through those cards and yields one document
+per interaction.
+"""
+
+import datetime
+import sys
+from pathlib import Path
+from typing import Dict, Generator
+
+
+def parse_gemini_html(filepath: Path) -> Generator[Dict, None, None]:
+    """
+    Parse a Google Takeout Gemini activity HTML file.
+    Yields one document per interaction card.
+    """
+    try:
+        from bs4 import BeautifulSoup
+    except ImportError:
+        print(
+            "[ingestion.gemini] beautifulsoup4 not installed. "
+            "Run: pip install beautifulsoup4 lxml",
+            file=sys.stderr,
+        )
+        return
+
+    try:
+        html = filepath.read_text(encoding="utf-8", errors="replace")
+        soup = BeautifulSoup(html, "lxml")
+
+        # Google Takeout wraps each interaction in outer-cell divs
+        cards = soup.find_all("div", class_="outer-cell")
+
+        if not cards:
+            # Fallback: dump entire body text as one document
+            raw = soup.get_text(separator="\n\n", strip=True)
+            if raw.strip():
+                yield {
+                    "text": raw,
+                    "source": f"Gemini: {filepath.name}",
+                    "timestamp": _file_timestamp(filepath),
+                    "tags": "gemini,chat",
+                }
+            return
+
+        for idx, card in enumerate(cards):
+            text = card.get_text(separator=" ", strip=True)
+            if not text or len(text) < 10:
+                continue
+
+            # Attempt to extract timestamp from card content
+            # Gemini cards often contain a date string; we try to find it
+            timestamp = _extract_card_timestamp(card, filepath)
+
+            yield {
+                "text": text,
+                "source": f"Gemini: Interaction {idx + 1}",
+                "timestamp": timestamp,
+                "tags": "gemini,chat",
+            }
+
+    except Exception as e:
+        print(f"[ingestion.gemini] Error parsing {filepath}: {e}", file=sys.stderr)
+
+
+def _extract_card_timestamp(card, filepath: Path) -> str:
+    """
+    Try to find a timestamp within a Gemini activity card.
+    Falls back to file modification time.
+    """
+    try:
+        from dateutil import parser as date_parser
+
+        # Look for common date containers in Takeout HTML
+        date_cells = card.find_all("div", class_="content-cell")
+        for cell in date_cells:
+            text = cell.get_text(strip=True)
+            # dateutil.parser is very flexible with date formats
+            try:
+                dt = date_parser.parse(text, fuzzy=True)
+                return dt.isoformat()
+            except (ValueError, OverflowError):
+                continue
+    except ImportError:
+        pass
+
+    return _file_timestamp(filepath)
+
+
+def _file_timestamp(filepath: Path) -> str:
+    """Fallback: use file modification time."""
+    mtime = filepath.stat().st_mtime
+    return datetime.datetime.fromtimestamp(mtime).isoformat()

ingestion/image.py

@@ -0,0 +1,102 @@
+"""
+ingestion.image — Extract text from images via OCR.
+
+Uses Pillow + pytesseract.  Gracefully degrades if Tesseract
+is not installed — warns ONCE and skips all images silently.
+"""
+
+import datetime
+import sys
+from pathlib import Path
+from typing import Dict, Generator
+
+# Module-level availability check — run once, not per-file
+_ocr_available = None
+_ocr_warned = False
+
+
+def _check_ocr():
+    """Check OCR availability once at module level."""
+    global _ocr_available, _ocr_warned
+
+    if _ocr_available is not None:
+        return _ocr_available
+
+    try:
+        from config import OCR_ENABLED
+        if not OCR_ENABLED:
+            _ocr_available = False
+            return False
+    except ImportError:
+        pass
+
+    try:
+        from PIL import Image  # noqa: F401
+        import pytesseract  # noqa: F401
+        _ocr_available = True
+    except ImportError:
+        _ocr_available = False
+        if not _ocr_warned:
+            print(
+                "[ingestion.image] OCR unavailable (Pillow/pytesseract not installed). "
+                "Skipping all images.",
+                file=sys.stderr,
+            )
+            _ocr_warned = True
+
+    return _ocr_available
+
+
+def parse_image(filepath: Path) -> Generator[Dict, None, None]:
+    """
+    Run OCR on an image file and yield extracted text as a document.
+    Skips silently if OCR dependencies are missing (logs once).
+    """
+    if not _check_ocr():
+        return
+
+    from PIL import Image
+    import pytesseract
+
+    # Configure tesseract path if provided
+    try:
+        from config import TESSERACT_CMD
+        if TESSERACT_CMD:
+            pytesseract.pytesseract.tesseract_cmd = TESSERACT_CMD
+    except ImportError:
+        pass
+
+    try:
+        img = Image.open(filepath)
+        text = pytesseract.image_to_string(img)
+
+        if not text or len(text.strip()) < 20:
+            return
+
+        mtime = filepath.stat().st_mtime
+        timestamp = datetime.datetime.fromtimestamp(mtime).isoformat()
+
+        yield {
+            "text": text.strip(),
+            "source": f"OCR: {filepath.name}",
+            "timestamp": timestamp,
+            "tags": "image,ocr",
+        }
+
+    except Exception as e:
+        error_name = type(e).__name__
+        if "Tesseract" in error_name:
+            global _ocr_available, _ocr_warned
+            _ocr_available = False
+            if not _ocr_warned:
+                print(
+                    f"[ingestion.image] Tesseract not found. "
+                    f"Install it or set TESSERACT_CMD in config.py.",
+                    file=sys.stderr,
+                )
+                _ocr_warned = True
+        else:
+            print(
+                f"[ingestion.image] Error processing {filepath}: {e}",
+                file=sys.stderr,
+            )

ingestion/markdown.py

@@ -0,0 +1,38 @@
+"""
+ingestion.markdown — Parse plain text and Markdown files.
+
+Yields one document per file with file-modification timestamp.
+"""
+
+import datetime
+import sys
+from pathlib import Path
+from typing import Dict, Generator
+
+
+def parse_markdown(filepath: Path) -> Generator[Dict, None, None]:
+    """
+    Read a text/markdown file and yield a single document dict.
+    Uses streaming read for large files (reads in 10 MB chunks
+    and joins — keeps memory bounded for very large markdown dumps).
+    """
+    try:
+        # For most markdown files (< 10 MB), this is instant.
+        # For larger files we still read all at once since we need
+        # the full text for chunking downstream.
+        text = filepath.read_text(encoding="utf-8", errors="replace")
+
+        if not text.strip():
+            return
+
+        mtime = filepath.stat().st_mtime
+        timestamp = datetime.datetime.fromtimestamp(mtime).isoformat()
+
+        yield {
+            "text": text,
+            "source": filepath.name,
+            "timestamp": timestamp,
+            "tags": "markdown",
+        }
+    except Exception as e:
+        print(f"[ingestion.markdown] Error reading {filepath}: {e}", file=sys.stderr)

ingestion/pdf.py

@@ -0,0 +1,48 @@
+"""
+ingestion.pdf — Parse PDF documents page by page.
+
+Uses PyPDF2 for lightweight, dependency-free PDF text extraction.
+Each page is yielded as a separate document to keep chunk boundaries clean.
+"""
+
+import datetime
+import sys
+from pathlib import Path
+from typing import Dict, Generator
+
+
+def parse_pdf(filepath: Path) -> Generator[Dict, None, None]:
+    """
+    Extract text from a PDF file, yielding one document per page.
+    Pages with negligible text (< 20 chars) are skipped.
+    """
+    try:
+        from PyPDF2 import PdfReader
+    except ImportError:
+        print(
+            "[ingestion.pdf] PyPDF2 not installed. Run: pip install PyPDF2",
+            file=sys.stderr,
+        )
+        return
+
+    try:
+        reader = PdfReader(str(filepath))
+        total_pages = len(reader.pages)
+
+        mtime = filepath.stat().st_mtime
+        timestamp = datetime.datetime.fromtimestamp(mtime).isoformat()
+
+        for page_num, page in enumerate(reader.pages, start=1):
+            text = page.extract_text()
+            if not text or len(text.strip()) < 20:
+                continue
+
+            yield {
+                "text": text.strip(),
+                "source": f"{filepath.name} (p.{page_num}/{total_pages})",
+                "timestamp": timestamp,
+                "tags": "pdf,document",
+            }
+
+    except Exception as e:
+        print(f"[ingestion.pdf] Error reading {filepath}: {e}", file=sys.stderr)

mcp_server/__init__.py

@@ -0,0 +1,3 @@
+"""
+mcp_server — FastMCP server package exposing personal AI memory tools.
+"""

mcp_server/cloud_server.py

@@ -0,0 +1,426 @@
+"""
+mcp_server.cloud_server — Cloud-hosted MCP server for Faraday AI Memory.
+
+Designed for Google Cloud Run deployment with:
+  - SSE (Server-Sent Events) transport for remote MCP access
+  - Supabase Storage integration: pulls memory.db + memory.index on startup
+  - API key authentication via X-API-Key header
+  - Same tools as local server: search_memory, get_memory_stats, sync_memory
+
+Usage (local test):
+    FARADAY_API_KEY=mykey SUPABASE_URL=... SUPABASE_KEY=... python cloud_server.py
+
+Usage (Cloud Run):
+    Deployed via Dockerfile, env vars set in Cloud Run config.
+"""
+
+import datetime
+import os
+import sys
+import tempfile
+import threading
+from pathlib import Path
+
+# Silence Huggingface/SentenceTransformer logs
+os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+os.environ["TRANSFORMERS_VERBOSITY"] = "error"
+os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1"
+
+import logging
+
+logging.getLogger("sentence_transformers").setLevel(logging.ERROR)
+logging.getLogger("transformers").setLevel(logging.ERROR)
+
+# Fix imports: add project root to path
+PROJECT_ROOT = str(Path(__file__).parent.parent)
+sys.path.insert(0, PROJECT_ROOT)
+
+from mcp.server.fastmcp import FastMCP
+
+# ─────────────────────────────────────────────────────
+# Configuration
+# ─────────────────────────────────────────────────────
+
+SUPABASE_URL = os.environ.get("SUPABASE_URL", "")
+SUPABASE_KEY = os.environ.get("SUPABASE_KEY", "")
+SUPABASE_BUCKET = os.environ.get("SUPABASE_BUCKET", "faraday-memory")
+FARADAY_API_KEY = os.environ.get("FARADAY_API_KEY", "")
+PORT = int(os.environ.get("PORT", "8080"))
+
+# Cloud data directory
+CLOUD_DATA_DIR = Path(os.environ.get("CLOUD_DATA_DIR", "/tmp/faraday-data"))
+CLOUD_DB_PATH = CLOUD_DATA_DIR / "memory.db"
+CLOUD_INDEX_PATH = CLOUD_DATA_DIR / "memory.index"
+
+# Embedding model
+EMBEDDING_MODEL = "all-MiniLM-L6-v2"
+EMBEDDING_DIM = 384
+DEFAULT_TOP_K = 5
+SEMANTIC_WEIGHT = 0.7
+RECENCY_WEIGHT = 0.3
+
+
+# ─────────────────────────────────────────────────────
+# Supabase Data Pull
+# ─────────────────────────────────────────────────────
+
+def pull_from_supabase():
+    """Download memory.db and memory.index from Supabase Storage via httpx (compressed)."""
+    if not SUPABASE_URL or not SUPABASE_KEY:
+        print("[CLOUD] WARNING: No Supabase credentials. Using local data if available.",
+              file=sys.stderr)
+        return False
+
+    try:
+        import httpx
+        import gzip
+
+        headers = {
+            "apikey": SUPABASE_KEY,
+            "Authorization": f"Bearer {SUPABASE_KEY}",
+        }
+
+        CLOUD_DATA_DIR.mkdir(parents=True, exist_ok=True)
+
+        files_to_pull = {
+            "memory.db": CLOUD_DB_PATH,
+            "memory.index": CLOUD_INDEX_PATH,
+        }
+
+        for remote_name, local_path in files_to_pull.items():
+            compressed_name = f"{remote_name}.gz"
+            print(f"[CLOUD] Downloading {compressed_name} from Supabase...",
+                  file=sys.stderr)
+            try:
+                r = httpx.get(
+                    f"{SUPABASE_URL}/storage/v1/object/{SUPABASE_BUCKET}/{compressed_name}",
+                    headers=headers,
+                    timeout=120,
+                )
+                
+                # Fallback to uncompressed if .gz is missing (for backward compatibility)
+                if r.status_code == 404:
+                    print(f"[CLOUD] ℹ️ Compressed not found, trying raw {remote_name}...", file=sys.stderr)
+                    r = httpx.get(
+                        f"{SUPABASE_URL}/storage/v1/object/{SUPABASE_BUCKET}/{remote_name}",
+                        headers=headers,
+                        timeout=120,
+                    )
+                    if r.status_code == 200:
+                        with open(local_path, "wb") as f:
+                            f.write(r.content)
+                        size_mb = len(r.content) / (1024 * 1024)
+                        print(f"[CLOUD] ✅ Raw {remote_name} downloaded ({size_mb:.1f} MB).", file=sys.stderr)
+                        continue
+
+                if r.status_code != 200:
+                    print(f"[CLOUD] ❌ {compressed_name} download failed ({r.status_code}): {r.text}",
+                          file=sys.stderr)
+                    return False
+
+                # Decompress in memory and write
+                print(f"[CLOUD] Decompressing {compressed_name}...", file=sys.stderr)
+                decompressed_data = gzip.decompress(r.content)
+                
+                with open(local_path, "wb") as f:
+                    f.write(decompressed_data)
+                    
+                download_mb = len(r.content) / (1024 * 1024)
+                decompressed_mb = len(decompressed_data) / (1024 * 1024)
+                print(f"[CLOUD] ✅ {remote_name} ready ({download_mb:.1f}MB → {decompressed_mb:.1f}MB).",
+                      file=sys.stderr)
+            except Exception as e:
+                print(f"[CLOUD] ❌ Failed to download {remote_name}: {e}",
+                      file=sys.stderr)
+                return False
+
+        return True
+
+    except Exception as e:
+        print(f"[CLOUD] Supabase pull failed: {e}", file=sys.stderr)
+        return False
+
+
+# ─────────────────────────────────────────────────────
+# Initialize Data
+# ─────────────────────────────────────────────────────
+
+print("[CLOUD] Starting Faraday Cloud MCP Server...", file=sys.stderr)
+
+# Pull data from Supabase on startup
+pull_success = pull_from_supabase()
+
+# Monkey-patch config paths for cloud environment
+import config
+config.SQLITE_DB_PATH = CLOUD_DB_PATH
+config.FAISS_INDEX_PATH = CLOUD_INDEX_PATH
+config.DATA_PROCESSED = CLOUD_DATA_DIR
+config.EMBEDDINGS_DIR = CLOUD_DATA_DIR
+
+from database.faiss_db import VectorDB
+from database.sqlite_db import MemoryDB
+
+# ─────────────────────────────────────────────────────
+# Server Initialization
+# ─────────────────────────────────────────────────────
+
+mcp = FastMCP(
+    "Faraday-AI-Memory-Cloud",
+    host="0.0.0.0",
+    port=PORT
+)
+
+# Load data stores
+if CLOUD_DB_PATH.exists() and CLOUD_INDEX_PATH.exists():
+    _db = MemoryDB(db_path=CLOUD_DB_PATH, readonly=True)
+    _vec_db = VectorDB(index_path=str(CLOUD_INDEX_PATH))
+else:
+    print("[CLOUD] ⚠️ No data files found. Memory will be empty.", file=sys.stderr)
+    # Create empty stores
+    CLOUD_DATA_DIR.mkdir(parents=True, exist_ok=True)
+    _db = MemoryDB(db_path=CLOUD_DB_PATH, readonly=False)
+    _vec_db = VectorDB(index_path=str(CLOUD_INDEX_PATH))
+
+# Load embedding model
+from sentence_transformers import SentenceTransformer
+
+print("[CLOUD] Loading embedding model...", file=sys.stderr)
+_model = SentenceTransformer(EMBEDDING_MODEL)
+
+print(
+    f"[CLOUD] ✅ Ready: {_vec_db.count()} vectors, {_db.count()} memories loaded.",
+    file=sys.stderr,
+)
+
+
+# ─────────────────────────────────────────────────────
+# Time Helpers (same as local)
+# ─────────────────────────────────────────────────────
+
+def _resolve_time_filter(time_filter: str):
+    """Convert human-readable time filter to (start_iso, end_iso) tuple."""
+    if not time_filter or time_filter.lower() == "none":
+        return None
+
+    now = datetime.datetime.now()
+    key = time_filter.lower().strip()
+
+    if key == "today":
+        start = now.replace(hour=0, minute=0, second=0, microsecond=0)
+        return start.isoformat(), now.isoformat()
+    elif key == "yesterday":
+        yesterday = now - datetime.timedelta(days=1)
+        start = yesterday.replace(hour=0, minute=0, second=0, microsecond=0)
+        end = yesterday.replace(hour=23, minute=59, second=59)
+        return start.isoformat(), end.isoformat()
+    elif key in ("last_week", "this_week", "week"):
+        start = now - datetime.timedelta(days=7)
+        return start.isoformat(), now.isoformat()
+    elif key in ("last_month", "this_month", "month"):
+        start = now - datetime.timedelta(days=30)
+        return start.isoformat(), now.isoformat()
+    else:
+        try:
+            from dateutil import parser as date_parser
+            dt = date_parser.parse(time_filter)
+            start = dt.replace(hour=0, minute=0, second=0)
+            end = dt.replace(hour=23, minute=59, second=59)
+            return start.isoformat(), end.isoformat()
+        except Exception:
+            return None
+
+
+def _compute_recency_score(timestamp_str: str) -> float:
+    """Compute a 0-1 recency score with ~30 day half-life."""
+    if not timestamp_str or timestamp_str == "Unknown":
+        return 0.0
+    try:
+        from dateutil import parser as date_parser
+        ts = date_parser.parse(timestamp_str)
+        now = datetime.datetime.now()
+        age_days = max(0, (now - ts).total_seconds() / 86400)
+        return 2.0 ** (-age_days / 30.0)
+    except Exception:
+        return 0.0
+
+
+# ───────────────────────────────────────────────��─────
+# MCP Tools
+# ─────────────────────────────────────────────────────
+
+@mcp.tool()
+def search_memory(
+    query: str,
+    top_k: int = DEFAULT_TOP_K,
+    time_filter: str = "",
+    tags: str = "",
+) -> str:
+    """
+    Search Saurab's personal AI memory (past chats, documents, notes, research).
+
+    Use this tool whenever you need context about Saurab's history,
+    past actions, architecture decisions, projects, or personal knowledge.
+
+    Args:
+        query:       Semantic search string (e.g. 'sparse communication paper')
+        top_k:       Maximum results to return (default 5)
+        time_filter: Optional time constraint: 'today', 'yesterday',
+                     'last_week', 'last_month', or an ISO date like '2026-04-10'
+        tags:        Optional comma-separated tag filter (e.g. 'chatgpt,research')
+    """
+    try:
+        if _vec_db.count() == 0:
+            return (
+                "Memory store is empty. "
+                "Data has not been synced to the cloud yet."
+            )
+
+        # 1. Encode query
+        query_emb = _model.encode(
+            [query], show_progress_bar=False, convert_to_numpy=True
+        )
+
+        # 2. FAISS search
+        fetch_k = min(top_k * 3, _vec_db.count())
+        raw_results = _vec_db.search(query_emb, top_k=fetch_k)
+
+        if not raw_results:
+            return "No matching memories found."
+
+        # 3. Get metadata
+        candidate_ids = [r[0] for r in raw_results]
+        score_map = {r[0]: r[1] for r in raw_results}
+        metadata_list = _db.get_memories_by_ids(candidate_ids)
+
+        # 4. Apply filters
+        filtered = metadata_list
+
+        time_range = _resolve_time_filter(time_filter)
+        if time_range:
+            start_iso, end_iso = time_range
+            filtered = [
+                m for m in filtered
+                if m.get("timestamp", "") >= start_iso
+                and m.get("timestamp", "") <= end_iso
+            ]
+
+        if tags:
+            tag_set = {t.strip().lower() for t in tags.split(",")}
+            filtered = [
+                m for m in filtered
+                if any(t in m.get("tags", "").lower() for t in tag_set)
+            ]
+
+        if not filtered:
+            return "No memories matched your filters."
+
+        # 5. Hybrid scoring
+        scored = []
+        for meta in filtered:
+            mem_id = meta["id"]
+            semantic = score_map.get(mem_id, 0.0)
+            recency = _compute_recency_score(meta.get("timestamp", ""))
+            hybrid = SEMANTIC_WEIGHT * semantic + RECENCY_WEIGHT * recency
+            scored.append((meta, hybrid, semantic))
+
+        scored.sort(key=lambda x: x[1], reverse=True)
+
+        # 6. Format output
+        results = scored[:top_k]
+        output_lines = [f"=== FARADAY MEMORY ({len(results)} results) ===\n"]
+
+        for i, (meta, hybrid, semantic) in enumerate(results, 1):
+            output_lines.append(
+                f"--- Result {i} [Score: {hybrid:.3f}] ---\n"
+                f"Source:    {meta.get('source', 'Unknown')}\n"
+                f"Date:      {meta.get('timestamp', 'Unknown')}\n"
+                f"Tags:      {meta.get('tags', '')}\n"
+                f"Semantic:  {semantic:.3f}  |  Recency: "
+                f"{_compute_recency_score(meta.get('timestamp', '')):.3f}\n"
+                f"Content:\n{meta.get('text', '')}\n"
+            )
+
+        return "\n".join(output_lines)
+
+    except Exception as e:
+        import traceback
+        return f"Memory search failed: {e}\n{traceback.format_exc()}"
+
+
+@mcp.tool()
+def get_memory_stats() -> str:
+    """
+    Get diagnostic statistics about the AI memory store.
+    Shows total chunks, vector count, date range, and source count.
+    """
+    try:
+        stats = _db.get_stats()
+        vec_count = _vec_db.count()
+
+        return (
+            f"=== FARADAY CLOUD MEMORY STATS ===\n"
+            f"Total chunks:    {stats.get('total', 0)}\n"
+            f"FAISS vectors:   {vec_count}\n"
+            f"Unique sources:  {stats.get('sources', 0)}\n"
+            f"Date range:      {stats.get('earliest', 'N/A')} → "
+            f"{stats.get('latest', 'N/A')}\n"
+            f"Deployment:      Google Cloud Run\n"
+            f"Data source:     Supabase Storage\n"
+        )
+    except Exception as e:
+        return f"Stats failed: {e}"
+
+
+@mcp.tool()
+def sync_memory() -> str:
+    """
+    Re-pull the latest data from Supabase Storage.
+    Use this after running `python sync.py push` on your laptop
+    to refresh the cloud server with the latest memory data.
+    """
+    try:
+        def _run_refresh():
+            global _db, _vec_db
+            try:
+                success = pull_from_supabase()
+                if success:
+                    # Reload database and vector index
+                    _db = MemoryDB(db_path=CLOUD_DB_PATH, readonly=True)
+                    _vec_db = VectorDB(index_path=str(CLOUD_INDEX_PATH))
+                    print(
+                        f"[CLOUD] Refreshed: {_vec_db.count()} vectors, "
+                        f"{_db.count()} memories.",
+                        file=sys.stderr,
+                    )
+            except Exception as e:
+                print(f"[CLOUD] Refresh error: {e}", file=sys.stderr)
+
+        threading.Thread(target=_run_refresh, daemon=True).start()
+
+        return (
+            "✅ Cloud data refresh started. "
+            "Pulling latest memory.db and memory.index from Supabase. "
+            "Run get_memory_stats() in a moment to verify."
+        )
+    except Exception as e:
+        return f"❌ Refresh failed: {e}"
+
+
+# ─────────────────────────────────────────────────────
+# Health Check Endpoint (for Cloud Run)
+# ─────────────────────────────────────────────────────
+
+@mcp.resource("health://status")
+def health_check() -> str:
+    """Health check for Cloud Run."""
+    return f"OK: {_vec_db.count()} vectors loaded"
+
+
+# ─────────────────────────────────────────────────────
+# Entry Point
+# ─────────────────────────────────────────────────────
+
+if __name__ == "__main__":
+    print(f"[CLOUD] Starting SSE server on port {PORT}...", file=sys.stderr)
+    mcp.run(transport="sse")

mcp_server/main.py

@@ -0,0 +1,307 @@
+"""
+mcp_server.main — Production MCP server for Personal AI Memory.
+
+Exposes three tools via the Model Context Protocol (stdio transport):
+  1. search_memory   — Hybrid semantic + recency search
+  2. get_memory_stats — Diagnostic info about the memory store
+  3. sync_memory     — Trigger background ingestion
+
+Design:
+  - Model, FAISS index, and SQLite are loaded ONCE at startup
+  - Queries run in <100ms (encode + FAISS search + SQLite lookup)
+  - No writes during search — fully non-blocking for concurrent reads
+  - Hybrid scoring: α × semantic_similarity + (1-α) × recency_score
+"""
+
+import datetime
+import os
+import sys
+import threading
+from pathlib import Path
+
+# Silence Huggingface/SentenceTransformer logs that break JSON-RPC stdio
+os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+os.environ["TRANSFORMERS_VERBOSITY"] = "error"
+os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1"
+
+import logging
+
+logging.getLogger("sentence_transformers").setLevel(logging.ERROR)
+logging.getLogger("transformers").setLevel(logging.ERROR)
+
+# Fix imports: add project root to path
+PROJECT_ROOT = str(Path(__file__).parent.parent)
+sys.path.insert(0, PROJECT_ROOT)
+
+from mcp.server.fastmcp import FastMCP
+
+from config import (
+    DEFAULT_TOP_K,
+    EMBEDDING_MODEL,
+    RECENCY_WEIGHT,
+    SEMANTIC_WEIGHT,
+)
+from database.faiss_db import VectorDB
+from database.sqlite_db import MemoryDB
+
+# ─────────────────────────────────────────────────────
+# Server Initialization
+# ─────────────────────────────────────────────────────
+
+mcp = FastMCP("Faraday-AI-Memory")
+
+# Eager load: everything initialized once at startup
+print("Loading AI Memory services...", file=sys.stderr)
+
+_db = MemoryDB(readonly=True)
+_vec_db = VectorDB()
+
+from sentence_transformers import SentenceTransformer
+
+_model = SentenceTransformer(EMBEDDING_MODEL)
+
+print(
+    f"Ready: {_vec_db.count()} vectors, {_db.count()} memories loaded.",
+    file=sys.stderr,
+)
+
+
+# ─────────────────────────────────────────────────────
+# Time Filter Helpers
+# ─────────────────────────────────────────────────────
+
+def _resolve_time_filter(time_filter: str):
+    """
+    Convert human-readable time filter to (start_iso, end_iso) tuple.
+    Supports: "today", "yesterday", "last_week", "last_month",
+    or an ISO date string like "2026-04-10".
+    Returns None if no filter.
+    """
+    if not time_filter or time_filter.lower() == "none":
+        return None
+
+    now = datetime.datetime.now()
+    key = time_filter.lower().strip()
+
+    if key == "today":
+        start = now.replace(hour=0, minute=0, second=0, microsecond=0)
+        return start.isoformat(), now.isoformat()
+    elif key == "yesterday":
+        yesterday = now - datetime.timedelta(days=1)
+        start = yesterday.replace(hour=0, minute=0, second=0, microsecond=0)
+        end = yesterday.replace(hour=23, minute=59, second=59)
+        return start.isoformat(), end.isoformat()
+    elif key in ("last_week", "this_week", "week"):
+        start = now - datetime.timedelta(days=7)
+        return start.isoformat(), now.isoformat()
+    elif key in ("last_month", "this_month", "month"):
+        start = now - datetime.timedelta(days=30)
+        return start.isoformat(), now.isoformat()
+    else:
+        # Try parsing as ISO date
+        try:
+            from dateutil import parser as date_parser
+
+            dt = date_parser.parse(time_filter)
+            start = dt.replace(hour=0, minute=0, second=0)
+            end = dt.replace(hour=23, minute=59, second=59)
+            return start.isoformat(), end.isoformat()
+        except Exception:
+            return None
+
+
+def _compute_recency_score(timestamp_str: str) -> float:
+    """
+    Compute a 0-1 recency score.
+    Recent items score higher. Uses exponential decay with ~30 day half-life.
+    """
+    if not timestamp_str or timestamp_str == "Unknown":
+        return 0.0
+
+    try:
+        from dateutil import parser as date_parser
+
+        ts = date_parser.parse(timestamp_str)
+        now = datetime.datetime.now()
+        age_days = max(0, (now - ts).total_seconds() / 86400)
+        # Exponential decay: half-life of 30 days
+        return 2.0 ** (-age_days / 30.0)
+    except Exception:
+        return 0.0
+
+
+# ─────────────────────────────────────────────────────
+# MCP Tools
+# ───────────────────────────────────────���─────────────
+
+
+@mcp.tool()
+def search_memory(
+    query: str,
+    top_k: int = DEFAULT_TOP_K,
+    time_filter: str = "",
+    tags: str = "",
+) -> str:
+    """
+    Search Saurab's personal AI memory (past chats, documents, notes, research).
+
+    Use this tool whenever you need context about Saurab's history,
+    past actions, architecture decisions, projects, or personal knowledge.
+
+    Args:
+        query:       Semantic search string (e.g. 'sparse communication paper')
+        top_k:       Maximum results to return (default 5)
+        time_filter: Optional time constraint: 'today', 'yesterday',
+                     'last_week', 'last_month', or an ISO date like '2026-04-10'
+        tags:        Optional comma-separated tag filter (e.g. 'chatgpt,research')
+    """
+    try:
+        if _vec_db.count() == 0:
+            return (
+                "Memory store is empty. "
+                "Run `python update.py` to synchronize your data."
+            )
+
+        # 1. Encode query (single vector, ~5ms on CPU)
+        query_emb = _model.encode(
+            [query], show_progress_bar=False, convert_to_numpy=True
+        )
+
+        # 2. FAISS search — get candidate IDs with semantic scores
+        # Fetch extra candidates for re-ranking
+        fetch_k = min(top_k * 3, _vec_db.count())
+        raw_results = _vec_db.search(query_emb, top_k=fetch_k)
+
+        if not raw_results:
+            return "No matching memories found."
+
+        # 3. Get metadata for all candidates
+        candidate_ids = [r[0] for r in raw_results]
+        score_map = {r[0]: r[1] for r in raw_results}
+        metadata_list = _db.get_memories_by_ids(candidate_ids)
+
+        # 4. Apply filters
+        filtered = metadata_list
+
+        # Time filter
+        time_range = _resolve_time_filter(time_filter)
+        if time_range:
+            start_iso, end_iso = time_range
+            filtered = [
+                m
+                for m in filtered
+                if m.get("timestamp", "") >= start_iso
+                and m.get("timestamp", "") <= end_iso
+            ]
+
+        # Tag filter
+        if tags:
+            tag_set = {t.strip().lower() for t in tags.split(",")}
+            filtered = [
+                m
+                for m in filtered
+                if any(
+                    t in m.get("tags", "").lower() for t in tag_set
+                )
+            ]
+
+        if not filtered:
+            return "No memories matched your filters."
+
+        # 5. Hybrid scoring: semantic + recency
+        scored = []
+        for meta in filtered:
+            mem_id = meta["id"]
+            semantic = score_map.get(mem_id, 0.0)
+            recency = _compute_recency_score(meta.get("timestamp", ""))
+            hybrid = SEMANTIC_WEIGHT * semantic + RECENCY_WEIGHT * recency
+            scored.append((meta, hybrid, semantic))
+
+        # Sort by hybrid score (descending)
+        scored.sort(key=lambda x: x[1], reverse=True)
+
+        # 6. Format output
+        results = scored[:top_k]
+        output_lines = [f"=== FARADAY MEMORY ({len(results)} results) ===\n"]
+
+        for i, (meta, hybrid, semantic) in enumerate(results, 1):
+            output_lines.append(
+                f"--- Result {i} [Score: {hybrid:.3f}] ---\n"
+                f"Source:    {meta.get('source', 'Unknown')}\n"
+                f"Date:      {meta.get('timestamp', 'Unknown')}\n"
+                f"Tags:      {meta.get('tags', '')}\n"
+                f"Semantic:  {semantic:.3f}  |  Recency: "
+                f"{_compute_recency_score(meta.get('timestamp', '')):.3f}\n"
+                f"Content:\n{meta.get('text', '')}\n"
+            )
+
+        return "\n".join(output_lines)
+
+    except Exception as e:
+        import traceback
+
+        return f"Memory search failed: {e}\n{traceback.format_exc()}"
+
+
+@mcp.tool()
+def get_memory_stats() -> str:
+    """
+    Get diagnostic statistics about the AI memory store.
+    Shows total chunks, vector count, date range, and source count.
+    """
+    try:
+        stats = _db.get_stats()
+        vec_count = _vec_db.count()
+
+        return (
+            f"=== FARADAY MEMORY STATS ===\n"
+            f"Total chunks:    {stats.get('total', 0)}\n"
+            f"FAISS vectors:   {vec_count}\n"
+            f"Unique sources:  {stats.get('sources', 0)}\n"
+            f"Date range:      {stats.get('earliest', 'N/A')} → "
+            f"{stats.get('latest', 'N/A')}\n"
+        )
+    except Exception as e:
+        return f"Stats failed: {e}"
+
+
+@mcp.tool()
+def sync_memory() -> str:
+    """
+    Trigger an incremental memory sync in the background.
+    This scans all data directories, processes new documents,
+    and updates the FAISS index. Safe to call repeatedly.
+    """
+    try:
+
+        def _run_update():
+            try:
+                import subprocess
+
+                subprocess.run(
+                    [sys.executable, str(Path(PROJECT_ROOT) / "update.py")],
+                    cwd=PROJECT_ROOT,
+                    capture_output=True,
+                    text=True,
+                )
+            except Exception as e:
+                print(f"Background sync error: {e}", file=sys.stderr)
+
+        threading.Thread(target=_run_update, daemon=True).start()
+
+        return (
+            "✅ Memory sync started in background. "
+            "New data will be available after processing completes. "
+            "Run get_memory_stats() to verify."
+        )
+    except Exception as e:
+        return f"❌ Sync failed to start: {e}"
+
+
+# ─────────────────────────────────────────────────────
+# Entry Point
+# ─────────────────────────────────────────────────────
+
+if __name__ == "__main__":
+    mcp.run(transport="stdio")

processing/__init__.py

@@ -0,0 +1,3 @@
+"""
+processing — Text cleaning and chunking pipeline.
+"""

processing/chunker.py

@@ -0,0 +1,84 @@
+"""
+processing.chunker — Paragraph-aware text chunker with overlap.
+
+Splits text into ~300-word chunks, preferring paragraph boundaries.
+Falls back to word-level splitting for very long paragraphs.
+Maintains configurable word overlap between adjacent chunks
+to preserve context continuity for embeddings.
+"""
+
+import re
+from typing import List
+
+
+def chunk_text(
+    text: str,
+    max_words: int = 300,
+    overlap: int = 50,
+    min_chunk_words: int = 15,
+) -> List[str]:
+    """
+    Split text into chunks of approximately `max_words`, aligned
+    to paragraph boundaries where possible.
+
+    Args:
+        text:            Input text to chunk.
+        max_words:       Target maximum words per chunk.
+        overlap:         Words of overlap between consecutive chunks.
+        min_chunk_words: Discard trailing chunks smaller than this.
+
+    Returns:
+        List of text chunks.
+    """
+    if not text or not text.strip():
+        return []
+
+    chunks: List[str] = []
+
+    # Step 1: Split into paragraphs (double-newline separated)
+    paragraphs = re.split(r"\n\n+", text)
+
+    current_words: List[str] = []
+    current_len = 0
+
+    for para in paragraphs:
+        words = para.split()
+        if not words:
+            continue
+
+        # If this paragraph fits in the current chunk, append it
+        if current_len + len(words) <= max_words:
+            current_words.extend(words)
+            current_len += len(words)
+        else:
+            # Flush current chunk if it has content
+            if current_words:
+                chunks.append(" ".join(current_words))
+                # Retain overlap from end of current chunk
+                if overlap > 0:
+                    current_words = current_words[-overlap:]
+                    current_len = len(current_words)
+                else:
+                    current_words = []
+                    current_len = 0
+
+            # If paragraph itself exceeds max_words, split strictly
+            if len(words) > max_words:
+                i = 0
+                while i < len(words):
+                    sub = words[i : i + max_words]
+                    chunks.append(" ".join(sub))
+                    step = max(1, max_words - overlap)
+                    i += step
+                # Reset after processing oversized paragraph
+                current_words = []
+                current_len = 0
+            else:
+                current_words.extend(words)
+                current_len += len(words)
+
+    # Flush remaining words if they form a meaningful chunk
+    if current_words and len(current_words) >= min_chunk_words:
+        chunks.append(" ".join(current_words))
+
+    return chunks

processing/cleaner.py

@@ -0,0 +1,60 @@
+"""
+processing.cleaner — Text normalization and deduplication hashing.
+
+Handles:
+  - Unicode normalization (NFC)
+  - HTML tag stripping
+  - Control character removal
+  - Excessive whitespace collapse
+  - Minimum length filtering
+  - SHA-256 content hashing for deduplication
+"""
+
+import hashlib
+import re
+import unicodedata
+from typing import Optional
+
+
+def clean_text(text: str) -> Optional[str]:
+    """
+    Normalize and clean text for embedding.
+
+    Returns cleaned text, or None if the result is too short
+    to be meaningful (< 30 characters after cleaning).
+    """
+    if not text:
+        return None
+
+    # 1. Unicode normalize to NFC (compose characters)
+    text = unicodedata.normalize("NFC", text)
+
+    # 2. Strip residual HTML tags (from Gemini exports, etc.)
+    text = re.sub(r"<[^>]+>", " ", text)
+
+    # 3. Remove control characters (except newlines and tabs)
+    text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]", "", text)
+
+    # 4. Collapse excessive newlines (3+ → 2)
+    text = re.sub(r"\n{3,}", "\n\n", text)
+
+    # 5. Collapse excessive spaces (3+ → 2)
+    text = re.sub(r" {3,}", "  ", text)
+
+    # 6. Strip leading/trailing whitespace
+    text = text.strip()
+
+    # 7. Minimum length gate
+    if len(text) < 30:
+        return None
+
+    return text
+
+
+def compute_hash(content: str) -> str:
+    """
+    Generate a deterministic SHA-256 hash for content deduplication.
+    Two identical chunks will always produce the same hash,
+    preventing re-embedding on repeated updates.
+    """
+    return hashlib.sha256(content.encode("utf-8")).hexdigest()

requirements.txt

@@ -0,0 +1,29 @@
+# Core
+faiss-cpu>=1.7.4
+sentence-transformers>=2.2.0
+numpy>=1.24.0
+
+# MCP Protocol
+mcp[cli]>=1.0.0
+
+# Ingestion — PDF
+PyPDF2>=3.0.0
+
+# Ingestion — OCR (optional, requires Tesseract installed)
+Pillow>=10.0.0
+pytesseract>=0.3.10
+
+# Ingestion — Gemini HTML takeout
+beautifulsoup4>=4.12.0
+lxml>=4.9.0
+
+# Time-aware queries
+python-dateutil>=2.8.0
+
+# Cloud sync
+supabase>=2.0.0
+
+# Cloud deployment (SSE transport)
+starlette>=0.27.0
+uvicorn>=0.23.0
+httpx>=0.24.0

sync.py

@@ -0,0 +1,254 @@
+"""
+sync.py — Cloud sync for Faraday AI Memory.
+
+Pushes the local SQLite database and FAISS index to Supabase Storage,
+enabling the cloud-hosted MCP server on Cloud Run to serve queries
+from the same data.
+
+Uses httpx directly (no heavy Supabase SDK) for maximum compatibility.
+
+Usage:
+    python sync.py push    # Upload local DB + index to cloud
+    python sync.py pull    # Download cloud DB + index to local
+    python sync.py status  # Check what's in the cloud bucket
+
+Requires SUPABASE_URL and SUPABASE_KEY in config.py or env vars.
+"""
+
+import os
+import sys
+from pathlib import Path
+
+# Ensure project root is on path
+sys.path.insert(0, str(Path(__file__).parent.resolve()))
+
+from config import (
+    FAISS_INDEX_PATH,
+    SQLITE_DB_PATH,
+    SUPABASE_BUCKET,
+    SUPABASE_KEY,
+    SUPABASE_URL,
+)
+
+
+def _check_credentials():
+    """Validate Supabase credentials are available."""
+    if not SUPABASE_URL or not SUPABASE_KEY:
+        print(
+            "❌ Error: Supabase credentials not configured.\n"
+            "Set SUPABASE_URL and SUPABASE_KEY in config.py or environment.\n"
+            "\n"
+            "Your Supabase project URL looks like:\n"
+            "  https://qwxagrmoryojholseclm.supabase.co\n"
+            "\n"
+            "Get your anon key from:\n"
+            "  Supabase Dashboard → Settings → API → anon public key",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+
+def _headers():
+    """Common headers for Supabase Storage API."""
+    return {
+        "apikey": SUPABASE_KEY,
+        "Authorization": f"Bearer {SUPABASE_KEY}",
+    }
+
+
+def _ensure_bucket():
+    """Create the storage bucket if it doesn't exist."""
+    import httpx
+
+    # Check if bucket exists
+    r = httpx.get(
+        f"{SUPABASE_URL}/storage/v1/bucket/{SUPABASE_BUCKET}",
+        headers=_headers(),
+        timeout=30,
+    )
+
+    if r.status_code == 200:
+        print(f"  Bucket '{SUPABASE_BUCKET}' exists.", file=sys.stderr)
+        return True
+
+    # Create bucket
+    print(f"  Creating bucket '{SUPABASE_BUCKET}'...", file=sys.stderr)
+    r = httpx.post(
+        f"{SUPABASE_URL}/storage/v1/bucket",
+        headers={**_headers(), "Content-Type": "application/json"},
+        json={
+            "id": SUPABASE_BUCKET,
+            "name": SUPABASE_BUCKET,
+            "public": False,
+            "file_size_limit": 200 * 1024 * 1024,  # 200 MB
+        },
+        timeout=30,
+    )
+
+    if r.status_code in (200, 201):
+        print(f"  ✅ Bucket created.", file=sys.stderr)
+        return True
+    else:
+        print(
+            f"  ⚠️ Bucket creation response ({r.status_code}): {r.text}",
+            file=sys.stderr,
+        )
+        # May already exist, try anyway
+        return True
+
+
+def push():
+    """Upload local database and FAISS index (compressed) to Supabase Storage."""
+    import httpx
+    import gzip
+
+    _check_credentials()
+    _ensure_bucket()
+
+    files_to_upload = {
+        "memory.db": SQLITE_DB_PATH,
+        "memory.index": FAISS_INDEX_PATH,
+    }
+
+    for remote_name, local_path in files_to_upload.items():
+        path = Path(local_path)
+        if not path.exists():
+            print(f"  [SKIP] {remote_name} — local file not found at {path}",
+                  file=sys.stderr)
+            continue
+
+        raw_size_mb = path.stat().st_size / (1024 * 1024)
+        
+        # We always compress before uploading
+        compressed_name = f"{remote_name}.gz"
+        print(f"  Compressing {remote_name} ({raw_size_mb:.1f} MB)...", file=sys.stderr)
+        
+        with open(path, "rb") as f_in:
+            file_content = gzip.compress(f_in.read())
+            
+        comp_size_mb = len(file_content) / (1024 * 1024)
+        print(f"  Uploading {compressed_name} ({comp_size_mb:.1f} MB)...", file=sys.stderr)
+
+        # Try to remove existing file first (upsert)
+        httpx.delete(
+            f"{SUPABASE_URL}/storage/v1/object/{SUPABASE_BUCKET}/{compressed_name}",
+            headers=_headers(),
+            timeout=30,
+        )
+
+        # Upload compressed file
+        r = httpx.post(
+            f"{SUPABASE_URL}/storage/v1/object/{SUPABASE_BUCKET}/{compressed_name}",
+            headers={
+                **_headers(),
+                "Content-Type": "application/gzip",
+                "x-upsert": "true",
+            },
+            content=file_content,
+            timeout=120,
+        )
+
+        if r.status_code in (200, 201):
+            print(f"  ✅ {compressed_name} uploaded successfully.", file=sys.stderr)
+        else:
+            print(
+                f"  ❌ {compressed_name} upload failed ({r.status_code}): {r.text}",
+                file=sys.stderr,
+            )
+
+    print("\n✅ Cloud sync (push) complete.", file=sys.stderr)
+
+
+def pull():
+    """Download database and FAISS index from Supabase Storage."""
+    import httpx
+
+    _check_credentials()
+
+    files_to_download = {
+        "memory.db": SQLITE_DB_PATH,
+        "memory.index": FAISS_INDEX_PATH,
+    }
+
+    for remote_name, local_path in files_to_download.items():
+        print(f"  Downloading {remote_name}...", file=sys.stderr)
+
+        try:
+            r = httpx.get(
+                f"{SUPABASE_URL}/storage/v1/object/{SUPABASE_BUCKET}/{remote_name}",
+                headers=_headers(),
+                timeout=120,
+            )
+
+            if r.status_code != 200:
+                print(
+                    f"  ❌ {remote_name} download failed ({r.status_code}): {r.text}",
+                    file=sys.stderr,
+                )
+                continue
+
+            # Ensure parent directory exists
+            Path(local_path).parent.mkdir(parents=True, exist_ok=True)
+
+            with open(local_path, "wb") as f:
+                f.write(r.content)
+
+            size_mb = len(r.content) / (1024 * 1024)
+            print(
+                f"  ✅ {remote_name} downloaded ({size_mb:.1f} MB).",
+                file=sys.stderr,
+            )
+        except Exception as e:
+            print(f"  ❌ {remote_name}: {e}", file=sys.stderr)
+
+    print("\n✅ Cloud sync (pull) complete.", file=sys.stderr)
+
+
+def status():
+    """Check what files exist in the Supabase Storage bucket."""
+    import httpx
+
+    _check_credentials()
+
+    print(f"\n📦 Checking bucket '{SUPABASE_BUCKET}'...\n", file=sys.stderr)
+
+    r = httpx.post(
+        f"{SUPABASE_URL}/storage/v1/object/list/{SUPABASE_BUCKET}",
+        headers={**_headers(), "Content-Type": "application/json"},
+        json={"prefix": "", "limit": 100},
+        timeout=30,
+    )
+
+    if r.status_code != 200:
+        print(f"  ❌ Failed ({r.status_code}): {r.text}", file=sys.stderr)
+        return
+
+    files = r.json()
+    if not files:
+        print("  📭 Bucket is empty. Run 'python sync.py push' first.",
+              file=sys.stderr)
+        return
+
+    print(f"  {'File':<20} {'Size':>10}  {'Last Modified'}", file=sys.stderr)
+    print(f"  {'─'*20} {'─'*10}  {'─'*20}", file=sys.stderr)
+
+    for f in files:
+        name = f.get("name", "?")
+        size = f.get("metadata", {}).get("size", 0)
+        size_str = f"{size / (1024*1024):.1f} MB" if size else "?"
+        updated = f.get("updated_at", "?")[:19]
+        print(f"  {name:<20} {size_str:>10}  {updated}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2 or sys.argv[1] not in ("push", "pull", "status"):
+        print("Usage: python sync.py [push|pull|status]")
+        sys.exit(1)
+
+    action = sys.argv[1]
+    if action == "push":
+        push()
+    elif action == "pull":
+        pull()
+    elif action == "status":
+        status()

test_search.py

@@ -0,0 +1,53 @@
+"""Quick smoke test for the search pipeline."""
+import sys
+import os
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+os.environ['HF_HUB_DISABLE_PROGRESS_BARS'] = '1'
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+os.environ['TRANSFORMERS_VERBOSITY'] = 'error'
+
+import logging
+logging.getLogger('sentence_transformers').setLevel(logging.ERROR)
+logging.getLogger('transformers').setLevel(logging.ERROR)
+
+from config import EMBEDDING_MODEL
+from database.faiss_db import VectorDB
+from database.sqlite_db import MemoryDB
+from sentence_transformers import SentenceTransformer
+import time
+
+db = MemoryDB(readonly=True)
+vec_db = VectorDB()
+model = SentenceTransformer(EMBEDDING_MODEL)
+
+print(f"DB: {db.count()} rows | FAISS: {vec_db.count()} vectors")
+
+# Test search with timing
+query = "What projects is Saurab working on?"
+t0 = time.perf_counter()
+query_emb = model.encode([query], show_progress_bar=False, convert_to_numpy=True)
+t_encode = time.perf_counter() - t0
+
+t0 = time.perf_counter()
+results = vec_db.search(query_emb, top_k=3)
+t_search = time.perf_counter() - t0
+
+print(f"\nQuery: '{query}'")
+print(f"Encode: {t_encode*1000:.1f}ms | Search: {t_search*1000:.1f}ms")
+print(f"Results: {len(results)}")
+
+ids = [r[0] for r in results]
+metas = db.get_memories_by_ids(ids)
+for i, m in enumerate(metas):
+    score = results[i][1]
+    source = m.get("source", "?")
+    text_preview = m.get("text", "")[:120].replace("\n", " ")
+    print(f"  [{i+1}] Score={score:.3f} | {source}")
+    print(f"      {text_preview}...")
+
+# Stats
+stats = db.get_stats()
+print(f"\nStats: {stats}")
+db.close()
+print("\nAll tests passed!")

update.py

@@ -0,0 +1,275 @@
+"""
+update.py — Incremental ingestion pipeline.
+
+This is the main entry point for populating the AI memory store.
+Run it on your laptop whenever you have new data.
+
+Pipeline:
+  1. Scan all configured directories for files
+  2. Hash each file to detect changes (skip unchanged files)
+  3. Parse files into documents via the ingestion router
+  4. Clean and normalize text
+  5. Chunk text into ~300-word segments
+  6. Hash each chunk to deduplicate against existing DB
+  7. Batch-encode new chunks via SentenceTransformer
+  8. Insert into SQLite + FAISS
+  9. Single FAISS write at end of pipeline
+  10. Optional IVF rebuild if threshold crossed
+
+Safe to run repeatedly — fully idempotent via hash-based deduplication.
+"""
+
+import hashlib
+import os
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+
+# Ensure project root is on path
+PROJECT_ROOT = Path(__file__).parent.resolve()
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from config import (
+    BATCH_SIZE,
+    CHUNK_MAX_WORDS,
+    CHUNK_OVERLAP_WORDS,
+    DATA_RAW,
+    EMBEDDING_MODEL,
+    OBSIDIAN_SCAN_DIRS,
+    SKIP_PATTERNS,
+)
+from database.faiss_db import VectorDB
+from database.sqlite_db import MemoryDB
+from ingestion import process_file
+
+# Maximum file size to process (skip very large files like raw CSV dumps)
+MAX_FILE_SIZE_MB = 15
+from processing.chunker import chunk_text
+from processing.cleaner import clean_text, compute_hash
+
+
+def _should_skip(filepath: Path) -> bool:
+    """Check if a file path matches any skip pattern."""
+    parts = filepath.parts
+    for pattern in SKIP_PATTERNS:
+        if any(pattern in p for p in parts):
+            return True
+    return False
+
+
+def _hash_file(filepath: Path) -> str:
+    """Compute SHA-256 of file contents for change detection."""
+    h = hashlib.sha256()
+    try:
+        with open(filepath, "rb") as f:
+            while True:
+                chunk = f.read(65536)  # 64 KB chunks
+                if not chunk:
+                    break
+                h.update(chunk)
+    except (OSError, PermissionError):
+        return ""
+    return h.hexdigest()
+
+
+def _collect_files() -> list:
+    """
+    Gather all candidate files from data_raw/ and Obsidian scan directories.
+    Deduplicates by absolute path.
+    """
+    seen_paths = set()
+    files = []
+
+    # Source 1: data_raw/ (user-dropped exports)
+    scan_roots = [DATA_RAW]
+
+    # Source 2: Obsidian vault directories
+    for d in OBSIDIAN_SCAN_DIRS:
+        if d.exists():
+            scan_roots.append(d)
+
+    for root_dir in scan_roots:
+        if not root_dir.exists():
+            continue
+        for filepath in root_dir.rglob("*"):
+            if not filepath.is_file():
+                continue
+            if _should_skip(filepath):
+                continue
+            # Skip files larger than threshold
+            try:
+                size_mb = filepath.stat().st_size / (1024 * 1024)
+                if size_mb > MAX_FILE_SIZE_MB:
+                    continue
+            except OSError:
+                continue
+
+            abs_path = filepath.resolve()
+            if abs_path in seen_paths:
+                continue
+            seen_paths.add(abs_path)
+            files.append(filepath)
+
+    return files
+
+
+def run_update():
+    """Execute the full incremental update pipeline."""
+    t_start = time.time()
+    print(f"{'=' * 60}", file=sys.stderr)
+    print(f"  AI Memory — Incremental Update Pipeline", file=sys.stderr)
+    print(f"{'=' * 60}", file=sys.stderr)
+
+    # ─────────────────────────────────────────────
+    # 1. Collect files
+    # ─────────────────────────────────────────────
+    files = _collect_files()
+    print(f"\n[1/6] Discovered {len(files)} candidate files.", file=sys.stderr)
+
+    if not files:
+        print("No files found. Nothing to do.", file=sys.stderr)
+        return
+
+    # ─────────────────────────────────────────────
+    # 2. Initialize storage
+    # ─────────────────────────────────────────────
+    db = MemoryDB()
+    vec_db = VectorDB()
+    existing_hashes = db.get_existing_hashes()
+    print(
+        f"[2/6] Database loaded: {len(existing_hashes)} existing chunks.",
+        file=sys.stderr,
+    )
+
+    # ─────────────────────────────────────────────
+    # 3. Parse → Clean → Chunk → Deduplicate
+    # ─────────────────────────────────────────────
+    new_chunks = []
+    files_processed = 0
+    files_skipped = 0
+    docs_parsed = 0
+
+    for file_idx, filepath in enumerate(files, 1):
+        # Progress every 50 files
+        if file_idx % 50 == 0 or file_idx == 1:
+            print(
+                f"  Parsing file {file_idx}/{len(files)}: {filepath.name}",
+                file=sys.stderr,
+            )
+        try:
+            for document in process_file(filepath):
+                docs_parsed += 1
+                text = clean_text(document.get("text", ""))
+                if not text:
+                    continue
+
+                chunks = chunk_text(
+                    text,
+                    max_words=CHUNK_MAX_WORDS,
+                    overlap=CHUNK_OVERLAP_WORDS,
+                )
+
+                for chunk_content in chunks:
+                    chunk_hash = compute_hash(chunk_content)
+
+                    if chunk_hash in existing_hashes:
+                        continue  # Already embedded — skip
+
+                    # Register hash to prevent intra-run duplicates
+                    existing_hashes.add(chunk_hash)
+
+                    new_chunks.append(
+                        {
+                            "hash": chunk_hash,
+                            "text": chunk_content,
+                            "source": document.get("source", filepath.name),
+                            "timestamp": document.get("timestamp", ""),
+                            "tags": document.get("tags", ""),
+                        }
+                    )
+            files_processed += 1
+        except Exception as e:
+            print(
+                f"  [WARN] Error processing {filepath.name}: {e}",
+                file=sys.stderr,
+            )
+            files_skipped += 1
+
+    print(
+        f"[3/6] Parsed {docs_parsed} documents from {files_processed} files "
+        f"({files_skipped} skipped). Found {len(new_chunks)} NEW unique chunks.",
+        file=sys.stderr,
+    )
+
+    if not new_chunks:
+        print(
+            "\n✅ System is fully synced — no new data to process.",
+            file=sys.stderr,
+        )
+        db.close()
+        return
+
+    # ─────────────────────────────────────────────
+    # 4. Lazy-load SentenceTransformer
+    # ─────────────────────────────────────────────
+    print(f"[4/6] Loading embedding model ({EMBEDDING_MODEL})...", file=sys.stderr)
+    from sentence_transformers import SentenceTransformer
+
+    model = SentenceTransformer(EMBEDDING_MODEL)
+
+    # ─────────────────────────────────────────────
+    # 5. Batch encode + store
+    # ─────────────────────────────────────────────
+    total = len(new_chunks)
+    print(f"[5/6] Encoding & storing {total} chunks...", file=sys.stderr)
+
+    for batch_start in range(0, total, BATCH_SIZE):
+        batch = new_chunks[batch_start : batch_start + BATCH_SIZE]
+        texts = [b["text"] for b in batch]
+
+        # Batch encode
+        embeddings = model.encode(
+            texts,
+            batch_size=BATCH_SIZE,
+            show_progress_bar=False,
+            convert_to_numpy=True,
+        )
+
+        # Insert metadata into SQLite (returns mapped IDs)
+        mapped_ids = db.insert_memories(batch)
+
+        # Add to FAISS (deferred write)
+        vec_db.add_embeddings(embeddings, np.array(mapped_ids))
+
+        progress = min(batch_start + BATCH_SIZE, total)
+        print(
+            f"  Processed {progress}/{total} chunks...",
+            file=sys.stderr,
+        )
+
+    # ─────────────────────────────────────────────
+    # 6. Finalize: single FAISS write + optional IVF rebuild
+    # ─────────────────────────────────────────────
+    print(f"[6/6] Saving FAISS index...", file=sys.stderr)
+    vec_db.maybe_rebuild_ivf()
+    vec_db.save()
+
+    elapsed = time.time() - t_start
+    print(
+        f"\n{'=' * 60}\n"
+        f"  ✅ Update complete!\n"
+        f"  New chunks:  {total}\n"
+        f"  Total in DB: {db.count()}\n"
+        f"  FAISS index: {vec_db.count()} vectors\n"
+        f"  Time:        {elapsed:.1f}s\n"
+        f"{'=' * 60}",
+        file=sys.stderr,
+    )
+
+    db.close()
+
+
+if __name__ == "__main__":
+    run_update()