# Faraday Memory: Data Ingestion Guide

Your AI memory system has built-in smart parsers designed to ingest various file formats. You do not need to convert your documents manually — the ingestion pipeline handles extraction, chunking, and vector embedding automatically based on file extensions.

## 📥 Where to Drop Files

You have two primary designated "Drop Zones". The `sync.py push` command recursively scans these directories and all their subfolders:

1. **Your Obsidian Vault's Inbox or Reference Folders**
   - `00 - Inbox/`
   - `01 - Raw Sources/`
2. **The Dedicated Raw Data Directory**
   - `Faraday/ai-memory-mcp/data_raw/`

## 📄 Supported Formats & Naming Rules

### 1. ChatGPT Exports (JSON)
*   **Format:** `.json`
*   **Rule:** The filename **must** contain the word `conversations`.
*   **Example:** `chatgpt_conversations.json` or `conversations_2025.json`
*   *Behavior:* Explodes the dump and parses it specifically tracking Human vs AI messages.

### 2. Gemini Exports (HTML / Takeout)
*   **Format:** `.html`
*   **Rule:** The filename **must** contain the word `activity` or `gemini`.
*   **Example:** `My_Gemini_Activity.html`
*   *Behavior:* Strips out Google tracking headers and parses out prompts and responses cleanly.

### 3. PDF Documents (Coursework, Papers, E-books)
*   **Format:** `.pdf`
*   **Rule:** Standard PDF documents (text-based). 
*   **Example:** `Linear_Algebra_Notes.pdf` or `Attention_Is_All_You_Need.pdf`
*   *Behavior:* Extracts multi-page text using PDFMiner/PyMuPDF into logical reading chunks.

### 4. Images & Scans (Diagrams, Receipts, Screenshots)
*   **Format:** `.png`, `.jpg`, `.jpeg`, `.webp`, `.bmp`, `.tiff`
*   **Rule:** Can be named anything.
*   **Requirements:** You must have [Tesseract-OCR](https://github.com/UB-Mannheim/tesseract/wiki) installed on your system. 
*   *Behavior:* Automatically performs Optical Character Recognition (OCR) to rip the text out of the image and inject it into your vector database.

### 5. Plain Text, Emails, Code, CSVs, and Markdown
*   **Format:** `.md`, `.txt`, `.csv`, `.log`, `.rst`
*   **Rule:** Can be named anything.
*   **Example:** `Meeting_Notes.md` or `email_invoice.txt`
*   *Behavior:* The default fallback. Reads the file as plain structured text and intelligently slices it up. Large CSVs are skipped if they exceed 15MB to prevent clogging the embedding memory.

---

## 🚀 How to Add It to the Cloud

Once your files are dropped into the folders:
1. Open your terminal in the `ai-memory-mcp` folder.
2. Run the ingestion and cloud synchronization command:
   ```bash
   python sync.py push
   ```
3. The script will:
   - Identify *only* the new files.
   - Run the appropriate parser (OCR, HTML extractor, etc.).
   - Break them down into semantic chunks and generate embeddings.
   - Compress the new database.
   - Securely upload the compressed database to Supabase.
   
Your Claude mobile app (and Antigravity!) will automatically have access to this new data upon their next query!