RAG_SYSTEM_DOCS.md

# LoomChat RAG — Complete System Documentation

---

## 1. Architecture Overview

```
                        HF Space: idnameraj/rag-vs
                        (cpu-basic: 2 vCPU, 16 GB RAM)
+----------------------------------------------------------+
|                                                          |
|   FastAPI (uvicorn, port 7860)                           |
|   +------------+  +--------------+  +----------------+   |
|   | /upload    |  | /query       |  | /reset         |   |
|   | (ingest)   |  | (RAG search) |  | (clear DB)     |   |
|   +-----+------+  +------+-------+  +----------------+   |
|         |                |                               |
|   +-----v------+  +------v-------+                       |
|   | Sentence   |  | LanceDB      |                       |
|   | Transformer|  | (vector DB)  |                       |
|   | MiniLM-L6  |  | /tmp (fast)  |                       |
|   +------------+  +------+-------+                       |
|                          | incremental sync              |
|                   +------v-------+                       |
|                   | /data bucket |                       |
|                   | (FUSE mount) |                       |
|                   +--------------+                       |
|                                                          |
|   +------------------+                                   |
|   | Keep-Alive Loop  |  self-ping every 15 min           |
|   +------------------+                                   |
+----------------------------------------------------------+
                           |
                    HTTP (httpx)
                           |
              +------------v------------+
              | HF Space:               |
              | idnameraj/              |
              |   loomchat-ollama-new   |
              |                         |
              | Ollama server           |
              | Model: qwen2.5:1.5b     |
              | Endpoint: /api/generate |
              +-------------------------+
```

---

## 2. Infrastructure (Two HF Spaces)

| Component | HF Space | Hardware | Role |
|-----------|----------|----------|------|
| **RAG API** | `idnameraj/rag-vs` | cpu-basic (2 vCPU, 16 GB RAM) | Embedding, vector search, document ingestion, API |
| **Ollama LLM** | `idnameraj/loomchat-ollama-new` | Separate Space | Hosts qwen2.5:1.5b for answer generation |
| **Storage Bucket** | `idnameraj/loomchat-rag-data` | FUSE-mounted at `/data` | Persistent vector DB storage across restarts |

---

## 2.1 Authentication — `RAG_API_SECRET`

Sensitive routes on this service (`/query`, `/upload`, `/reset`, deletes, `/debug-storage`, …) require HTTP header **`X-API-Key`** when the environment variable **`RAG_API_SECRET`** is set (non-empty). The LoomChat backend must use the **same** value in its **`RAG_API_SECRET`** setting so outbound `/query` and `/upload` succeed.

| Where | What to do |
|-------|------------|
| **Hugging Face Space** | **Settings → Repository secrets** → name `RAG_API_SECRET`, value = generated secret → reboot Space |
| **Backend** (Docker / `.env`) | `RAG_API_SECRET=<same string>` (see root `docker-compose.yml`, `.env.docker`, `backend/.env.example`) |
| **Local dev** | Leave **empty on both** the Space and the backend to disable auth (no-op guard in `main.py`) |

---

## 3. File Structure

```
hf-deploy/rag-api/
├── main.py            # FastAPI app — all RAG logic (345 lines)
├── Dockerfile         # Docker build for HF Spaces deployment
├── requirements.txt   # Python dependencies (10 packages)
├── test_ui.py         # Streamlit validation UI (133 lines)
└── RAG_SYSTEM_DOCS.md # This documentation file
```

---

## 4. Module-by-Module Breakdown

---

### 4.1 Dockerfile

```dockerfile
FROM python:3.11-slim
```

| Line | What | Why |
|------|------|-----|
| `FROM python:3.11-slim` | Base image | Minimal footprint (~150 MB vs ~900 MB full) |
| `ENV HF_HOME="/tmp/hf_cache"` | HuggingFace cache dir | Bucket FUSE mount doesn't support symlinks needed by HF hub cache; `/tmp` does |
| `pip install torch --index-url .../cpu` | CPU-only PyTorch | Saves ~2.5 GB by skipping CUDA libraries (no GPU on free tier) |
| `EXPOSE 7860` | HF Spaces port | Standard port required by HF Spaces Docker SDK |
| Runs as **root** (no `USER` directive) | | Bucket mount at `/data` requires root write permissions |

---

### 4.2 Dependencies (requirements.txt)

| Package | Version | Purpose |
|---------|---------|---------|
| `fastapi` | latest | Web framework for the API |
| `uvicorn[standard]` | latest | ASGI server to run FastAPI |
| `python-multipart` | latest | File upload parsing (`multipart/form-data`) |
| `lancedb>=0.20` | 0.20+ | Embedded vector database (columnar, Lance format) |
| `sentence-transformers` | latest | Embedding model loader (wraps HuggingFace transformers + PyTorch) |
| `pandas` | latest | DataFrame for LanceDB ingestion |
| `pypdf` | latest | PDF text extraction |
| `python-docx` | latest | DOCX text extraction |
| `httpx` | latest | Async HTTP client for Ollama calls |
| `huggingface-hub` | latest | Model download from HF Hub |

---

### 4.3 Main Application (main.py)

#### 4.3.1 Constants & Globals (Lines 24-41)

```python
BUCKET_PATH = "/data"                     # FUSE-mounted HF Storage Bucket
LOCAL_DB_PATH = "/tmp/lancedb_store"       # Fast ephemeral local disk
BUCKET_DB_PATH = "/data/lancedb_store"     # Persistent bucket path
TABLE_NAME = "documents"                   # Single LanceDB table name

OLLAMA_URL = "https://idnameraj-loomchat-ollama-new.hf.space"
OLLAMA_MODEL = "qwen2.5:1.5b"
ALLOWED_EXTENSIONS = {".pdf", ".txt", ".csv", ".docx"}
```

Both `OLLAMA_URL` and `OLLAMA_MODEL` are overridable via environment variables/secrets in HF Space settings.

#### 4.3.2 Persistence — Sync Functions (Lines 44-97)

**`sync_from_bucket()`** — Called once at startup
- Copies entire `/data/lancedb_store/` to `/tmp/lancedb_store/` using `shutil.copytree`
- Wrapped in `try/except OSError` — if the FUSE mount has I/O issues, it logs a warning and starts fresh instead of crashing
- Full copy is acceptable here since it runs only once

**`sync_to_bucket()`** — Called after every upload and on shutdown
- **Incremental sync** — walks the local directory, compares each file against the bucket copy
- Only copies files where: destination is missing, OR size differs, OR source mtime is newer
- Removes stale bucket files that no longer exist locally (handles LanceDB compaction)
- Much faster than the old `rmtree` + `copytree` approach as the DB grows

```
STARTUP:   /data/lancedb_store/ --full copy--> /tmp/lancedb_store/
RUNTIME:   All reads/writes hit /tmp (fast local disk)
ON WRITE:  /tmp/lancedb_store/ --incremental--> /data/lancedb_store/
SHUTDOWN:  Final incremental sync to /data
```

#### 4.3.3 Keep-Alive Loop (Lines 100-112)

```python
KEEP_ALIVE_INTERVAL = 15 * 60  # 15 minutes
```

- Background `asyncio.Task` that pings `http://localhost:7860/` every 15 minutes
- Prevents HF free-tier inactivity sleep (which triggers after ~48h of no traffic)
- Non-blocking, failure-tolerant (silent `except`)
- Cancelled gracefully on shutdown

#### 4.3.4 Lifespan Manager (Lines 115-135)

FastAPI's `lifespan` context manager handles startup and shutdown:

**Startup sequence:**
1. `sync_from_bucket()` — restore data from persistent storage
2. `lancedb.connect(LOCAL_DB_PATH)` — open vector DB on fast local disk
3. `SentenceTransformer("all-MiniLM-L6-v2")` — load embedding model (~80 MB, 384-dim)
4. `asyncio.create_task(keep_alive_loop())` — start background ping

**Shutdown sequence:**
1. Cancel keep-alive task
2. `sync_to_bucket()` — final persistence
3. Delete models, `gc.collect()` — free memory

#### 4.3.5 Text Extractors (Lines 169-197)

| Format | Extractor | Library | Method |
|--------|-----------|---------|--------|
| `.pdf` | `extract_text_from_pdf` | `pypdf.PdfReader` | Iterates all pages, extracts text |
| `.txt` | `extract_text_from_txt` | Built-in `open()` | Reads entire file with UTF-8, ignores errors |
| `.csv` | `extract_text_from_csv` | Built-in `csv.reader` | Joins each row with commas, joins rows with newlines |
| `.docx` | `extract_text_from_docx` | `python-docx` | Extracts text from all paragraphs |

Registered in `EXTRACTORS` dict, dispatched by file extension.

#### 4.3.6 Chunking (Lines 211-218)

```python
def chunk_text(text, chunk_size=500, overlap=50):
```

- Splits extracted text into **500-character chunks** with **50-character overlap**
- Overlap ensures context isn't lost at chunk boundaries
- Strips whitespace, removes empty chunks
- A typical 10-page PDF produces ~40-80 chunks

#### 4.3.7 Embedding (Lines 261-267, inside upload)

```python
embed_model.encode(batch)  # -> 384-dim float vector per chunk
```

- Model: `sentence-transformers/all-MiniLM-L6-v2` (~22M params, ~80 MB)
- Output: 384-dimensional dense float vector per chunk
- Batch size: 16 chunks at a time (memory-defensive)
- Runs on CPU — ~50ms per batch of 16

#### 4.3.8 Vector Storage — LanceDB

- **Embedded** database (no separate server process)
- Stores data in columnar Lance format (`.lance` files)
- Single table `"documents"` with columns: `text` (string), `vector` (float[384])
- Search: approximate nearest neighbor (ANN) using L2 distance
- API compatibility helper `get_table_names()` handles both old (`table_names()`) and new (`list_tables()`) LanceDB APIs

#### 4.3.9 LLM Generation — Ollama (Lines 221-232)

```python
async def ollama_generate(prompt: str) -> str:
```

- Sends prompt to `https://idnameraj-loomchat-ollama-new.hf.space/api/generate`
- Model: **qwen2.5:1.5b** (1.5 billion parameters, ~1 GB)
- Non-streaming mode (`"stream": False`)
- Timeout: 120 seconds
- Uses `httpx.AsyncClient` with `verify=False` (HF internal SSL)

---

### 4.4 API Endpoints

#### `GET /` — Health Check (Lines 141-163)

Returns system status:
```json
{
  "status": "ok",
  "ollama": "connected",
  "ollama_model": "qwen2.5:1.5b",
  "accepted_formats": [".pdf", ".docx", ".txt", ".csv"],
  "chunks_in_db": 6
}
```
Pings Ollama (5s timeout) and counts rows in LanceDB.

#### `POST /upload` — Document Ingestion (Lines 238-295)

**Flow:**
1. Validate file extension
2. Save to temp file (`tempfile.NamedTemporaryFile`)
3. Extract text using format-specific extractor
4. Chunk text (500 chars, 50 overlap)
5. Embed in batches of 16 -> 384-dim vectors
6. Store in LanceDB (create table or append)
7. `gc.collect()` to free memory
8. Incremental sync to bucket
9. Return `{"chunks_stored": N, "filename": "..."}`

#### `POST /query` — RAG Query (Lines 305-333)

**Flow:**
1. Embed the query -> 384-dim vector
2. Vector search in LanceDB -> top 3 nearest chunks
3. Build prompt with retrieved context
4. Send to Ollama for generation
5. Return `{"answer": "...", "sources": [chunk1, chunk2, chunk3]}`

**Prompt template:**
```
Using only the context below, answer the question.
If the context does not contain the answer, say "I don't know."

Context:
{top 3 chunks joined by newlines}

Question: {user query}

Answer:
```

#### `POST /reset` — Clear Database (Lines 339-344)

Drops all LanceDB tables, syncs empty state to bucket.

---

### 4.5 Test UI (test_ui.py)

Streamlit app pointing at `https://idnameraj-rag-vs.hf.space`:
- Health dashboard (Ollama status, model name, chunk count)
- File upload widget (PDF, TXT, CSV, DOCX)
- Chat interface with message history (`st.session_state`)
- Source chunks displayed in expandable sections

---

## 5. Complete Data Flow

```
  User uploads file.pdf
         |
         v
  POST /upload (multipart)
         |
         v
  Save to /tmp/tmpXXXX.pdf
         |
         v
  pypdf extracts text --> "The quick brown fox..."
         |
         v
  chunk_text(500, 50) --> ["The quick brown...", "brown fox jumped...", ...]
         |
         v
  embed_model.encode() --> [[0.12, -0.34, ...], [0.56, 0.78, ...], ...]
  (batches of 16)           (384-dim each)
         |
         v
  LanceDB table.add() --> /tmp/lancedb_store/documents.lance/
         |
         v
  sync_to_bucket() --> /data/lancedb_store/ (incremental, only changed files)
         |
         v
  Return {"chunks_stored": 12}


  User asks "What did the fox do?"
         |
         v
  POST /query {"query": "What did the fox do?"}
         |
         v
  embed_model.encode("What did the fox do?") --> [0.45, -0.12, ...]
         |
         v
  LanceDB table.search(vector).limit(3) --> Top 3 nearest chunks
         |
         v
  Build prompt with context + question
         |
         v
  httpx POST to Ollama /api/generate --> qwen2.5:1.5b generates answer
         |
         v
  Return {"answer": "The fox jumped over...", "sources": [...]}
```

---

## 6. Maximum Usage & Thresholds

### RAG API Space (idnameraj/rag-vs — cpu-basic)

| Resource | Limit | Current Usage | Notes |
|----------|-------|---------------|-------|
| **vCPU** | 2 cores | ~0.1 idle, spikes to 2 during embed | Embedding is CPU-bound |
| **RAM** | 16 GB | ~2.5 GB baseline (PyTorch + model + LanceDB) | ~13 GB available for data |
| **Disk (`/tmp`)** | ~50 GB (shared ephemeral) | Minimal | LanceDB files grow here |
| **Bucket (`/data`)** | **20 GB** (free-tier HF Storage Bucket) | ~few MB | Hard limit for persistent data |
| **Max file upload** | ~100 MB (practical) | | FastAPI/uvicorn default body limits |
| **Concurrent requests** | 1 worker (uvicorn default) | | Async, but CPU-bound embedding blocks |

### Ollama Space (idnameraj/loomchat-ollama-new)

| Resource | Limit | Notes |
|----------|-------|-------|
| **Model** | qwen2.5:1.5b (~1 GB RAM) | Small enough for free-tier CPU |
| **Context window** | **32,768 tokens** | qwen2.5 native context length |
| **Effective input** | ~1,500 chars (3 chunks x 500) + query + prompt template = **~600 tokens** | Well within limit |
| **Generation timeout** | 120 seconds | Set in httpx client |
| **Inference speed** | ~5-15 tokens/sec on CPU | Varies by load |
| **Max answer length** | ~500-1000 tokens typical | No explicit `num_predict` set (model default) |

### Calculated Capacity Limits

| Metric | Estimate | How Calculated |
|--------|----------|----------------|
| **Max documents** | **~2,000-3,000 PDFs** (10 pages each) | 20 GB bucket / ~7-10 MB per 10-page PDF in LanceDB |
| **Max chunks in DB** | **~200,000-500,000** | 20 GB / (~40-100 KB per chunk with 384-dim vector) |
| **Upload speed** | ~5-10 seconds per 10-page PDF | Extract + chunk + embed (16 batch) + sync |
| **Query latency** | **3-15 seconds** total | ~50ms embed + ~10ms search + 3-15s Ollama generation |
| **Embedding throughput** | ~300 chunks/min | 16 chunks/batch, ~3s/batch on 2 vCPU |
| **Keep-alive duration** | Indefinite (with self-ping) | 15-min ping prevents sleep; HF may still restart for maintenance |

### Hard Limits to Watch

| Limit | Threshold | What Happens |
|-------|-----------|--------------|
| **Bucket storage hits 20 GB** | `sync_to_bucket()` fails with `OSError` | New uploads fail to persist; data lost on restart |
| **RAM hits 16 GB** | OOM kill (exit code 137) | Container restarts, data in `/tmp` lost (bucket has last sync) |
| **Ollama Space sleeps** | Query returns 502 | First query after sleep takes ~30-60s (cold start) |
| **Both Spaces sleep** | Full cold start | RAG: ~60-90s (model download + bucket sync), Ollama: ~30-60s |
| **Concurrent heavy uploads** | RAM spike from embedding | 16-batch limit helps, but 5+ simultaneous large files could OOM |

### Recommendations for Scaling Beyond Free Tier

| Bottleneck | Solution | Cost |
|------------|----------|------|
| Slow inference | Upgrade Ollama Space to GPU (T4) | ~$0.60/hr |
| More storage | Upgrade bucket or use external DB | $5-10/mo |
| Higher throughput | Upgrade RAG Space to cpu-upgrade (8 vCPU) | ~$0.03/hr |
| Always-on guarantee | Enable "Persistent" Space setting (paid) | Included with hardware upgrade |