Hugging Face's logo

Spaces:
idnameraj
/
rag-vs
Sleeping

App Files Community
rag-vs / RAG_SYSTEM_DOCS.md
idnameraj's picture
idnameraj
Upload 4 files
b67cce4 verified
preview code
|
Raw
History Blame Contribute Delete
16 kB

LoomChat RAG — Complete System Documentation

1. Architecture Overview 

                        HF Space: idnameraj/rag-vs
                        (cpu-basic: 2 vCPU, 16 GB RAM)
+----------------------------------------------------------+
|                                                          |
|   FastAPI (uvicorn, port 7860)                           |
|   +------------+  +--------------+  +----------------+   |
|   | /upload    |  | /query       |  | /reset         |   |
|   | (ingest)   |  | (RAG search) |  | (clear DB)     |   |
|   +-----+------+  +------+-------+  +----------------+   |
|         |                |                               |
|   +-----v------+  +------v-------+                       |
|   | Sentence   |  | LanceDB      |                       |
|   | Transformer|  | (vector DB)  |                       |
|   | MiniLM-L6  |  | /tmp (fast)  |                       |
|   +------------+  +------+-------+                       |
|                          | incremental sync              |
|                   +------v-------+                       |
|                   | /data bucket |                       |
|                   | (FUSE mount) |                       |
|                   +--------------+                       |
|                                                          |
|   +------------------+                                   |
|   | Keep-Alive Loop  |  self-ping every 15 min           |
|   +------------------+                                   |
+----------------------------------------------------------+
                           |
                    HTTP (httpx)
                           |
              +------------v------------+
              | HF Space:               |
              | idnameraj/              |
              |   loomchat-ollama-new   |
              |                         |
              | Ollama server           |
              | Model: qwen2.5:1.5b     |
              | Endpoint: /api/generate |
              +-------------------------+

2. Infrastructure (Two HF Spaces)

Component HF Space Hardware Role
RAG API idnameraj/rag-vs cpu-basic (2 vCPU, 16 GB RAM) Embedding, vector search, document ingestion, API
Ollama LLM idnameraj/loomchat-ollama-new Separate Space Hosts qwen2.5:1.5b for answer generation
Storage Bucket idnameraj/loomchat-rag-data FUSE-mounted at /data Persistent vector DB storage across restarts

2.1 Authentication — RAG_API_SECRET

Sensitive routes on this service (/query, /upload, /reset, deletes, /debug-storage, …) require HTTP header X-API-Key when the environment variable RAG_API_SECRET is set (non-empty). The LoomChat backend must use the same value in its RAG_API_SECRET setting so outbound /query and /upload succeed.

Where What to do
Hugging Face Space Settings → Repository secrets → name RAG_API_SECRET, value = generated secret → reboot Space
Backend (Docker / .env) RAG_API_SECRET=<same string> (see root docker-compose.yml, .env.docker, backend/.env.example)
Local dev Leave empty on both the Space and the backend to disable auth (no-op guard in main.py)

3. File Structure 

hf-deploy/rag-api/
├── main.py            # FastAPI app — all RAG logic (345 lines)
├── Dockerfile         # Docker build for HF Spaces deployment
├── requirements.txt   # Python dependencies (10 packages)
├── test_ui.py         # Streamlit validation UI (133 lines)
└── RAG_SYSTEM_DOCS.md # This documentation file

4. Module-by-Module Breakdown

4.1 Dockerfile 

FROM python:3.11-slim
Line What Why
FROM python:3.11-slim Base image Minimal footprint (~150 MB vs ~900 MB full)
ENV HF_HOME="/tmp/hf_cache" HuggingFace cache dir Bucket FUSE mount doesn't support symlinks needed by HF hub cache; /tmp does
pip install torch --index-url .../cpu CPU-only PyTorch Saves ~2.5 GB by skipping CUDA libraries (no GPU on free tier)
EXPOSE 7860 HF Spaces port Standard port required by HF Spaces Docker SDK
Runs as root (no USER directive) Bucket mount at /data requires root write permissions

4.2 Dependencies (requirements.txt)

Package Version Purpose
fastapi latest Web framework for the API
uvicorn[standard] latest ASGI server to run FastAPI
python-multipart latest File upload parsing (multipart/form-data)
lancedb>=0.20 0.20+ Embedded vector database (columnar, Lance format)
sentence-transformers latest Embedding model loader (wraps HuggingFace transformers + PyTorch)
pandas latest DataFrame for LanceDB ingestion
pypdf latest PDF text extraction
python-docx latest DOCX text extraction
httpx latest Async HTTP client for Ollama calls
huggingface-hub latest Model download from HF Hub

4.3 Main Application (main.py)

4.3.1 Constants & Globals (Lines 24-41) 

BUCKET_PATH = "/data"                     # FUSE-mounted HF Storage Bucket
LOCAL_DB_PATH = "/tmp/lancedb_store"       # Fast ephemeral local disk
BUCKET_DB_PATH = "/data/lancedb_store"     # Persistent bucket path
TABLE_NAME = "documents"                   # Single LanceDB table name

OLLAMA_URL = "https://idnameraj-loomchat-ollama-new.hf.space"
OLLAMA_MODEL = "qwen2.5:1.5b"
ALLOWED_EXTENSIONS = {".pdf", ".txt", ".csv", ".docx"}

Both OLLAMA_URL and OLLAMA_MODEL are overridable via environment variables/secrets in HF Space settings.

4.3.2 Persistence — Sync Functions (Lines 44-97)

sync_from_bucket() — Called once at startup

  • Copies entire /data/lancedb_store/ to /tmp/lancedb_store/ using shutil.copytree
  • Wrapped in try/except OSError — if the FUSE mount has I/O issues, it logs a warning and starts fresh instead of crashing
  • Full copy is acceptable here since it runs only once

sync_to_bucket() — Called after every upload and on shutdown

  • Incremental sync — walks the local directory, compares each file against the bucket copy
  • Only copies files where: destination is missing, OR size differs, OR source mtime is newer
  • Removes stale bucket files that no longer exist locally (handles LanceDB compaction)
  • Much faster than the old rmtree + copytree approach as the DB grows
STARTUP:   /data/lancedb_store/ --full copy--> /tmp/lancedb_store/
RUNTIME:   All reads/writes hit /tmp (fast local disk)
ON WRITE:  /tmp/lancedb_store/ --incremental--> /data/lancedb_store/
SHUTDOWN:  Final incremental sync to /data

4.3.3 Keep-Alive Loop (Lines 100-112) 

KEEP_ALIVE_INTERVAL = 15 * 60  # 15 minutes
  • Background asyncio.Task that pings http://localhost:7860/ every 15 minutes
  • Prevents HF free-tier inactivity sleep (which triggers after ~48h of no traffic)
  • Non-blocking, failure-tolerant (silent except)
  • Cancelled gracefully on shutdown

4.3.4 Lifespan Manager (Lines 115-135)

FastAPI's lifespan context manager handles startup and shutdown:

Startup sequence:

  1. sync_from_bucket() — restore data from persistent storage
  2. lancedb.connect(LOCAL_DB_PATH) — open vector DB on fast local disk
  3. SentenceTransformer("all-MiniLM-L6-v2") — load embedding model (~80 MB, 384-dim)
  4. asyncio.create_task(keep_alive_loop()) — start background ping

Shutdown sequence:

  1. Cancel keep-alive task
  2. sync_to_bucket() — final persistence
  3. Delete models, gc.collect() — free memory

4.3.5 Text Extractors (Lines 169-197)

Format Extractor Library Method
.pdf extract_text_from_pdf pypdf.PdfReader Iterates all pages, extracts text
.txt extract_text_from_txt Built-in open() Reads entire file with UTF-8, ignores errors
.csv extract_text_from_csv Built-in csv.reader Joins each row with commas, joins rows with newlines
.docx extract_text_from_docx python-docx Extracts text from all paragraphs

Registered in EXTRACTORS dict, dispatched by file extension.

4.3.6 Chunking (Lines 211-218) 

def chunk_text(text, chunk_size=500, overlap=50):
  • Splits extracted text into 500-character chunks with 50-character overlap
  • Overlap ensures context isn't lost at chunk boundaries
  • Strips whitespace, removes empty chunks
  • A typical 10-page PDF produces ~40-80 chunks

4.3.7 Embedding (Lines 261-267, inside upload) 

embed_model.encode(batch)  # -> 384-dim float vector per chunk
  • Model: sentence-transformers/all-MiniLM-L6-v2 (~22M params, ~80 MB)
  • Output: 384-dimensional dense float vector per chunk
  • Batch size: 16 chunks at a time (memory-defensive)
  • Runs on CPU — ~50ms per batch of 16

4.3.8 Vector Storage — LanceDB

  • Embedded database (no separate server process)
  • Stores data in columnar Lance format (.lance files)
  • Single table "documents" with columns: text (string), vector (float[384])
  • Search: approximate nearest neighbor (ANN) using L2 distance
  • API compatibility helper get_table_names() handles both old (table_names()) and new (list_tables()) LanceDB APIs

4.3.9 LLM Generation — Ollama (Lines 221-232) 

async def ollama_generate(prompt: str) -> str:
  • Sends prompt to https://idnameraj-loomchat-ollama-new.hf.space/api/generate
  • Model: qwen2.5:1.5b (1.5 billion parameters, ~1 GB)
  • Non-streaming mode ("stream": False)
  • Timeout: 120 seconds
  • Uses httpx.AsyncClient with verify=False (HF internal SSL)

4.4 API Endpoints

GET / — Health Check (Lines 141-163)

Returns system status:

{
  "status": "ok",
  "ollama": "connected",
  "ollama_model": "qwen2.5:1.5b",
  "accepted_formats": [".pdf", ".docx", ".txt", ".csv"],
  "chunks_in_db": 6
}

Pings Ollama (5s timeout) and counts rows in LanceDB.

POST /upload — Document Ingestion (Lines 238-295)

Flow:

  1. Validate file extension
  2. Save to temp file (tempfile.NamedTemporaryFile)
  3. Extract text using format-specific extractor
  4. Chunk text (500 chars, 50 overlap)
  5. Embed in batches of 16 -> 384-dim vectors
  6. Store in LanceDB (create table or append)
  7. gc.collect() to free memory
  8. Incremental sync to bucket
  9. Return {"chunks_stored": N, "filename": "..."}

POST /query — RAG Query (Lines 305-333)

Flow:

  1. Embed the query -> 384-dim vector
  2. Vector search in LanceDB -> top 3 nearest chunks
  3. Build prompt with retrieved context
  4. Send to Ollama for generation
  5. Return {"answer": "...", "sources": [chunk1, chunk2, chunk3]}

Prompt template:

Using only the context below, answer the question.
If the context does not contain the answer, say "I don't know."

Context:
{top 3 chunks joined by newlines}

Question: {user query}

Answer:

POST /reset — Clear Database (Lines 339-344)

Drops all LanceDB tables, syncs empty state to bucket.

4.5 Test UI (test_ui.py)

Streamlit app pointing at https://idnameraj-rag-vs.hf.space:

  • Health dashboard (Ollama status, model name, chunk count)
  • File upload widget (PDF, TXT, CSV, DOCX)
  • Chat interface with message history (st.session_state)
  • Source chunks displayed in expandable sections

5. Complete Data Flow 

  User uploads file.pdf
         |
         v
  POST /upload (multipart)
         |
         v
  Save to /tmp/tmpXXXX.pdf
         |
         v
  pypdf extracts text --> "The quick brown fox..."
         |
         v
  chunk_text(500, 50) --> ["The quick brown...", "brown fox jumped...", ...]
         |
         v
  embed_model.encode() --> [[0.12, -0.34, ...], [0.56, 0.78, ...], ...]
  (batches of 16)           (384-dim each)
         |
         v
  LanceDB table.add() --> /tmp/lancedb_store/documents.lance/
         |
         v
  sync_to_bucket() --> /data/lancedb_store/ (incremental, only changed files)
         |
         v
  Return {"chunks_stored": 12}


  User asks "What did the fox do?"
         |
         v
  POST /query {"query": "What did the fox do?"}
         |
         v
  embed_model.encode("What did the fox do?") --> [0.45, -0.12, ...]
         |
         v
  LanceDB table.search(vector).limit(3) --> Top 3 nearest chunks
         |
         v
  Build prompt with context + question
         |
         v
  httpx POST to Ollama /api/generate --> qwen2.5:1.5b generates answer
         |
         v
  Return {"answer": "The fox jumped over...", "sources": [...]}

6. Maximum Usage & Thresholds

RAG API Space (idnameraj/rag-vs — cpu-basic)

Resource Limit Current Usage Notes
vCPU 2 cores ~0.1 idle, spikes to 2 during embed Embedding is CPU-bound
RAM 16 GB ~2.5 GB baseline (PyTorch + model + LanceDB) ~13 GB available for data
Disk (/tmp) ~50 GB (shared ephemeral) Minimal LanceDB files grow here
Bucket (/data) 20 GB (free-tier HF Storage Bucket) ~few MB Hard limit for persistent data
Max file upload ~100 MB (practical) FastAPI/uvicorn default body limits
Concurrent requests 1 worker (uvicorn default) Async, but CPU-bound embedding blocks

Ollama Space (idnameraj/loomchat-ollama-new)

Resource Limit Notes
Model qwen2.5:1.5b (~1 GB RAM) Small enough for free-tier CPU
Context window 32,768 tokens qwen2.5 native context length
Effective input 1,500 chars (3 chunks x 500) + query + prompt template = **600 tokens** Well within limit
Generation timeout 120 seconds Set in httpx client
Inference speed ~5-15 tokens/sec on CPU Varies by load
Max answer length ~500-1000 tokens typical No explicit num_predict set (model default)

Calculated Capacity Limits

Metric Estimate How Calculated
Max documents ~2,000-3,000 PDFs (10 pages each) 20 GB bucket / ~7-10 MB per 10-page PDF in LanceDB
Max chunks in DB ~200,000-500,000 20 GB / (~40-100 KB per chunk with 384-dim vector)
Upload speed ~5-10 seconds per 10-page PDF Extract + chunk + embed (16 batch) + sync
Query latency 3-15 seconds total ~50ms embed + ~10ms search + 3-15s Ollama generation
Embedding throughput ~300 chunks/min 16 chunks/batch, ~3s/batch on 2 vCPU
Keep-alive duration Indefinite (with self-ping) 15-min ping prevents sleep; HF may still restart for maintenance

Hard Limits to Watch

Limit Threshold What Happens
Bucket storage hits 20 GB sync_to_bucket() fails with OSError New uploads fail to persist; data lost on restart
RAM hits 16 GB OOM kill (exit code 137) Container restarts, data in /tmp lost (bucket has last sync)
Ollama Space sleeps Query returns 502 First query after sleep takes ~30-60s (cold start)
Both Spaces sleep Full cold start RAG: ~60-90s (model download + bucket sync), Ollama: ~30-60s
Concurrent heavy uploads RAM spike from embedding 16-batch limit helps, but 5+ simultaneous large files could OOM

Recommendations for Scaling Beyond Free Tier

Bottleneck Solution Cost
Slow inference Upgrade Ollama Space to GPU (T4) ~$0.60/hr
More storage Upgrade bucket or use external DB $5-10/mo
Higher throughput Upgrade RAG Space to cpu-upgrade (8 vCPU) ~$0.03/hr
Always-on guarantee Enable "Persistent" Space setting (paid) Included with hardware upgrade