DEPLOYMENT.md

Deployment Guide

Overview

This RAG system is a Python backend application that requires:

• Python runtime (3.9+)
• Ollama service running locally or remotely
• Persistent storage for ChromaDB
• 4GB+ RAM

It cannot be deployed as a static site.

Why GitHub Pages Doesn't Work

GitHub Pages serves static HTML/CSS/JS files only. This project requires:

• Python interpreter
• Long-running processes (Ollama, ChromaDB)
• Server-side document processing
• Dynamic request handling

Verdict: GitHub Pages is incompatible with this architecture.

Supported Deployment Options

1. Local Development (Recommended for Testing)

Pros:

• Full control
• No cost
• Fast iteration
• Privacy (documents stay local)

Cons:

• Not accessible remotely
• Requires manual setup

Setup:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2

# Run application
python main.py

Access at: http://localhost:7860

---

2. Hugging Face Spaces (Best for Demos)

Pros:

• Free tier available
• Gradio native support
• Public URL
• No server management

Cons:

• CPU-only (slow inference)
• Limited RAM (7GB max on free tier)
• Ephemeral storage (documents reset on restart)
• Cold start delays

Requirements:

• Create app.py (rename main.py)
• Add requirements.txt
• Configure Spaces to use Gradio SDK
• Important: Ollama must run in same container or use external endpoint

Limitations:

• Ollama models are large (~4GB for llama3.2)
• May exceed free tier storage
• Consider using smaller models (e.g., llama3.2:1b)

Example README.md for Spaces: ```yaml

title: RAG Document QA emoji: 📚 colorFrom: blue colorTo: green sdk: gradio sdk_version: 4.44.1 app_file: app.py pinned: false

---

### 3. Cloud Platforms (Production-Ready)

#### Render

**Pros**:
- Persistent storage
- Custom Docker support
- Automatic deployments from Git

**Cons**:
- Paid plans required for sufficient resources
- ~$7/month minimum for 1GB RAM

**Setup**:
1. Create `render.yaml`:
```yaml
services:
  - type: web
    name: rag-system
    runtime: python3
    buildCommand: pip install -r requirements.txt
    startCommand: python main.py
    envVars:
      - key: OLLAMA_HOST
        value: http://localhost:11434

2. Add Ollama as separate service or use external endpoint

---

Fly.io

Pros:

• Generous free tier
• Global edge deployment
• Docker-based

Cons:

• Requires Dockerfile
• Complex setup for multi-service apps

Setup:

FROM python:3.9-slim

# Install Ollama
RUN curl -fsSL https://ollama.com/install.sh | sh

# Copy application
COPY . /app
WORKDIR /app

RUN pip install -r requirements.txt

# Start Ollama and app
CMD ollama serve & python main.py

---

Railway

Pros:

• Simple Git integration
• Automatic HTTPS
• Database support

Cons:

• Free tier limited to 500 hours/month
• Resource constraints on free tier

Setup:

• Connect GitHub repository
• Set environment variables
• Deploy from main branch

---

4. Docker (Self-Hosted)

Best for: VPS, home server, enterprise deployment

Dockerfile:

FROM python:3.9-slim

# Install Ollama
RUN curl -fsSL https://ollama.com/install.sh | sh

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY . /app
WORKDIR /app

# Expose Gradio port
EXPOSE 7860

# Start services
CMD ["sh", "-c", "ollama serve & sleep 5 && ollama pull llama3.2 && python main.py"]

Docker Compose:

version: '3.8'

services:
  rag-system:
    build: .
    ports:
      - "7860:7860"
    volumes:
      - ./documents:/app/documents
      - ./chroma_db:/app/chroma_db
    environment:
      - OLLAMA_HOST=http://localhost:11434

---

Deployment Checklist

Before deploying, ensure:

• Ollama is accessible (local or remote endpoint)
• Required model is pulled (llama3.2 or alternative)
• Environment variables are set (see .env.example)
• Sufficient RAM available (4GB minimum, 8GB recommended)
• Persistent storage configured for chroma_db/
• Documents directory is populated or upload mechanism exists
• Security considerations addressed (see below)

Security Considerations

This application is not production-hardened. Before public deployment:

1. Add Authentication

   • Gradio supports basic auth: demo.launch(auth=("username", "password"))
   • Consider OAuth for multi-user scenarios

2. Rate Limiting

   • Implement request throttling
   • Prevent abuse of LLM inference

3. Input Validation

   • Sanitize uploaded documents
   • Limit file sizes and types

4. Network Security

   • Use HTTPS (reverse proxy with nginx/Caddy)
   • Restrict Ollama endpoint access

5. Resource Limits

   • Set memory limits in Docker
   • Implement query timeouts

Performance Optimization

For production deployments:

1. Use GPU Acceleration

   • Ollama supports CUDA/ROCm
   • 10-50x faster inference

2. Caching

   • Cache embeddings for frequently accessed documents
   • Implement query result caching

3. Model Selection

   • Smaller models (1B-3B params) for faster responses
   • Quantized models (Q4, Q5) for reduced memory

4. Horizontal Scaling

   • Run multiple Ollama instances
   • Load balance with nginx

Cost Estimates

Platform	Free Tier	Paid (Minimum)	Notes
Local	$0	$0	Electricity costs only
HF Spaces	Limited	$0	CPU-only, slow
Render	No	~$7/month	1GB RAM insufficient
Fly.io	500hrs	~$5/month	Requires optimization
Railway	500hrs	~$5/month	Good for demos
VPS (Hetzner)	No	~$5/month	Full control

Monitoring

Recommended monitoring for production:

• Application Logs: Track query latency, errors
• Resource Usage: RAM, CPU, disk I/O
• Ollama Metrics: Model load time, inference speed
• ChromaDB Stats: Collection size, query performance

Backup Strategy

Critical data to backup:

• chroma_db/ - Vector database
• documents/ - Source documents
• config.py - Configuration
• .env - Environment variables (encrypted)

Troubleshooting Deployments

Issue: Ollama connection refused

Solution: Ensure Ollama is running before application starts

# Add to startup script
ollama serve &
sleep 5  # Wait for Ollama to initialize
python main.py

Issue: Out of memory

Solution: Reduce model size or increase RAM

# Use smaller model
OLLAMA_MODEL_NAME = "llama3.2:1b"

# Reduce chunk retrieval
DEFAULT_N_RESULTS = 3

Issue: Slow cold starts

Solution: Keep models pre-loaded

# In Dockerfile
RUN ollama pull llama3.2

Alternative: API-Only Deployment

For advanced users, deploy as REST API instead of Gradio:

1. Replace Gradio with FastAPI
2. Create separate frontend (Vue.js, React)
3. Deploy frontend to Vercel/Netlify
4. Deploy backend to Render/Fly.io

See ARCHITECTURE.md for API design considerations.

---

Recommendation: Start with local deployment, then move to Hugging Face Spaces for demos, and finally to Render/Fly.io for production.