# Julia — WinnCare RAG Assistant: Technical Documentation

**Stack:** Python 3.12 · FastAPI · Qdrant · BGE-M3 · BGE Reranker · Groq (LLaMA 3.3-70b) · React · Vite  
**Deployment:** HuggingFace Spaces (Docker, 2 vCPU / 16 GB RAM, no GPU)  
**Live URL:** `https://esra2001-winncare.hf.space`

---

## Table of Contents

1. [System Overview](#1-system-overview)
2. [Architecture](#2-architecture)
3. [Request Lifecycle](#3-request-lifecycle)
4. [Ingestion Pipeline](#4-ingestion-pipeline)
5. [Retrieval Pipeline](#5-retrieval-pipeline)
6. [Answer Generation](#6-answer-generation)
7. [API Reference](#7-api-reference)
8. [Configuration Reference](#8-configuration-reference)
9. [Frontend](#9-frontend)
10. [Authentication](#10-authentication)
11. [Conversation Persistence](#11-conversation-persistence)
12. [Evaluation Harness (RAGAS)](#12-evaluation-harness-ragas)
13. [Deployment](#13-deployment)
14. [Local Development](#14-local-development)
15. [Operational Notes](#15-operational-notes)
16. [Known Limitations and Trade-offs](#16-known-limitations-and-trade-offs)

---

## 1. System Overview

Julia is a multilingual Retrieval-Augmented Generation (RAG) chatbot built for WinnCare Tunisia. It answers questions about internal documents — product specifications, procedures, compliance reports, price lists, capacity data — by retrieving relevant content from an indexed corpus and generating grounded, cited answers.

**Supported document types:** PDF (any layout including scans, screenshots, tables, charts, forms), Excel (.xlsx, .xls, .xlsm)  
**Supported languages:** French, Arabic, English (any mix per message)  
**Deployment model:** Single-tenant, shared corpus. All authenticated users query the same document set.

### What Julia does (and doesn't do)

| Does | Does not |
|------|----------|
| Answer factual questions grounded in uploaded documents | Make up information not in documents |
| Extract data from scanned PDFs using vision LLM | Answer from general internet knowledge |
| Query Excel sheets using generated SQL | Execute arbitrary SQL or modify data |
| Hold a 5-turn conversation with context | Maintain separate per-user memory |
| Abstain when evidence is weak or absent | Guess when uncertain |
| Answer in the user's language | Force a fixed language |

---

## 2. Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│  BROWSER                                                                │
│  React SPA (Vite)                                                       │
│    ChatView ── SSE token stream ──────────────────────────┐            │
│    UploadView ── multipart POST ──────────────────────┐   │            │
└──────────────────────────────────────────────────────────────────────── │
                                                         │   │
┌───────────────────────────── FASTAPI (port 7860) ───────│───│─────────┐
│                                                         │   │         │
│  /ingest/*  ──── upload.py ──────────────────────────── ┘   │         │
│                      │                                       │         │
│                 ┌────▼──────────────────────────────┐        │         │
│                 │ INGESTION PIPELINE                │        │         │
│                 │  parsers.py   (PDF → pages)       │        │         │
│                 │  chunking.py  (pages → chunks)    │        │         │
│                 │  embeddings.py (BGE-M3)            │        │         │
│                 │  store.py     (→ Qdrant)           │        │         │
│                 │  tabular.py   (Excel → DuckDB)     │        │         │
│                 └──────────────────────────────────┘        │         │
│                                                              │         │
│  /query/ask/stream ── router.py ──────────────────────────── ┘         │
│                           │                                             │
│                    ┌──────▼──────────────────────────────────────┐     │
│                    │ QUERY PIPELINE                               │     │
│                    │  condense.py   (history-aware rewrite)       │     │
│                    │  classify.py   (casual / descriptive / SQL)  │     │
│                    │  retrieval.py  (BGE-M3 + Qdrant + reranker)  │     │
│                    │  generate.py   (Groq LLaMA / GPT-4o-mini)    │     │
│                    │  text_to_sql.py (DuckDB SQL path)            │     │
│                    └──────────────────────────────────────────────┘    │
│                                                                         │
│  /auth/*         auth/router.py  (JWT, 30-day tokens)                  │
│  /conversations  conversations/  (SQLite / Turso)                      │
│  /health         system health check                                    │
└─────────────────────────────────────────────────────────────────────────┘

External services:
  Qdrant Cloud  — vector + sparse index
  Groq API      — primary LLM (LLaMA 3.3-70b, free tier 100K TPD)
  OpenAI API    — vision LLM for PDF parsing (VISION_API_KEY)
                  fallback LLM when Groq quota exhausted (FALLBACK_API_KEY)
  Turso         — cloud SQLite for conversation storage (optional)
```

---

## 3. Request Lifecycle

A complete streaming chat request from browser to response:

```
User sends message
        │
        ▼
[1] condense.py — history-aware query rewrite
    If history exists, rewrite follow-up ("et la clause ?")
    into a standalone query ("Quelle clause ISO 13485 concerne X ?").
    Uses Groq (cheap, 60 max_tokens). Skipped on first turn.
        │
        ▼
[2] classify.py — intent classification
    Labels the condensed query as: casual | descriptive | numeric
    · casual     → skip retrieval, Julia answers from personality
    · descriptive → vector search path
    · numeric    → SQL path (if tabular data exists)
    Uses Groq (5 max_tokens, one word output).
        │
     ┌──┴──────────────────────────┐
     │ casual path                 │ descriptive/numeric path
     ▼                             ▼
[3a] stream_answer()           [3b] retrieve()
     chunks = []                    BGE-M3 embed query + keyword variant
     Julia replies from             Qdrant hybrid search (dense + sparse RRF)
     system personality             BGE cross-encoder rerank
                                    MMR deduplication
                                    Source-window expansion
                                    Abstain if best score < 0.08
        │                               │
        │                    ┌──────────┘
        │                    │ descriptive   │ numeric + tabular data
        │                    ▼               ▼
        │               [4] generate.py  [4b] text_to_sql.py
        │               Build context       Generate SQL → DuckDB
        │               [S1][S2] IDs        Render result as text
        │               Groq streaming      Single LLM call
        │               → SSE tokens
        │                    │
        └────────────────────┘
                             │
                             ▼
[5] SSE done event
    { citations, source_map, chunks_used, path, provider }
    Citations verified against source_map (no hallucinated filenames)
```

---

## 4. Ingestion Pipeline

### Entry point: `POST /ingest/upload`

The upload endpoint saves the file, creates a job record, and runs ingestion in a thread-pool executor (non-blocking). PDF and Excel follow separate paths.

### PDF path

```
File on disk
    │
    ▼
parsers.parse_pdf()        — iterates pages with PyMuPDF
    │
    ├── For each page:
    │     PyMuPDF text extraction (fast, free)
    │     has_images = bool(page.get_images())
    │
    │     if has_images OR text < 2000 chars:
    │         Vision LLM path (GPT-4o-mini)
    │         render page to PNG at 200 DPI → base64
    │         send to OpenAI-compatible vision API
    │         returns markdown: headings, tables, form fields, chart data
    │         if vision_text > pymupdf_text → use vision result
    │
    │     elif text < 50 chars:
    │         Tesseract OCR (ara+fra+eng) as last resort
    │
    │     else:
    │         PyMuPDF text used as-is
    │
    ▼
chunking.chunk_pages()
    │
    ├── Tables from PyMuPDF path → atomic chunks (never split)
    │   prefix "[TABLE]\n" + markdown rows
    │
    └── Body text → paragraph-boundary splitting
        _CHUNK_CHARS   = 2500 chars
        _OVERLAP_CHARS = 300 chars (last paragraph repeated at next chunk start)
        Each chunk gets: chunk_id (UUID), content_hash (SHA-256[:16]),
        chunk_index (doc-wide 0-based position), source_language (langdetect)
    │
    ▼
embeddings.embed_texts()   — BGE-M3 (BAAI/bge-m3)
    Each chunk → dense vector (1024-dim) + sparse vector (lexical weights)
    Batch size 12, max_length 1024 tokens, CPU inference
    │
    ▼
store.upsert_chunks()      — Qdrant upsert
    Payload per point: doc_id, filename, page, source_language,
                       category, text, content_hash, chunk_index
    Indexed fields: doc_id, filename, category, source_language
```

### Excel path

```
File on disk
    │
    ▼
tabular.load_excel_to_duckdb()
    For each sheet:
        _find_header_row() — detects actual header (not always row 0)
        _coerce_numeric_columns() — object → numeric when ≥80% convertible
        CREATE TABLE tbl_<doc_id_prefix>_<sheet_name> AS SELECT * FROM df
        Register in _table_registry
    │
    ▼
tabular.get_prose_chunks_for_qdrant()
    Converts each sheet to "col: value | col: value" text lines
    Chunks and embeds into Qdrant (same pipeline as PDF body text)
    page = sheet_name (string) so citations read as "[file, sheet SheetName]"
```

### Duplicate detection

Before any ingestion, `store.delete_by_filename(filename)` removes all existing Qdrant points for that filename. Re-uploading the same document replaces, not duplicates, the previous chunks.

---

## 5. Retrieval Pipeline

File: `backend/app/query/retrieval.py`

```python
# Score thresholds
_MIN_SCORE     = 0.05   # drop chunks with reranker score below this
_ABSTAIN_SCORE = 0.08   # if best surviving chunk < this, return [] → model abstains

# MMR diversity
_MMR_OVERLAP_THRESHOLD = 0.70   # Jaccard similarity above this = duplicate → drop

# Window expansion
_EXPAND_WINDOW = True   # fetch same-doc adjacent-page chunks for top result
```

### Step-by-step

**1. Query expansion**  
Two variants are embedded: the original query and a stop-word-stripped keyword form.  
Example: "quelle est la composition du PB2050683" → also embeds "composition PB2050683"

**2. BGE-M3 embedding**  
Both variants embedded in one batch. Results cached in a 256-entry LRU dict keyed by query string.  
Returns: `dense` (1024-float list) + `sparse` (token_id → weight dict)

**3. Qdrant hybrid search**  
One `Prefetch` pair (dense + sparse) per query variant. Qdrant fuses all prefetches using Reciprocal Rank Fusion (RRF). Candidate pool = `top_k × 6` when reranking is enabled.

**4. BGE reranker**  
Cross-encoder `BAAI/bge-reranker-v2-m3` scores every (query, chunk) pair. `normalize=True` maps scores to [0, 1].

**5. Abstention check**  
If the highest reranker score is below `_ABSTAIN_SCORE = 0.08`, return an empty list. The generator then tells the model "no relevant information found" and the model correctly abstains.

**6. MMR deduplication**  
After sorting by score, walk the list. Drop any chunk whose character k-gram Jaccard similarity with a higher-ranked already-selected chunk exceeds 0.70. Prevents 3 near-identical paragraphs consuming context budget.

**7. Source-window expansion**  
For the top-ranked chunk, fetch all other chunks from the same `doc_id` on adjacent pages (page ± 1) from Qdrant. Append them to the context. Gives the model surrounding paragraphs without full parent-child retrieval.

---

## 6. Answer Generation

File: `backend/app/query/generate.py`

### Context construction

Retrieved chunks are formatted as:

```
[S1] procedure_ISO13485.pdf, page 12
<chunk text>

---

[S2] procedure_ISO13485.pdf, page 13
<chunk text>

---

[S3] fiche_produit_PB2050683.pdf, page 2
<chunk text>
```

The source IDs `[S1]`, `[S2]`, `[S3]` are stable — the model can only reference IDs it sees in the context. It cannot invent a filename. `extract_citations()` maps `[S1]` back to the real `{filename, page, snippet}` after generation.

Context budget: **10,000 chars** total (~2,500 tokens). Per-chunk ceiling: **2,000 chars**.

### System prompt structure

The system prompt is structured in sections:

- **Identity** — Julia, WinnCare Tunisia assistant
- **Conversation style** — casual messages need no citations
- **Document Q&A rules** — ground claims in context; cite with [S1]; abstain when missing; no cross-product contamination; ask clarifying questions when ambiguous
- **Formatting** — Markdown, bold, bullet lists
- **Language rule** (non-negotiable) — reply in the EXACT language of the LATEST user message, ignoring history language

### Groq streaming + fallback

```
Primary: Groq API (LLaMA 3.3-70b-versatile)
    stream=True → token-by-token SSE
    temperature=0.0, max_tokens=1200

    If 429 / rate_limit / quota / 5xx:
        ▼
Fallback: FALLBACK_API_KEY provider (default gpt-4o-mini)
    stream=False → complete response
    Simulated streaming: word-by-word with 12ms delay

    If fallback also fails:
        ▼
Error message: "⚠️ Limite quotidienne atteinte..." with retry-after time
```

Done event payload:
```json
{
  "type": "done",
  "citations": [{"filename": "...", "page": 12, "snippet": "..."}],
  "source_map": [...],
  "chunks_used": 6,
  "path": "vector",
  "provider": "groq"
}
```

### Conversation memory

Last **10 messages (5 turns)** are included in every request as `history`. The condensation step (`condense.py`) also uses 5 turns to rewrite follow-up queries.

---

## 7. API Reference

All routes require `Authorization: Bearer <token>` except `/auth/login` and `/health`.

### Authentication

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/auth/login` | `{email, password}` → `{token, email}`. Token is HS256 JWT, 30-day expiry. |
| `GET`  | `/auth/me` | Returns `{email}` for the bearer token. |

### Ingestion

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/ingest/upload` | Multipart: `file` (PDF/Excel), `category` (string, optional). Returns `{job_id, doc_id, status}`. Ingestion is async. |
| `GET`  | `/ingest/status/{job_id}` | Returns job status: `queued \| processing \| done \| failed`, `chunks_stored`, `error`. |
| `GET`  | `/ingest/documents` | Lists all indexed documents from Qdrant + DuckDB. |
| `DELETE` | `/ingest/documents/{doc_id}` | Delete all chunks for a document from Qdrant and DuckDB. |
| `DELETE` | `/ingest/documents` | Drop and recreate the Qdrant collection; clear DuckDB; delete uploaded files. |

### Query

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/query/ask/stream` | **Primary endpoint.** SSE stream. Body: `{question, history, top_k, category}`. Emits `status`, `token`, `done`, `error` events. |
| `POST` | `/query/ask` | Non-streaming fallback. Same body, returns full `AskResponse`. Used by RAGAS eval. |

#### SSE event types

```
data: {"type": "status", "status": "searching"}
data: {"type": "status", "status": "generating"}
data: {"type": "token", "content": "La composition"}
data: {"type": "token", "content": " est 49%"}
data: {"type": "done", "citations": [...], "source_map": [...], "chunks_used": 6, "path": "vector", "provider": "groq"}
data: {"type": "error", "message": "..."}
```

SSE headers prevent proxy buffering:
```
Cache-Control: no-cache
X-Accel-Buffering: no
Connection: keep-alive
```

#### AskResponse schema

```typescript
{
  answer:      string
  citations:   Array<{ filename: string; page: number | string; snippet?: string }>
  source_map:  Array<{ filename: string; page: number | string; snippet: string }>
  chunks_used: number
  path:        "vector" | "sql"
  sql:         string | null
  contexts:    string[]   // raw chunk texts (used by RAGAS eval harness only)
}
```

### Conversations

| Method | Path | Description |
|--------|------|-------------|
| `GET`    | `/conversations` | List user's conversations (id, title, timestamps). |
| `POST`   | `/conversations` | Create: `{title, messages}`. Returns conversation object. |
| `GET`    | `/conversations/{id}` | Get conversation with full message array. |
| `PUT`    | `/conversations/{id}` | Update title and messages. |
| `DELETE` | `/conversations/{id}` | Delete conversation. |

### System

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Returns env, qdrant_url, embedding_model, groq_model. No auth required. |

---

## 8. Configuration Reference

All settings live in `backend/app/config.py` and are read from environment variables (or `backend/.env` for local development). **Never commit `backend/.env`** — it contains real API keys.

### Required in production

| Variable | Description |
|----------|-------------|
| `GROQ_API_KEY` | Groq API key. Free tier: 100K tokens/day. |
| `JWT_SECRET` | Random string for HS256 JWT signing. Use `openssl rand -hex 32`. |
| `AUTH_USERS` | JSON object `{"email@example.com": "password"}`. Supports multiple users. |
| `QDRANT_URL` | Qdrant server URL. E.g. `https://xxx.cloud.qdrant.io:6333` |
| `QDRANT_API_KEY` | Qdrant Cloud API key (blank for local). |

### Optional but recommended

| Variable | Default | Description |
|----------|---------|-------------|
| `GROQ_MODEL` | `llama-3.3-70b-versatile` | Primary LLM. LLaMA 3.3-70b is free on Groq and supports 128K context. |
| `VISION_API_KEY` | `` | API key for vision LLM (PDF parsing). Uses `VISION_BASE_URL` endpoint. If blank, vision path is disabled and PyMuPDF + Tesseract are used. |
| `VISION_BASE_URL` | `https://api.openai.com/v1` | Vision API endpoint (OpenAI-compatible). |
| `VISION_MODEL` | `gpt-4o-mini` | Vision model. gpt-4o-mini: ~$0.01/40-page document. |
| `FALLBACK_API_KEY` | `` | API key for text-generation fallback when Groq quota is exhausted. |
| `FALLBACK_BASE_URL` | `https://api.openai.com/v1` | Fallback LLM endpoint. |
| `FALLBACK_MODEL` | `gpt-4o-mini` | Fallback model name. |
| `RERANK_ENABLED` | `true` | Set `false` to skip cross-encoder reranking (faster, lower precision). |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | Cross-encoder model. |
| `TURSO_URL` | `` | Turso database URL for persistent conversation storage. If blank, uses local SQLite (lost on container restart). |
| `TURSO_TOKEN` | `` | Turso auth token. |

### Less commonly changed

| Variable | Default | Description |
|----------|---------|-------------|
| `QDRANT_COLLECTION` | `rag_documents` | Qdrant collection name. |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | Embedding model. Do not change without re-ingesting all documents. |
| `EMBEDDING_DEVICE` | `cpu` | `cpu`, `cuda`, `mps`. HF Spaces has no GPU — always `cpu`. |
| `DUCKDB_PATH` | `data/tabular.duckdb` | DuckDB file path (relative to working dir). |
| `UPLOAD_DIR` | `data/uploads` | Uploaded source files directory. |
| `LOG_LEVEL` | `INFO` | Python logging level. |
| `APP_ENV` | `development` | Visible in `/health`. |

---

## 9. Frontend

**Stack:** React 18, TypeScript, Vite, react-markdown with remark-gfm

### Component structure

```
App.tsx
├── LoginPage.tsx         — email + password form, token stored in localStorage
│
└── (authenticated)
    ├── Sidebar.tsx       — conversation list, new chat, sign out
    │
    └── main content
        ├── ChatView.tsx  — message thread, SSE streaming, citation chips
        └── UploadView.tsx — file drop zone, ingestion status polling
```

### Key frontend behaviors

**Streaming:** `askQuestionStream()` in `api.ts` reads the SSE response using `ReadableStream`. Status events update the thinking indicator ("Recherche dans les documents…" / "Rédaction de la réponse…"). Token events append to the message in real time.

**Citations:** Citation chips appear below each assistant message. `page` can be `number` (PDF page) or `string` (Excel sheet name). Hovering a chip shows the first 300 chars of that source chunk as a tooltip.

**Markdown:** `ReactMarkdown` with `remark-gfm` renders tables, bold, code, blockquotes. Vision LLM pages return markdown tables which render correctly.

**Conversation persistence:** After each assistant response, the full message array is saved via `PUT /conversations/{id}`. On sidebar click, `GET /conversations/{id}` restores the thread.

**Languages:** UI strings are in `i18n.ts`, supporting `en` and `fr`. Language preference stored in `localStorage`.

### Build output

Vite builds to `frontend/dist/`. The Dockerfile copies this to `backend/static/`. FastAPI serves `static/assets/` and falls back to `static/index.html` for any path not matched by an API route (SPA client-side routing).

---

## 10. Authentication

**Mechanism:** HS256 JWT, 30-day expiry.

**User management:** Users are defined in the `AUTH_USERS` environment variable as a JSON object:
```json
{"alice@winncare.tn": "password1", "bob@winncare.tn": "password2"}
```

There is no self-registration or password reset. Add/remove users by updating the secret.

**Token storage:** Stored in `localStorage` under key `julia_token`. Auto-cleared on 401 response and page reloads to the login screen.

**Authorization on API routes:** All query, ingestion, and conversation routes depend on `get_current_user()`. A missing or invalid token returns HTTP 401 immediately.

> **Note:** This is a simple single-tenant auth model. For multi-tenant deployments with data isolation, each user would need their own Qdrant namespace or collection, and the `doc_id` / `filename` payload indexes would need a `tenant_id` field.

---

## 11. Conversation Persistence

File: `backend/app/conversations/store.py`

Conversations are stored in SQLite (local dev) or Turso (production).

**Schema:**
```sql
CREATE TABLE conversations (
    id          TEXT PRIMARY KEY,
    user_email  TEXT NOT NULL,
    title       TEXT NOT NULL,
    messages    TEXT NOT NULL DEFAULT '[]',   -- JSON array
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL
)
```

Messages are stored as a JSON blob. Each message: `{role: "user"|"assistant", text: string, meta?: AskResponse}`.

**Turso:** Turso is a cloud SQLite service with a free tier. Set `TURSO_URL` and `TURSO_TOKEN` in secrets. Without it, conversations are stored in `data/conversations.db` which is ephemeral on HF Spaces (lost on container restart).

---

## 12. Evaluation Harness (RAGAS)

File: `backend/app/eval/ragas_eval.py`  
Questions: `backend/app/eval/data/questions.jsonl`  
Results: `backend/app/eval/results/report.json`

### Metrics

| Metric | What it measures | Target |
|--------|-----------------|--------|
| `faithfulness` | Fraction of Julia's claims that are verifiable from retrieved chunks. 1.0 = zero hallucination. | ≥ 0.85 |
| `context_precision` | Were retrieved chunks actually useful? Penalizes noise at the top of the ranking. | ≥ 0.75 |
| `context_recall` | Did retrieval cover everything needed to answer? Requires `ground_truth`. | ≥ 0.80 |

### Running the eval

The backend must be running. Run in a separate virtual environment (dependency conflict between FlagEmbedding and RAGAS on `datasets` version):

```bash
cd backend
python -m venv .venv-eval
.venv-eval\Scripts\activate          # Windows
# or: source .venv-eval/bin/activate  # Mac/Linux
pip install -r requirements-eval.txt
python -m app.eval.ragas_eval
```

### Questions file format

```jsonl
{"question": "Quelle est la composition du PB2050683 ?", "ground_truth": "49% PE - 51% PU"}
{"question": "Quel est le seuil AQL pour les défauts critiques ?", "ground_truth": "AQL 0,065"}
```

Lines starting with `#` are comments.

### Interpreting results

```
faithfulness      < 0.80 → Julia hallucinates — tighten the system prompt or reduce context noise
context_precision < 0.70 → Retrieval is noisy — tune top_k, chunking, or reranker threshold
context_recall    < 0.70 → Retrieval misses info — check chunking size, vision threshold, re-ingest
```

The report surfaces the 3 lowest-scoring questions by combined faithfulness + recall. Fix these first.

---

## 13. Deployment

### HuggingFace Spaces (current)

The app runs as a Docker container on HF Spaces. Push to the `main` branch of the HF Space repository triggers a rebuild.

**Remote:** `https://huggingface.co/spaces/esra2001/winncare` (git remote: `origin`)

**Build:** Two-stage Docker build. Stage 1 builds the React SPA. Stage 2 is Python 3.12-slim, installs backend deps, copies the Vite `dist/` into `backend/static/`, pre-downloads BGE-M3 (~2 GB baked into image layer so the first request is fast).

**Exposed port:** 7860 (HF Spaces requirement).

**Secrets (set in HF Space dashboard, not in git):**

```
GROQ_API_KEY
JWT_SECRET
AUTH_USERS
QDRANT_URL
QDRANT_API_KEY
VISION_API_KEY
VISION_BASE_URL
VISION_MODEL
FALLBACK_API_KEY
FALLBACK_BASE_URL
FALLBACK_MODEL
RERANK_ENABLED
RERANKER_MODEL
GROQ_MODEL
TURSO_URL
TURSO_TOKEN
```

**Persistent storage:** HF Spaces ephemeral disk — `data/uploads/` and `data/conversations.db` are lost on restarts. Use Qdrant Cloud for vectors and Turso for conversations to achieve full persistence across restarts.

**Triggering a rebuild without code changes:**
```bash
git commit --allow-empty -m "trigger rebuild" && git push origin HEAD:main
```

### Production (self-hosted)

Use `docker-compose.yml` for local Qdrant and Redis, then run the app container separately:

```bash
docker-compose up -d           # starts Qdrant (port 6333) and Redis (port 6379)
docker build -t julia-rag .
docker run -p 7860:7860 \
  -e GROQ_API_KEY=... \
  -e JWT_SECRET=... \
  -e AUTH_USERS='{"admin@company.com":"password"}' \
  -e QDRANT_URL=http://host.docker.internal:6333 \
  julia-rag
```

---

## 14. Local Development

### Backend

```bash
cd backend

# Create and activate virtual environment
python -m venv .venv
.venv\Scripts\activate       # Windows
# source .venv/bin/activate  # Mac/Linux

pip install -r requirements.txt

# Start infrastructure
docker-compose up -d   # Qdrant + Redis

# Configure
cp .env.example .env   # fill in GROQ_API_KEY at minimum

# Run
uvicorn app.main:app --reload --port 8000
```

Minimal `.env` for local dev:
```env
GROQ_API_KEY=gsk_...
JWT_SECRET=any-random-string
AUTH_USERS={"dev@local.com": "dev"}
QDRANT_URL=http://localhost:6333
```

### Frontend

```bash
cd frontend
npm install
npm run dev     # Vite dev server on http://localhost:5173
```

Vite proxies `/query`, `/ingest`, `/auth`, `/conversations` to `http://localhost:8000` (configured in `vite.config.ts`). The SPA runs on its own port; no CORS issues in dev.

### Building for production (manual)

```bash
cd frontend && npm run build      # → frontend/dist/
cp -r frontend/dist backend/static
cd backend && uvicorn app.main:app --port 7860
```

---

## 15. Operational Notes

### After changing chunking or parsing logic

Re-ingest all documents. Old chunks in Qdrant do not reflect new chunk sizes, overlap, content_hash, or chunk_index. Steps:
1. Go to the Upload tab → "Clear all documents"
2. Re-upload each document

### After changing the embedding model

The new model produces incompatible vectors. You must recreate the collection:
1. Delete collection in Qdrant dashboard (or clear all via UI)
2. Re-ingest all documents
3. Update `EMBEDDING_MODEL` in secrets before re-ingesting

### Groq daily quota (free tier)

Groq free tier: **100,000 tokens/day**. At typical usage (question + 5-turn history + 10K context + 1200 token answer ≈ 14K tokens/request), that's ~7 full requests before hitting the daily limit.

Options:
- Set `FALLBACK_API_KEY` + `FALLBACK_MODEL` to automatically failover to OpenAI/Gemini
- Upgrade Groq plan
- Switch `GROQ_MODEL` to `llama-3.1-8b-instant` (6K TPM limit but uses fewer tokens)

The 429 error message shown to users includes a retry-after time extracted from the Groq error response.

### Vision LLM costs

At GPT-4o-mini pricing (~$0.075 per 1M input tokens, ~$0.30 per 1M output tokens), a 40-page document where 20 pages go through vision costs roughly $0.02–$0.05. Budget accordingly for large corpora.

### Reranker memory

`BAAI/bge-reranker-v2-m3` loads ~550 MB into RAM on first query. On HF Spaces (16 GB), this is fine alongside BGE-M3 (~2 GB). Total model footprint: ~2.6 GB.

### SSE and proxy buffering

The `/query/ask/stream` endpoint sends three headers to prevent proxy buffering:
- `X-Accel-Buffering: no` — disables nginx buffering (critical for HF Spaces)
- `Cache-Control: no-cache` — tells intermediate proxies not to cache the stream
- `Connection: keep-alive` — prevents idle-timeout drops on long responses

If responses appear cut or arrive all at once on a specific network, a local proxy is the cause. These headers resolve it for standard nginx-based setups.

---

## 16. Known Limitations and Trade-offs

### Free-tier constraints

| Constraint | Impact | Workaround |
|-----------|--------|------------|
| Groq: 100K tokens/day | ~7 full requests before quota | Set FALLBACK_API_KEY |
| HF Spaces: 2 vCPU, no GPU | BGE-M3 embed ~3-5s per chunk batch; reranker ~2-4s per request | Pre-download models in Dockerfile (already done) |
| HF Spaces: ephemeral disk | Uploads and local conversations lost on restart | Use Qdrant Cloud + Turso |

### Architecture trade-offs

**No parent-child retrieval.** Chunks are 2500 chars with 300 chars overlap. Source-window expansion (fetch adjacent page chunks for top result) partially compensates. True parent-child would require dual-index ingestion at different granularities.

**Single tenant.** All users share one Qdrant collection and one DuckDB file. No per-user document isolation. Adding `tenant_id` to every chunk payload and filtering on it is the path to multi-tenancy.

**Vision LLM at ingestion time only.** PDF parsing with vision happens when a document is uploaded, not at query time. Changing the vision model requires re-ingesting documents to benefit.

**SQL path for aggregations only.** `classify.py` routes to DuckDB only for `numeric` questions (sum, count, average, max, min across many rows). Single-value lookups ("what is the price of X?") go through the vector path even when the value is in an Excel sheet.

**No streaming fallback.** When Groq fails and the fallback provider responds, the answer is simulated as word-by-word streaming (12ms delay per word) rather than true token streaming. The experience is slightly less responsive but visually similar.

### Accuracy targets (current RAGAS baseline)

| Metric | Baseline | Target |
|--------|----------|--------|
| faithfulness | 0.50 | ≥ 0.85 |
| context_precision | 0.67 | ≥ 0.75 |
| context_recall | 0.71 | ≥ 0.80 |

The main drivers of low faithfulness are: weak context (already addressed by abstention threshold) and the model summarizing across documents rather than citing specific claims. Running RAGAS after re-ingestion with the Phase 1 changes should show measurable improvement in all three metrics.