Spaces:
Running
Running
| # Julia — WinnCare RAG Assistant: Technical Documentation | |
| **Stack:** Python 3.12 · FastAPI · Qdrant · BGE-M3 · BGE Reranker · Groq (LLaMA 3.3-70b) · React · Vite | |
| **Deployment:** HuggingFace Spaces (Docker, 2 vCPU / 16 GB RAM, no GPU) | |
| **Live URL:** `https://esra2001-winncare.hf.space` | |
| --- | |
| ## Table of Contents | |
| 1. [System Overview](#1-system-overview) | |
| 2. [Architecture](#2-architecture) | |
| 3. [Request Lifecycle](#3-request-lifecycle) | |
| 4. [Ingestion Pipeline](#4-ingestion-pipeline) | |
| 5. [Retrieval Pipeline](#5-retrieval-pipeline) | |
| 6. [Answer Generation](#6-answer-generation) | |
| 7. [API Reference](#7-api-reference) | |
| 8. [Configuration Reference](#8-configuration-reference) | |
| 9. [Frontend](#9-frontend) | |
| 10. [Authentication](#10-authentication) | |
| 11. [Conversation Persistence](#11-conversation-persistence) | |
| 12. [Evaluation Harness (RAGAS)](#12-evaluation-harness-ragas) | |
| 13. [Deployment](#13-deployment) | |
| 14. [Local Development](#14-local-development) | |
| 15. [Operational Notes](#15-operational-notes) | |
| 16. [Known Limitations and Trade-offs](#16-known-limitations-and-trade-offs) | |
| --- | |
| ## 1. System Overview | |
| Julia is a multilingual Retrieval-Augmented Generation (RAG) chatbot built for WinnCare Tunisia. It answers questions about internal documents — product specifications, procedures, compliance reports, price lists, capacity data — by retrieving relevant content from an indexed corpus and generating grounded, cited answers. | |
| **Supported document types:** PDF (any layout including scans, screenshots, tables, charts, forms), Excel (.xlsx, .xls, .xlsm) | |
| **Supported languages:** French, Arabic, English (any mix per message) | |
| **Deployment model:** Single-tenant, shared corpus. All authenticated users query the same document set. | |
| ### What Julia does (and doesn't do) | |
| | Does | Does not | | |
| |------|----------| | |
| | Answer factual questions grounded in uploaded documents | Make up information not in documents | | |
| | Extract data from scanned PDFs using vision LLM | Answer from general internet knowledge | | |
| | Query Excel sheets using generated SQL | Execute arbitrary SQL or modify data | | |
| | Hold a 5-turn conversation with context | Maintain separate per-user memory | | |
| | Abstain when evidence is weak or absent | Guess when uncertain | | |
| | Answer in the user's language | Force a fixed language | | |
| --- | |
| ## 2. Architecture | |
| ``` | |
| ┌─────────────────────────────────────────────────────────────────────────┐ | |
| │ BROWSER │ | |
| │ React SPA (Vite) │ | |
| │ ChatView ── SSE token stream ──────────────────────────┐ │ | |
| │ UploadView ── multipart POST ──────────────────────┐ │ │ | |
| └──────────────────────────────────────────────────────────────────────── │ | |
| │ │ | |
| ┌───────────────────────────── FASTAPI (port 7860) ───────│───│─────────┐ | |
| │ │ │ │ | |
| │ /ingest/* ──── upload.py ──────────────────────────── ┘ │ │ | |
| │ │ │ │ | |
| │ ┌────▼──────────────────────────────┐ │ │ | |
| │ │ INGESTION PIPELINE │ │ │ | |
| │ │ parsers.py (PDF → pages) │ │ │ | |
| │ │ chunking.py (pages → chunks) │ │ │ | |
| │ │ embeddings.py (BGE-M3) │ │ │ | |
| │ │ store.py (→ Qdrant) │ │ │ | |
| │ │ tabular.py (Excel → DuckDB) │ │ │ | |
| │ └──────────────────────────────────┘ │ │ | |
| │ │ │ | |
| │ /query/ask/stream ── router.py ──────────────────────────── ┘ │ | |
| │ │ │ | |
| │ ┌──────▼──────────────────────────────────────┐ │ | |
| │ │ QUERY PIPELINE │ │ | |
| │ │ condense.py (history-aware rewrite) │ │ | |
| │ │ classify.py (casual / descriptive / SQL) │ │ | |
| │ │ retrieval.py (BGE-M3 + Qdrant + reranker) │ │ | |
| │ │ generate.py (Groq LLaMA / GPT-4o-mini) │ │ | |
| │ │ text_to_sql.py (DuckDB SQL path) │ │ | |
| │ └──────────────────────────────────────────────┘ │ | |
| │ │ | |
| │ /auth/* auth/router.py (JWT, 30-day tokens) │ | |
| │ /conversations conversations/ (SQLite / Turso) │ | |
| │ /health system health check │ | |
| └─────────────────────────────────────────────────────────────────────────┘ | |
| External services: | |
| Qdrant Cloud — vector + sparse index | |
| Groq API — primary LLM (LLaMA 3.3-70b, free tier 100K TPD) | |
| OpenAI API — vision LLM for PDF parsing (VISION_API_KEY) | |
| fallback LLM when Groq quota exhausted (FALLBACK_API_KEY) | |
| Turso — cloud SQLite for conversation storage (optional) | |
| ``` | |
| --- | |
| ## 3. Request Lifecycle | |
| A complete streaming chat request from browser to response: | |
| ``` | |
| User sends message | |
| │ | |
| ▼ | |
| [1] condense.py — history-aware query rewrite | |
| If history exists, rewrite follow-up ("et la clause ?") | |
| into a standalone query ("Quelle clause ISO 13485 concerne X ?"). | |
| Uses Groq (cheap, 60 max_tokens). Skipped on first turn. | |
| │ | |
| ▼ | |
| [2] classify.py — intent classification | |
| Labels the condensed query as: casual | descriptive | numeric | |
| · casual → skip retrieval, Julia answers from personality | |
| · descriptive → vector search path | |
| · numeric → SQL path (if tabular data exists) | |
| Uses Groq (5 max_tokens, one word output). | |
| │ | |
| ┌──┴──────────────────────────┐ | |
| │ casual path │ descriptive/numeric path | |
| ▼ ▼ | |
| [3a] stream_answer() [3b] retrieve() | |
| chunks = [] BGE-M3 embed query + keyword variant | |
| Julia replies from Qdrant hybrid search (dense + sparse RRF) | |
| system personality BGE cross-encoder rerank | |
| MMR deduplication | |
| Source-window expansion | |
| Abstain if best score < 0.08 | |
| │ │ | |
| │ ┌──────────┘ | |
| │ │ descriptive │ numeric + tabular data | |
| │ ▼ ▼ | |
| │ [4] generate.py [4b] text_to_sql.py | |
| │ Build context Generate SQL → DuckDB | |
| │ [S1][S2] IDs Render result as text | |
| │ Groq streaming Single LLM call | |
| │ → SSE tokens | |
| │ │ | |
| └────────────────────┘ | |
| │ | |
| ▼ | |
| [5] SSE done event | |
| { citations, source_map, chunks_used, path, provider } | |
| Citations verified against source_map (no hallucinated filenames) | |
| ``` | |
| --- | |
| ## 4. Ingestion Pipeline | |
| ### Entry point: `POST /ingest/upload` | |
| The upload endpoint saves the file, creates a job record, and runs ingestion in a thread-pool executor (non-blocking). PDF and Excel follow separate paths. | |
| ### PDF path | |
| ``` | |
| File on disk | |
| │ | |
| ▼ | |
| parsers.parse_pdf() — iterates pages with PyMuPDF | |
| │ | |
| ├── For each page: | |
| │ PyMuPDF text extraction (fast, free) | |
| │ has_images = bool(page.get_images()) | |
| │ | |
| │ if has_images OR text < 2000 chars: | |
| │ Vision LLM path (GPT-4o-mini) | |
| │ render page to PNG at 200 DPI → base64 | |
| │ send to OpenAI-compatible vision API | |
| │ returns markdown: headings, tables, form fields, chart data | |
| │ if vision_text > pymupdf_text → use vision result | |
| │ | |
| │ elif text < 50 chars: | |
| │ Tesseract OCR (ara+fra+eng) as last resort | |
| │ | |
| │ else: | |
| │ PyMuPDF text used as-is | |
| │ | |
| ▼ | |
| chunking.chunk_pages() | |
| │ | |
| ├── Tables from PyMuPDF path → atomic chunks (never split) | |
| │ prefix "[TABLE]\n" + markdown rows | |
| │ | |
| └── Body text → paragraph-boundary splitting | |
| _CHUNK_CHARS = 2500 chars | |
| _OVERLAP_CHARS = 300 chars (last paragraph repeated at next chunk start) | |
| Each chunk gets: chunk_id (UUID), content_hash (SHA-256[:16]), | |
| chunk_index (doc-wide 0-based position), source_language (langdetect) | |
| │ | |
| ▼ | |
| embeddings.embed_texts() — BGE-M3 (BAAI/bge-m3) | |
| Each chunk → dense vector (1024-dim) + sparse vector (lexical weights) | |
| Batch size 12, max_length 1024 tokens, CPU inference | |
| │ | |
| ▼ | |
| store.upsert_chunks() — Qdrant upsert | |
| Payload per point: doc_id, filename, page, source_language, | |
| category, text, content_hash, chunk_index | |
| Indexed fields: doc_id, filename, category, source_language | |
| ``` | |
| ### Excel path | |
| ``` | |
| File on disk | |
| │ | |
| ▼ | |
| tabular.load_excel_to_duckdb() | |
| For each sheet: | |
| _find_header_row() — detects actual header (not always row 0) | |
| _coerce_numeric_columns() — object → numeric when ≥80% convertible | |
| CREATE TABLE tbl_<doc_id_prefix>_<sheet_name> AS SELECT * FROM df | |
| Register in _table_registry | |
| │ | |
| ▼ | |
| tabular.get_prose_chunks_for_qdrant() | |
| Converts each sheet to "col: value | col: value" text lines | |
| Chunks and embeds into Qdrant (same pipeline as PDF body text) | |
| page = sheet_name (string) so citations read as "[file, sheet SheetName]" | |
| ``` | |
| ### Duplicate detection | |
| Before any ingestion, `store.delete_by_filename(filename)` removes all existing Qdrant points for that filename. Re-uploading the same document replaces, not duplicates, the previous chunks. | |
| --- | |
| ## 5. Retrieval Pipeline | |
| File: `backend/app/query/retrieval.py` | |
| ```python | |
| # Score thresholds | |
| _MIN_SCORE = 0.05 # drop chunks with reranker score below this | |
| _ABSTAIN_SCORE = 0.08 # if best surviving chunk < this, return [] → model abstains | |
| # MMR diversity | |
| _MMR_OVERLAP_THRESHOLD = 0.70 # Jaccard similarity above this = duplicate → drop | |
| # Window expansion | |
| _EXPAND_WINDOW = True # fetch same-doc adjacent-page chunks for top result | |
| ``` | |
| ### Step-by-step | |
| **1. Query expansion** | |
| Two variants are embedded: the original query and a stop-word-stripped keyword form. | |
| Example: "quelle est la composition du PB2050683" → also embeds "composition PB2050683" | |
| **2. BGE-M3 embedding** | |
| Both variants embedded in one batch. Results cached in a 256-entry LRU dict keyed by query string. | |
| Returns: `dense` (1024-float list) + `sparse` (token_id → weight dict) | |
| **3. Qdrant hybrid search** | |
| One `Prefetch` pair (dense + sparse) per query variant. Qdrant fuses all prefetches using Reciprocal Rank Fusion (RRF). Candidate pool = `top_k × 6` when reranking is enabled. | |
| **4. BGE reranker** | |
| Cross-encoder `BAAI/bge-reranker-v2-m3` scores every (query, chunk) pair. `normalize=True` maps scores to [0, 1]. | |
| **5. Abstention check** | |
| If the highest reranker score is below `_ABSTAIN_SCORE = 0.08`, return an empty list. The generator then tells the model "no relevant information found" and the model correctly abstains. | |
| **6. MMR deduplication** | |
| After sorting by score, walk the list. Drop any chunk whose character k-gram Jaccard similarity with a higher-ranked already-selected chunk exceeds 0.70. Prevents 3 near-identical paragraphs consuming context budget. | |
| **7. Source-window expansion** | |
| For the top-ranked chunk, fetch all other chunks from the same `doc_id` on adjacent pages (page ± 1) from Qdrant. Append them to the context. Gives the model surrounding paragraphs without full parent-child retrieval. | |
| --- | |
| ## 6. Answer Generation | |
| File: `backend/app/query/generate.py` | |
| ### Context construction | |
| Retrieved chunks are formatted as: | |
| ``` | |
| [S1] procedure_ISO13485.pdf, page 12 | |
| <chunk text> | |
| --- | |
| [S2] procedure_ISO13485.pdf, page 13 | |
| <chunk text> | |
| --- | |
| [S3] fiche_produit_PB2050683.pdf, page 2 | |
| <chunk text> | |
| ``` | |
| The source IDs `[S1]`, `[S2]`, `[S3]` are stable — the model can only reference IDs it sees in the context. It cannot invent a filename. `extract_citations()` maps `[S1]` back to the real `{filename, page, snippet}` after generation. | |
| Context budget: **10,000 chars** total (~2,500 tokens). Per-chunk ceiling: **2,000 chars**. | |
| ### System prompt structure | |
| The system prompt is structured in sections: | |
| - **Identity** — Julia, WinnCare Tunisia assistant | |
| - **Conversation style** — casual messages need no citations | |
| - **Document Q&A rules** — ground claims in context; cite with [S1]; abstain when missing; no cross-product contamination; ask clarifying questions when ambiguous | |
| - **Formatting** — Markdown, bold, bullet lists | |
| - **Language rule** (non-negotiable) — reply in the EXACT language of the LATEST user message, ignoring history language | |
| ### Groq streaming + fallback | |
| ``` | |
| Primary: Groq API (LLaMA 3.3-70b-versatile) | |
| stream=True → token-by-token SSE | |
| temperature=0.0, max_tokens=1200 | |
| If 429 / rate_limit / quota / 5xx: | |
| ▼ | |
| Fallback: FALLBACK_API_KEY provider (default gpt-4o-mini) | |
| stream=False → complete response | |
| Simulated streaming: word-by-word with 12ms delay | |
| If fallback also fails: | |
| ▼ | |
| Error message: "⚠️ Limite quotidienne atteinte..." with retry-after time | |
| ``` | |
| Done event payload: | |
| ```json | |
| { | |
| "type": "done", | |
| "citations": [{"filename": "...", "page": 12, "snippet": "..."}], | |
| "source_map": [...], | |
| "chunks_used": 6, | |
| "path": "vector", | |
| "provider": "groq" | |
| } | |
| ``` | |
| ### Conversation memory | |
| Last **10 messages (5 turns)** are included in every request as `history`. The condensation step (`condense.py`) also uses 5 turns to rewrite follow-up queries. | |
| --- | |
| ## 7. API Reference | |
| All routes require `Authorization: Bearer <token>` except `/auth/login` and `/health`. | |
| ### Authentication | |
| | Method | Path | Description | | |
| |--------|------|-------------| | |
| | `POST` | `/auth/login` | `{email, password}` → `{token, email}`. Token is HS256 JWT, 30-day expiry. | | |
| | `GET` | `/auth/me` | Returns `{email}` for the bearer token. | | |
| ### Ingestion | |
| | Method | Path | Description | | |
| |--------|------|-------------| | |
| | `POST` | `/ingest/upload` | Multipart: `file` (PDF/Excel), `category` (string, optional). Returns `{job_id, doc_id, status}`. Ingestion is async. | | |
| | `GET` | `/ingest/status/{job_id}` | Returns job status: `queued \| processing \| done \| failed`, `chunks_stored`, `error`. | | |
| | `GET` | `/ingest/documents` | Lists all indexed documents from Qdrant + DuckDB. | | |
| | `DELETE` | `/ingest/documents/{doc_id}` | Delete all chunks for a document from Qdrant and DuckDB. | | |
| | `DELETE` | `/ingest/documents` | Drop and recreate the Qdrant collection; clear DuckDB; delete uploaded files. | | |
| ### Query | |
| | Method | Path | Description | | |
| |--------|------|-------------| | |
| | `POST` | `/query/ask/stream` | **Primary endpoint.** SSE stream. Body: `{question, history, top_k, category}`. Emits `status`, `token`, `done`, `error` events. | | |
| | `POST` | `/query/ask` | Non-streaming fallback. Same body, returns full `AskResponse`. Used by RAGAS eval. | | |
| #### SSE event types | |
| ``` | |
| data: {"type": "status", "status": "searching"} | |
| data: {"type": "status", "status": "generating"} | |
| data: {"type": "token", "content": "La composition"} | |
| data: {"type": "token", "content": " est 49%"} | |
| data: {"type": "done", "citations": [...], "source_map": [...], "chunks_used": 6, "path": "vector", "provider": "groq"} | |
| data: {"type": "error", "message": "..."} | |
| ``` | |
| SSE headers prevent proxy buffering: | |
| ``` | |
| Cache-Control: no-cache | |
| X-Accel-Buffering: no | |
| Connection: keep-alive | |
| ``` | |
| #### AskResponse schema | |
| ```typescript | |
| { | |
| answer: string | |
| citations: Array<{ filename: string; page: number | string; snippet?: string }> | |
| source_map: Array<{ filename: string; page: number | string; snippet: string }> | |
| chunks_used: number | |
| path: "vector" | "sql" | |
| sql: string | null | |
| contexts: string[] // raw chunk texts (used by RAGAS eval harness only) | |
| } | |
| ``` | |
| ### Conversations | |
| | Method | Path | Description | | |
| |--------|------|-------------| | |
| | `GET` | `/conversations` | List user's conversations (id, title, timestamps). | | |
| | `POST` | `/conversations` | Create: `{title, messages}`. Returns conversation object. | | |
| | `GET` | `/conversations/{id}` | Get conversation with full message array. | | |
| | `PUT` | `/conversations/{id}` | Update title and messages. | | |
| | `DELETE` | `/conversations/{id}` | Delete conversation. | | |
| ### System | |
| | Method | Path | Description | | |
| |--------|------|-------------| | |
| | `GET` | `/health` | Returns env, qdrant_url, embedding_model, groq_model. No auth required. | | |
| --- | |
| ## 8. Configuration Reference | |
| All settings live in `backend/app/config.py` and are read from environment variables (or `backend/.env` for local development). **Never commit `backend/.env`** — it contains real API keys. | |
| ### Required in production | |
| | Variable | Description | | |
| |----------|-------------| | |
| | `GROQ_API_KEY` | Groq API key. Free tier: 100K tokens/day. | | |
| | `JWT_SECRET` | Random string for HS256 JWT signing. Use `openssl rand -hex 32`. | | |
| | `AUTH_USERS` | JSON object `{"email@example.com": "password"}`. Supports multiple users. | | |
| | `QDRANT_URL` | Qdrant server URL. E.g. `https://xxx.cloud.qdrant.io:6333` | | |
| | `QDRANT_API_KEY` | Qdrant Cloud API key (blank for local). | | |
| ### Optional but recommended | |
| | Variable | Default | Description | | |
| |----------|---------|-------------| | |
| | `GROQ_MODEL` | `llama-3.3-70b-versatile` | Primary LLM. LLaMA 3.3-70b is free on Groq and supports 128K context. | | |
| | `VISION_API_KEY` | `` | API key for vision LLM (PDF parsing). Uses `VISION_BASE_URL` endpoint. If blank, vision path is disabled and PyMuPDF + Tesseract are used. | | |
| | `VISION_BASE_URL` | `https://api.openai.com/v1` | Vision API endpoint (OpenAI-compatible). | | |
| | `VISION_MODEL` | `gpt-4o-mini` | Vision model. gpt-4o-mini: ~$0.01/40-page document. | | |
| | `FALLBACK_API_KEY` | `` | API key for text-generation fallback when Groq quota is exhausted. | | |
| | `FALLBACK_BASE_URL` | `https://api.openai.com/v1` | Fallback LLM endpoint. | | |
| | `FALLBACK_MODEL` | `gpt-4o-mini` | Fallback model name. | | |
| | `RERANK_ENABLED` | `true` | Set `false` to skip cross-encoder reranking (faster, lower precision). | | |
| | `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | Cross-encoder model. | | |
| | `TURSO_URL` | `` | Turso database URL for persistent conversation storage. If blank, uses local SQLite (lost on container restart). | | |
| | `TURSO_TOKEN` | `` | Turso auth token. | | |
| ### Less commonly changed | |
| | Variable | Default | Description | | |
| |----------|---------|-------------| | |
| | `QDRANT_COLLECTION` | `rag_documents` | Qdrant collection name. | | |
| | `EMBEDDING_MODEL` | `BAAI/bge-m3` | Embedding model. Do not change without re-ingesting all documents. | | |
| | `EMBEDDING_DEVICE` | `cpu` | `cpu`, `cuda`, `mps`. HF Spaces has no GPU — always `cpu`. | | |
| | `DUCKDB_PATH` | `data/tabular.duckdb` | DuckDB file path (relative to working dir). | | |
| | `UPLOAD_DIR` | `data/uploads` | Uploaded source files directory. | | |
| | `LOG_LEVEL` | `INFO` | Python logging level. | | |
| | `APP_ENV` | `development` | Visible in `/health`. | | |
| --- | |
| ## 9. Frontend | |
| **Stack:** React 18, TypeScript, Vite, react-markdown with remark-gfm | |
| ### Component structure | |
| ``` | |
| App.tsx | |
| ├── LoginPage.tsx — email + password form, token stored in localStorage | |
| │ | |
| └── (authenticated) | |
| ├── Sidebar.tsx — conversation list, new chat, sign out | |
| │ | |
| └── main content | |
| ├── ChatView.tsx — message thread, SSE streaming, citation chips | |
| └── UploadView.tsx — file drop zone, ingestion status polling | |
| ``` | |
| ### Key frontend behaviors | |
| **Streaming:** `askQuestionStream()` in `api.ts` reads the SSE response using `ReadableStream`. Status events update the thinking indicator ("Recherche dans les documents…" / "Rédaction de la réponse…"). Token events append to the message in real time. | |
| **Citations:** Citation chips appear below each assistant message. `page` can be `number` (PDF page) or `string` (Excel sheet name). Hovering a chip shows the first 300 chars of that source chunk as a tooltip. | |
| **Markdown:** `ReactMarkdown` with `remark-gfm` renders tables, bold, code, blockquotes. Vision LLM pages return markdown tables which render correctly. | |
| **Conversation persistence:** After each assistant response, the full message array is saved via `PUT /conversations/{id}`. On sidebar click, `GET /conversations/{id}` restores the thread. | |
| **Languages:** UI strings are in `i18n.ts`, supporting `en` and `fr`. Language preference stored in `localStorage`. | |
| ### Build output | |
| Vite builds to `frontend/dist/`. The Dockerfile copies this to `backend/static/`. FastAPI serves `static/assets/` and falls back to `static/index.html` for any path not matched by an API route (SPA client-side routing). | |
| --- | |
| ## 10. Authentication | |
| **Mechanism:** HS256 JWT, 30-day expiry. | |
| **User management:** Users are defined in the `AUTH_USERS` environment variable as a JSON object: | |
| ```json | |
| {"alice@winncare.tn": "password1", "bob@winncare.tn": "password2"} | |
| ``` | |
| There is no self-registration or password reset. Add/remove users by updating the secret. | |
| **Token storage:** Stored in `localStorage` under key `julia_token`. Auto-cleared on 401 response and page reloads to the login screen. | |
| **Authorization on API routes:** All query, ingestion, and conversation routes depend on `get_current_user()`. A missing or invalid token returns HTTP 401 immediately. | |
| > **Note:** This is a simple single-tenant auth model. For multi-tenant deployments with data isolation, each user would need their own Qdrant namespace or collection, and the `doc_id` / `filename` payload indexes would need a `tenant_id` field. | |
| --- | |
| ## 11. Conversation Persistence | |
| File: `backend/app/conversations/store.py` | |
| Conversations are stored in SQLite (local dev) or Turso (production). | |
| **Schema:** | |
| ```sql | |
| CREATE TABLE conversations ( | |
| id TEXT PRIMARY KEY, | |
| user_email TEXT NOT NULL, | |
| title TEXT NOT NULL, | |
| messages TEXT NOT NULL DEFAULT '[]', -- JSON array | |
| created_at TEXT NOT NULL, | |
| updated_at TEXT NOT NULL | |
| ) | |
| ``` | |
| Messages are stored as a JSON blob. Each message: `{role: "user"|"assistant", text: string, meta?: AskResponse}`. | |
| **Turso:** Turso is a cloud SQLite service with a free tier. Set `TURSO_URL` and `TURSO_TOKEN` in secrets. Without it, conversations are stored in `data/conversations.db` which is ephemeral on HF Spaces (lost on container restart). | |
| --- | |
| ## 12. Evaluation Harness (RAGAS) | |
| File: `backend/app/eval/ragas_eval.py` | |
| Questions: `backend/app/eval/data/questions.jsonl` | |
| Results: `backend/app/eval/results/report.json` | |
| ### Metrics | |
| | Metric | What it measures | Target | | |
| |--------|-----------------|--------| | |
| | `faithfulness` | Fraction of Julia's claims that are verifiable from retrieved chunks. 1.0 = zero hallucination. | ≥ 0.85 | | |
| | `context_precision` | Were retrieved chunks actually useful? Penalizes noise at the top of the ranking. | ≥ 0.75 | | |
| | `context_recall` | Did retrieval cover everything needed to answer? Requires `ground_truth`. | ≥ 0.80 | | |
| ### Running the eval | |
| The backend must be running. Run in a separate virtual environment (dependency conflict between FlagEmbedding and RAGAS on `datasets` version): | |
| ```bash | |
| cd backend | |
| python -m venv .venv-eval | |
| .venv-eval\Scripts\activate # Windows | |
| # or: source .venv-eval/bin/activate # Mac/Linux | |
| pip install -r requirements-eval.txt | |
| python -m app.eval.ragas_eval | |
| ``` | |
| ### Questions file format | |
| ```jsonl | |
| {"question": "Quelle est la composition du PB2050683 ?", "ground_truth": "49% PE - 51% PU"} | |
| {"question": "Quel est le seuil AQL pour les défauts critiques ?", "ground_truth": "AQL 0,065"} | |
| ``` | |
| Lines starting with `#` are comments. | |
| ### Interpreting results | |
| ``` | |
| faithfulness < 0.80 → Julia hallucinates — tighten the system prompt or reduce context noise | |
| context_precision < 0.70 → Retrieval is noisy — tune top_k, chunking, or reranker threshold | |
| context_recall < 0.70 → Retrieval misses info — check chunking size, vision threshold, re-ingest | |
| ``` | |
| The report surfaces the 3 lowest-scoring questions by combined faithfulness + recall. Fix these first. | |
| --- | |
| ## 13. Deployment | |
| ### HuggingFace Spaces (current) | |
| The app runs as a Docker container on HF Spaces. Push to the `main` branch of the HF Space repository triggers a rebuild. | |
| **Remote:** `https://huggingface.co/spaces/esra2001/winncare` (git remote: `origin`) | |
| **Build:** Two-stage Docker build. Stage 1 builds the React SPA. Stage 2 is Python 3.12-slim, installs backend deps, copies the Vite `dist/` into `backend/static/`, pre-downloads BGE-M3 (~2 GB baked into image layer so the first request is fast). | |
| **Exposed port:** 7860 (HF Spaces requirement). | |
| **Secrets (set in HF Space dashboard, not in git):** | |
| ``` | |
| GROQ_API_KEY | |
| JWT_SECRET | |
| AUTH_USERS | |
| QDRANT_URL | |
| QDRANT_API_KEY | |
| VISION_API_KEY | |
| VISION_BASE_URL | |
| VISION_MODEL | |
| FALLBACK_API_KEY | |
| FALLBACK_BASE_URL | |
| FALLBACK_MODEL | |
| RERANK_ENABLED | |
| RERANKER_MODEL | |
| GROQ_MODEL | |
| TURSO_URL | |
| TURSO_TOKEN | |
| ``` | |
| **Persistent storage:** HF Spaces ephemeral disk — `data/uploads/` and `data/conversations.db` are lost on restarts. Use Qdrant Cloud for vectors and Turso for conversations to achieve full persistence across restarts. | |
| **Triggering a rebuild without code changes:** | |
| ```bash | |
| git commit --allow-empty -m "trigger rebuild" && git push origin HEAD:main | |
| ``` | |
| ### Production (self-hosted) | |
| Use `docker-compose.yml` for local Qdrant and Redis, then run the app container separately: | |
| ```bash | |
| docker-compose up -d # starts Qdrant (port 6333) and Redis (port 6379) | |
| docker build -t julia-rag . | |
| docker run -p 7860:7860 \ | |
| -e GROQ_API_KEY=... \ | |
| -e JWT_SECRET=... \ | |
| -e AUTH_USERS='{"admin@company.com":"password"}' \ | |
| -e QDRANT_URL=http://host.docker.internal:6333 \ | |
| julia-rag | |
| ``` | |
| --- | |
| ## 14. Local Development | |
| ### Backend | |
| ```bash | |
| cd backend | |
| # Create and activate virtual environment | |
| python -m venv .venv | |
| .venv\Scripts\activate # Windows | |
| # source .venv/bin/activate # Mac/Linux | |
| pip install -r requirements.txt | |
| # Start infrastructure | |
| docker-compose up -d # Qdrant + Redis | |
| # Configure | |
| cp .env.example .env # fill in GROQ_API_KEY at minimum | |
| # Run | |
| uvicorn app.main:app --reload --port 8000 | |
| ``` | |
| Minimal `.env` for local dev: | |
| ```env | |
| GROQ_API_KEY=gsk_... | |
| JWT_SECRET=any-random-string | |
| AUTH_USERS={"dev@local.com": "dev"} | |
| QDRANT_URL=http://localhost:6333 | |
| ``` | |
| ### Frontend | |
| ```bash | |
| cd frontend | |
| npm install | |
| npm run dev # Vite dev server on http://localhost:5173 | |
| ``` | |
| Vite proxies `/query`, `/ingest`, `/auth`, `/conversations` to `http://localhost:8000` (configured in `vite.config.ts`). The SPA runs on its own port; no CORS issues in dev. | |
| ### Building for production (manual) | |
| ```bash | |
| cd frontend && npm run build # → frontend/dist/ | |
| cp -r frontend/dist backend/static | |
| cd backend && uvicorn app.main:app --port 7860 | |
| ``` | |
| --- | |
| ## 15. Operational Notes | |
| ### After changing chunking or parsing logic | |
| Re-ingest all documents. Old chunks in Qdrant do not reflect new chunk sizes, overlap, content_hash, or chunk_index. Steps: | |
| 1. Go to the Upload tab → "Clear all documents" | |
| 2. Re-upload each document | |
| ### After changing the embedding model | |
| The new model produces incompatible vectors. You must recreate the collection: | |
| 1. Delete collection in Qdrant dashboard (or clear all via UI) | |
| 2. Re-ingest all documents | |
| 3. Update `EMBEDDING_MODEL` in secrets before re-ingesting | |
| ### Groq daily quota (free tier) | |
| Groq free tier: **100,000 tokens/day**. At typical usage (question + 5-turn history + 10K context + 1200 token answer ≈ 14K tokens/request), that's ~7 full requests before hitting the daily limit. | |
| Options: | |
| - Set `FALLBACK_API_KEY` + `FALLBACK_MODEL` to automatically failover to OpenAI/Gemini | |
| - Upgrade Groq plan | |
| - Switch `GROQ_MODEL` to `llama-3.1-8b-instant` (6K TPM limit but uses fewer tokens) | |
| The 429 error message shown to users includes a retry-after time extracted from the Groq error response. | |
| ### Vision LLM costs | |
| At GPT-4o-mini pricing (~$0.075 per 1M input tokens, ~$0.30 per 1M output tokens), a 40-page document where 20 pages go through vision costs roughly $0.02–$0.05. Budget accordingly for large corpora. | |
| ### Reranker memory | |
| `BAAI/bge-reranker-v2-m3` loads ~550 MB into RAM on first query. On HF Spaces (16 GB), this is fine alongside BGE-M3 (~2 GB). Total model footprint: ~2.6 GB. | |
| ### SSE and proxy buffering | |
| The `/query/ask/stream` endpoint sends three headers to prevent proxy buffering: | |
| - `X-Accel-Buffering: no` — disables nginx buffering (critical for HF Spaces) | |
| - `Cache-Control: no-cache` — tells intermediate proxies not to cache the stream | |
| - `Connection: keep-alive` — prevents idle-timeout drops on long responses | |
| If responses appear cut or arrive all at once on a specific network, a local proxy is the cause. These headers resolve it for standard nginx-based setups. | |
| --- | |
| ## 16. Known Limitations and Trade-offs | |
| ### Free-tier constraints | |
| | Constraint | Impact | Workaround | | |
| |-----------|--------|------------| | |
| | Groq: 100K tokens/day | ~7 full requests before quota | Set FALLBACK_API_KEY | | |
| | HF Spaces: 2 vCPU, no GPU | BGE-M3 embed ~3-5s per chunk batch; reranker ~2-4s per request | Pre-download models in Dockerfile (already done) | | |
| | HF Spaces: ephemeral disk | Uploads and local conversations lost on restart | Use Qdrant Cloud + Turso | | |
| ### Architecture trade-offs | |
| **No parent-child retrieval.** Chunks are 2500 chars with 300 chars overlap. Source-window expansion (fetch adjacent page chunks for top result) partially compensates. True parent-child would require dual-index ingestion at different granularities. | |
| **Single tenant.** All users share one Qdrant collection and one DuckDB file. No per-user document isolation. Adding `tenant_id` to every chunk payload and filtering on it is the path to multi-tenancy. | |
| **Vision LLM at ingestion time only.** PDF parsing with vision happens when a document is uploaded, not at query time. Changing the vision model requires re-ingesting documents to benefit. | |
| **SQL path for aggregations only.** `classify.py` routes to DuckDB only for `numeric` questions (sum, count, average, max, min across many rows). Single-value lookups ("what is the price of X?") go through the vector path even when the value is in an Excel sheet. | |
| **No streaming fallback.** When Groq fails and the fallback provider responds, the answer is simulated as word-by-word streaming (12ms delay per word) rather than true token streaming. The experience is slightly less responsive but visually similar. | |
| ### Accuracy targets (current RAGAS baseline) | |
| | Metric | Baseline | Target | | |
| |--------|----------|--------| | |
| | faithfulness | 0.50 | ≥ 0.85 | | |
| | context_precision | 0.67 | ≥ 0.75 | | |
| | context_recall | 0.71 | ≥ 0.80 | | |
| The main drivers of low faithfulness are: weak context (already addressed by abstention threshold) and the model summarizing across documents rather than citing specific claims. Running RAGAS after re-ingestion with the Phase 1 changes should show measurable improvement in all three metrics. | |