Spaces:

esra2001
/

winncare

Running

App Files Files Community

winncare / DOCUMENTATION.md

esra2001

add comprehensive technical documentation

ff25e0e 17 days ago

preview code

Raw

History Blame Contribute Delete

33.1 kB

Julia — WinnCare RAG Assistant: Technical Documentation

Stack: Python 3.12 · FastAPI · Qdrant · BGE-M3 · BGE Reranker · Groq (LLaMA 3.3-70b) · React · Vite
Deployment: HuggingFace Spaces (Docker, 2 vCPU / 16 GB RAM, no GPU)
Live URL: https://esra2001-winncare.hf.space

System Overview
Architecture
Request Lifecycle
Ingestion Pipeline
Retrieval Pipeline
Answer Generation
API Reference
Configuration Reference
Frontend
Authentication
Conversation Persistence
Evaluation Harness (RAGAS)
Deployment
Local Development
Operational Notes
Known Limitations and Trade-offs

1. System Overview

Julia is a multilingual Retrieval-Augmented Generation (RAG) chatbot built for WinnCare Tunisia. It answers questions about internal documents — product specifications, procedures, compliance reports, price lists, capacity data — by retrieving relevant content from an indexed corpus and generating grounded, cited answers.

Supported document types: PDF (any layout including scans, screenshots, tables, charts, forms), Excel (.xlsx, .xls, .xlsm)
Supported languages: French, Arabic, English (any mix per message)
Deployment model: Single-tenant, shared corpus. All authenticated users query the same document set.

What Julia does (and doesn't do)

Does	Does not
Answer factual questions grounded in uploaded documents	Make up information not in documents
Extract data from scanned PDFs using vision LLM	Answer from general internet knowledge
Query Excel sheets using generated SQL	Execute arbitrary SQL or modify data
Hold a 5-turn conversation with context	Maintain separate per-user memory
Abstain when evidence is weak or absent	Guess when uncertain
Answer in the user's language	Force a fixed language

2. Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│  BROWSER                                                                │
│  React SPA (Vite)                                                       │
│    ChatView ── SSE token stream ──────────────────────────┐            │
│    UploadView ── multipart POST ──────────────────────┐   │            │
└──────────────────────────────────────────────────────────────────────── │
                                                         │   │
┌───────────────────────────── FASTAPI (port 7860) ───────│───│─────────┐
│                                                         │   │         │
│  /ingest/*  ──── upload.py ──────────────────────────── ┘   │         │
│                      │                                       │         │
│                 ┌────▼──────────────────────────────┐        │         │
│                 │ INGESTION PIPELINE                │        │         │
│                 │  parsers.py   (PDF → pages)       │        │         │
│                 │  chunking.py  (pages → chunks)    │        │         │
│                 │  embeddings.py (BGE-M3)            │        │         │
│                 │  store.py     (→ Qdrant)           │        │         │
│                 │  tabular.py   (Excel → DuckDB)     │        │         │
│                 └──────────────────────────────────┘        │         │
│                                                              │         │
│  /query/ask/stream ── router.py ──────────────────────────── ┘         │
│                           │                                             │
│                    ┌──────▼──────────────────────────────────────┐     │
│                    │ QUERY PIPELINE                               │     │
│                    │  condense.py   (history-aware rewrite)       │     │
│                    │  classify.py   (casual / descriptive / SQL)  │     │
│                    │  retrieval.py  (BGE-M3 + Qdrant + reranker)  │     │
│                    │  generate.py   (Groq LLaMA / GPT-4o-mini)    │     │
│                    │  text_to_sql.py (DuckDB SQL path)            │     │
│                    └──────────────────────────────────────────────┘    │
│                                                                         │
│  /auth/*         auth/router.py  (JWT, 30-day tokens)                  │
│  /conversations  conversations/  (SQLite / Turso)                      │
│  /health         system health check                                    │
└─────────────────────────────────────────────────────────────────────────┘

External services:
  Qdrant Cloud  — vector + sparse index
  Groq API      — primary LLM (LLaMA 3.3-70b, free tier 100K TPD)
  OpenAI API    — vision LLM for PDF parsing (VISION_API_KEY)
                  fallback LLM when Groq quota exhausted (FALLBACK_API_KEY)
  Turso         — cloud SQLite for conversation storage (optional)

3. Request Lifecycle

A complete streaming chat request from browser to response:

User sends message
        │
        ▼
[1] condense.py — history-aware query rewrite
    If history exists, rewrite follow-up ("et la clause ?")
    into a standalone query ("Quelle clause ISO 13485 concerne X ?").
    Uses Groq (cheap, 60 max_tokens). Skipped on first turn.
        │
        ▼
[2] classify.py — intent classification
    Labels the condensed query as: casual | descriptive | numeric
    · casual     → skip retrieval, Julia answers from personality
    · descriptive → vector search path
    · numeric    → SQL path (if tabular data exists)
    Uses Groq (5 max_tokens, one word output).
        │
     ┌──┴──────────────────────────┐
     │ casual path                 │ descriptive/numeric path
     ▼                             ▼
[3a] stream_answer()           [3b] retrieve()
     chunks = []                    BGE-M3 embed query + keyword variant
     Julia replies from             Qdrant hybrid search (dense + sparse RRF)
     system personality             BGE cross-encoder rerank
                                    MMR deduplication
                                    Source-window expansion
                                    Abstain if best score < 0.08
        │                               │
        │                    ┌──────────┘
        │                    │ descriptive   │ numeric + tabular data
        │                    ▼               ▼
        │               [4] generate.py  [4b] text_to_sql.py
        │               Build context       Generate SQL → DuckDB
        │               [S1][S2] IDs        Render result as text
        │               Groq streaming      Single LLM call
        │               → SSE tokens
        │                    │
        └────────────────────┘
                             │
                             ▼
[5] SSE done event
    { citations, source_map, chunks_used, path, provider }
    Citations verified against source_map (no hallucinated filenames)

4. Ingestion Pipeline

Entry point: `POST /ingest/upload`

The upload endpoint saves the file, creates a job record, and runs ingestion in a thread-pool executor (non-blocking). PDF and Excel follow separate paths.

PDF path

File on disk
    │
    ▼
parsers.parse_pdf()        — iterates pages with PyMuPDF
    │
    ├── For each page:
    │     PyMuPDF text extraction (fast, free)
    │     has_images = bool(page.get_images())
    │
    │     if has_images OR text < 2000 chars:
    │         Vision LLM path (GPT-4o-mini)
    │         render page to PNG at 200 DPI → base64
    │         send to OpenAI-compatible vision API
    │         returns markdown: headings, tables, form fields, chart data
    │         if vision_text > pymupdf_text → use vision result
    │
    │     elif text < 50 chars:
    │         Tesseract OCR (ara+fra+eng) as last resort
    │
    │     else:
    │         PyMuPDF text used as-is
    │
    ▼
chunking.chunk_pages()
    │
    ├── Tables from PyMuPDF path → atomic chunks (never split)
    │   prefix "[TABLE]\n" + markdown rows
    │
    └── Body text → paragraph-boundary splitting
        _CHUNK_CHARS   = 2500 chars
        _OVERLAP_CHARS = 300 chars (last paragraph repeated at next chunk start)
        Each chunk gets: chunk_id (UUID), content_hash (SHA-256[:16]),
        chunk_index (doc-wide 0-based position), source_language (langdetect)
    │
    ▼
embeddings.embed_texts()   — BGE-M3 (BAAI/bge-m3)
    Each chunk → dense vector (1024-dim) + sparse vector (lexical weights)
    Batch size 12, max_length 1024 tokens, CPU inference
    │
    ▼
store.upsert_chunks()      — Qdrant upsert
    Payload per point: doc_id, filename, page, source_language,
                       category, text, content_hash, chunk_index
    Indexed fields: doc_id, filename, category, source_language

Excel path

File on disk
    │
    ▼
tabular.load_excel_to_duckdb()
    For each sheet:
        _find_header_row() — detects actual header (not always row 0)
        _coerce_numeric_columns() — object → numeric when ≥80% convertible
        CREATE TABLE tbl_<doc_id_prefix>_<sheet_name> AS SELECT * FROM df
        Register in _table_registry
    │
    ▼
tabular.get_prose_chunks_for_qdrant()
    Converts each sheet to "col: value | col: value" text lines
    Chunks and embeds into Qdrant (same pipeline as PDF body text)
    page = sheet_name (string) so citations read as "[file, sheet SheetName]"

Duplicate detection

Before any ingestion, store.delete_by_filename(filename) removes all existing Qdrant points for that filename. Re-uploading the same document replaces, not duplicates, the previous chunks.

5. Retrieval Pipeline

File: backend/app/query/retrieval.py

# Score thresholds
_MIN_SCORE     = 0.05   # drop chunks with reranker score below this
_ABSTAIN_SCORE = 0.08   # if best surviving chunk < this, return [] → model abstains

# MMR diversity
_MMR_OVERLAP_THRESHOLD = 0.70   # Jaccard similarity above this = duplicate → drop

# Window expansion
_EXPAND_WINDOW = True   # fetch same-doc adjacent-page chunks for top result

Step-by-step

1. Query expansion
Two variants are embedded: the original query and a stop-word-stripped keyword form.
Example: "quelle est la composition du PB2050683" → also embeds "composition PB2050683"

2. BGE-M3 embedding
Both variants embedded in one batch. Results cached in a 256-entry LRU dict keyed by query string.
Returns: dense (1024-float list) + sparse (token_id → weight dict)

3. Qdrant hybrid search
One Prefetch pair (dense + sparse) per query variant. Qdrant fuses all prefetches using Reciprocal Rank Fusion (RRF). Candidate pool = top_k × 6 when reranking is enabled.

4. BGE reranker
Cross-encoder BAAI/bge-reranker-v2-m3 scores every (query, chunk) pair. normalize=True maps scores to [0, 1].

5. Abstention check
If the highest reranker score is below _ABSTAIN_SCORE = 0.08, return an empty list. The generator then tells the model "no relevant information found" and the model correctly abstains.

6. MMR deduplication
After sorting by score, walk the list. Drop any chunk whose character k-gram Jaccard similarity with a higher-ranked already-selected chunk exceeds 0.70. Prevents 3 near-identical paragraphs consuming context budget.

7. Source-window expansion
For the top-ranked chunk, fetch all other chunks from the same doc_id on adjacent pages (page ± 1) from Qdrant. Append them to the context. Gives the model surrounding paragraphs without full parent-child retrieval.

6. Answer Generation

File: backend/app/query/generate.py

Context construction

Retrieved chunks are formatted as:

[S1] procedure_ISO13485.pdf, page 12
<chunk text>

---

[S2] procedure_ISO13485.pdf, page 13
<chunk text>

---

[S3] fiche_produit_PB2050683.pdf, page 2
<chunk text>

The source IDs [S1], [S2], [S3] are stable — the model can only reference IDs it sees in the context. It cannot invent a filename. extract_citations() maps [S1] back to the real {filename, page, snippet} after generation.

Context budget: 10,000 chars total (~2,500 tokens). Per-chunk ceiling: 2,000 chars.

System prompt structure

The system prompt is structured in sections:

Identity — Julia, WinnCare Tunisia assistant
Conversation style — casual messages need no citations
Document Q&A rules — ground claims in context; cite with [S1]; abstain when missing; no cross-product contamination; ask clarifying questions when ambiguous
Formatting — Markdown, bold, bullet lists
Language rule (non-negotiable) — reply in the EXACT language of the LATEST user message, ignoring history language

Groq streaming + fallback

Primary: Groq API (LLaMA 3.3-70b-versatile)
    stream=True → token-by-token SSE
    temperature=0.0, max_tokens=1200

    If 429 / rate_limit / quota / 5xx:
        ▼
Fallback: FALLBACK_API_KEY provider (default gpt-4o-mini)
    stream=False → complete response
    Simulated streaming: word-by-word with 12ms delay

    If fallback also fails:
        ▼
Error message: "⚠️ Limite quotidienne atteinte..." with retry-after time

Done event payload:

{
  "type": "done",
  "citations": [{"filename": "...", "page": 12, "snippet": "..."}],
  "source_map": [...],
  "chunks_used": 6,
  "path": "vector",
  "provider": "groq"
}

Conversation memory

Last 10 messages (5 turns) are included in every request as history. The condensation step (condense.py) also uses 5 turns to rewrite follow-up queries.

7. API Reference

All routes require Authorization: Bearer <token> except /auth/login and /health.

Authentication

Method	Path	Description
`POST`	`/auth/login`	`{email, password}` → `{token, email}`. Token is HS256 JWT, 30-day expiry.
`GET`	`/auth/me`	Returns `{email}` for the bearer token.

Ingestion

Method	Path	Description
`POST`	`/ingest/upload`	Multipart: `file` (PDF/Excel), `category` (string, optional). Returns `{job_id, doc_id, status}`. Ingestion is async.
`GET`	`/ingest/status/{job_id}`	Returns job status: `queued \| processing \| done \| failed`, `chunks_stored`, `error`.
`GET`	`/ingest/documents`	Lists all indexed documents from Qdrant + DuckDB.
`DELETE`	`/ingest/documents/{doc_id}`	Delete all chunks for a document from Qdrant and DuckDB.
`DELETE`	`/ingest/documents`	Drop and recreate the Qdrant collection; clear DuckDB; delete uploaded files.

Query

Method	Path	Description
`POST`	`/query/ask/stream`	Primary endpoint. SSE stream. Body: `{question, history, top_k, category}`. Emits `status`, `token`, `done`, `error` events.
`POST`	`/query/ask`	Non-streaming fallback. Same body, returns full `AskResponse`. Used by RAGAS eval.

SSE event types

data: {"type": "status", "status": "searching"}
data: {"type": "status", "status": "generating"}
data: {"type": "token", "content": "La composition"}
data: {"type": "token", "content": " est 49%"}
data: {"type": "done", "citations": [...], "source_map": [...], "chunks_used": 6, "path": "vector", "provider": "groq"}
data: {"type": "error", "message": "..."}

SSE headers prevent proxy buffering:

Cache-Control: no-cache
X-Accel-Buffering: no
Connection: keep-alive

AskResponse schema

{
  answer:      string
  citations:   Array<{ filename: string; page: number | string; snippet?: string }>
  source_map:  Array<{ filename: string; page: number | string; snippet: string }>
  chunks_used: number
  path:        "vector" | "sql"
  sql:         string | null
  contexts:    string[]   // raw chunk texts (used by RAGAS eval harness only)
}

Conversations

Method	Path	Description
`GET`	`/conversations`	List user's conversations (id, title, timestamps).
`POST`	`/conversations`	Create: `{title, messages}`. Returns conversation object.
`GET`	`/conversations/{id}`	Get conversation with full message array.
`PUT`	`/conversations/{id}`	Update title and messages.
`DELETE`	`/conversations/{id}`	Delete conversation.

System

Method	Path	Description
`GET`	`/health`	Returns env, qdrant_url, embedding_model, groq_model. No auth required.

8. Configuration Reference

All settings live in backend/app/config.py and are read from environment variables (or backend/.env for local development). Never commit backend/.env — it contains real API keys.

Required in production

Variable	Description
`GROQ_API_KEY`	Groq API key. Free tier: 100K tokens/day.
`JWT_SECRET`	Random string for HS256 JWT signing. Use `openssl rand -hex 32`.
`AUTH_USERS`	JSON object `{"email@example.com": "password"}`. Supports multiple users.
`QDRANT_URL`	Qdrant server URL. E.g. `https://xxx.cloud.qdrant.io:6333`
`QDRANT_API_KEY`	Qdrant Cloud API key (blank for local).

Optional but recommended

Variable	Default	Description
`GROQ_MODEL`	`llama-3.3-70b-versatile`	Primary LLM. LLaMA 3.3-70b is free on Groq and supports 128K context.
`VISION_API_KEY`	``	API key for vision LLM (PDF parsing). Uses `VISION_BASE_URL` endpoint. If blank, vision path is disabled and PyMuPDF + Tesseract are used.
`VISION_BASE_URL`	`https://api.openai.com/v1`	Vision API endpoint (OpenAI-compatible).
`VISION_MODEL`	`gpt-4o-mini`	Vision model. gpt-4o-mini: ~$0.01/40-page document.
`FALLBACK_API_KEY`	``	API key for text-generation fallback when Groq quota is exhausted.
`FALLBACK_BASE_URL`	`https://api.openai.com/v1`	Fallback LLM endpoint.
`FALLBACK_MODEL`	`gpt-4o-mini`	Fallback model name.
`RERANK_ENABLED`	`true`	Set `false` to skip cross-encoder reranking (faster, lower precision).
`RERANKER_MODEL`	`BAAI/bge-reranker-v2-m3`	Cross-encoder model.
`TURSO_URL`	``	Turso database URL for persistent conversation storage. If blank, uses local SQLite (lost on container restart).
`TURSO_TOKEN`	``	Turso auth token.

Less commonly changed

Variable	Default	Description
`QDRANT_COLLECTION`	`rag_documents`	Qdrant collection name.
`EMBEDDING_MODEL`	`BAAI/bge-m3`	Embedding model. Do not change without re-ingesting all documents.
`EMBEDDING_DEVICE`	`cpu`	`cpu`, `cuda`, `mps`. HF Spaces has no GPU — always `cpu`.
`DUCKDB_PATH`	`data/tabular.duckdb`	DuckDB file path (relative to working dir).
`UPLOAD_DIR`	`data/uploads`	Uploaded source files directory.
`LOG_LEVEL`	`INFO`	Python logging level.
`APP_ENV`	`development`	Visible in `/health`.

9. Frontend

Stack: React 18, TypeScript, Vite, react-markdown with remark-gfm

Component structure

App.tsx
├── LoginPage.tsx         — email + password form, token stored in localStorage
│
└── (authenticated)
    ├── Sidebar.tsx       — conversation list, new chat, sign out
    │
    └── main content
        ├── ChatView.tsx  — message thread, SSE streaming, citation chips
        └── UploadView.tsx — file drop zone, ingestion status polling

Key frontend behaviors

Streaming: askQuestionStream() in api.ts reads the SSE response using ReadableStream. Status events update the thinking indicator ("Recherche dans les documents…" / "Rédaction de la réponse…"). Token events append to the message in real time.

Citations: Citation chips appear below each assistant message. page can be number (PDF page) or string (Excel sheet name). Hovering a chip shows the first 300 chars of that source chunk as a tooltip.

Markdown: ReactMarkdown with remark-gfm renders tables, bold, code, blockquotes. Vision LLM pages return markdown tables which render correctly.

Conversation persistence: After each assistant response, the full message array is saved via PUT /conversations/{id}. On sidebar click, GET /conversations/{id} restores the thread.

Languages: UI strings are in i18n.ts, supporting en and fr. Language preference stored in localStorage.

Build output

Vite builds to frontend/dist/. The Dockerfile copies this to backend/static/. FastAPI serves static/assets/ and falls back to static/index.html for any path not matched by an API route (SPA client-side routing).

10. Authentication

Mechanism: HS256 JWT, 30-day expiry.

User management: Users are defined in the AUTH_USERS environment variable as a JSON object:

{"alice@winncare.tn": "password1", "bob@winncare.tn": "password2"}

There is no self-registration or password reset. Add/remove users by updating the secret.

Token storage: Stored in localStorage under key julia_token. Auto-cleared on 401 response and page reloads to the login screen.

Authorization on API routes: All query, ingestion, and conversation routes depend on get_current_user(). A missing or invalid token returns HTTP 401 immediately.

Note: This is a simple single-tenant auth model. For multi-tenant deployments with data isolation, each user would need their own Qdrant namespace or collection, and the doc_id / filename payload indexes would need a tenant_id field.

11. Conversation Persistence

File: backend/app/conversations/store.py

Conversations are stored in SQLite (local dev) or Turso (production).

Schema:

CREATE TABLE conversations (
    id          TEXT PRIMARY KEY,
    user_email  TEXT NOT NULL,
    title       TEXT NOT NULL,
    messages    TEXT NOT NULL DEFAULT '[]',   -- JSON array
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL
)

Messages are stored as a JSON blob. Each message: {role: "user"|"assistant", text: string, meta?: AskResponse}.

Turso: Turso is a cloud SQLite service with a free tier. Set TURSO_URL and TURSO_TOKEN in secrets. Without it, conversations are stored in data/conversations.db which is ephemeral on HF Spaces (lost on container restart).

12. Evaluation Harness (RAGAS)

File: backend/app/eval/ragas_eval.py
Questions: backend/app/eval/data/questions.jsonl
Results: backend/app/eval/results/report.json

Metrics

Metric	What it measures	Target
`faithfulness`	Fraction of Julia's claims that are verifiable from retrieved chunks. 1.0 = zero hallucination.	≥ 0.85
`context_precision`	Were retrieved chunks actually useful? Penalizes noise at the top of the ranking.	≥ 0.75
`context_recall`	Did retrieval cover everything needed to answer? Requires `ground_truth`.	≥ 0.80

Running the eval

The backend must be running. Run in a separate virtual environment (dependency conflict between FlagEmbedding and RAGAS on datasets version):

cd backend
python -m venv .venv-eval
.venv-eval\Scripts\activate          # Windows
# or: source .venv-eval/bin/activate  # Mac/Linux
pip install -r requirements-eval.txt
python -m app.eval.ragas_eval

Questions file format

{"question": "Quelle est la composition du PB2050683 ?", "ground_truth": "49% PE - 51% PU"}
{"question": "Quel est le seuil AQL pour les défauts critiques ?", "ground_truth": "AQL 0,065"}

Lines starting with # are comments.

Interpreting results

faithfulness      < 0.80 → Julia hallucinates — tighten the system prompt or reduce context noise
context_precision < 0.70 → Retrieval is noisy — tune top_k, chunking, or reranker threshold
context_recall    < 0.70 → Retrieval misses info — check chunking size, vision threshold, re-ingest

The report surfaces the 3 lowest-scoring questions by combined faithfulness + recall. Fix these first.

13. Deployment

HuggingFace Spaces (current)

The app runs as a Docker container on HF Spaces. Push to the main branch of the HF Space repository triggers a rebuild.

Remote: https://huggingface.co/spaces/esra2001/winncare (git remote: origin)

Build: Two-stage Docker build. Stage 1 builds the React SPA. Stage 2 is Python 3.12-slim, installs backend deps, copies the Vite dist/ into backend/static/, pre-downloads BGE-M3 (~2 GB baked into image layer so the first request is fast).

Exposed port: 7860 (HF Spaces requirement).

Secrets (set in HF Space dashboard, not in git):

GROQ_API_KEY
JWT_SECRET
AUTH_USERS
QDRANT_URL
QDRANT_API_KEY
VISION_API_KEY
VISION_BASE_URL
VISION_MODEL
FALLBACK_API_KEY
FALLBACK_BASE_URL
FALLBACK_MODEL
RERANK_ENABLED
RERANKER_MODEL
GROQ_MODEL
TURSO_URL
TURSO_TOKEN

Persistent storage: HF Spaces ephemeral disk — data/uploads/ and data/conversations.db are lost on restarts. Use Qdrant Cloud for vectors and Turso for conversations to achieve full persistence across restarts.

Triggering a rebuild without code changes:

git commit --allow-empty -m "trigger rebuild" && git push origin HEAD:main

Production (self-hosted)

Use docker-compose.yml for local Qdrant and Redis, then run the app container separately:

docker-compose up -d           # starts Qdrant (port 6333) and Redis (port 6379)
docker build -t julia-rag .
docker run -p 7860:7860 \
  -e GROQ_API_KEY=... \
  -e JWT_SECRET=... \
  -e AUTH_USERS='{"admin@company.com":"password"}' \
  -e QDRANT_URL=http://host.docker.internal:6333 \
  julia-rag

14. Local Development

Backend

cd backend

# Create and activate virtual environment
python -m venv .venv
.venv\Scripts\activate       # Windows
# source .venv/bin/activate  # Mac/Linux

pip install -r requirements.txt

# Start infrastructure
docker-compose up -d   # Qdrant + Redis

# Configure
cp .env.example .env   # fill in GROQ_API_KEY at minimum

# Run
uvicorn app.main:app --reload --port 8000

Minimal .env for local dev:

GROQ_API_KEY=gsk_...
JWT_SECRET=any-random-string
AUTH_USERS={"dev@local.com": "dev"}
QDRANT_URL=http://localhost:6333

Frontend

cd frontend
npm install
npm run dev     # Vite dev server on http://localhost:5173

Vite proxies /query, /ingest, /auth, /conversations to http://localhost:8000 (configured in vite.config.ts). The SPA runs on its own port; no CORS issues in dev.

Building for production (manual)

cd frontend && npm run build      # → frontend/dist/
cp -r frontend/dist backend/static
cd backend && uvicorn app.main:app --port 7860

15. Operational Notes

After changing chunking or parsing logic

Re-ingest all documents. Old chunks in Qdrant do not reflect new chunk sizes, overlap, content_hash, or chunk_index. Steps:

Go to the Upload tab → "Clear all documents"
Re-upload each document

After changing the embedding model

The new model produces incompatible vectors. You must recreate the collection:

Delete collection in Qdrant dashboard (or clear all via UI)
Re-ingest all documents
Update EMBEDDING_MODEL in secrets before re-ingesting

Groq daily quota (free tier)

Groq free tier: 100,000 tokens/day. At typical usage (question + 5-turn history + 10K context + 1200 token answer ≈ 14K tokens/request), that's ~7 full requests before hitting the daily limit.

Options:

Set FALLBACK_API_KEY + FALLBACK_MODEL to automatically failover to OpenAI/Gemini
Upgrade Groq plan
Switch GROQ_MODEL to llama-3.1-8b-instant (6K TPM limit but uses fewer tokens)

The 429 error message shown to users includes a retry-after time extracted from the Groq error response.

Vision LLM costs

At GPT-4o-mini pricing (~$0.075 per 1M input tokens, ~$0.30 per 1M output tokens), a 40-page document where 20 pages go through vision costs roughly $0.02–$0.05. Budget accordingly for large corpora.

Reranker memory

BAAI/bge-reranker-v2-m3 loads ~~550 MB into RAM on first query. On HF Spaces (16 GB), this is fine alongside BGE-M3 (~~2 GB). Total model footprint: ~2.6 GB.

SSE and proxy buffering

The /query/ask/stream endpoint sends three headers to prevent proxy buffering:

X-Accel-Buffering: no — disables nginx buffering (critical for HF Spaces)
Cache-Control: no-cache — tells intermediate proxies not to cache the stream
Connection: keep-alive — prevents idle-timeout drops on long responses

If responses appear cut or arrive all at once on a specific network, a local proxy is the cause. These headers resolve it for standard nginx-based setups.

16. Known Limitations and Trade-offs

Free-tier constraints

Constraint	Impact	Workaround
Groq: 100K tokens/day	~7 full requests before quota	Set FALLBACK_API_KEY
HF Spaces: 2 vCPU, no GPU	BGE-M3 embed ~3-5s per chunk batch; reranker ~2-4s per request	Pre-download models in Dockerfile (already done)
HF Spaces: ephemeral disk	Uploads and local conversations lost on restart	Use Qdrant Cloud + Turso

Architecture trade-offs

No parent-child retrieval. Chunks are 2500 chars with 300 chars overlap. Source-window expansion (fetch adjacent page chunks for top result) partially compensates. True parent-child would require dual-index ingestion at different granularities.

Single tenant. All users share one Qdrant collection and one DuckDB file. No per-user document isolation. Adding tenant_id to every chunk payload and filtering on it is the path to multi-tenancy.

Vision LLM at ingestion time only. PDF parsing with vision happens when a document is uploaded, not at query time. Changing the vision model requires re-ingesting documents to benefit.

SQL path for aggregations only. classify.py routes to DuckDB only for numeric questions (sum, count, average, max, min across many rows). Single-value lookups ("what is the price of X?") go through the vector path even when the value is in an Excel sheet.

No streaming fallback. When Groq fails and the fallback provider responds, the answer is simulated as word-by-word streaming (12ms delay per word) rather than true token streaming. The experience is slightly less responsive but visually similar.

Accuracy targets (current RAGAS baseline)

Metric	Baseline	Target
faithfulness	0.50	≥ 0.85
context_precision	0.67	≥ 0.75
context_recall	0.71	≥ 0.80

The main drivers of low faithfulness are: weak context (already addressed by abstention threshold) and the model summarizing across documents rather than citing specific claims. Running RAGAS after re-ingestion with the Phase 1 changes should show measurable improvement in all three metrics.