docs: add architecture flow and change safety contract

PRESERVE_EXISTING_FUNCTIONALITY.md

@@ -0,0 +1,112 @@
+# Preserve Existing Functionality
+
+This file is the change-safety contract for IRIS FINBOT. Read it before making any new code, deployment, workflow, or UI changes.
+
+## Goal
+
+New changes must not overwrite saved Hugging Face deployment fixes, break the cache-backed backend fallback, remove page screenshot assets, or regress existing user workflows.
+
+## Protected Runtime Behavior
+
+1. Browser requests must use relative `/api/proxy/*` URLs.
+2. The browser must never call `localhost:8000` or `127.0.0.1:8000` directly.
+3. `/api/proxy/api/health` must return `200` even when the FastAPI process is warming up or unavailable.
+4. `/api/proxy/api/chat/query` must return cached IR answers for supported financial questions when FastAPI is unavailable.
+5. `/api/proxy/api/documents` must return indexed document metadata from local cache when FastAPI is unavailable.
+6. `/api/proxy/api/visuals/<doc_id>/<page>` must return exact PDF page screenshots from `backend/data/pages`.
+7. Legacy `/pages/<doc_id>/pages/<image>.png` links must continue to work through the Next.js proxy.
+8. `docker-entrypoint.sh` must keep the FastAPI backend in a restart loop and keep the public Next.js server alive.
+9. Hugging Face sync must not overwrite HF-specific `Dockerfile`, `docker-entrypoint.sh`, or `README.md` from GitHub automation.
+
+## Protected Files
+
+Treat these files as high-risk. Do not edit them casually, and never replace them wholesale without checking the behavior above.
+
+- `src/app/api/proxy/[...path]/route.ts`
+- `src/lib/api.ts`
+- `docker-entrypoint.sh`
+- `Dockerfile`
+- `next.config.ts`
+- `sync-to-hf.sh`
+- `.github/workflows/sync-to-hf.yml`
+- `backend/lightweight_server.py`
+- `backend/data/response_cache/**`
+- `backend/data/pages/**`
+- `README.md`
+
+## Required Checks Before Pushing
+
+Run these checks after any change touching runtime, API, deployment, evidence, or navigation:
+
+```bash
+npm run build
+npx eslint 'src/app/api/proxy/[...path]/route.ts' src/lib/api.ts src/components/layout/ResponseWorkspace.tsx
+bash -n docker-entrypoint.sh sync-to-hf.sh
+```
+
+For backend-fallback changes, also run a forced fallback smoke test by starting Next.js with an unreachable backend:
+
+```bash
+PORT=4012 HOSTNAME=127.0.0.1 BACKEND_INTERNAL_URL=http://127.0.0.1:65531 node .next/standalone/server.js
+```
+
+Then verify:
+
+```bash
+curl -s http://127.0.0.1:4012/api/proxy/api/health
+curl -s http://127.0.0.1:4012/api/proxy/api/documents
+curl -s -H 'Content-Type: application/json' \
+  -d '{"question":"What was the total profit in Q1 2026?","doc_ids":["emiratesnbd_investor_presentation_2026_q1"]}' \
+  http://127.0.0.1:4012/api/proxy/api/chat/query
+curl -I http://127.0.0.1:4012/api/proxy/api/visuals/emiratesnbd_investor_presentation_2026_q1/1
+curl -I http://127.0.0.1:4012/pages/emiratesnbd_investor_presentation_2026_q1/pages/page_0001_colpali.png
+```
+
+Expected result:
+
+- health: `200`
+- documents: `200`
+- chat: `200` with `response_type: "ir_response"` for supported cached questions
+- visual image: `200 image/png`
+- legacy page image: `200 image/png`
+
+## Change Rules
+
+1. Keep edits narrow and local to the requested behavior.
+2. Do not remove fallback logic just because FastAPI works locally.
+3. Do not remove cache files or screenshot folders from HF uploads.
+4. Do not reintroduce plain Git force-push for HF syncs that include PNG files; use `hf upload`.
+5. Do not change `NEXT_PUBLIC_BACKEND_URL` to a direct backend URL for production.
+6. Do not show user-facing "Backend not detected" or "Demo Mode" messages when the cache-backed proxy is working.
+7. Do not change response schemas without updating both FastAPI and Next.js fallback responses.
+8. Do not alter document/page URL formats without preserving legacy normalization.
+9. If a workflow syncs GitHub to HF, keep HF-specific deployment files excluded unless the sync path intentionally owns them.
+
+## HF Deployment Notes
+
+Use targeted uploads for small fixes:
+
+```bash
+hf upload rajvivan/iris-ir-platform <local_file> <repo_file> --repo-type space
+```
+
+Use `./sync-to-hf.sh` only when syncing the full HF-ready bundle. The script uses `hf upload` to avoid binary screenshot push failures.
+
+After pushing, verify:
+
+```bash
+curl -s https://huggingface.co/api/spaces/rajvivan/iris-ir-platform
+curl -L -s -o /tmp/hf-home.html -w '%{http_code}\n' https://rajvivan-iris-ir-platform.hf.space/
+curl -L -s https://rajvivan-iris-ir-platform.hf.space/api/proxy/api/health
+```
+
+## Review Checklist
+
+Before accepting any future change, confirm:
+
+- Existing chat questions still produce `ir_response` where cache coverage exists.
+- Source footnotes still open the correct page.
+- Screen popup still opens as a full-screen overlay.
+- Page screenshots still load in HF.
+- Guardrail cards still render unsupported and insufficient states correctly.
+- HF runtime status is `RUNNING` after rebuild.

README.md

@@ -7,8 +7,6 @@ sdk: docker
 pinned: false
 short_description: Multimodal RAG platform for financial document intelligence
 thumbnail: https://huggingface.co/spaces/rajvivan/iris-ir-platform/resolve/main/screenshot.png
-app_port: 7860
-startup_duration_timeout: 1h
 ---
 
 # IRIS — Investor Relations Intelligence System
@@ -31,46 +29,36 @@ IRIS is a state-of-the-art, fully local, multimodal Retrieval-Augmented Generati
 
 ## 🏛️ System Architecture
 
+```mermaid
+flowchart TD
+  S1["1. User opens HF Space or local app"]
+  S2["2. Next.js UI renders chat, document viewer, KPI cards, and evidence panels"]
+  S3["3. Browser calls relative API routes under /api/proxy/*"]
+  S4["4. Next.js proxy checks internal FastAPI at BACKEND_INTERNAL_URL, default 127.0.0.1:8000"]
+  S5{"5. Is FastAPI live?"}
+  S6["6A. Live path: FastAPI receives /api/health, /api/documents, /api/chat/query, /api/visuals/*"]
+  S7["6B. Fallback path: Next.js serves the same API from backend/data cache and page screenshots"]
+  S8["7. Guardrail and intent detection classify unsupported, insufficient, or IR financial query"]
+  S9["8. Retrieval/answer layer loads verified KPI cache, response_cache JSON, tables, and exact page images"]
+  S10["9. UI returns grounded answer with executive summary, KPI rows, key drivers, source footnotes, and visual evidence"]
+
+  S1 --> S2 --> S3 --> S4 --> S5
+  S5 -- yes --> S6 --> S8
+  S5 -- "no / warming / restarting" --> S7 --> S8
+  S8 --> S9 --> S10
 ```
-┌─────────────────────────────────────────────────────────────────────┐
-│               Next.js Frontend / HF Web Server (3000/7860)          │
-│          Chat UI · PDF Viewer · KPI Cards · Visual Evidence         │
-└──────────────────────────────┬──────────────────────────────────────┘
-                               │  /api/proxy/*
-                               ▼
-┌─────────────────────────────────────────────────────────────────────┐
-│                 Resilient Next.js API Proxy                         │
-│                                                                     │
-│  1. Try internal FastAPI at BACKEND_INTERNAL_URL / localhost:8000   │
-│  2. If unavailable, serve the same API from checked-in cache files  │
-│  3. Keep /api/health green so office networks do not block usage    │
-└──────────────────────────────┬──────────────────────────────────────┘
-                               │  internal container call
-                               ▼
-┌─────────────────────────────────────────────────────────────────────┐
-│                    FastAPI Backend (Port 8000)                       │
-│                                                                     │
-│  ┌──────────────────── Response Pipeline ─────────────────────────┐ │
-│  │                                                                 │ │
-│  │  Layer 1 · Smart Engine           ~1–50ms  ✅ ANY trained PDF  │ │
-│  │  ├── IntentClassifier     detect financial intent               │ │
-│  │  ├── KPIContextBuilder    load verified KPIs from ground truth  │ │
-│  │  └── SmartResponseEngine  fill template / load cached response  │ │
-│  │                                                                 │ │
-│  │  Layer 2 · Static Cache           ~10ms   (Emirates NBD fallback│ │
-│  │                                                                 │ │
-│  │  Layer 3 · Ollama RAG Pipeline    ~30–120s (unknown questions)  │ │
-│  │  ├── HybridRetriever (Text + Table + ColPali)                   │ │
-│  │  └── FinancialAnalystAgent (Ollama generation)                  │ │
-│  └─────────────────────────────────────────────────────────────────┘ │
-│                                                                     │
-│  ┌──────────── Retrieval Legs (Hybrid RRF Fusion) ────────────────┐ │
-│  │  Dense Text  │  Structured Tables  │  ColPali Visual (MaxSim)  │ │
-│  │  ChromaDB    │  tables.json index  │  Page patch embeddings     │ │
-│  │  BGE Embeds  │  Keyword + metadata │  PaliGemma vision model    │ │
-│  └─────────────────────────────────────────────────────────────────┘ │
-└─────────────────────────────────────────────────────────────────────┘
-```
+
+### 9-Step Runtime Flow
+
+1. User opens the Hugging Face Space or local app.
+2. Next.js renders the chat workspace, document panels, source viewer, and evidence cards.
+3. The browser calls only relative `/api/proxy/*` routes, so office networks never need direct access to `localhost:8000`.
+4. The Next.js proxy tries the private in-container FastAPI backend at `BACKEND_INTERNAL_URL`, defaulting to `http://127.0.0.1:8000`.
+5. If FastAPI is live, requests are streamed through to the backend.
+6. If FastAPI is warming, blocked, or restarting, Next.js serves the same public API contract from checked-in cache files under `backend/data`.
+7. Domain guardrails and intent detection decide whether the query is unsupported, insufficient, or a valid Investor Relations financial question.
+8. The answer layer loads verified KPI data, precomputed `response_cache` JSON, source metadata, and exact PDF page screenshots.
+9. The UI returns an evidence-grounded response with executive summary, KPI rows, key drivers, source footnotes, and visual evidence.
 
 ---
 
@@ -93,6 +81,12 @@ This means office networks only need to reach the single Hugging Face HTTPS URL.
 
 ---
 
+## Change Safety Contract
+
+Before changing runtime, sync, proxy, cache, or evidence-rendering behavior, read [PRESERVE_EXISTING_FUNCTIONALITY.md](./PRESERVE_EXISTING_FUNCTIONALITY.md). It lists protected files, required checks, and the expected HF-safe deployment behavior.
+
+---
+
 ## ⚡ Smart Response Optimizer (New)
 
 The core performance innovation. All common financial questions are handled before any LLM inference:
@@ -301,9 +295,8 @@ The 9-step training pipeline runs:
 5. Generate training pairs & calibrate retrieval weights
 6. (Optional) Fine-tune embedding model
 7. Embed & index chunks in ChromaDB
-8. Index tables with enriched metadata
-9. **[8b] Auto-generate smart response cache** ← all intent templates pre-filled
-10. Render pages + ColPali visual indexing
+8. Index tables, render page screenshots, and build ColPali visual indexing
+9. Auto-generate smart response cache ← all intent templates pre-filled
 
 After training, all supported financial intents are instantly cached for the new document. First question → template generation (<10ms). Subsequent questions → cache hit (<5ms).