codebase.md

# GeminiRAG — Codebase Reference
**Every file, what it does, and how they connect.**

---

## Directory Tree

```
geminirag/
├── app/
│   ├── main.py
│   ├── config.py
│   ├── deps.py
│   ├── security.py
│   ├── limiter.py
│   ├── api/
│   │   ├── auth.py
│   │   ├── files.py
│   │   ├── jobs.py
│   │   ├── documents.py
│   │   ├── query.py
│   │   ├── admin.py
│   │   └── agent.py
│   ├── models/
│   │   └── db.py
│   ├── processors/
│   │   ├── base.py
│   │   ├── pdf.py
│   │   ├── docx_proc.py
│   │   ├── xlsx_proc.py
│   │   ├── image.py
│   │   └── video.py
│   ├── rag/
│   │   ├── engine.py
│   │   ├── chunker.py
│   │   ├── embedder.py
│   │   └── vectorstore.py
│   ├── workers/
│   │   ├── celery_app.py
│   │   └── tasks.py
│   ├── agent/
│   │   ├── agent.py
│   │   └── tools.py
│   ├── evaluation/
│   │   └── ragas_eval.py
│   └── observability/
│       ├── logging.py
│       └── tracing.py
├── frontend/
│   ├── src/
│   │   ├── main.tsx
│   │   ├── App.tsx
│   │   ├── index.css
│   │   ├── vite-env.d.ts
│   │   ├── api/
│   │   │   └── client.ts
│   │   ├── context/
│   │   │   ├── AuthContext.tsx
│   │   │   └── ToastContext.tsx
│   │   ├── components/
│   │   │   ├── NavBar.tsx
│   │   │   └── PrivateRoute.tsx
│   │   ├── hooks/
│   │   │   └── useToast.ts
│   │   └── pages/
│   │       ├── LoginPage.tsx
│   │       ├── RegisterPage.tsx
│   │       ├── UploadPage.tsx
│   │       ├── QueryPage.tsx
│   │       ├── JobsPage.tsx
│   │       ├── AdminPage.tsx
│   │       └── AgentPage.tsx
│   ├── package.json
│   ├── vite.config.ts
│   ├── tailwind.config.js
│   ├── tsconfig.json
│   └── index.html
├── scripts/
│   ├── seed_admin.py
│   ├── ragas_baseline.py
│   └── download_ragas_datasets.py
├── tests/
│   ├── conftest.py
│   ├── test_api.py
│   ├── test_processors.py
│   ├── test_rag.py
│   ├── test_query.py
│   └── test_agent.py
├── migrations/             ← Alembic migration versions
├── Data set/
│   └── ragas_eval/
│       ├── ms_marco_samples.json
│       └── natural_questions_samples.json
├── .env                    ← gitignored
├── .env.example
├── pyproject.toml
├── docker-compose.yml
├── docker-compose.prod.yml
├── Dockerfile
├── alembic.ini
├── README.md
├── HANDOVER.md
├── DEMO_SCRIPT.md
├── context.md              ← this project's session context
└── codebase.md             ← this file
```

---

## Backend Files (`app/`)

---

### `app/main.py` — App Factory

**What it does:** Creates the FastAPI application, wires up all middleware and routers, exposes `/health`.

**Key contents:**
- `create_app() → FastAPI` — the factory function
- CORS middleware using `settings.allowed_origins_list` (env-configurable)
- slowapi rate limiter exception handler
- HTTP request logging middleware — logs `request_id`, `user_id`, `endpoint`, `method`, `status_code`, `latency_ms` for every request
- `/health` GET — pings PostgreSQL (`SELECT 1`) and ChromaDB (`heartbeat()`); returns `{"status":"ok","database":"ok","chromadb":"ok"}` or 503 if either is down
- Registers 7 routers: `auth`, `files`, `jobs`, `documents`, `query`, `admin`, `agent`
- `app = create_app()` — singleton used by uvicorn

**Connects to:** `config.py`, `limiter.py`, `observability/logging.py`, `observability/tracing.py`, all `api/` modules, `models/db.py`, `rag/vectorstore.py`

---

### `app/config.py` — Settings

**What it does:** Loads and validates all environment variables. The app crashes on startup if P0 vars are missing or still set to placeholder values.

**Key contents:**
- `class Settings(BaseSettings)` — Pydantic settings model
- P0 fields (required, app exits if missing): `GEMINI_API_KEY`, `DATABASE_URL`, `REDIS_URL`, `SECRET_KEY`
- P1 fields (have defaults): `CHROMA_HOST/PORT/COLLECTION`, `ACCESS_TOKEN_EXPIRE_MINUTES`, `ALGORITHM`, `UPLOAD_DIR`, `GEMINI_MODEL`, `GEMINI_EMBEDDING_MODEL`, `CHUNK_SIZE` (800), `CHUNK_OVERLAP` (100), `RAG_TOP_K` (5), `CONFIDENCE_THRESHOLD` (0.65), `CELERY_MAX_RETRIES`, `CELERY_RETRY_BACKOFF`, `OTEL_*`, `ALLOWED_ORIGINS`
- `allowed_origins_list` property — splits `ALLOWED_ORIGINS` by comma for CORS
- `model_post_init()` — validates no placeholder values remain
- `settings` singleton — imported everywhere via `from app.config import settings`

**Connects to:** imported by virtually every other module

---

### `app/deps.py` — FastAPI Dependencies

**What it does:** Provides reusable dependency-injected objects for route handlers.

**Key contents:**
- `oauth2_scheme = OAuth2PasswordBearer(tokenUrl="/auth/login")`
- `get_db()` — yields a `Session(get_engine())`, used as `Depends(get_db)` in routes
- `get_current_user(token, db)` — decodes JWT, loads `User` from DB, checks `is_active`, updates `last_active_at`, raises 401 if anything fails
- `require_admin(current_user)` — checks `current_user.role == UserRole.admin`, raises 403 if not

**Connects to:** `security.py` (decode_token), `models/db.py` (User, get_engine)

---

### `app/security.py` — Auth Utilities

**What it does:** Password hashing and JWT encoding/decoding. No FastAPI dependencies — pure utility functions.

**Key contents:**
- `hash_password(password) → str` — bcrypt hash via passlib
- `verify_password(plain, hashed) → bool` — bcrypt comparison
- `create_access_token(data, expires_minutes) → str` — JWT encode with `exp` claim, signed with `settings.SECRET_KEY`
- `decode_token(token) → dict` — JWT decode, raises HTTP 401 on JWTError or expired token

**Connects to:** `config.py` (SECRET_KEY, ALGORITHM), used by `api/auth.py` and `deps.py`

---

### `app/limiter.py` — Rate Limiter

**What it does:** Single module that instantiates the slowapi `Limiter` so it can be imported without circular dependencies.

**Key contents:**
- `limiter = Limiter(key_func=get_remote_address)`

**Connects to:** `main.py` (registers exception handler), `api/auth.py` (decorates /login)

---

## API Route Handlers (`app/api/`)

---

### `app/api/auth.py` — Authentication

**Routes:**
- `POST /auth/register` — creates User record with hashed password
- `POST /auth/login` (rate limit: 10/min) — verifies credentials, returns JWT `access_token`

**Connects to:** `security.py`, `limiter.py`, `models/db.py` (User), `deps.py` (get_db)

---

### `app/api/files.py` — File Upload

**Routes:**
- `POST /v1/files/upload` — multipart upload, validates type/size, creates Job (PENDING), saves file to disk, enqueues Celery task

**Key logic:**
- `EXTENSION_MAP` — maps file extensions to internal type strings (pdf, docx, xlsx, csv, image, video, audio)
- `MAX_FILE_SIZE_BYTES = 500 * 1024 * 1024` (500 MB)
- Saves file to `UPLOAD_DIR/{job_id}/{original_filename}`
- Creates `Job` in `PENDING` state
- Calls `process_file.delay(str(job.id))`
- Returns 202 immediately with `{job_id, filename, file_type, status: "PENDING"}`

**Connects to:** `models/db.py` (Job, JobStatus), `workers/tasks.py` (process_file), `deps.py`, `observability/logging.py`

---

### `app/api/jobs.py` — Job Management

**Routes:**
- `GET /v1/jobs/{job_id}` — fetch single job (owner or admin)
- `GET /v1/jobs` — list jobs (user sees own, admin sees all)
- `POST /v1/jobs/{job_id}/reprocess` — reset error fields, re-queue via `process_file.delay()`

**Connects to:** `models/db.py` (Job, JobStatus, UserRole), `workers/tasks.py` (process_file), `deps.py`

---

### `app/api/documents.py` — Document Retrieval

**Routes:**
- `GET /v1/documents` — list COMPLETED jobs (these are "documents")
- `GET /v1/documents/{job_id}/summary` — return `job.result` parsed as JSON

**Connects to:** `models/db.py` (Job, JobStatus, UserRole), `deps.py`

---

### `app/api/query.py` — RAG Queries

**Routes:**
- `POST /v1/query` — standard RAG query, waits for full answer
- `POST /v1/query/stream` — same RAG retrieval, but streams answer token-by-token via SSE

**Key logic:**
- `_resolve_chunks_and_context()` — shared helper: embeds question, searches ChromaDB, applies confidence gate, returns `{early_return: bool, payload: ..., chunks: [...], user_prompt: str}`
- Streaming uses `StreamingResponse(event_stream(), media_type="text/event-stream")` — yields `data: {json}\n\n` events of type `chunk` (text fragment) and `done` (final answer + citations)
- Frontend uses `fetch` + `ReadableStream` (not `EventSource`) to handle auth headers

**Connects to:** `rag/engine.py`, `rag/embedder.py`, `rag/vectorstore.py`, `models/db.py`, `deps.py`, `google-genai` SDK

---

### `app/api/admin.py` — Admin Analytics

**Routes:**
- `GET /v1/admin/usage` — token counts, latency trends, per-user breakdown
- `GET /v1/admin/ragas` — RAGAS averages + low-scoring queries
- `GET /v1/admin/users` — user list with stats; `PATCH` to toggle `is_active`
- `GET /v1/admin/logs` — paginated raw UsageLog entries

**Connects to:** `models/db.py` (UsageLog, QueryHistory, User, UserRole), `deps.py` (require_admin)

---

### `app/api/agent.py` — Agent Chat

**Routes:**
- `POST /v1/agent/chat` — sends message to ADK agent, returns `{response, tool_calls_made, session_id, prompt_tokens, completion_tokens}`

**Connects to:** `agent/agent.py` (run_agent), `deps.py`

---

## Database Models (`app/models/`)

---

### `app/models/db.py` — ORM Tables

**What it does:** Defines all four database tables using SQLModel (SQLAlchemy under the hood).

**Tables:**

| Table | Primary Fields |
|---|---|
| `users` | id (UUID), email (unique), hashed_password, role (admin/user), is_active, created_at, last_active_at |
| `jobs` | id (UUID), user_id (FK), filename, file_type, file_path, status, step, retry_count, error_type, error_message, result (JSON str), chunk_count, created_at, updated_at |
| `usage_logs` | id (UUID), user_id, job_id, endpoint, model, prompt/completion/total_tokens, latency_ms, query_text, llm_response_preview, created_at |
| `query_history` | id (UUID), user_id, question, answer, citations (JSON), job_ids_queried (JSON), chunk_count_retrieved, avg_similarity_score, confidence_gate_passed, prompt/completion_tokens, latency_ms, ragas_scores (JSON), ragas_computed_at, created_at |

**Key functions:**
- `get_engine()` — lazy singleton with `pool_size=10, max_overflow=20, pool_pre_ping=True`
- `create_db_and_tables()` — creates all tables (used in startup or tests)

**Connects to:** everything — all API handlers, tasks, and scripts import from here

---

## File Processors (`app/processors/`)

All processors follow the same pattern: extend `BaseProcessor`, implement `extract()` and `summarise()`, call via `processor.run(db)`.

---

### `app/processors/base.py` — Abstract Base

**What it does:** Defines the interface all processors must implement, and provides the Gemini API call wrappers.

**Key contents:**
- `RateLimitError`, `InvalidInputError` — custom exceptions for error classification
- `BaseProcessor(ABC)` abstract class:
  - `extract() → str` — abstract, extract raw text from file
  - `summarise(text, db) → dict` — abstract, call Gemini and return JSON summary
  - `run(db) → (str, dict)` — template method: calls extract(), summarise(), stores JSON in `job.result`, returns both
  - `_call_gemini_json(prompt, db)` — calls Gemini with `response_mime_type="application/json"`, handles 429/400/503, logs to UsageLog
  - `_call_gemini_vision_json(prompt, image_data, mime_type, db)` — multimodal Gemini call (image + text)

**Connects to:** `observability/logging.py` (log_llm_call), `config.py` (settings), `google-genai` SDK

---

### `app/processors/pdf.py` — PDF Processor

- **extract():** pdfplumber → page text + tables → `[Page N]` prefixed concatenation
- **summarise():** Gemini JSON → `{title, document_type, summary, key_points, risks, entities, tables_found}`
- **Library:** pdfplumber

---

### `app/processors/docx_proc.py` — DOCX Processor

- **extract():** python-docx → paragraphs + tables → markdown
- **summarise():** Gemini JSON → `{title, document_type, summary, key_points, risks, sections, entities}`
- **Library:** python-docx

---

### `app/processors/xlsx_proc.py` — XLSX/CSV Processor

- **extract():** openpyxl (XLSX) or csv.reader (CSV) → markdown tables, `[Sheet: name]` prefixed, capped at 500 rows
- **summarise():** Gemini JSON → `{title, summary, sheets, column_descriptions, key_insights, row_count}`
- **Libraries:** openpyxl, csv

---

### `app/processors/image.py` — Image Processor

- **extract():** returns `""` — no text extraction step
- **summarise():** reads file as bytes → `_call_gemini_vision_json()` → `{image_type, ocr_text, language, business_card: {name, title, company, email, phone, address, website}, summary}`
- **Supported MIME types:** image/png, image/jpeg, image/webp

---

### `app/processors/video.py` — Audio/Video Processor

- **extract():** uploads file to Gemini Files API, polls until ACTIVE (300s timeout), stores `uploaded_file` reference
- **summarise():** multimodal Gemini call with diarization prompt → `{duration_seconds, speaker_count, speakers, segments: [{speaker, timestamp, text}], full_transcript, summary, action_items, key_decisions, topics_discussed}`
- **Note:** Handles both audio (.mp3/.wav/.m4a) and video (.mp4/.mov). Diarization accuracy depends on audio quality.

---

## RAG Layer (`app/rag/`)

---

### `app/rag/engine.py` — RAG Orchestration

**What it does:** The core query brain. Connects all RAG components together.

**Key contents:**
- `RAG_SYSTEM_PROMPT` — instructs Gemini to only answer from context, cite sources as [1][2], and refuse out-of-scope questions
- `_resolve_chunks_and_context(question, job_ids, settings)` — shared by both `/query` and `/query/stream`: embed question → search ChromaDB → confidence gate → format user prompt. Returns `{early_return, payload}` or `{chunks, user_prompt}`
- `query(question, job_ids, user_id, db, settings)` — full pipeline: call `_resolve_chunks_and_context()`, call Gemini for answer, parse citations, log to UsageLog + QueryHistory, enqueue `compute_ragas.delay()`, return result dict

**Confidence gate:** If `avg_similarity_score < CONFIDENCE_THRESHOLD (0.65)` → returns canned "I don't have enough information" answer without calling Gemini.

**Connects to:** `rag/embedder.py` (embed_query), `rag/vectorstore.py` (search), `observability/logging.py` (log_llm_call), `models/db.py` (QueryHistory), `workers/tasks.py` (compute_ragas.delay), `google-genai` SDK

---

### `app/rag/chunker.py` — Text Chunking

**What it does:** Splits extracted text into overlapping chunks suitable for embedding.

**Key functions:**
- `chunk_text(text, job_id, filename, file_type, chunk_size=800, overlap=100)` — splits on whitespace, sliding window (800 words, 100-word overlap), extracts `[Page N]` markers, skips chunks < 50 words. Returns list of `{text, job_id, filename, file_type, chunk_index, metadata: {page_or_segment}}`
- `chunk_video_segments(segments, job_id, filename)` — converts `[{speaker, timestamp, text}]` from Gemini diarization output into chunks with speaker/timestamp metadata

**Connects to:** `workers/tasks.py` (called during CHUNKING step)

---

### `app/rag/embedder.py` — Embedding Generation

**What it does:** Converts text chunks and queries into 768-dimensional vectors using Gemini.

**Key functions:**
- `embed_chunks(chunks, user_id, job_id, settings, db)` — batches 100 chunks at a time, calls `genai.embed_content()` with `task_type="RETRIEVAL_DOCUMENT"`, retries on 429 with delays [60, 120, 240]s, logs each batch to UsageLog
- `embed_query(question, settings)` — embeds single query with `task_type="RETRIEVAL_QUERY"`, returns 768-dim vector

**Connects to:** `observability/logging.py` (log_llm_call), `config.py` (GEMINI_EMBEDDING_MODEL), `google-genai` SDK

---

### `app/rag/vectorstore.py` — ChromaDB Operations

**What it does:** All interactions with ChromaDB vector database.

**Key functions:**
- `get_chroma_client(settings)` → `chromadb.HttpClient(host, port)`
- `get_or_create_collection(client, settings)` → cosine-distance collection named `settings.CHROMA_COLLECTION`
- `add_chunks(collection, chunks, embeddings)` — upserts with 3x retry + 5s backoff. IDs: `{job_id}_{chunk_index}`. Stores text, embeddings, metadata (job_id, filename, file_type, chunk_index, page_or_segment, speaker, timestamp)
- `search(collection, query_embedding, top_k=5, job_ids=None)` → list of `{text, score, filename, page_or_segment, job_id}`, similarity = 1 - cosine_distance
- `delete_job_chunks(collection, job_id)` — removes all chunks for a job (called on reprocess)

**Connects to:** `config.py` (CHROMA_HOST/PORT/COLLECTION), ChromaDB client library

---

## Celery Workers (`app/workers/`)

---

### `app/workers/celery_app.py` — Celery Configuration

**What it does:** Creates and configures the Celery application instance.

**Key configuration:**
- broker: `settings.REDIS_URL` (Redis)
- backend: `settings.DATABASE_URL` (PostgreSQL)
- `task_serializer / result_serializer`: `"json"`
- `worker_prefetch_multiplier: 1` — process one task at a time
- beat_schedule: `cleanup_old_uploads` runs every 86400 seconds (daily)

**Connects to:** `config.py` (REDIS_URL, DATABASE_URL)

---

### `app/workers/tasks.py` — Task Definitions

**What it does:** The three Celery task functions that do the actual background work.

**Key functions:**

`process_file(self, job_id)` — max_retries=3, bound task:
1. Dispatches to correct processor by `job.file_type`
2. Calls `processor.run(db)` → `(extracted_text, summary)`
3. `chunk_text()` or `chunk_video_segments()`
4. `embed_chunks()` → vectors
5. `add_chunks()` to ChromaDB
6. Updates job to COMPLETED
7. On error: `classify_error()` → retry if retryable (max 3 times, exponential backoff), else FAILED_PERMANENT + push to Redis `geminirag:dead_letter` list

`compute_ragas(query_history_id)` — max_retries=2:
- Re-embeds question, re-searches ChromaDB
- Calls `compute_ragas_scores()`
- Saves scores to `QueryHistory.ragas_scores`

`cleanup_old_uploads()` — scheduled daily:
- Finds COMPLETED/FAILED_PERMANENT jobs older than 7 days
- Deletes upload directories from `UPLOAD_DIR`

**Helper functions:**
- `update_job_state(db, job_id, status, step, ...)` — atomic DB update with logging
- `classify_error(exc) → (error_type_str, is_retryable)` — "429"/"quota"/"rate" → RATE_LIMIT (retryable), "400"/"invalid" → INVALID_INPUT (not retryable), else UNKNOWN (retryable)

**Connects to:** all processors, all rag modules, `models/db.py`, `celery_app.py`, `observability/logging.py`, `observability/tracing.py`

---

## ADK Agent (`app/agent/`)

---

### `app/agent/agent.py` — Agent Runner

**What it does:** Creates and runs the Google ADK conversational agent.

**Key contents:**
- `AGENT_SYSTEM_PROMPT` — instructs agent on capabilities (process files, check status, query RAG, cite sources)
- `_agent = Agent(model="gemini-2.0-flash", tools=[ingest_file, get_job_status, query_rag, list_documents, summarize_document])`
- `_session_service = InMemorySessionService()` — conversation history (resets on restart)
- `_runner = Runner(app_name="geminirag", agent=_agent, session_service=_session_service)`
- `run_agent(message, user_id, session_id?) → dict` — sets user context, calls runner, collects tool_calls and final text, returns `{response, tool_calls_made, session_id, prompt_tokens, completion_tokens}`

**Connects to:** `agent/tools.py` (5 tools), `google.adk` library, `observability/logging.py`

---

### `app/agent/tools.py` — MCP Tools

**What it does:** Implements the 5 tools the agent can call.

**Context:** `_current_user_id: ContextVar[str]` — set by `run_agent()` so tools know which user is calling.

**Tools:**

| Tool | Input | What it does | Returns |
|---|---|---|---|
| `ingest_file` | file_path: str | Creates Job, copies file to upload dir, queues process_file | {job_id, status, message} |
| `get_job_status` | job_id: str | Looks up Job in DB | {job_id, status, step, chunk_count, error_message} |
| `query_rag` | question, job_ids?, use_job_context | Calls engine.query() | {answer, citations, confidence_gate_passed, scores} |
| `list_documents` | — | Returns all COMPLETED jobs | {documents: [{job_id, filename, file_type, chunk_count}]} |
| `summarize_document` | job_id: str | Returns job.result JSON | {job_id, filename, summary: {...}} |

**Connects to:** `rag/engine.py` (query_rag), `workers/tasks.py` (ingest_file queues process_file), `models/db.py` (Job, User), `observability/logging.py`

---

## Evaluation (`app/evaluation/`)

---

### `app/evaluation/ragas_eval.py` — RAGAS Metrics

**What it does:** Computes 5 RAG quality metrics using the RAGAS library.

**Key functions:**
- `get_ragas_llm(settings)` → `ChatGoogleGenerativeAI` — LangChain wrapper for Gemini
- `get_ragas_embeddings(settings)` → `GoogleGenerativeAIEmbeddings`
- `compute_ragas_scores(question, answer, contexts, ground_truth, settings) → dict` — runs RAGAS evaluation:
  - Always: Faithfulness, AnswerRelevancy, ContextPrecision
  - If ground_truth provided: ContextRecall, AnswerCorrectness
  - Returns `{faithfulness, answer_relevancy, context_precision, context_recall, answer_correctness}` or `{error: str}`

**RAGAS target scores:** Faithfulness ≥ 0.80, Context Precision ≥ 0.60

**Connects to:** `config.py` (GEMINI_MODEL, GEMINI_EMBEDDING_MODEL), ragas library, langchain-google-genai

---

## Observability (`app/observability/`)

---

### `app/observability/logging.py` — Structured Logging

**What it does:** Configures structlog and provides the `log_llm_call()` helper that writes to both structlog and the `usage_logs` database table.

**Key functions:**
- `configure_logging()` — structlog setup with JSONRenderer, TimeStamper, StackInfoRenderer
- `get_logger()` → structlog bound logger
- `log_llm_call(user_id, job_id, endpoint, model, prompt_tokens, completion_tokens, latency_ms, query_text, llm_response_preview, db)` — creates `UsageLog` record in DB + logs to structlog

**Used by:** processors (every Gemini call), embedder, rag/engine, agent/tools

---

### `app/observability/tracing.py` — OpenTelemetry

**What it does:** Configures OpenTelemetry distributed tracing.

**Usage:** `from app.observability.tracing import tracer` then `with tracer.start_as_current_span("process_file") as span: span.set_attribute("job_id", ...)`

**Exporter:** stdout (configurable via `OTEL_EXPORTER` env var)

**Connected to:** `workers/tasks.py` (`process_file` task uses spans with job_id, file_type, user_id attributes), `main.py` (configures on startup)

---

## Frontend (`frontend/src/`)

---

### `frontend/src/main.tsx` — Entry Point

Renders `<App />` into `#root` DOM element. No logic here.

---

### `frontend/src/App.tsx` — Router

**What it does:** Sets up React Router with 7 lazy-loaded pages, wrapped in providers.

**Structure:**
```tsx
<AuthProvider>
  <ToastProvider>
    <BrowserRouter>
      <Suspense fallback={<PageLoader />}>
        <Routes> … </Routes>
      </Suspense>
    </BrowserRouter>
  </ToastProvider>
</AuthProvider>
```

**Pages (all lazy-loaded):** LoginPage, RegisterPage, UploadPage, QueryPage, AgentPage, JobsPage, AdminPage  
**Guards:** `<PrivateRoute>` wraps authenticated routes; `<PrivateRoute requireAdmin>` for /admin

**Connects to:** All page components, `context/AuthContext.tsx`, `context/ToastContext.tsx`, `components/PrivateRoute.tsx`

---

### `frontend/src/api/client.ts` — HTTP Client

**What it does:** Axios instance pre-configured for the API with JWT auth injection.

**Key exports:**
- `api` — default Axios instance with `baseURL = VITE_API_URL || "http://localhost:8000"`
- Request interceptor: attaches `Authorization: Bearer <token>` from `_getToken()`
- Response interceptor: catches 401 → calls `_onUnauthorized()` (logout + redirect)
- `setTokenGetter(fn)` / `setUnauthorizedHandler(fn)` — called by AuthContext to inject token source and logout callback

**Connects to:** `context/AuthContext.tsx` (calls setters on login/logout), used by every page

---

### `frontend/src/context/AuthContext.tsx` — Auth State

**What it does:** Global JWT authentication context. Manages logged-in user state.

**Key exports:**
- `AuthProvider` — wraps app, persists token in localStorage
- `useAuth()` → `{user: AuthUser | null, login(email, password), logout}`
- `AuthUser` = `{id, email, role, token}`

**Login flow:** POST /auth/login → decode JWT payload (base64 split on ".") → extract {sub, role} → store `AuthUser` in state + localStorage

**Connects to:** `api/client.ts` (injects token getter + unauthorized handler), `components/NavBar.tsx`, `components/PrivateRoute.tsx`, all pages

---

### `frontend/src/context/ToastContext.tsx` — Toast Notifications

**What it does:** App-wide toast notification system (no external library).

**Key exports:**
- `ToastProvider` — renders toast container fixed bottom-right, manages queue
- `useToastContext()` → `{addToast(message, type: "success"|"error"|"info"|"warning")}`

**Connects to:** `hooks/useToast.ts` (uses internal hook), used by all pages for success/error feedback

---

### `frontend/src/hooks/useToast.ts` — Toast Hook

**What it does:** Manages the toast array state, auto-removes after 4000ms.

**Key exports:**
- `useToast()` → `{toasts, addToast(message, type), removeToast(id)}`
- Each toast: `{id: number, message: string, type: ToastType}`

**Connects to:** `context/ToastContext.tsx` (used internally)

---

### `frontend/src/components/NavBar.tsx` — Navigation Bar

**What it does:** Top navigation bar with links to all pages and logout button.

**Active link:** Uses `useLocation()` to highlight current page (`border-b-2 border-white`)

**Connects to:** `context/AuthContext.tsx` (logout), React Router (useLocation, Link)

---

### `frontend/src/components/PrivateRoute.tsx` — Route Guard

**What it does:** Wraps routes that require authentication (or admin role).

**Logic:** If `!user` → redirect to `/login`. If `requireAdmin && user.role !== "admin"` → redirect to `/upload`.

**Connects to:** `context/AuthContext.tsx` (useAuth)

---

### `frontend/src/pages/LoginPage.tsx`

Email + password form → `useAuth().login()` → redirect to `/upload` on success.

---

### `frontend/src/pages/RegisterPage.tsx`

Email + password form → `POST /auth/register` → redirect to `/login`.

---

### `frontend/src/pages/UploadPage.tsx` — Upload & Job Management

**What it does:** The main file upload page and job status dashboard.

**Features:**
- Drag-and-drop zone + file input button
- `EXT_TO_TYPE` map for client-side validation before upload
- Polls `GET /v1/jobs/{job_id}` every 3 seconds until COMPLETED or FAILED
- Job cards with color-coded status badges
- Expandable card shows document summary (from `GET /v1/documents/{id}/summary`)
- Retry button re-uploads file via `POST /v1/files/upload` with stored File reference
- Empty state when no jobs exist
- Toast notifications on upload success/failure

**Connects to:** `api/client.ts`, `context/ToastContext.tsx`

---

### `frontend/src/pages/QueryPage.tsx` — RAG Query Interface

**What it does:** The primary user interface for asking questions against uploaded documents.

**Features:**
- Loads document list from `GET /v1/documents` for document selector
- Multi-select documents to scope query (or query all)
- Toggle between standard and streaming mode
- Standard: `POST /v1/query` → displays full answer when ready
- Streaming: `POST /v1/query/stream` via Fetch API + ReadableStream → streams tokens as they arrive
- Citation rendering: `[1]`, `[2]` superscripts in answer text → clickable → highlights citation card
- RAGAS score badges (green ≥ 0.8, amber 0.6–0.8, red < 0.6)
- Copy-to-clipboard button on answer
- Query history sidebar

**Why Fetch not EventSource:** EventSource API doesn't support POST or custom headers. The streaming endpoint needs both (POST body + JWT Bearer token).

**Connects to:** `api/client.ts`, `context/ToastContext.tsx`

---

### `frontend/src/pages/JobsPage.tsx` — Jobs Table

**What it does:** Full table view of all jobs with detailed status and re-process capability.

**Features:**
- `GET /v1/jobs` → table with all columns (filename, type, status, step, chunks, timestamps)
- Expandable rows with error details
- Re-process button → `POST /v1/jobs/{id}/reprocess` for failed jobs
- Horizontal scroll for wide table on mobile

**Connects to:** `api/client.ts`, `context/ToastContext.tsx`

---

### `frontend/src/pages/AdminPage.tsx` — Admin Dashboard

**What it does:** Three-tab analytics dashboard for administrators only.

**Tabs:**
1. **Usage** — today's tokens, avg latency, 7-day token trend chart (Recharts LineChart), endpoint breakdown, per-user table
2. **RAGAS** — metric averages with pass/fail indicators, 7-day RAGAS trend chart, low-scoring queries table (faith < 0.8 or relevance < 0.7)
3. **Users** — all users with query/token/job counts, last_active_at, toggle is_active button (guards self-deactivation)

**Connects to:** `api/client.ts` (admin endpoints), `context/AuthContext.tsx` (useAuth for self-deactivation guard), Recharts

---

### `frontend/src/pages/AgentPage.tsx` — Agent Chat

**What it does:** Conversational UI for the ADK agent with tool call visibility.

**Features:**
- Chat messages (user on right, agent on left with avatar)
- `POST /v1/agent/chat` with `{message, session_id}` for multi-turn conversation
- Left sidebar shows tool calls made in each response (name + icon from TOOL_ICONS map)
- Shift+Enter = newline, Enter = submit
- Markdown rendering in responses (bold, inline code, citation links)
- Token count footer
- Clear conversation button (resets session_id)

**Tool icons:** ingest_file 📎, get_job_status 🔍, query_rag 💬, list_documents 📋, summarize_document 📄

**Connects to:** `api/client.ts`, `context/ToastContext.tsx`

---

## Scripts (`scripts/`)

---

### `scripts/seed_admin.py`

```bash
py scripts/seed_admin.py --email admin@test.com --password Admin1234!
```
Creates an admin user in PostgreSQL. Exits gracefully if email already exists.

**Connects to:** `app/config.py`, `app/models/db.py` (User, UserRole), `app/security.py` (hash_password)

---

### `scripts/ragas_baseline.py`

```bash
py scripts/ragas_baseline.py --test-set C:/tmp/ragas_test_set.json
```
**Input:** JSON array of `{question, ground_truth, job_id}` (default: `C:/tmp/ragas_test_set.json`)  
**Process:** Runs RAG engine for each Q&A pair → computes RAGAS scores → prints table → saves to `C:/tmp/ragas_baseline.json`  
**Output:** Table with columns: Question | Faith | AnswRel | CtxPrec | CtxRec | AnsCorr; plus averages vs targets.

**Connects to:** `app/config.py`, `app/rag/engine.py`, `app/evaluation/ragas_eval.py`, `app/models/db.py`

---

### `scripts/download_ragas_datasets.py`

```bash
py scripts/download_ragas_datasets.py
```
Downloads 50 Q&A pairs from MS MARCO v1.1 validation and Natural Questions dev via HuggingFace `datasets` (streaming mode — does not download full dataset).

**Output:**
- `Data set/ragas_eval/ms_marco_samples.json` — 50 × `{question, ground_truth}`
- `Data set/ragas_eval/natural_questions_samples.json` — 50 × `{question, ground_truth}`

**Next step after downloading:** Add `job_id` to entries matching uploaded documents, then run `ragas_baseline.py`.

---

## Tests (`tests/`)

---

### `tests/conftest.py`

**Fixtures:**
- `engine` — SQLite `test_geminirag.db`, creates all tables, drops after session
- `db` — `Session(engine)` per test
- `client` — `TestClient(app)` with dependency override to inject test DB engine, rate limiter reset

---

### `tests/test_api.py`

Tests for authentication and file upload routes. Includes:
- `test_register_and_login` — register user, login, get token
- `test_upload_unsupported_type` — upload .xyz file → 415 error
- `test_upload_file_too_large` — upload >500MB → 413 error
- `test_get_job_wrong_user_403` — user A cannot see user B's job
- `test_login_inactive_user` — deactivated user gets 401
- `test_health` — accepts 200 or 503 (depends on ChromaDB/DB availability in test env)

---

### `tests/test_processors.py`

Tests for all 5 processor classes with mocked Gemini API calls.

---

### `tests/test_rag.py`

Tests for `chunk_text()`, `embed_chunks()` (mocked), and `vectorstore` operations.

---

### `tests/test_query.py`

Tests for `/v1/query` endpoint including confidence gate behavior and citation format.

---

### `tests/test_agent.py`

Tests for ADK agent tool invocations and response format.

---

## Top-Level Config Files

| File | Purpose |
|---|---|
| `.env` | Secrets — gitignored. Contains GEMINI_API_KEY, DB/Redis/Secret credentials |
| `.env.example` | Template for .env — committed to repo |
| `pyproject.toml` | Python 3.11+ project metadata and all dependencies |
| `alembic.ini` | Alembic migration config — points to `DATABASE_URL` env var |
| `docker-compose.yml` | Dev orchestration — 5 services (api, worker, postgres, redis, chromadb) |
| `docker-compose.prod.yml` | Production variant — no --reload, resource limits, ALLOWED_ORIGINS from env |
| `Dockerfile` | python:3.11-slim, installs deps, exposes 8000 |
| `README.md` | Full project documentation — architecture, setup, API reference, observability |
| `HANDOVER.md` | Client handover doc — setup, admin seed, RAGAS baseline, limitations, key files |
| `DEMO_SCRIPT.md` | 10-min demo guide with timing, talking points, and "if something goes wrong" table |
| `context.md` | Session context for next Claude conversation |
| `codebase.md` | This file |

---

## Data Flow Summary

```
Browser (localhost:5173)
  │
  ├── POST /auth/login ──────────────────→ api/auth.py → security.py → DB (users)
  │                                        ← JWT token
  │
  ├── POST /v1/files/upload ─────────────→ api/files.py → DB (jobs) → Redis → Celery
  │   (multipart, JWT)                     ← {job_id, status: "PENDING"}
  │
  │                                        Celery worker: process_file()
  │                                          → processors/*.py → Gemini API
  │                                          → rag/chunker.py
  │                                          → rag/embedder.py → Gemini embeddings
  │                                          → rag/vectorstore.py → ChromaDB
  │                                          → DB (jobs.status = COMPLETED)
  │
  ├── GET /v1/jobs/{id} (poll) ──────────→ api/jobs.py → DB (jobs)
  │                                        ← {status, step, chunk_count}
  │
  ├── POST /v1/query ────────────────────→ api/query.py → rag/engine.py
  │   (JWT, {question, job_ids?})              → embedder.py → Gemini
  │                                            → vectorstore.py → ChromaDB
  │                                            → Gemini RAG call
  │                                            → DB (query_history)
  │                                            → Redis → Celery (compute_ragas)
  │                                        ← {answer, citations, scores}
  │
  └── POST /v1/agent/chat ─────────────→ api/agent.py → agent/agent.py (ADK)
      (JWT, {message, session_id})            → agent/tools.py (5 tools)
                                          ← {response, tool_calls_made}
```

---

## Import Graph (simplified)

```
config.py ←────────────── everything
models/db.py ←─────────── api/, workers/, scripts/, agent/tools.py
security.py ←──────────── api/auth.py, deps.py
deps.py ←──────────────── all api/ route handlers
observability/logging.py ← processors/, rag/embedder.py, rag/engine.py, agent/tools.py
rag/engine.py ←──────────  api/query.py, agent/tools.py, scripts/ragas_baseline.py
rag/vectorstore.py ←─────  rag/engine.py, workers/tasks.py
rag/embedder.py ←────────  workers/tasks.py, rag/engine.py (via compute_ragas)
processors/base.py ←─────  processors/pdf.py, docx_proc.py, xlsx_proc.py, image.py, video.py
workers/tasks.py ←───────  api/files.py (.delay()), api/jobs.py (.delay()), rag/engine.py (.delay())
evaluation/ragas_eval.py ← workers/tasks.py (compute_ragas), scripts/ragas_baseline.py
agent/tools.py ←─────────  agent/agent.py
```