ARCHITECTURE.md

# Author RAG — Architecture Reference

---

## 🤖 AI AGENT MANDATORY PROTOCOL

> **THIS BLOCK IS FOR AI AGENTS (including Antigravity / Gemini). FOLLOW ON EVERY PROMPT.**

### On EVERY new request you must:

1. **READ this file first** before reading any source file or writing any code.
2. **CHECK §9 Invariants** — verify your planned change does not violate any rule.
3. **CHECK §8 Where to Add Features** — confirm you are editing the correct file for the task.
4. **After completing any change** that adds a file, route, model column, config value, or service — **UPDATE this document** to reflect the new state.
5. **Push the updated `ARCHITECTURE.md`** in the same commit as the code change. They must never go out of sync.

### What counts as a required update:

| Change made | Section to update |
|---|---|
| New file created | §4 Full File Tree |
| New route added | §4 (correct sub-section) |
| New DB column added | §4 Models table + §8 |
| New config value added | §7 Configuration |
| New invariant established | §9 Invariants |
| New feature area added | §8 Where to Add Features |
| Architecture restructured | §3 Layer diagram + §4 |

### NEVER do this:
- Edit source files without checking §9 first.
- Add a new file without adding it to §4.
- Define a new guard/detector outside `guards.py`.
- Add prompt text outside `prompter.py`.
- Skip updating this document after a structural change.

---

> **Purpose:** This document is the single source of truth for any AI agent or human
> developer joining this codebase. Read this file first before opening any source file.
> It answers: *What does each file do? How does a request flow? Where do I add X?*

---

## 1. What This System Does

**Author RAG** is a multi-tenant SaaS platform. Each customer (an *author*) gets an
AI-powered chatbot that lives on their website and answers questions about their books.
The chatbot is a persuasive sales tool — it understands reader intent, retrieves precise
passages from the book, and surfaces purchase links at the right moment.

**Key numbers:**
- Runtime: Python 3.11, FastAPI async
- Database: SQLite (dev) / PostgreSQL (prod) via SQLAlchemy async
- Vector store: ChromaDB (local persistent)
- LLM: OpenAI GPT-4o-mini (configurable)
- Embeddings: `text-embedding-3-small`

---

## 2. Top-Level Directory Layout

```
Author RAG/
├── app/                    ← All application code (see §4)
├── static/                 ← Compiled frontend (admin SPA, widget JS)
├── requirements.txt        ← Python dependencies
├── Dockerfile              ← HuggingFace Spaces deployment
├── ARCHITECTURE.md         ← THIS FILE
└── .env                    ← Secrets (never committed)
```

---

## 3. Layer Architecture

The codebase follows a strict 5-layer architecture. **Data only flows downward.**
A layer may never import from a layer above it.

```
┌─────────────────────────────────────────────────────┐
│  Layer 1 — API / Routers                            │
│  app/api/, app/admin/routers/, app/superadmin/      │
│  • Accepts HTTP requests                            │
│  • Validates input (Pydantic schemas)               │
│  • Delegates to services — no business logic here   │
├─────────────────────────────────────────────────────┤
│  Layer 2 — Services / Pipeline                      │
│  app/services/                                      │
│  • All business logic lives here                    │
│  • The RAG pipeline is the core of this layer       │
│  • May call repositories and other services         │
├─────────────────────────────────────────────────────┤
│  Layer 3 — Repositories                             │
│  app/repositories/                                  │
│  • All database queries live here                   │
│  • Returns model objects — never raw SQL            │
│  • No business logic                                │
├─────────────────────────────────────────────────────┤
│  Layer 4 — Models                                   │
│  app/models/                                        │
│  • SQLAlchemy ORM table definitions only            │
│  • No methods with logic                            │
├─────────────────────────────────────────────────────┤
│  Layer 5 — Core / Infrastructure                    │
│  app/core/, app/config.py, app/dependencies.py      │
│  • DB engine, Redis, ChromaDB clients               │
│  • JWT / token crypto                               │
│  • No domain logic                                  │
└─────────────────────────────────────────────────────┘
```

---

## 4. Full File Tree with Descriptions

### `app/` — Root

| File | Role |
|------|------|
| `main.py` | FastAPI app factory. Registers middleware, exception handlers, routers. Startup delegates to `core/startup/`. |
| `config.py` | All env-var settings via Pydantic `BaseSettings`. **One source of truth for every config value.** |
| `dependencies.py` | FastAPI `Depends()` providers: DB session, Redis, auth guards (`get_current_author_scoped`, `get_subscription_author`, `get_current_superadmin`). |

---

### `app/core/` — Infrastructure

| File | Role |
|------|------|
| `core/startup/db.py` | `init_db()` — creates all tables + runs incremental ALTER TABLE migrations on startup. |
| `core/startup/seeder.py` | `seed_superadmin()` — bootstraps the superadmin account from env vars on first run. |
| `core/access/token_crypto.py` | Creates and verifies signed subscription tokens (widget embed). |
| `core/access/subscription.py` | Subscription validation logic (expiry, budget, revocation). |
| `core/access/jwt_blacklist.py` | Redis-backed JWT revocation list. |
| `core/access/totp.py` | TOTP 2FA for SuperAdmin login. |
| `core/chroma_client.py` | Singleton ChromaDB client. |
| `core/security.py` | `hash_password()`, `verify_password()`. |

---

### `app/api/` — Public Widget API

These routes are called by the embedded widget on the author's website.
All require `X-Subscription-Token` header (validated via `get_subscription_author`).

| File | Routes | Role |
|------|--------|------|
| `api/chat.py` | `POST /api/chat/{slug}` | Main chat endpoint — calls `run_pipeline()`. |
| | `POST /api/chat/{slug}/session/init` | Creates a new visitor session. |
| | `GET  /api/chat/{slug}/session/history/{id}` | Returns chat history. |
| | `POST /api/chat/{slug}/session/farewell` | Graceful session close + summarization. |
| | `POST /api/chat/{slug}/session/rate` | Star rating. |
| | `POST /api/chat/{slug}/track-click` | Records buy-link click. |
| | `POST /api/chat/{slug}/feedback` | 👍/👎 message feedback. |
| | `GET  /api/chat/{slug}/events` | SSE stream for ingestion progress. |
| `api/ingest.py` | `POST /api/admin/{slug}/books/upload` | Book PDF upload + async embedding pipeline. |
| `api/widget.py` | `POST /api/widget/token` | Returns a signed subscription token for the embed script. |
| `api/schemas_router.py` | `/api/auth/*` | Login, register, token refresh. |

---

### `app/admin/` — Author Admin API

Prefix: `/api/admin/{author_slug}/`
Auth: Bearer JWT (`get_current_author_scoped`)

| File | Routes | Role |
|------|--------|------|
| `admin/router.py` | — | 20-line aggregator. Mounts all 7 sub-routers. |
| `admin/routers/dashboard.py` | `/sessions`, `/sessions/search`, `/sessions/{id}/transcript`, `/sessions/{id}/block`, `/sessions/{id}/messages/{mid}/annotate` | Session management and message review. |
| `admin/routers/books.py` | `/books`, `/books/{id}`, `/books/{id}/cover` | Book list, delete, cover upload. |
| `admin/routers/analytics.py` | `/analytics`, `/analytics/funnel`, `/analytics/intents`, `/analytics/visitors`, `/analytics/geo`, `/analytics/devices`, `/analytics/sessions/stats` | Dashboard charts, funnel data, visitor analytics. |
| `admin/routers/settings.py` | `/password`, `/widget-config`, `/profile`, `/personality`, `/notifications`, `/embed-token`, `/token-usage` | All author account settings. |
| `admin/routers/links.py` | `/smart-links/{book_id}` | Buy/preview URL management. |
| `admin/routers/qa.py` | `/qa`, `/qa/import`, `/qa/export`, `/qa/{id}` | Custom Q&A training data (CRUD + CSV). |
| `admin/routers/exports.py` | `/export/sessions`, `/export/analytics`, `/export/conversations` | CSV data exports. |

---

### `app/superadmin/` — SuperAdmin API

Prefix: `/api/super/`
Auth: Bearer JWT + TOTP (`get_current_superadmin`)

| File | Routes | Role |
|------|--------|------|
| `superadmin/router.py` | — | 15-line aggregator. Mounts 3 sub-routers. |
| `superadmin/routers/platform.py` | `/`, `/diag`, `/health`, `/audit`, `/announce`, `/backup` | Platform diagnostics, audit log, announcements. |
| `superadmin/routers/authors.py` | `/authors`, `/authors/{id}`, `/authors/{id}/suspend`, `/authors/{id}/grant`, `/authors/{id}/embed-token` | Full author account lifecycle. |
| `superadmin/routers/grants.py` | `/grants`, `/grants/{id}/revoke`, `/grants/{id}/bonus-tokens`, `/grants/{id}/extend`, `/grants/{id}/reset-tokens`, `/grant` | Subscription token budget management. |
| `superadmin/routers/_utils.py` | — | Shared `_err()` error handler (no routes). |

---

### `app/services/pipeline/` — The RAG Pipeline ⭐

This is the core of the product. Every visitor message flows through `run_pipeline()`.

```
app/services/pipeline/
├── __init__.py       ← Public API: run_pipeline, PipelineResult, invalidate_book_cache
├── core.py           ← 12-step orchestrator (the conductor, ~200 lines)
├── generation.py     ← Steps 8–10: LLM call + faithfulness retry + safety scrub
├── handlers.py       ← Short-circuit response functions (greeting, catalog, piracy...)
├── helpers.py        ← Pure stateless utilities (call_llm, format_history, get_book_links...)
├── guards.py         ← Boolean detectors (is_greeting, is_full_story_request, ...) — SINGLE SOURCE OF TRUTH
├── cache.py          ← LRU answer cache (256 slots, MD5 key)
└── dedup.py          ← Chunk deduplication by Jaccard overlap
```

#### The 12-Step Pipeline (`core.py`)

```
Step 0   sanitize_user_input()         Strip dangerous/empty input
Step 1   check_boundary()              Jailbreak / piracy / off-topic guard
Step 1.5 check_custom_qa()            Short-circuit: exact Q&A match (no LLM)
Step 2   classify_intent()            3-tier: rules → cache → LLM-as-fallback
Step 3   Book resolution              Greeting / catalog / book-select short-circuits
         LRU cache lookup             Skip steps 4-12 on cache hit
Step 4   rewrite_query()              Generate 3 query variations for multi-pass retrieval
Step 5   retrieve_chunks()            ChromaDB vector search (top_k=15 default)
Step 6   rerank_chunks()              Cross-encoder re-ranking (top_n=5 default)
Step 6.5 deduplicate_chunks()         Remove near-duplicate overlapping windows
Step 7   build_context()              Token-aware context assembly (max 2000 tokens)
Step 8   MASTER_SYSTEM_PROMPT.format  Assemble full prompt with history + interest + context
         call_llm()                   OpenAI chat completions
Step 9   check_faithfulness()         NLI guardrail — retry with stricter prompt if hallucinating
Step 10  scrub_unsafe_response()      Output safety scrub + is_response_safe() check
Step 11  UpsellEngine.select_strategy() Choose upsell strategy based on intent + session state
Step 12  ResponseFormatter.format()   Assemble final JSON response with optional buy button
         cache_set()                  Store result for future identical questions
```

**Where to look for what:**

| Task | File |
|------|------|
| Change the LLM model or temperature | `config.py` → `OPENAI_CHAT_MODEL`, `RAG_TEMPERATURE` |
| Edit the system prompt | `services/prompter.py` |
| Change what counts as a greeting | `services/pipeline/guards.py` → `_GREETINGS` |
| Change upsell logic | `services/upsell_engine.py` |
| Change faithfulness threshold | `services/faithfulness.py` |
| Change how queries are rewritten | `services/rewriter.py` |
| Change intent labels / rules | `services/intent.py` |
| Change chunking strategy | `services/chunker.py` |
| Change re-ranking | `services/reranker.py` |
| Change context token limit | `config.py` → `RAG_CONTEXT_MAX_TOKENS` |

---

### `app/services/` — All Other Services

| File | Role |
|------|------|
| `rag_pipeline.py` | **Compatibility shim only.** Imports from `pipeline/` and re-exports. |
| `intent.py` | 3-tier intent classifier: keyword rules → session cache → LLM fallback. |
| `rewriter.py` | Query rewriter — generates 3 variations using conversation context. |
| `guardrails.py` | Input sanitization, boundary checks, output safety, response scrubbing. |
| `prompter.py` | All prompt templates: `MASTER_SYSTEM_PROMPT`, boundary responses, upsell hooks. |
| `faithfulness.py` | NLI-based faithfulness scoring against retrieved context. |
| `formatter.py` | `ResponseFormatter` — assembles the final JSON `{text, links, has_links}` dict. |
| `upsell_engine.py` | `UpsellEngine` — selects upsell strategy, decides if link should show. |
| `context_builder.py` | Token-aware context string builder from ranked chunks. |
| `vector_store.py` | ChromaDB retrieval — multi-query search + score filtering. |
| `reranker.py` | Cross-encoder re-ranking of retrieved chunks. |
| `embeddings.py` | Generates and stores OpenAI embeddings into ChromaDB. |
| `chunker.py` | Splits book text into overlapping sliding-window chunks. |
| `parser.py` | Extracts clean text from PDF uploads. |
| `auth_service.py` | Register, login, token refresh logic. |
| `superadmin_service.py` | Business logic for author lifecycle (suspend, delete, grant). |
| `token_budget.py` | Token usage tracking and budget enforcement. |
| `email_service.py` | Transactional email sending. |
| `rate_limiter.py` | Per-IP / per-author rate limiting. |
| `file_validator.py` | PDF upload validation (size, MIME type, header). |
| `session_store.py` | Redis-backed session state persistence. |
| `summarizer.py` | End-of-session conversation summarizer. |
| `notifications.py` | In-app notification helpers. |
| `analytics.py` | Analytics event recording facade. |

#### `services/analytics_core/`
| File | Role |
|------||------|
| `tracker.py` | Records `AnalyticsEvent` rows after each chat turn. |
| `aggregator.py` | Pre-aggregates daily analytics for dashboard charts. |
| `geo.py` | IP → country/region/city lookup (MaxMind GeoLite2). Exposes `get_real_ip()` (proxy-aware). |
| `visitor_tracker.py` | `record_visitor()` — upserts `Visitor` table on every `session/init`. Deduplicates via localStorage UUID then fingerprint fallback. |

#### `services/session_core/`
| File | Role |
|------|------|
| `manager.py` | `SessionManager` + `SessionContext` — the session state object passed through the pipeline. |
| `context.py` | Interest score and tag tracking across turns. |
| `fingerprint.py` | Visitor fingerprint generation (SHA-256, no PII). |

---

### `app/models/` — Database Tables

| File | Table | Key columns |
|------|-------|-------------|
| `user.py` | `users` | `id`, `email`, `role` (`author`/`superadmin`), `bot_name`, `chatbot_is_active` |
| `book.py` | `books` | `id`, `author_id`, `title`, `status`, `chroma_collection_id`, `chunk_count`, `buy_url` |
| `chat_session.py` | `chat_sessions`, `chat_messages` | Session: `visitor_fingerprint`, `visitor_uid`, `turn_count`, `rating`, `blocked`. Message: `role`, `intent`, `faithfulness_score`, `hallucination_detected` |
| `client_access.py` | `client_access` | `author_id`, `plan`, `token_budget`, `tokens_used`, `expires_at`, `is_revoked` |
| `custom_qa.py` | `custom_qa` | `author_id`, `question`, `answer`, `match_threshold`, `priority`, `match_count` |
| `analytics.py` | `analytics_events` | `author_id`, `session_id`, `intent`, `link_clicked`, `prompt_tokens`, `response_ms` |
| `link.py` | `links` | `author_id`, `book_id`, `purchase_url`, `preview_url` |
| `document.py` | `documents` | `book_id`, `filename`, `status`, `pages` |
| `visitor.py` | `visitors` | `author_id`, `visitor_uid` (UUID, unique per author), `fingerprint`, `first_seen`, `last_seen`, `page_views`, `country_code`, `region`, `city`, `device_type`, `browser`, `os` |

---

### `app/repositories/` — Database Access Layer

| File | Role |
|------||------|
| `base.py` | `BaseRepository` — shared `get_by_id`, `save`, `delete` methods. |
| `user_repo.py` | `UserRepository` — lookup by email/slug, list all authors. |
| `book_repo.py` | `BookRepository` — `list_active_for_author()`, `list_for_author()`. |
| `access_repo.py` | `AccessRepository` — `get_active_for_author()` — used everywhere for budget checks. |
| `link_repo.py` | `LinkRepository` — `get_for_book()`, `upsert_for_book()`. |
| `audit_repo.py` | `AuditRepository` — append-only `log()`, `list_recent()`. |
| `document_repo.py` | `DocumentRepository` — book document tracking. |
| `visitor_repo.py` | `VisitorRepository` — `get_by_uid()`, `get_by_fingerprint()`, `create()`, `touch()`, distribution queries for analytics. |

---

### `app/schemas/` — Pydantic Request/Response Schemas

| File | Role |
|------|------|
| `chatbot.py` | `ChatRequest`, `ChatResponse`, `SessionInitResponse`, `FarewellRequest` |
| `admin.py` | `AnnotateRequest`, `FlagRequest`, `PasswordChangeRequest`, `ProfileUpdate`, `WidgetConfigUpdate` |
| `superadmin.py` | `CreateAuthorRequest`, `GrantAccessRequest`, `RevokeAccessRequest`, `AddBonusTokensRequest` |
| `auth.py` | `LoginRequest`, `RegisterRequest`, `TokenResponse` |

---

### `app/middleware/` — Request Middleware

| File | Role |
|------|------|
| `logging_middleware.py` | Structured request/response logging with `structlog`. |
| `rate_limit_middleware.py` | Sliding-window rate limiting (configurable per route type). |
| `security_headers.py` | CSP, HSTS, X-Frame-Options headers. |
| `metrics.py` | Prometheus metrics collection (optional). |

---

### `app/tasks/` — Background Tasks

| File | Role |
|------|------|
| `ingestion_task.py` | Async book processing: parse → chunk → embed → store in ChromaDB. |
| `analytics_task.py` | Periodic analytics aggregation. |
| `email_task.py` | Queued email sending. |
| `backup_task.py` | SQLite + ChromaDB backup. |
| `expiry_check_task.py` | Subscription expiry warnings. |
| `geo_update_task.py` | Retroactive geo enrichment for sessions. |
| `link_health_task.py` | Checks buy links for 404s. |
| `celery_app.py` | Celery app instance (optional — tasks can run inline). |

---

## 5. A Complete Request Trace

**Scenario:** A reader on an author's website asks *"Is the ending happy?"*

```
Browser (widget JS)
  │  POST /api/chat/{author_slug}
  │  Headers: X-Subscription-Token: <signed JWT>
  │  Body: { "message": "Is the ending happy?", "session_id": "..." }
  ▼
app/api/chat.py → chat()
  │  1. get_subscription_author()      Verify token, check budget, load author
  │  2. SessionManager.load()          Load session from Redis (history, book selection)
  │  3. run_pipeline(query, author, session_context, db)
  ▼
app/services/pipeline/core.py → run_pipeline()
  │  Step 0:  sanitize_user_input()       → "Is the ending happy?"
  │  Step 1:  check_boundary()            → no violation
  │  Step 1.5: check_custom_qa()          → no match
  │  Step 2:  classify_intent()           → intent="question", confidence=0.92
  │  Step 3:  BookRepository.list_active_for_author()  → [Book("The Last Signal")]
  │           is_greeting()              → False
  │           is_catalog_question()      → False
  │           LRU cache lookup           → miss
  │  Step 4:  rewrite_query()            → ["Is the ending happy?", "How does it end?", "resolution of the story"]
  │  Step 5:  retrieve_chunks()          → 15 chunks from ChromaDB
  │  Step 6:  rerank_chunks()            → top 5 chunks scored by cross-encoder
  │  Step 6.5: deduplicate_chunks()      → 4 unique chunks
  │  Step 7:  build_context()            → 1,840 tokens of context
  ▼
app/services/pipeline/generation.py → generate_response()
  │  Step 8:  MASTER_SYSTEM_PROMPT.format(...)
  │           call_llm()                 → "Without giving anything away, I can say..."
  │  Step 9:  check_faithfulness()       → faithful=True, score=0.91
  │  Step 10: is_response_safe()         → True
  │           return (response, 0.91, False, 312, 87)
  ▼
core.py (continues)
  │  Step 11: UpsellEngine.select_strategy()   → "SOFT_MENTION"
  │           should_include_link()            → True (turn 4, high interest)
  │           get_book_links()                 → purchase_url="https://amazon.com/..."
  │  Step 12: ResponseFormatter.format()       → {text, links:[{label:"Get the book", url:...}], has_links:True}
  │           cache_set()                      → stored for next identical query
  ▼
app/api/chat.py (continues)
  │  SessionManager.save()             Update session: turn_count++, history appended
  │  analytics.record_event()          Log intent, tokens, latency, link_shown
  │  token_budget.deduct()             Deduct prompt+completion tokens from grant
  ▼
HTTP 200 { "message": "Without giving anything away...", "links": [...] }
  ▼
Browser (widget JS)
  Renders response + "Get the book" button
```

---

## 6. Authentication Model

Three distinct auth tiers with separate dependencies:

```
Tier 1: Visitor (Widget)
  → Header: X-Subscription-Token (signed JWT, no login needed)
  → Dependency: get_subscription_author()
  → Checks: token signature, expiry, token budget, author active

Tier 2: Author (Admin)
  → Header: Authorization: Bearer <JWT>
  → Dependency: get_current_author_scoped()
  → Checks: JWT validity, author role, slug matches token

Tier 3: SuperAdmin
  → Header: Authorization: Bearer <JWT>
  → Dependency: get_current_superadmin()
  → Checks: JWT validity, superadmin role, TOTP verified
```

---

## 7. Key Configuration Values

All in `app/config.py` (loaded from `.env`):

| Variable | Default | Effect |
|----------|---------|--------|
| `OPENAI_CHAT_MODEL` | `gpt-4o-mini` | LLM used for response generation |
| `OPENAI_EMBED_MODEL` | `text-embedding-3-small` | Embedding model |
| `RAG_RETRIEVAL_TOP_K` | `15` | Chunks fetched from ChromaDB |
| `RAG_RERANK_TOP_N` | `5` | Chunks kept after re-ranking |
| `RAG_RERANK_MIN_SCORE` | `0.3` | Min cross-encoder score to keep |
| `RAG_CONTEXT_MAX_TOKENS` | `2000` | Max tokens in assembled context |
| `RAG_MAX_RESPONSE_TOKENS` | `300` | Max tokens in LLM response |
| `RAG_TEMPERATURE` | `0.4` | LLM temperature |
| `RAG_BOOK_CONFIDENCE_THRESHOLD` | `0.7` | Min score to route to a specific book |
| `SUPERADMIN_EMAIL` | — | Seeded on first startup |
| `SUPERADMIN_PASSWORD` | — | Seeded on first startup |

---

## 8. Where to Add New Features

| Task | Where to add it | Notes |
|------|-----------------|-------|
| New author admin route | `app/admin/routers/<closest>.py` | Or create a new file + mount in `admin/router.py` |
| New superadmin route | `app/superadmin/routers/<platform|authors|grants>.py` | |
| New intent label | `app/services/intent.py` → `_RULE_MAP` | Add keyword rules first; LLM only as fallback |
| New guard/detector | `app/services/pipeline/guards.py` | **Never define is_greeting elsewhere** |
| New short-circuit response | `app/services/pipeline/handlers.py` | Must return `PipelineResult` |
| New prompt template | `app/services/prompter.py` | Keep ALL prompt text in one file |
| New DB column | `app/models/<model>.py` + `app/core/startup/db.py` → `_NEW_COLUMNS` | No Alembic needed |
| New background task | `app/tasks/<name>_task.py` | Register in `tasks/celery_app.py` if async |
| New config value | `app/config.py` | Always typed, always has a default |

---

## 9. Invariants — Rules That Must Never Be Broken

1. **Services never import from routers.** Data flows down only.
2. **`guards.py` is the single source of truth** for all boolean detectors. Never re-define `is_greeting()` elsewhere.
3. **All prompt text lives in `prompter.py`.** No inline f-strings with user-facing text in other files.
4. **All pipeline results return `PipelineResult`.** Never return a raw dict from `run_pipeline()`.
5. **Repositories never contain business logic.** If it's a decision, it belongs in a service.
6. **Never skip a pipeline step.** Steps 0–12 run for every message; short-circuits return early from `core.py`, not by skipping steps.
7. **Cache only cacheable intents.** `purchase_intent`, `complaint`, `greeting` are never cached (time/person-sensitive).
8. **Token usage must always be recorded.** Every LLM call goes through `call_llm()` in `helpers.py` which returns token counts.
9. **All new DB columns go into `core/startup/db.py → _NEW_COLUMNS`.** This keeps migrations consistent across environments.
10. **All exceptions have typed exception classes** in `app/exceptions/`. Never `raise HTTPException` directly in service layer.
11. **This document must be updated on every structural change.** See §AI AGENT MANDATORY PROTOCOL above.

---

## 10. Change Log

> AI agents: append an entry here after every significant change. Format: `YYYY-MM-DD | What changed | Files affected`.

| Date | Change | Files Affected |
|------|--------|----------------|
| 2026-06-19 | Initial project build — auth, chat API, admin panel, ingestion pipeline | All |
| 2026-06-22 | Intelligence overhaul — Python-first intent classifier, hybrid rewriter, LRU cache, chunk dedup, natural upsell | `intent.py`, `rewriter.py`, `upsell_engine.py`, `rag_pipeline.py` |
| 2026-06-23 | Modularity refactor Phase 1 — `admin/router.py` split into 7 sub-routers, `rag_pipeline.py` split into `pipeline/` package | `admin/router.py`, `admin/routers/*`, `services/pipeline/*`, `services/rag_pipeline.py` |
| 2026-06-23 | Modularity refactor Phase 2 — `superadmin/router.py` split into 3 sub-routers, `main.py` startup extracted to `core/startup/`, `pipeline/core.py` generation extracted to `pipeline/generation.py` | `superadmin/router.py`, `superadmin/routers/*`, `core/startup/*`, `services/pipeline/generation.py`, `main.py` |
| 2026-06-23 | Added `ARCHITECTURE.md` with AI Agent Mandatory Protocol | `ARCHITECTURE.md` |
| 2026-06-23 | **Production Improvements Phases 1–5:** 8 bugs fixed, 7 features added. B1 fingerprint null crash, B2 silent truncation, B3 OpenAI client per-call, B4 no LLM retry, B5 piracy→jailbreak wrong intent, B6 history only 3 turns, B8 no budget warning, B9 sync ChromaDB in async loop. New: per-visitor rate limit, input cap, tenacity retry, intent rule expansion (+30% coverage), history fix, budget email warning, async health check. | `config.py`, `middleware/rate_limit_middleware.py`, `api/chat.py`, `services/pipeline/core.py`, `services/pipeline/helpers.py`, `services/intent.py`, `dependencies.py`, `services/token_budget.py`, `requirements.txt` |
| 2026-06-23 | **Visitor Analytics System:** Replaced session-based visitor counting with proper deduplication. New `Visitor` model (one row per browser per author), `VisitorRepository`, `visitor_tracker.py` service. `session/init` now accepts `visitor_uid` (localStorage UUID) in request body — HF-Spaces-safe (no cookies). 4 new analytics endpoints: `/visitors`, `/geo`, `/devices`, `/sessions/stats`. `geo.py` updated with region extraction and X-Real-IP header support. | `models/visitor.py`, `repositories/visitor_repo.py`, `services/analytics_core/visitor_tracker.py`, `services/analytics_core/geo.py`, `api/chat.py`, `admin/routers/analytics.py`, `schemas/chatbot.py`, `models/chat_session.py`, `core/startup/db.py`, `models/__init__.py` |