implementation_plan.md

# Author RAG — Implementation Plan
## AI-Powered Book Sales Chatbot (Multi-Tenant SaaS)

**Target directory:** `C:\Users\DELL\Desktop\Author RAG`
**Reference project:** `C:\Users\DELL\Desktop\RAGv1v2\RAG 1.2_pushed` (READ-ONLY — no changes will be made)

---

## Background

The Author RAG project is a brand-new SaaS platform that deploys a persuasive AI chatbot on any author's website. The bot knows the author's full book catalog (from uploaded summaries/documents), engages visitors in brief conversations, and strategically converts curiosity into purchases — using Cialdini's six principles of persuasion and a 12-step RAG pipeline adapted from the proven HRM chatbot architecture.

Unlike the HRM chatbot (which is informational), this bot's **primary KPI is conversion** — turning a casual website visitor into a buyer in 1-3 messages.

---

## Persuasion & Book-Selling Strategy (Research-Backed)

The following strategies are baked into the system prompt, upsell engine, and intent pipeline:

### The 6 Cialdini Principles (applied per turn)
| Principle | How It's Applied |
|---|---|
| **Reciprocity** | Bot offers a free teaser quote or hook line before any CTA |
| **Scarcity** | Surfaces discount expiry or signed-copy limits from author config |
| **Authority** | Injects author accolades, bestseller tags, awards in first response |
| **Social Proof** | Cites reader count, reviews, or star ratings from author config |
| **Liking** | Speaks in the author's own voice/tone to feel personal |
| **Commitment** | Starts with a tiny ask ("Want to hear the first twist?") before the buy CTA |

### Conversion Micro-Tactics
1. **Open Loop / Cliffhanger**: Never reveal the ending; always leave an open question.
2. **Pain → Solution Frame**: "If you've ever felt X, this book is written for you."
3. **Future Pacing**: "Imagine finishing this book in a weekend and finally understanding Y."
4. **Binary Choice Close**: "Would you prefer the ebook now or paperback delivered?"
5. **Urgency Injection**: Time-limited offers only fire if author configured a discount.
6. **Story Interruption**: Share 2 sentences of a gripping scene, then say "that's just Chapter 1."
7. **Reader Identity Anchor**: "Readers like you who love Z typically rate this 5 stars."
8. **The One-Line Pitch**: Every book gets a 1-sentence punchy pitch stored in the author's config.

### What the Bot Will NEVER Do
- Reveal the full plot / ending of any book
- Fabricate reviews, awards, or claims not in the source document
- Pressure users with aggressive language
- Discuss competitors' books or redirect to external sites without author permission

---

## Roles & Access Control

### Role Hierarchy

```
SuperAdmin
    └── Author (one per website)
            └── Reader (end-user, anonymous)
```

### Role: SuperAdmin
- Full platform control
- Can create / suspend / delete Author accounts
- Can set global token budgets (per Author)
- Can view cross-author analytics
- Can revoke API tokens instantly (Redis blacklist)
- Requires **TOTP 2FA** on every sensitive action
- All actions write to an **immutable audit log** (append-only table, no DELETE)
- Can impersonate any Author panel for support
- Can configure platform-level rate limits

### Role: Author
- Isolated tenant — sees ONLY their own data
- Can upload books (PDF, EPUB, DOCX, TXT) — async ingestion
- Can view their own chat sessions / reader analytics
- Can configure their bot persona (name, avatar, welcome message, tone)
- Can configure smart links (buy URL, preview URL, discount code)
- Can set purchase urgency messages
- Can set book-specific pitches, awards, review snippets
- Can manage their own admin password (enforced strength policy)
- Can export chat logs (Excel / JSON)
- Can block/unblock individual reader sessions
- Cannot access other authors' data
- Subject to token budget enforced by SuperAdmin

### Role: Reader (End User)
- Anonymous — identified by SHA-256 fingerprint (IP + UA), no PII stored
- Can chat with the bot (subject to rate limiting)
- Can rate the conversation (1-5 stars)
- Can start new chat threads

---

## Complete Feature List

### 🔐 Security & Authentication
1. SuperAdmin TOTP 2FA (pyotp) — required for all sensitive actions
2. Author login: username + password + OTP via email (SMTP)
3. JWT session tokens for Author panel (short-lived + refresh)
4. HMAC-signed widget tokens (validates domain origin)
5. Redis revocation blacklist for instant token invalidation
6. 5-layer API token validation: HMAC → Expiry → Redis blacklist → DB record → Token budget
7. Timing-safe credential comparison (`secrets.compare_digest`)
8. Brute-force lockout after 5 failed login attempts (per IP, 15-minute lockout)
9. Immutable SuperAdmin audit log (append-only SQLite table)
10. Fail-safe rate limiter: Redis sliding window, falls back to in-memory if Redis crashes
11. CORS domain whitelist — widget API only accepts requests from the Author's registered domain
12. Input guardrails: blocks 10+ jailbreak / prompt-injection patterns
13. Output boundary check: ensures system prompt was never leaked

### 📚 Document Ingestion
14. Upload: PDF, EPUB, DOCX, TXT — validated by magic bytes (not just extension)
15. Async processing via background threads (Celery-ready, threading fallback)
16. Real-time SSE progress bar (chunking → embedding → indexing)
17. Per-book ChromaDB collection (isolated per Author tenant)
18. Configurable chunk size & overlap per Author
19. Auto-generated book summary on upload (LLM-powered, stored as metadata)
20. Support for multiple books per Author (multi-book catalog)
21. Book metadata: title, genre, author name, publication year, price, buy URL, preview URL, discount code, one-line pitch, awards, review snippets
22. Delete / re-ingest a book (full re-embedding)

### 🤖 12-Step RAG Pipeline (adapted for book sales)
23. **Step 1 — Context Injection**: Author tone, persona, style preferences, active books
24. **Step 2 — Intent Classification**: Detects query type (book Q&A, price inquiry, character question, plot question, buy intent, comparison, series/sequel question, general greeting)
25. **Step 3 — Query Rewriting**: Optimizes raw query for vector search
26. **Step 4 — Input Guardrails**: Blocks jailbreak / off-topic patterns
27. **Step 5 — Retrieval**: `all-MiniLM-L6-v2` + ChromaDB, per-tenant collection
28. **Step 6 — Reranking**: CrossEncoder rescoring (`cross-encoder/ms-marco-MiniLM-L-6-v2`)
29. **Step 7 — Prompt Assembly**: System prompt + history + ranked chunks + upsell config
30. **Step 8 — LLM Generation**: Streaming via OpenAI / Groq / Fireworks / HuggingFace
31. **Step 9 — Faithfulness Check (NLI)**: `deberta-v3-small` verifies answer is grounded in book text
32. **Step 10 — Output Boundary Check**: Detects system prompt leakage or competitor mentions
33. **Step 11 — Upsell Engine**: Analyzes conversation turn number + detected buy intent → injects relevant CTA, smart link, or discount code
34. **Step 12 — Formatter**: Cleans markdown, enforces short/concise reply style

### 🎯 Upsell Engine Logic
35. Turn 1: Focus on curiosity / hook (no hard sell)
36. Turn 2: Introduce value + social proof (soft CTA)
37. Turn 3+: Active CTA with buy link + urgency message (if configured)
38. Buy-intent signal detected at any turn → immediate smart link injection
39. Price question → instant buy link + value justification
40. Series/sequel question → cross-sell to next book
41. Configurable: Author can disable/enable upsell per book
42. A/B testing slot for CTA variants (stored, switchable from admin)

### 📊 Analytics & Tracking
43. Anonymous fingerprinting: SHA-256(IP + UA) — no PII
44. Per-session tracking: intent detected, tokens used, latency, faithfulness score
45. Per-book analytics: views, clicks, buy-link clicks (UTM tracking)
46. Geo-location: MaxMind GeoLite2 (country-level only)
47. Hourly aggregation Celery Beat job → daily dashboard charts
48. Conversion funnel: impression → engagement → buy-link click
49. Export analytics: CSV / Excel
50. SuperAdmin global analytics dashboard (cross-author)

### 🎨 Chat Widget (Author-Customizable)
51. Bot name, avatar image upload
52. Welcome message (custom per book landing page)
53. Widget position: bottom-right / bottom-left / center
54. Theme: light / dark / custom hex colors
55. Fallback behavior when bot can't answer
56. Widget embed via `<script>` tag with Author's signed token
57. Mobile-responsive widget
58. Typing indicator animation
59. Message delivery ticks
60. Star rating at end of conversation

### 👥 Multi-Tenant Author Admin Panel
61. Dashboard: total readers, total conversations, conversion rate, avg session length
62. Book manager: upload, edit metadata, delete, re-ingest
63. Session viewer: read all reader transcripts
64. Block / unblock readers
65. Reply to readers manually (live agent takeover)
66. End chat / archive session
67. Widget config (name, avatar, colors, position, welcome message)
68. Smart Links manager (buy URL, preview URL, discount code, urgency message)
69. Analytics page (charts: daily readers, top books, top intents, geo map)
70. Export chats (Excel / JSON)
71. Password change (self-service)
72. API settings (LLM provider choice per Author)
73. Subscription info (token budget, expiry — read-only, managed by SuperAdmin)

### 🛡️ SuperAdmin Panel
74. Manage Author accounts: create, suspend, delete
75. Set token budgets per Author (monthly / total)
76. Extend / revoke Author subscriptions
77. Impersonate Author panel (read-only support mode)
78. View cross-author analytics
79. Platform-wide rate limit settings
80. Audit log viewer (immutable — no delete)
81. Platform health dashboard (Redis status, ChromaDB status, LLM status)
82. Broadcast announcement to all Authors (shown in their admin panel)

### 🛠️ Deployment & Infrastructure
83. Hugging Face Spaces ready (custom Dockerfile)
84. Pre-downloaded models baked into Docker image (no cold-start)
85. ChromaDB for vector storage (per-tenant collections)
86. Redis for rate limiting, token blacklist, pub/sub (SSE progress)
87. SQLite for Author accounts, audit log, subscription data
88. Full async: FastAPI + asyncpg (or aiosqlite) + Redis.asyncio
89. Celery (with Redis broker) for async ingestion + scheduled aggregation
90. Environment-based config (`.env`) — no secrets in code

---

## Architecture Overview

```
┌────────────────────────────────────────────────────────────────────────────┐
│                          Author RAG Platform                               │
├──────────────────────────┬─────────────────────────────────────────────────┤
│   SuperAdmin Panel       │  Author Admin Panel (per tenant)                │
│   /superadmin            │  /admin/{author_slug}                           │
│   TOTP 2FA required      │  JWT + OTP auth                                 │
├──────────────────────────┴─────────────────────────────────────────────────┤
│                        FastAPI Backend                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────────┐ │
│  │ Chat API     │  │ Ingestion API│  │ Admin API    │  │ SuperAdmin API │ │
│  │ /api/chat    │  │ /api/ingest  │  │ /api/admin/* │  │ /api/super/*   │ │
│  └──────┬───────┘  └──────┬───────┘  └──────────────┘  └────────────────┘ │
│         │                 │                                                  │
│  ┌──────▼───────────────────────────────────────────────────────────────┐  │
│  │                    12-Step RAG Pipeline                               │  │
│  │  Context→Intent→Rewrite→Guard→Retrieve→Rerank→Assemble→LLM→         │  │
│  │  Faithfulness→BoundaryCheck→Upsell→Format                            │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
├──────────────────────────────────────────────────────────────────────────── ┤
│  Storage                                                                    │
│  ChromaDB (per-tenant)  │  SQLite (accounts, audit)  │  Redis (cache/rate) │
└────────────────────────────────────────────────────────────────────────────┘
```

---

## Proposed File Structure

```
C:\Users\DELL\Desktop\Author RAG\
│
├── app/
│   ├── __init__.py
│   ├── main.py                        # FastAPI app factory, startup
│   │
│   ├── core/
│   │   ├── __init__.py
│   │   ├── config.py                  # Pydantic settings (env-based)
│   │   ├── database.py                # SQLite/aiosqlite setup + migrations
│   │   └── security.py                # TOTP, JWT, HMAC, password hashing utils
│   │
│   ├── models/
│   │   ├── __init__.py
│   │   ├── author.py                  # Author account model
│   │   ├── book.py                    # Book metadata model
│   │   ├── session.py                 # Reader session model
│   │   └── audit.py                   # Immutable audit log model
│   │
│   ├── services/
│   │   ├── __init__.py
│   │   ├── embeddings.py              # SentenceTransformer wrapper
│   │   ├── vector_store.py            # ChromaDB per-tenant client
│   │   ├── reranker.py                # CrossEncoder reranker
│   │   ├── llm.py                     # Multi-provider LLM service
│   │   ├── rag_pipeline.py            # 12-step pipeline
│   │   ├── upsell_engine.py           # Upsell logic & CTA injection
│   │   ├── faithfulness.py            # DeBERTa NLI faithfulness check
│   │   ├── guardrails.py              # Input/output guard patterns
│   │   ├── ingestion.py               # Async file parsing + chunking
│   │   ├── session_store.py           # Reader session CRUD (JSON files)
│   │   ├── rate_limiter.py            # Redis sliding window + fallback
│   │   ├── analytics.py               # Event tracking + aggregation
│   │   └── notifications.py           # Email (SMTP) notifications
│   │
│   ├── api/
│   │   ├── __init__.py
│   │   ├── schemas.py                 # Pydantic request/response schemas
│   │   ├── chat.py                    # /api/chat endpoints
│   │   ├── ingest.py                  # /api/ingest endpoints (SSE progress)
│   │   └── widget.py                  # /api/widget/token (signed embed tokens)
│   │
│   ├── superadmin/
│   │   ├── __init__.py
│   │   ├── router.py                  # /api/super/* endpoints
│   │   └── templates/
│   │       └── superadmin.html        # SuperAdmin SPA
│   │
│   └── admin/
│       ├── __init__.py
│       ├── router.py                  # /api/admin/* endpoints (per-author)
│       └── templates/
│           └── admin.html             # Author Admin SPA
│
├── static/
│   ├── widget.js                      # Embeddable chat widget script
│   ├── widget.css                     # Widget styles
│   └── addon.html                     # Standalone widget test page
│
├── data/
│   ├── index/                         # ChromaDB persistence
│   ├── sessions/                      # Reader session JSON files
│   └── author_rag.db                  # SQLite: authors, books, audit log
│
├── docs/                              # Default book document storage
│
├── Dockerfile                         # HF Spaces compatible
├── docker-compose.yml                 # Local dev with Redis
├── requirements.txt
├── .env.example
├── app.py                             # Entry point (uvicorn)
└── README.md
```

---

## Proposed Changes (Component by Component)

---

### Core Infrastructure

#### [NEW] `app/core/config.py`
Extended settings for multi-tenant Author RAG:
- `SUPERADMIN_USER`, `SUPERADMIN_PASS`, `SUPERADMIN_TOTP_SECRET`
- `JWT_SECRET_KEY`, `JWT_ALGORITHM`, `JWT_EXPIRE_MINUTES`
- `REDIS_URL` (for rate limiting, blacklist, pub/sub)
- `CHROMA_PERSIST_DIR` (ChromaDB storage path)
- `DB_PATH` (SQLite path)
- `MAXMIND_DB_PATH` (GeoLite2)
- All LLM provider keys (same as reference project)
- `FAITHFULNESS_MODEL` = `cross-encoder/nli-deberta-v3-small`
- `EMBEDDING_MODEL` = `all-MiniLM-L6-v2`

#### [NEW] `app/core/database.py`
SQLite schema via aiosqlite:
- `authors` table: id, slug, name, email, password_hash, totp_secret, token_budget, tokens_used, subscription_expiry, suspended, created_at
- `books` table: id, author_id, title, genre, pub_year, price, buy_url, preview_url, discount_code, pitch, awards, reviews_json, ingested_at, status
- `audit_log` table: id, actor, action, target, detail, timestamp — **NO DELETE** allowed via app layer
- `announcements` table: id, message, created_at, created_by

#### [NEW] `app/core/security.py`
- `hash_password()` / `verify_password()` via bcrypt
- `create_jwt()` / `verify_jwt()`
- `generate_totp_secret()` / `verify_totp(token, secret)`
- `sign_widget_token(author_slug, domain)` / `verify_widget_token()`
- `hmac_sign()` / `hmac_verify()`

---

### Services Layer

#### [NEW] `app/services/rag_pipeline.py`
The 12-step pipeline adapted for book sales:
- Replaces HRM-specific intent map with a **book-sales intent map**: `BOOK_QA`, `PRICE_INQUIRY`, `BUY_INTENT`, `CHARACTER_QUESTION`, `PLOT_QUESTION`, `SERIES_QUESTION`, `COMPARISON`, `GENERAL_GREETING`
- Author tone + persona injected at Step 1
- Faithfulness check at Step 9 (DeBERTa NLI)
- Upsell Engine called at Step 11

#### [NEW] `app/services/upsell_engine.py`
```
Rules:
  - Turn 1:  Hook → free teaser quote → NO buy link
  - Turn 2:  Value frame + social proof → soft CTA ("learn more")
  - Turn 3+: Hard CTA with buy_url + urgency_message (if set)
  - Any turn: BUY_INTENT detected → immediate smart link
  - Any turn: PRICE_INQUIRY → buy_url + value justification
  - SERIES_QUESTION: cross-sell next book in series
```

#### [NEW] `app/services/faithfulness.py`
- Uses `cross-encoder/nli-deberta-v3-small`
- Returns entailment probability score
- If score < threshold → strips the hallucinated claim from reply

#### [NEW] `app/services/guardrails.py`
- Input patterns: same 10+ jailbreak patterns from reference + book-specific off-topic blocks (e.g., "write this book for free", "give me a pirated copy")
- Output patterns: detects system prompt leakage, competitor book mentions

#### [NEW] `app/services/vector_store.py`
- ChromaDB client (replaces FAISS from reference)
- Per-author, per-book collection naming: `author_{slug}_book_{id}`
- Supports multi-book search (queries all author's collections)
- Hybrid search: dense (embedding) + sparse (BM25) + RRF fusion

#### [NEW] `app/services/ingestion.py`
- Magic byte validation (PDF=`%PDF`, EPUB=PK header, DOCX=PK header)
- Parse: PyPDF2 (PDF), ebooklib (EPUB), python-docx (DOCX), plain text
- Chunking: sentence-aware, 350 tokens, 80 overlap
- Redis Pub/Sub SSE events: `{stage: "chunking", progress: 45}`
- Auto-summary generation via LLM after ingestion

#### [NEW] `app/services/analytics.py`
- `track_event(author_id, book_id, event_type, session_id, metadata)`
- Event types: `chat_start`, `message`, `buy_link_click`, `rating`
- SHA-256 fingerprinting: `hash(ip + user_agent)`
- MaxMind GeoLite2 lookup (country only)
- `aggregate_daily()` — Celery Beat task, hourly

---

### API Layer

#### [NEW] `app/api/chat.py`
- `POST /api/chat/{author_slug}` — main chat endpoint (widget calls this)
- `POST /api/chat/{author_slug}/session/init`
- `GET  /api/chat/{author_slug}/session/history/{session_id}`
- `POST /api/chat/{author_slug}/session/rate`
- `POST /api/chat/{author_slug}/session/reset`
- `GET  /api/chat/{author_slug}/events` — SSE for ingestion progress

#### [NEW] `app/api/ingest.py`
- `POST /api/admin/{author_slug}/books/upload`
- `GET  /api/admin/{author_slug}/books/upload/progress` — SSE stream

#### [NEW] `app/api/widget.py`
- `GET  /api/widget/token?slug={author_slug}&domain={domain}` — returns signed widget token
- `GET  /widget/{author_slug}` — serves the embeddable widget HTML

---

### Admin Panel (Per Author)

#### [NEW] `app/admin/router.py`
All routes require valid JWT for the specific Author:
- Dashboard, session list/detail, block/unblock, takeover, reply
- Book CRUD, widget config, smart links, analytics, exports
- Password change endpoint

#### [NEW] `app/admin/templates/admin.html`
Single-page application with:
- Sidebar nav: Dashboard, Books, Readers, Widget Config, Analytics, Settings
- Book upload drag-and-drop with live progress bar
- Reader session list with search + filter
- Real-time chart (Chart.js): daily readers, top books, conversion funnel
- Widget configurator: live preview of chat widget

---

### SuperAdmin Panel

#### [NEW] `app/superadmin/router.py`
All routes require SuperAdmin JWT + TOTP verification header:
- Author CRUD (create, suspend, delete)
- Token budget management
- Impersonation (read-only)
- Audit log viewer
- Platform health dashboard
- Broadcast announcements

#### [NEW] `app/superadmin/templates/superadmin.html`
- Minimal, high-security design
- All destructive actions require re-entry of TOTP code
- Audit log displayed as immutable feed

---

### Chat Widget

#### [NEW] `static/widget.js`
- Embed via `<script src="https://your-space.hf.space/static/widget.js" data-author="slug" data-token="signed_token"></script>`
- Auto-initializes on page load
- Reads `data-author` and `data-token` attributes
- Validates signed token on first message (backend verifies HMAC + domain)
- Persistent session via localStorage
- SSE polling for admin live replies
- Typing indicator, delivery ticks, star rating

---

### Entry Point & Deployment

#### [NEW] `app/main.py`
- FastAPI app factory
- Mounts all routers
- Startup: loads ChromaDB, downloads models, creates DB tables
- Lifespan events (replaces deprecated `@app.on_event`)

#### [NEW] `Dockerfile`
```dockerfile
FROM python:3.11-slim
# Pre-download models during build
RUN python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"
RUN python -c "from sentence_transformers.cross_encoder import CrossEncoder; CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')"
RUN python -c "from sentence_transformers.cross_encoder import CrossEncoder; CrossEncoder('cross-encoder/nli-deberta-v3-small')"
```

#### [NEW] `requirements.txt`
```
fastapi
uvicorn[standard]
chromadb
sentence-transformers
pydantic-settings
pydantic[email]
pypdf2
ebooklib
python-docx
python-dotenv
numpy
httpx
python-multipart
pandas
openpyxl
rank_bm25
bcrypt
python-jose[cryptography]
pyotp
qrcode[pil]
aiosqlite
redis[asyncio]
celery[redis]
geoip2
maxminddb
```

---

## Verification Plan

### Automated Checks
- `uvicorn app.main:app --reload` — server starts without errors
- All import paths resolve (no circular imports)
- `.env.example` covers all required settings

### Manual Verification (Post-Build)
1. SuperAdmin login with TOTP → create Author account
2. Author login → upload a book (PDF) → see SSE progress bar
3. Open widget → chat as reader → verify upsell fires on turn 3
4. Admin panel → block reader → verify reader gets 403
5. Admin panel → live takeover → verify admin reply appears in widget
6. SuperAdmin → audit log shows all actions
7. Rate limiter → send >60 requests/min → verify 429

---

## Open Questions

> [!IMPORTANT]
> **Q1 — Widget Hosting**: Should the chat widget be hosted on the same HF Space URL, or should authors embed a separate JS snippet that calls back to the space? (Recommended: single space, embed via `<script>` tag with `data-author` attribute.)

> [!IMPORTANT]
> **Q2 — Multi-Book Search Strategy**: When a reader asks a general question not tied to a specific book, should the bot search ALL of the author's books and pick the best match, or ask the reader which book they mean first? (Recommended: search all, pick best — more seamless.)

> [!IMPORTANT]
> **Q3 — Redis Requirement**: Redis is needed for rate limiting, token blacklist, and SSE. Is Redis available on your deployment environment? If not, we can implement fallback in-memory alternatives for all three (same as reference project's in-memory rate limiter pattern).

> [!WARNING]
> **Q4 — Faithfulness Model Size**: `cross-encoder/nli-deberta-v3-small` adds ~180MB to the Docker image and ~200ms per response. If this is too slow, we can make it optional (admin toggle per Author). Shall we include it by default?

> [!NOTE]
> **Q5 — Gradio UI**: The reference project uses Gradio for a testing UI. Should we keep a Gradio test UI (`/ui` route) for this project as well, or skip it entirely since we have a custom widget?