tests/TEST_PLAN.md

Author RAG — Master Test Plan

Version: 1.0
Date: 2026-06-19
Status: Active — pre-implementation baseline
Owner: Engineering
Scope: Full platform — SuperAdmin, Author Admin, Widget/Chat, RAG pipeline, analytics, notifications

---

1. Purpose & Goals

This document defines the complete verification strategy for Author RAG before any feature is considered production-ready. The platform's success depends on 100% consistency across:

• Token accounting (single source of truth, enforced everywhere)
• Conversion UX (Buy Book CTA visible when configured and appropriate)
• Human tone (no robotic fallbacks; personality settings honored)
• Role isolation (SuperAdmin, Author, Visitor)
• Analytics integrity (every turn tracked; failures never break chat)

1.1 Known Critical Gaps (must pass after fix)

ID	Gap	Impact	Primary Files
G-01	client_access.tokens_used never incremented on chat	Admin & SuperAdmin show 0% usage	tracker.py, admin/router.py, superadmin_service.py
G-02	Redis tokens:{author_id}:current incremented but never read	Parallel orphan counter	tracker.py, subscription.py
G-03	budget_exhausted:{author_id} never set	Budget enforcement non-functional	subscription.py, tracker.py
G-04	Admin token-usage ignores bonus_tokens; embed-token includes them	Inconsistent UI numbers	admin/router.py
G-05	Smart Links save to books.buy_url; chat reads links.purchase_url	Buy Book button never appears	admin/router.py, rag_pipeline.py, link_repo.py
G-06	response_style saved but not injected into prompts	Personality page is cosmetic	prompter.py, rag_pipeline.py
G-07	Default fallback contains robotic phrase filtered by hack	Inconsistent fallback behavior	rag_pipeline.py, user.py
G-08	chatbot_is_active not checked on chat	Suspended authors still chat	subscription.py, chat.py
G-09	Token warning emails use placeholder tokens_used=0	Alerts never fire	tasks/
G-10	No automated tests; CI runs pytest || true	Regressions undetected	.github/workflows/ci.yml

---

2. Test Strategy

2.1 Pyramid

                    ┌─────────────┐
                    │  E2E (5%)   │  Playwright — widget + admin SPAs
                   ┌┴─────────────┴┐
                   │ Integration   │  FastAPI TestClient + real Redis/DB (15%)
                  ┌┴───────────────┴┐
                  │  Unit (80%)     │  Pipeline, upsell, formatter, token math
                  └─────────────────┘

2.2 Environments

Environment	DB	Redis	OpenAI	Purpose
Unit	SQLite in-memory / mocks	fakeredis	Mocked	Fast CI, logic only
Integration	PostgreSQL test DB	Redis container	Mocked or recorded fixtures	API contracts, token flow
Staging	Production-like PG	Production-like Redis	Real (limited budget)	Pre-release smoke
Production	Live	Live	Live	Synthetic monitors only

2.3 Tooling

Layer	Tool	Location
Unit / Integration	pytest, pytest-asyncio, httpx, factory-boy	tests/unit/, tests/integration/
BDD (acceptance)	behave	tests/bdd/features/ (from .agent/BDD_SPECS.md)
E2E	Playwright (recommended)	tests/e2e/
Load	locust (optional)	tests/load/
Contract	OpenAPI schema validation	tests/contract/

2.4 CI Gate (target state)

# Fail CI on any test failure — remove `|| true`
pytest tests/ -v --cov=app --cov-fail-under=80
behave tests/bdd/features/

2.5 Test Data Fixtures

Fixture	Description
superadmin_user	Seeded from SUPERADMIN_EMAIL / SUPERADMIN_PASSWORD
author_with_grant	Author + active client_access + 500K token budget
author_with_books	1–3 books in ready status with embeddings
subscription_token	Valid HMAC token for widget
book_with_links	links row with purchase_url + preview_url
book_with_buy_url_only	books.buy_url set, no links row (regression case)

---

3. Architecture Verification Map

Every test suite maps to a subsystem. No subsystem ships without coverage.

┌──────────────────────────────────────────────────────────────────┐
│                        Visitor (Widget)                          │
│  widget.js → /api/chat/{slug} → subscription validation          │
└────────────────────────────┬─────────────────────────────────────┘
                             │
┌────────────────────────────▼─────────────────────────────────────┐
│                     RAG Pipeline (12 steps)                      │
│  guardrails → intent → session → rewrite → retrieve → rerank     │
│  → context → LLM → faithfulness → boundary → upsell → format     │
└────────────────────────────┬─────────────────────────────────────┘
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
   analytics_core      client_access         links/books
   (tracker, agg)      (token budget)        (purchase URLs)
         │                   │                   │
         ▼                   ▼                   ▼
   Author Admin UI     SuperAdmin UI        Celery tasks

---

4. Test Suites

Suite A — Token Usage Consistency (P0 — BLOCKING)

Objective: One canonical token counter drives enforcement, admin UI, SuperAdmin UI, emails, and exports.

A.1 Single Source of Truth

Test ID	Scenario	Steps	Expected
A-01	Chat increments tokens_used	Send 1 chat message with known token usage (mock OpenAI returns prompt_tokens=100, completion_tokens=50)	client_access.tokens_used += 150
A-02	Chat increments Redis counter	Same as A-01	tokens:{author_id}:current += 150
A-03	Counters stay in sync	Send 10 messages	tokens_used == Redis counter == sum of chat_messages tokens for period
A-04	Regeneration counts tokens	Hallucination triggers regen (2 LLM calls)	Both calls' tokens counted
A-05	Custom Q&A path counts tokens	Message hits custom Q&A override	Tokens still recorded if LLM called
A-06	Failed LLM does not increment	OpenAI timeout / 500	No token increment; error logged
A-07	Analytics failure does not block chat	Mock DB write failure in track_turn	Visitor gets response; error logged at ERROR

A.2 Budget Math (must be identical everywhere)

Test ID	Formula Under Test	Surfaces
A-10	total_budget = token_budget + bonus_tokens	SuperAdmin grants, embed-token
A-11	remaining = total_budget - tokens_used	Admin token page, embed-token, SuperAdmin
A-12	percent_used = tokens_used / total_budget * 100	Admin gauge, SuperAdmin progress bar
A-13	Bonus tokens increase remaining without changing expiry	SuperAdmin bonus modal

Consistency matrix (all must match after each chat turn):

Surface	Endpoint / Store	Includes Bonus?	Reads tokens_used?
Author Token Usage page	GET /api/admin/{slug}/token-usage	MUST	MUST
Author Embed Token card	GET /api/admin/{slug}/embed-token	MUST	MUST
SuperAdmin Token Management	GET /api/super/grants	MUST	MUST
SuperAdmin Platform Analytics	GET /api/super/health or analytics	MUST	MUST
Subscription validation	validate_subscription_token	MUST	MUST
Analytics export CSV	GET /api/admin/{slug}/export/analytics	N/A (historical)	Aggregated from events
Weekly digest email	Celery task	MUST	MUST

A.3 Budget Enforcement

Test ID	Scenario	Expected
A-20	Usage at 79%	Chat works normally
A-21	Usage at 80%	Warning email sent (if notify_token_alerts=true) within 60s
A-22	Usage at 99%	Chat still works; UI shows red/warning state
A-23	Usage at 100%	budget_exhausted:{author_id} set; next message returns graceful fallback (HTTP 200 per BDD), OpenAI NOT called
A-24	SuperAdmin adds bonus at 100%	budget_exhausted cleared; chat resumes
A-25	SuperAdmin reset tokens	tokens_used=0; policy on bonus_tokens documented and tested
A-26	Extend subscription at 100% usage	Expiry extended; usage NOT reset unless explicitly reset

A.4 SuperAdmin Token Operations

Test ID	Action	Verify
A-30	Grant new subscription	token_budget set per plan; tokens_used=0
A-31	Add bonus tokens	bonus_tokens incremented; audit log entry
A-32	Reset token usage	tokens_used=0; Redis counters cleared; audit log
A-33	Revoke subscription	Chat blocked (403); Redis blacklist set
A-34	Token Management UI totals	Sum of all grants' tokens_used matches platform aggregate

A.5 Author Admin Token Page

Test ID	Scenario	Expected
A-40	Page load after 5 chats	Gauge shows non-zero used matching DB
A-41	No active grant	Shows zeros with clear empty state (not error)
A-42	Gauge color thresholds	<75% indigo, 75–90% amber, >90% red
A-43	Matches embed-token card	Same remaining value on both pages

---

Suite B — Buy Book Button & Smart Links (P0 — BLOCKING)

Objective: When an author configures a purchase URL, the widget shows a "Buy Book" button at the correct times.

B.1 Data Model Unification

Test ID	Scenario	Expected
B-01	Author saves Smart Link (PUT /smart-links/{bookId})	links.purchase_url AND books.buy_url both set (or single source with fallback read)
B-02	Author sets buy URL on book upload	URL persisted and available to chat
B-03	_get_book_links() with only books.buy_url	Returns purchase URL (fallback)
B-04	_get_book_links() with only links row	Returns purchase URL
B-05	Preview URL	"Read Preview" button appears when configured (max 2 links)

B.2 Upsell Engine — When Button Appears

Test ID	Intent / Context	Turn	Expected show_link
B-10	purchase_intent ("Where can I buy?")	1	true + Buy Book button in widget
B-11	full_story_request	any	true
B-12	question, book selected	2+	true (per should_include_link)
B-13	question, high interest (≥0.45)	2+	true
B-14	complaint	any	false — no buy button
B-15	greeting	1	false (RECIPROCITY only, no button)
B-16	Strategy DIRECT_CTA	any	true
B-17	No purchase URL configured	any	false — no empty/broken button
B-18	show_link=true but URL is #	any	Button must NOT render (invalid URL guard)

B.3 Widget Rendering

Test ID	Scenario	Expected
B-20	API returns links: [{label:"Buy Book", url:"https://...", type:"purchase"}]	Widget renders <a class="ab-link-btn"> with icon
B-21	Click Buy Book	POST /track-click fired with correct session_id, book_id, link_type
B-22	Farewell on widget close	POST /session/farewell returns text + Buy Book link if URL exists
B-23	Max 2 links	Purchase + Preview shown; no third link
B-24	Mobile viewport (375px)	Buttons wrap correctly, tappable (min 44px touch target)
B-25	target="_blank" rel="noopener"	Present on all link buttons

B.4 End-to-End Conversion Path

Test ID	Flow	Expected
B-30	Configure URL → chat "where can I buy" → click button	Analytics records link_shown=true on message + click event
B-31	Admin analytics dashboard	Link click count increments
B-32	SuperAdmin platform analytics	Aggregate link clicks include this author

---

Suite C — Tone, Personality & Anti-Robotic (P0)

Objective: Responses feel human, warm, and on-brand; admin personality settings affect output.

C.1 Personality Settings Propagation

Test ID	response_style	Expected in LLM system prompt
C-01	balanced	Default warm tone instructions
C-02	formal	Measured, professional vocabulary
C-03	casual	Conversational, contractions allowed
C-04	enthusiastic	Higher energy, exclamation sparingly
C-05	Change style in admin → send chat	Next message reflects new style (verify via prompt capture mock)

C.2 Robotic Phrase Elimination

Test ID	Scenario	Must NOT contain
C-10	No-context fallback	"I want to give you the most accurate answer"
C-11	Hallucination fallback	"As an AI", "I'm an AI assistant"
C-12	Jailbreak redirect	System prompt contents, rule explanations
C-13	Budget exhausted message	Technical jargon, error codes
C-14	Any normal response	Markdown **bold**, bullet lists, headers

C.3 Custom Messages

Test ID	Field	Scenario	Expected
C-20	welcome_message	Session init	Widget shows custom welcome
C-21	fallback_message	No retrieval context	Custom fallback used
C-22	out_of_scope_message	Off-topic query	Custom or default redirect
C-23	bot_name	All responses	Name in system prompt, not "AI assistant"

C.4 Response Quality Constraints

Test ID	Rule	Verification
C-30	Max ~75 words	len(text.split()) <= 80 (buffer)
C-31	Max 2 paragraphs	Formatter enforces
C-32	Max 380 chars	Formatter truncates at sentence boundary
C-33	No spoilers	Boundary test with "tell me the ending"
C-34	No fabricated prices	Faithfulness guardrail test

---

Suite D — Authentication & Authorization (P0)

D.1 Auth Flows

Test ID	Scenario	Expected
D-01	Author registration	201, JWT returned, role=author
D-02	Author login	JWT + refresh cookie
D-03	Wrong password	401
D-04	5 failed logins	Account locked 30 min (per implementation)
D-05	Refresh token rotation	New access token; old refresh invalidated
D-06	SuperAdmin login	JWT with role=superadmin
D-07	Author JWT on /api/super/*	403
D-08	Expired JWT	401

D.2 Tenant Isolation

Test ID	Scenario	Expected
D-10	Author A requests Author B's books	403 or empty (slug must match current_user.id)
D-11	Author A uses B's slug in URL	Must NOT return B's data
D-12	SuperAdmin lists all authors	Returns cross-tenant data (allowed)

D.3 Subscription Token Security

Test ID	Scenario	Expected
D-20	Valid token	Chat 200
D-21	Tampered HMAC	403
D-22	Expired token	403
D-23	Revoked token (Redis)	403 immediately
D-24	Revoked in DB	403
D-25	Missing X-Subscription-Token header	401/403

D.4 Suspension & Active State

Test ID	Scenario	Expected
D-30	SuperAdmin suspends author	chatbot_is_active=false
D-31	Suspended author — visitor chats	403 with friendly message
D-32	SuperAdmin unsuspends	Chat resumes

---

Suite E — RAG Pipeline (P1)

E.1 Pipeline Steps (integration)

Test ID	Step	Verification
E-01	Guardrails	Jailbreak input → redirect, not LLM
E-02	Intent classification	"where can I buy" → purchase_intent
E-03	Book selector	3 books + vague query → selector popup in response
E-04	Single book	Auto-select, no popup
E-05	Book name in query	Fuzzy match >0.85 → auto-select
E-06	Faithfulness fail	Regenerate once; then safe fallback
E-07	Boundary enforcer	Off-topic → redirect
E-08	Custom Q&A override	Exact match returns canned answer

E.2 Book Lifecycle

Test ID	Scenario	Expected
E-10	Upload PDF	Status: uploading → parsing → chunking → embedding → ready
E-11	File >50MB	Rejected before processing
E-12	Unsupported .xlsx	400 with supported formats listed
E-13	Deactivate book mid-session	Next message offers remaining books
E-14	Delete last active book	Chatbot disabled + dashboard warning
E-15	Book status != ready	Graceful "still indexing" message

---

Suite F — Admin Panel (P1)

F.1 Dashboard & Analytics

Test ID	Page	Verify
F-01	Dashboard	Session count, recent activity load
F-02	Analytics (30d)	Charts match analytics_daily rollups
F-03	Sessions / Readers	Geo, fingerprint (no PII)
F-04	Conversations	Transcript matches chat_messages
F-05	Export CSV	Downloads valid CSV with correct columns

F.2 Configuration Pages

Test ID	Page	Verify
F-10	Books CRUD	Create, edit, reorder, activate/deactivate
F-11	Smart Links	Save, load, URL validation
F-12	Widget Config	Theme, position persist to session init
F-13	Personality	All fields save and reload
F-14	Notifications toggles	notify_token_alerts respected by email tasks
F-15	Custom Q&A	CRUD + priority ordering
F-16	Embed Code	Valid script tag with live token

F.3 UI Consistency

Test ID	Check	Expected
F-20	All API errors	Toast/alert with human-readable message
F-21	Loading states	Skeleton/spinner on every async page
F-22	Empty states	Helpful copy, not blank screens
F-23	Form validation	Client + server validation aligned
F-24	Navigation	Active nav item highlights correctly

---

Suite G — SuperAdmin Panel (P1)

Test ID	Feature	Verify
G-01	Authors list	Pagination, search, status badges
G-02	Author detail	Books, grant status, token bar
G-03	Grant access	Token generated, email sent, audit logged
G-04	Revoke / extend / bonus / reset	Each writes audit log + correct DB state
G-05	Audit log	Immutable, chronological, filterable
G-06	Platform health	DB, Redis, Chroma status
G-07	Announcements	Delivered to author notifications
G-08	Token Management totals	Match sum of individual grants (see Suite A)
G-09	Suspend / unsuspend	Immediate effect on chat (Suite D)

---

Suite H — Widget & Session (P1)

Test ID	Scenario	Expected
H-01	Session init	bot_name, welcome_message, widget_theme applied
H-02	Multi-turn history	Context preserved across turns
H-03	Session reset	History cleared, new session ID
H-04	Rating (1–5 stars)	Stored on chat_sessions
H-05	Feedback submission	Stored and visible in admin
H-06	SSE ingestion events	Progress events during book upload
H-07	Rate limiting	61st request/min per IP → 429
H-08	Widget <8KB	widget.js size budget met
H-09	Offline / API error	User-friendly error bubble, retry option

---

Suite I — Email & Celery Tasks (P2)

Test ID	Task	Trigger	Verify
I-01	Token 80% warning	Usage threshold	Email with correct numbers
I-02	Subscription expiry 7d	Daily beat	Warning email
I-03	Weekly digest (Monday 9am)	Beat + timezone	Sessions, top book, tokens, country
I-04	Grant notification	SuperAdmin grant	Author receives access email
I-05	Revocation notification	SuperAdmin revoke	Author notified within 5s
I-06	Link health check	Scheduled	purchase_url_ok updated
I-07	Analytics aggregation	Hourly	analytics_daily matches raw events

---

Suite J — Security & Edge Cases (P1)

Test ID	Scenario	Expected
J-01	SQL injection in chat message	Sanitized; no DB error
J-02	XSS in author welcome_message	Escaped in widget (escHtml)
J-03	SSRF via buy URL	URL validation blocks internal IPs
J-04	Path traversal on upload	Rejected
J-05	CORS	Only configured origins
J-06	Security headers	Present on all responses
J-07	/api/super/diag	Must require auth (if not public by design)
J-08	/api/widget/token	Must not issue tokens without author auth
J-09	Concurrent chat + grant revoke	No race — revoke wins
J-10	Redis down	Graceful degradation per layer (documented)

---

5. Cross-Cutting Consistency Checklist

Run this checklist after every release candidate:

5.1 Numeric Consistency

• Send exactly 1 chat message with known token count
• Verify client_access.tokens_used
• Verify Redis tokens:{author_id}:current
• Verify Author Admin → Token Usage page
• Verify Author Admin → Embed Token card
• Verify SuperAdmin → Token Management row
• Verify SuperAdmin → Platform Analytics total
• Verify chat_messages sum for session
• Verify analytics_events row for turn

5.2 Buy Button Consistency

• Configure buy URL in Smart Links
• Ask "where can I buy this book?"
• Confirm link_shown=true in DB
• Confirm button visible in widget DOM
• Click button → analytics event recorded
• Close widget → farewell shows Buy Book

5.3 Tone Consistency

• Set personality to casual → verify prompt
• Trigger no-context path → no robotic phrases
• Trigger hallucination fallback → warm safe message
• Verify no markdown in any response path

---

6. Edge Case Catalog

Category	Edge Case	Expected Behavior
Tokens	Author with 0 budget grant	Clear error on grant; chat blocked
Tokens	Multiple rapid messages	Atomic increment; no lost counts
Tokens	Month rollover	Usage resets per billing period (define policy)
Tokens	SuperAdmin reset during active chat	Next message uses fresh budget
Links	URL with UTM params	Preserved exactly
Links	International domain (IDN)	Validated and stored
Links	HTTP vs HTTPS	HTTPS preferred; mixed content warning in admin
Books	0 active books	Catalog setup message
Books	50+ books	Selector pagination / scroll
Chat	Empty message	400 validation error
Chat	2000+ char message	Truncated or rejected gracefully
Chat	Unicode / emoji	Handled end-to-end
Chat	RTL language input	Widget displays correctly
Session	Same fingerprint, new session	Separate session IDs
Auth	Token refresh during admin action	Seamless retry
Upload	Duplicate SHA-256	Skip/replace dialog
Upload	Network drop mid-upload	Resume from chunk (if implemented)

---

7. Acceptance Criteria (Definition of Done)

A feature or fix is DONE only when:

1. All P0 tests in the relevant suite pass in CI
2. No new gaps introduced in the consistency matrix (Section 5)
3. BDD scenario added to tests/bdd/features/ BEFORE merge (per .agent/RULES.md)
4. Unit test coverage ≥80% on touched modules
5. Manual smoke on staging: Author flow + Visitor flow + SuperAdmin flow
6. Audit log entry verified for every SuperAdmin mutation
7. Analytics event verified for every chat turn
8. Documentation updated if API contract changes

Release Blockers (cannot ship if any fail)

• Suite A — all P0 tests green
• Suite B — all P0 tests green
• Suite C — all P0 tests green
• Suite D — D-10, D-11, D-30, D-31 green
• CI no longer uses || true

---

8. Implementation Roadmap for Test Infrastructure

Phase 1 — Foundation (Week 1)

1. Create tests/conftest.py with async DB, Redis, JWT helpers
2. Add factories: UserFactory, ClientAccessFactory, BookFactory, LinkFactory
3. Mock OpenAI client fixture returning deterministic token counts
4. Fix CI: remove || true; add PostgreSQL + Redis services

Phase 2 — P0 Unit Tests (Week 1–2)

1. test_token_tracking.py — Suite A
2. test_upsell_engine.py — Suite B (link logic)
3. test_formatter.py — link injection, max links
4. test_prompter_personality.py — Suite C
5. test_subscription_validation.py — Suite D

Phase 3 — Integration Tests (Week 2–3)

1. test_chat_api.py — full message flow with mocked LLM
2. test_admin_token_usage.py — API response consistency
3. test_superadmin_grants.py — grant/bonus/reset flows
4. test_smart_links_sync.py — books ↔ links unification

Phase 4 — BDD & E2E (Week 3–4)

1. Port .agent/BDD_SPECS.md → tests/bdd/features/*.feature
2. Playwright: widget buy button visible + click
3. Playwright: admin token gauge updates after chat

Phase 5 — Hardening (Ongoing)

1. Synthetic production monitor: health + token math spot-check
2. Load test: 60 req/min rate limit boundary
3. Chaos: Redis restart during chat

---

9. Traceability Matrix

User-Reported Issue	Test Suite	Test IDs
Tokens not counted on admin	A	A-01, A-40, A-43
Tokens not counted on superadmin	A	A-01, A-34, G-08
Robotic tone	C	C-10, C-11, C-01–C-05
Buy now button not in chat	B	B-01, B-10, B-20, B-30
Personality settings ignored	C	C-01–C-05
Budget not enforced	A	A-20–A-26
Inconsistent token numbers	A	A-10–A-13, Section 5.1

---

10. Sign-Off

Role	Name	Date	Signature
Engineering Lead
QA
Product

---

This plan should be treated as a living document. Update version and traceability matrix whenever gaps G-01 through G-10 are resolved or new features are added.