tests/TEST_PLAN.md

# Author RAG — Master Test Plan

**Version:** 1.0  
**Date:** 2026-06-19  
**Status:** Active — pre-implementation baseline  
**Owner:** Engineering  
**Scope:** Full platform — SuperAdmin, Author Admin, Widget/Chat, RAG pipeline, analytics, notifications

---

## 1. Purpose & Goals

This document defines the complete verification strategy for Author RAG before any feature is considered production-ready. The platform's success depends on **100% consistency** across:

- Token accounting (single source of truth, enforced everywhere)
- Conversion UX (Buy Book CTA visible when configured and appropriate)
- Human tone (no robotic fallbacks; personality settings honored)
- Role isolation (SuperAdmin, Author, Visitor)
- Analytics integrity (every turn tracked; failures never break chat)

### 1.1 Known Critical Gaps (must pass after fix)

| ID | Gap | Impact | Primary Files |
|----|-----|--------|---------------|
| G-01 | `client_access.tokens_used` never incremented on chat | Admin & SuperAdmin show 0% usage | `tracker.py`, `admin/router.py`, `superadmin_service.py` |
| G-02 | Redis `tokens:{author_id}:current` incremented but never read | Parallel orphan counter | `tracker.py`, `subscription.py` |
| G-03 | `budget_exhausted:{author_id}` never set | Budget enforcement non-functional | `subscription.py`, `tracker.py` |
| G-04 | Admin `token-usage` ignores `bonus_tokens`; embed-token includes them | Inconsistent UI numbers | `admin/router.py` |
| G-05 | Smart Links save to `books.buy_url`; chat reads `links.purchase_url` | Buy Book button never appears | `admin/router.py`, `rag_pipeline.py`, `link_repo.py` |
| G-06 | `response_style` saved but not injected into prompts | Personality page is cosmetic | `prompter.py`, `rag_pipeline.py` |
| G-07 | Default fallback contains robotic phrase filtered by hack | Inconsistent fallback behavior | `rag_pipeline.py`, `user.py` |
| G-08 | `chatbot_is_active` not checked on chat | Suspended authors still chat | `subscription.py`, `chat.py` |
| G-09 | Token warning emails use placeholder `tokens_used=0` | Alerts never fire | `tasks/` |
| G-10 | No automated tests; CI runs `pytest \|\| true` | Regressions undetected | `.github/workflows/ci.yml` |

---

## 2. Test Strategy

### 2.1 Pyramid

```
                    ┌─────────────┐
                    │  E2E (5%)   │  Playwright — widget + admin SPAs
                   ┌┴─────────────┴┐
                   │ Integration   │  FastAPI TestClient + real Redis/DB (15%)
                  ┌┴───────────────┴┐
                  │  Unit (80%)     │  Pipeline, upsell, formatter, token math
                  └─────────────────┘
```

### 2.2 Environments

| Environment | DB | Redis | OpenAI | Purpose |
|-------------|-----|-------|--------|---------|
| **Unit** | SQLite in-memory / mocks | fakeredis | Mocked | Fast CI, logic only |
| **Integration** | PostgreSQL test DB | Redis container | Mocked or recorded fixtures | API contracts, token flow |
| **Staging** | Production-like PG | Production-like Redis | Real (limited budget) | Pre-release smoke |
| **Production** | Live | Live | Live | Synthetic monitors only |

### 2.3 Tooling

| Layer | Tool | Location |
|-------|------|----------|
| Unit / Integration | `pytest`, `pytest-asyncio`, `httpx`, `factory-boy` | `tests/unit/`, `tests/integration/` |
| BDD (acceptance) | `behave` | `tests/bdd/features/` (from `.agent/BDD_SPECS.md`) |
| E2E | Playwright (recommended) | `tests/e2e/` |
| Load | `locust` (optional) | `tests/load/` |
| Contract | OpenAPI schema validation | `tests/contract/` |

### 2.4 CI Gate (target state)

```yaml
# Fail CI on any test failure — remove `|| true`
pytest tests/ -v --cov=app --cov-fail-under=80
behave tests/bdd/features/
```

### 2.5 Test Data Fixtures

| Fixture | Description |
|---------|-------------|
| `superadmin_user` | Seeded from `SUPERADMIN_EMAIL` / `SUPERADMIN_PASSWORD` |
| `author_with_grant` | Author + active `client_access` + 500K token budget |
| `author_with_books` | 1–3 books in `ready` status with embeddings |
| `subscription_token` | Valid HMAC token for widget |
| `book_with_links` | `links` row with `purchase_url` + `preview_url` |
| `book_with_buy_url_only` | `books.buy_url` set, no `links` row (regression case) |

---

## 3. Architecture Verification Map

Every test suite maps to a subsystem. No subsystem ships without coverage.

```
┌──────────────────────────────────────────────────────────────────┐
│                        Visitor (Widget)                          │
│  widget.js → /api/chat/{slug} → subscription validation          │
└────────────────────────────┬─────────────────────────────────────┘
                             │
┌────────────────────────────▼─────────────────────────────────────┐
│                     RAG Pipeline (12 steps)                      │
│  guardrails → intent → session → rewrite → retrieve → rerank     │
│  → context → LLM → faithfulness → boundary → upsell → format     │
└────────────────────────────┬─────────────────────────────────────┘
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
   analytics_core      client_access         links/books
   (tracker, agg)      (token budget)        (purchase URLs)
         │                   │                   │
         ▼                   ▼                   ▼
   Author Admin UI     SuperAdmin UI        Celery tasks
```

---

## 4. Test Suites

### Suite A — Token Usage Consistency (P0 — BLOCKING)

**Objective:** One canonical token counter drives enforcement, admin UI, SuperAdmin UI, emails, and exports.

#### A.1 Single Source of Truth

| Test ID | Scenario | Steps | Expected |
|---------|----------|-------|----------|
| A-01 | Chat increments `tokens_used` | Send 1 chat message with known token usage (mock OpenAI returns `prompt_tokens=100, completion_tokens=50`) | `client_access.tokens_used` += 150 |
| A-02 | Chat increments Redis counter | Same as A-01 | `tokens:{author_id}:current` += 150 |
| A-03 | Counters stay in sync | Send 10 messages | `tokens_used` == Redis counter == sum of `chat_messages` tokens for period |
| A-04 | Regeneration counts tokens | Hallucination triggers regen (2 LLM calls) | Both calls' tokens counted |
| A-05 | Custom Q&A path counts tokens | Message hits custom Q&A override | Tokens still recorded if LLM called |
| A-06 | Failed LLM does not increment | OpenAI timeout / 500 | No token increment; error logged |
| A-07 | Analytics failure does not block chat | Mock DB write failure in `track_turn` | Visitor gets response; error logged at ERROR |

#### A.2 Budget Math (must be identical everywhere)

| Test ID | Formula Under Test | Surfaces |
|---------|-------------------|----------|
| A-10 | `total_budget = token_budget + bonus_tokens` | SuperAdmin grants, embed-token |
| A-11 | `remaining = total_budget - tokens_used` | Admin token page, embed-token, SuperAdmin |
| A-12 | `percent_used = tokens_used / total_budget * 100` | Admin gauge, SuperAdmin progress bar |
| A-13 | Bonus tokens increase remaining without changing expiry | SuperAdmin bonus modal |

**Consistency matrix (all must match after each chat turn):**

| Surface | Endpoint / Store | Includes Bonus? | Reads `tokens_used`? |
|---------|------------------|-----------------|----------------------|
| Author Token Usage page | `GET /api/admin/{slug}/token-usage` | MUST | MUST |
| Author Embed Token card | `GET /api/admin/{slug}/embed-token` | MUST | MUST |
| SuperAdmin Token Management | `GET /api/super/grants` | MUST | MUST |
| SuperAdmin Platform Analytics | `GET /api/super/health` or analytics | MUST | MUST |
| Subscription validation | `validate_subscription_token` | MUST | MUST |
| Analytics export CSV | `GET /api/admin/{slug}/export/analytics` | N/A (historical) | Aggregated from events |
| Weekly digest email | Celery task | MUST | MUST |

#### A.3 Budget Enforcement

| Test ID | Scenario | Expected |
|---------|----------|----------|
| A-20 | Usage at 79% | Chat works normally |
| A-21 | Usage at 80% | Warning email sent (if `notify_token_alerts=true`) within 60s |
| A-22 | Usage at 99% | Chat still works; UI shows red/warning state |
| A-23 | Usage at 100% | `budget_exhausted:{author_id}` set; next message returns graceful fallback (HTTP 200 per BDD), OpenAI NOT called |
| A-24 | SuperAdmin adds bonus at 100% | `budget_exhausted` cleared; chat resumes |
| A-25 | SuperAdmin reset tokens | `tokens_used=0`; policy on `bonus_tokens` documented and tested |
| A-26 | Extend subscription at 100% usage | Expiry extended; usage NOT reset unless explicitly reset |

#### A.4 SuperAdmin Token Operations

| Test ID | Action | Verify |
|---------|--------|--------|
| A-30 | Grant new subscription | `token_budget` set per plan; `tokens_used=0` |
| A-31 | Add bonus tokens | `bonus_tokens` incremented; audit log entry |
| A-32 | Reset token usage | `tokens_used=0`; Redis counters cleared; audit log |
| A-33 | Revoke subscription | Chat blocked (403); Redis blacklist set |
| A-34 | Token Management UI totals | Sum of all grants' `tokens_used` matches platform aggregate |

#### A.5 Author Admin Token Page

| Test ID | Scenario | Expected |
|---------|----------|----------|
| A-40 | Page load after 5 chats | Gauge shows non-zero `used` matching DB |
| A-41 | No active grant | Shows zeros with clear empty state (not error) |
| A-42 | Gauge color thresholds | <75% indigo, 75–90% amber, >90% red |
| A-43 | Matches embed-token card | Same `remaining` value on both pages |

---

### Suite B — Buy Book Button & Smart Links (P0 — BLOCKING)

**Objective:** When an author configures a purchase URL, the widget shows a "Buy Book" button at the correct times.

#### B.1 Data Model Unification

| Test ID | Scenario | Expected |
|---------|----------|----------|
| B-01 | Author saves Smart Link (`PUT /smart-links/{bookId}`) | `links.purchase_url` AND `books.buy_url` both set (or single source with fallback read) |
| B-02 | Author sets buy URL on book upload | URL persisted and available to chat |
| B-03 | `_get_book_links()` with only `books.buy_url` | Returns purchase URL (fallback) |
| B-04 | `_get_book_links()` with only `links` row | Returns purchase URL |
| B-05 | Preview URL | "Read Preview" button appears when configured (max 2 links) |

#### B.2 Upsell Engine — When Button Appears

| Test ID | Intent / Context | Turn | Expected `show_link` |
|---------|------------------|------|----------------------|
| B-10 | `purchase_intent` ("Where can I buy?") | 1 | `true` + Buy Book button in widget |
| B-11 | `full_story_request` | any | `true` |
| B-12 | `question`, book selected | 2+ | `true` (per `should_include_link`) |
| B-13 | `question`, high interest (≥0.45) | 2+ | `true` |
| B-14 | `complaint` | any | `false` — no buy button |
| B-15 | `greeting` | 1 | `false` (RECIPROCITY only, no button) |
| B-16 | Strategy `DIRECT_CTA` | any | `true` |
| B-17 | No purchase URL configured | any | `false` — no empty/broken button |
| B-18 | `show_link=true` but URL is `#` | any | Button must NOT render (invalid URL guard) |

#### B.3 Widget Rendering

| Test ID | Scenario | Expected |
|---------|----------|----------|
| B-20 | API returns `links: [{label:"Buy Book", url:"https://...", type:"purchase"}]` | Widget renders `<a class="ab-link-btn">` with icon |
| B-21 | Click Buy Book | `POST /track-click` fired with correct `session_id`, `book_id`, `link_type` |
| B-22 | Farewell on widget close | `POST /session/farewell` returns text + Buy Book link if URL exists |
| B-23 | Max 2 links | Purchase + Preview shown; no third link |
| B-24 | Mobile viewport (375px) | Buttons wrap correctly, tappable (min 44px touch target) |
| B-25 | `target="_blank" rel="noopener"` | Present on all link buttons |

#### B.4 End-to-End Conversion Path

| Test ID | Flow | Expected |
|---------|------|----------|
| B-30 | Configure URL → chat "where can I buy" → click button | Analytics records `link_shown=true` on message + click event |
| B-31 | Admin analytics dashboard | Link click count increments |
| B-32 | SuperAdmin platform analytics | Aggregate link clicks include this author |

---

### Suite C — Tone, Personality & Anti-Robotic (P0)

**Objective:** Responses feel human, warm, and on-brand; admin personality settings affect output.

#### C.1 Personality Settings Propagation

| Test ID | `response_style` | Expected in LLM system prompt |
|---------|------------------|--------------------------------|
| C-01 | `balanced` | Default warm tone instructions |
| C-02 | `formal` | Measured, professional vocabulary |
| C-03 | `casual` | Conversational, contractions allowed |
| C-04 | `enthusiastic` | Higher energy, exclamation sparingly |
| C-05 | Change style in admin → send chat | Next message reflects new style (verify via prompt capture mock) |

#### C.2 Robotic Phrase Elimination

| Test ID | Scenario | Must NOT contain |
|---------|----------|------------------|
| C-10 | No-context fallback | "I want to give you the most accurate answer" |
| C-11 | Hallucination fallback | "As an AI", "I'm an AI assistant" |
| C-12 | Jailbreak redirect | System prompt contents, rule explanations |
| C-13 | Budget exhausted message | Technical jargon, error codes |
| C-14 | Any normal response | Markdown `**bold**`, bullet lists, headers |

#### C.3 Custom Messages

| Test ID | Field | Scenario | Expected |
|---------|-------|----------|----------|
| C-20 | `welcome_message` | Session init | Widget shows custom welcome |
| C-21 | `fallback_message` | No retrieval context | Custom fallback used |
| C-22 | `out_of_scope_message` | Off-topic query | Custom or default redirect |
| C-23 | `bot_name` | All responses | Name in system prompt, not "AI assistant" |

#### C.4 Response Quality Constraints

| Test ID | Rule | Verification |
|---------|------|--------------|
| C-30 | Max ~75 words | `len(text.split()) <= 80` (buffer) |
| C-31 | Max 2 paragraphs | Formatter enforces |
| C-32 | Max 380 chars | Formatter truncates at sentence boundary |
| C-33 | No spoilers | Boundary test with "tell me the ending" |
| C-34 | No fabricated prices | Faithfulness guardrail test |

---

### Suite D — Authentication & Authorization (P0)

#### D.1 Auth Flows

| Test ID | Scenario | Expected |
|---------|----------|----------|
| D-01 | Author registration | 201, JWT returned, `role=author` |
| D-02 | Author login | JWT + refresh cookie |
| D-03 | Wrong password | 401 |
| D-04 | 5 failed logins | Account locked 30 min (per implementation) |
| D-05 | Refresh token rotation | New access token; old refresh invalidated |
| D-06 | SuperAdmin login | JWT with `role=superadmin` |
| D-07 | Author JWT on `/api/super/*` | 403 |
| D-08 | Expired JWT | 401 |

#### D.2 Tenant Isolation

| Test ID | Scenario | Expected |
|---------|----------|----------|
| D-10 | Author A requests Author B's books | 403 or empty (slug must match `current_user.id`) |
| D-11 | Author A uses B's slug in URL | Must NOT return B's data |
| D-12 | SuperAdmin lists all authors | Returns cross-tenant data (allowed) |

#### D.3 Subscription Token Security

| Test ID | Scenario | Expected |
|---------|----------|----------|
| D-20 | Valid token | Chat 200 |
| D-21 | Tampered HMAC | 403 |
| D-22 | Expired token | 403 |
| D-23 | Revoked token (Redis) | 403 immediately |
| D-24 | Revoked in DB | 403 |
| D-25 | Missing `X-Subscription-Token` header | 401/403 |

#### D.4 Suspension & Active State

| Test ID | Scenario | Expected |
|---------|----------|----------|
| D-30 | SuperAdmin suspends author | `chatbot_is_active=false` |
| D-31 | Suspended author — visitor chats | 403 with friendly message |
| D-32 | SuperAdmin unsuspends | Chat resumes |

---

### Suite E — RAG Pipeline (P1)

#### E.1 Pipeline Steps (integration)

| Test ID | Step | Verification |
|---------|------|--------------|
| E-01 | Guardrails | Jailbreak input → redirect, not LLM |
| E-02 | Intent classification | "where can I buy" → `purchase_intent` |
| E-03 | Book selector | 3 books + vague query → selector popup in response |
| E-04 | Single book | Auto-select, no popup |
| E-05 | Book name in query | Fuzzy match >0.85 → auto-select |
| E-06 | Faithfulness fail | Regenerate once; then safe fallback |
| E-07 | Boundary enforcer | Off-topic → redirect |
| E-08 | Custom Q&A override | Exact match returns canned answer |

#### E.2 Book Lifecycle

| Test ID | Scenario | Expected |
|---------|----------|----------|
| E-10 | Upload PDF | Status: uploading → parsing → chunking → embedding → ready |
| E-11 | File >50MB | Rejected before processing |
| E-12 | Unsupported .xlsx | 400 with supported formats listed |
| E-13 | Deactivate book mid-session | Next message offers remaining books |
| E-14 | Delete last active book | Chatbot disabled + dashboard warning |
| E-15 | Book `status != ready` | Graceful "still indexing" message |

---

### Suite F — Admin Panel (P1)

#### F.1 Dashboard & Analytics

| Test ID | Page | Verify |
|---------|------|--------|
| F-01 | Dashboard | Session count, recent activity load |
| F-02 | Analytics (30d) | Charts match `analytics_daily` rollups |
| F-03 | Sessions / Readers | Geo, fingerprint (no PII) |
| F-04 | Conversations | Transcript matches `chat_messages` |
| F-05 | Export CSV | Downloads valid CSV with correct columns |

#### F.2 Configuration Pages

| Test ID | Page | Verify |
|---------|------|--------|
| F-10 | Books CRUD | Create, edit, reorder, activate/deactivate |
| F-11 | Smart Links | Save, load, URL validation |
| F-12 | Widget Config | Theme, position persist to session init |
| F-13 | Personality | All fields save and reload |
| F-14 | Notifications toggles | `notify_token_alerts` respected by email tasks |
| F-15 | Custom Q&A | CRUD + priority ordering |
| F-16 | Embed Code | Valid script tag with live token |

#### F.3 UI Consistency

| Test ID | Check | Expected |
|---------|-------|----------|
| F-20 | All API errors | Toast/alert with human-readable message |
| F-21 | Loading states | Skeleton/spinner on every async page |
| F-22 | Empty states | Helpful copy, not blank screens |
| F-23 | Form validation | Client + server validation aligned |
| F-24 | Navigation | Active nav item highlights correctly |

---

### Suite G — SuperAdmin Panel (P1)

| Test ID | Feature | Verify |
|---------|---------|--------|
| G-01 | Authors list | Pagination, search, status badges |
| G-02 | Author detail | Books, grant status, token bar |
| G-03 | Grant access | Token generated, email sent, audit logged |
| G-04 | Revoke / extend / bonus / reset | Each writes audit log + correct DB state |
| G-05 | Audit log | Immutable, chronological, filterable |
| G-06 | Platform health | DB, Redis, Chroma status |
| G-07 | Announcements | Delivered to author notifications |
| G-08 | Token Management totals | Match sum of individual grants (see Suite A) |
| G-09 | Suspend / unsuspend | Immediate effect on chat (Suite D) |

---

### Suite H — Widget & Session (P1)

| Test ID | Scenario | Expected |
|---------|----------|----------|
| H-01 | Session init | `bot_name`, `welcome_message`, `widget_theme` applied |
| H-02 | Multi-turn history | Context preserved across turns |
| H-03 | Session reset | History cleared, new session ID |
| H-04 | Rating (1–5 stars) | Stored on `chat_sessions` |
| H-05 | Feedback submission | Stored and visible in admin |
| H-06 | SSE ingestion events | Progress events during book upload |
| H-07 | Rate limiting | 61st request/min per IP → 429 |
| H-08 | Widget <8KB | `widget.js` size budget met |
| H-09 | Offline / API error | User-friendly error bubble, retry option |

---

### Suite I — Email & Celery Tasks (P2)

| Test ID | Task | Trigger | Verify |
|---------|------|---------|--------|
| I-01 | Token 80% warning | Usage threshold | Email with correct numbers |
| I-02 | Subscription expiry 7d | Daily beat | Warning email |
| I-03 | Weekly digest (Monday 9am) | Beat + timezone | Sessions, top book, tokens, country |
| I-04 | Grant notification | SuperAdmin grant | Author receives access email |
| I-05 | Revocation notification | SuperAdmin revoke | Author notified within 5s |
| I-06 | Link health check | Scheduled | `purchase_url_ok` updated |
| I-07 | Analytics aggregation | Hourly | `analytics_daily` matches raw events |

---

### Suite J — Security & Edge Cases (P1)

| Test ID | Scenario | Expected |
|---------|----------|----------|
| J-01 | SQL injection in chat message | Sanitized; no DB error |
| J-02 | XSS in author `welcome_message` | Escaped in widget (`escHtml`) |
| J-03 | SSRF via buy URL | URL validation blocks internal IPs |
| J-04 | Path traversal on upload | Rejected |
| J-05 | CORS | Only configured origins |
| J-06 | Security headers | Present on all responses |
| J-07 | `/api/super/diag` | Must require auth (if not public by design) |
| J-08 | `/api/widget/token` | Must not issue tokens without author auth |
| J-09 | Concurrent chat + grant revoke | No race — revoke wins |
| J-10 | Redis down | Graceful degradation per layer (documented) |

---

## 5. Cross-Cutting Consistency Checklist

Run this checklist after **every** release candidate:

### 5.1 Numeric Consistency

- [ ] Send exactly 1 chat message with known token count
- [ ] Verify `client_access.tokens_used`
- [ ] Verify Redis `tokens:{author_id}:current`
- [ ] Verify Author Admin → Token Usage page
- [ ] Verify Author Admin → Embed Token card
- [ ] Verify SuperAdmin → Token Management row
- [ ] Verify SuperAdmin → Platform Analytics total
- [ ] Verify `chat_messages` sum for session
- [ ] Verify `analytics_events` row for turn

### 5.2 Buy Button Consistency

- [ ] Configure buy URL in Smart Links
- [ ] Ask "where can I buy this book?"
- [ ] Confirm `link_shown=true` in DB
- [ ] Confirm button visible in widget DOM
- [ ] Click button → analytics event recorded
- [ ] Close widget → farewell shows Buy Book

### 5.3 Tone Consistency

- [ ] Set personality to `casual` → verify prompt
- [ ] Trigger no-context path → no robotic phrases
- [ ] Trigger hallucination fallback → warm safe message
- [ ] Verify no markdown in any response path

---

## 6. Edge Case Catalog

| Category | Edge Case | Expected Behavior |
|----------|-----------|-------------------|
| **Tokens** | Author with 0 budget grant | Clear error on grant; chat blocked |
| **Tokens** | Multiple rapid messages | Atomic increment; no lost counts |
| **Tokens** | Month rollover | Usage resets per billing period (define policy) |
| **Tokens** | SuperAdmin reset during active chat | Next message uses fresh budget |
| **Links** | URL with UTM params | Preserved exactly |
| **Links** | International domain (IDN) | Validated and stored |
| **Links** | HTTP vs HTTPS | HTTPS preferred; mixed content warning in admin |
| **Books** | 0 active books | Catalog setup message |
| **Books** | 50+ books | Selector pagination / scroll |
| **Chat** | Empty message | 400 validation error |
| **Chat** | 2000+ char message | Truncated or rejected gracefully |
| **Chat** | Unicode / emoji | Handled end-to-end |
| **Chat** | RTL language input | Widget displays correctly |
| **Session** | Same fingerprint, new session | Separate session IDs |
| **Auth** | Token refresh during admin action | Seamless retry |
| **Upload** | Duplicate SHA-256 | Skip/replace dialog |
| **Upload** | Network drop mid-upload | Resume from chunk (if implemented) |

---

## 7. Acceptance Criteria (Definition of Done)

A feature or fix is **DONE** only when:

1. All P0 tests in the relevant suite pass in CI
2. No new gaps introduced in the consistency matrix (Section 5)
3. BDD scenario added to `tests/bdd/features/` BEFORE merge (per `.agent/RULES.md`)
4. Unit test coverage ≥80% on touched modules
5. Manual smoke on staging: Author flow + Visitor flow + SuperAdmin flow
6. Audit log entry verified for every SuperAdmin mutation
7. Analytics event verified for every chat turn
8. Documentation updated if API contract changes

### Release Blockers (cannot ship if any fail)

- [ ] Suite A — all P0 tests green
- [ ] Suite B — all P0 tests green
- [ ] Suite C — all P0 tests green
- [ ] Suite D — D-10, D-11, D-30, D-31 green
- [ ] CI no longer uses `|| true`

---

## 8. Implementation Roadmap for Test Infrastructure

### Phase 1 — Foundation (Week 1)

1. Create `tests/conftest.py` with async DB, Redis, JWT helpers
2. Add factories: `UserFactory`, `ClientAccessFactory`, `BookFactory`, `LinkFactory`
3. Mock OpenAI client fixture returning deterministic token counts
4. Fix CI: remove `|| true`; add PostgreSQL + Redis services

### Phase 2 — P0 Unit Tests (Week 1–2)

1. `test_token_tracking.py` — Suite A
2. `test_upsell_engine.py` — Suite B (link logic)
3. `test_formatter.py` — link injection, max links
4. `test_prompter_personality.py` — Suite C
5. `test_subscription_validation.py` — Suite D

### Phase 3 — Integration Tests (Week 2–3)

1. `test_chat_api.py` — full message flow with mocked LLM
2. `test_admin_token_usage.py` — API response consistency
3. `test_superadmin_grants.py` — grant/bonus/reset flows
4. `test_smart_links_sync.py` — books ↔ links unification

### Phase 4 — BDD & E2E (Week 3–4)

1. Port `.agent/BDD_SPECS.md` → `tests/bdd/features/*.feature`
2. Playwright: widget buy button visible + click
3. Playwright: admin token gauge updates after chat

### Phase 5 — Hardening (Ongoing)

1. Synthetic production monitor: health + token math spot-check
2. Load test: 60 req/min rate limit boundary
3. Chaos: Redis restart during chat

---

## 9. Traceability Matrix

| User-Reported Issue | Test Suite | Test IDs |
|--------------------|------------|----------|
| Tokens not counted on admin | A | A-01, A-40, A-43 |
| Tokens not counted on superadmin | A | A-01, A-34, G-08 |
| Robotic tone | C | C-10, C-11, C-01–C-05 |
| Buy now button not in chat | B | B-01, B-10, B-20, B-30 |
| Personality settings ignored | C | C-01–C-05 |
| Budget not enforced | A | A-20–A-26 |
| Inconsistent token numbers | A | A-10–A-13, Section 5.1 |

---

## 10. Sign-Off

| Role | Name | Date | Signature |
|------|------|------|-----------|
| Engineering Lead | | | |
| QA | | | |
| Product | | | |

---

*This plan should be treated as a living document. Update version and traceability matrix whenever gaps G-01 through G-10 are resolved or new features are added.*