TESTING.md

Testing Guide

Three layers: automated pytest (unit + integration, including auth/role coverage), manual exploration through the officer console and client app, and a live smoke test script for post-deploy verification of the full officer/client flow.

1. Automated tests (pytest)

source venv/bin/activate
pip install -r requirements-dev.txt
pytest                     # everything
pytest tests/unit -q       # unit tests only (fast, no live DB writes)
pytest tests/integration -q  # integration tests (hits the real MySQL DB)

Unit tests (tests/unit/)

Small synthetic fixtures, no database:

• test_config.py — fail-loud .env validation (including the auth-related vars: SEED_OFFICER_EMAIL, SEED_OFFICER_PASSWORD, SESSION_EXPIRE_MINUTES)

• test_features.py — the shared feature-engineering module (app/features.py)

• test_train_split.py — time-respecting split logic

• test_scoring.py — the scoring service (app/scoring.py) against the real trained model, using hand-built transactions

• test_labeled_fixtures.py — replays tests/fixtures/labeled_transactions.json (58 hand-labeled synthetic transactions covering obvious fraud and legitimate patterns across simulated dates) through the real scoring service and asserts ≥90% overall accuracy and ≥90% fraud recall. Regenerate the fixture set with:

  python -m scripts.generate_fixtures
  

Integration tests (tests/integration/)

test_api.py exercises every endpoint through FastAPI's TestClient against the real, configured MySQL database, covering:

• Auth: login (correct/wrong password/unknown email — all the same generic error), logout invalidating the session, /api/auth/me
• Role boundaries: an officer route rejects a client (403) and an anonymous caller (401), and vice versa
• Officer flows: create client, list clients, client detail (transactions + alerts), cross-client alert queue, approve/reject/dismiss (reject reverses the transaction and restores the client's balance), dashboard/insights/model-performance stats, /api/predict + /api/transactions/batch
• Client flows: submit a transaction, can't overdraw, own transaction history, own alerts
• Data isolation: two client fixtures (two_clients — see below) confirm each client sees only their own transactions/alerts, and that a client hitting an officer-only route for another client's detail gets 403
• All HTML pages return 200 with the right content type

Fixtures (conftest.py): officer_identity logs in as the seed officer; two_clients creates two client accounts through the real /api/officer/clients endpoint (not inserted directly into the DB) and logs into both, giving you (client_a, client_b) — each a {"cookies": {...}, "user": {...}, "client_id": ...} dict. Because a single TestClient only has one cookie jar, tests pass cookies=identity["cookies"] explicitly per request to act as several logged-in identities at once without re-running the (model-loading) app lifespan per identity.

Isolation note: the configured DB user does not have CREATE DATABASE privilege, so these tests can't spin up a literal second schema. Instead, conftest.py wraps each test in a SQL SAVEPOINT-nested transaction that is always rolled back at the end of the test (SQLAlchemy's documented "join a Session into an external transaction" pattern) — route handlers' internal db.commit() calls only release the savepoint, never the outer transaction, so nothing a test writes is ever actually persisted. The one exception is the seed officer row itself: the app's startup lifespan creates it (idempotently) against the real engine, not the test's savepoint session — that's intentional, matching production behavior, and safe to re-run. If your database user does get CREATE DATABASE privilege later, point DATABASE_URL at a dedicated ..._test database instead for a literal second-schema setup — the tests themselves don't need to change.

2. Manual exploration

Start the API (uvicorn app.main:app --host 0.0.0.0 --port $APP_PORT) and open http://localhost:$APP_PORT/login.

As the officer (sign in with SEED_OFFICER_EMAIL/SEED_OFFICER_PASSWORD from .env):

1. /officer/clients → "+ Create client" — name, email, starting balance, account type. Copy the one-time temp password shown.
2. /officer/alerts — once a client has submitted a flagged transaction, click "Review" to expand the SHAP breakdown, then Approve, Reject & reverse, or Dismiss it. Reject asks for confirmation and reverses the transaction (excluded from the client's balance).
3. /officer/insights and /officer/model — the EDA and model-selection story behind the score.

As a client (log out, sign in with the email + temp password from step 1):

1. /client/transactions/new — pick a type, amount, destination account, and date/time.
2. To reproduce the classic fraud pattern: enter your full current balance as the amount for a Transfer or Cash out. This drains the account to exactly zero — the same signature the model learned from real PaySim fraud rows. The result reads "being reviewed for your security," not a raw probability.
3. /client/alerts — read-only, plain-language status; no admin actions available to a client.

Every client-submitted transaction flows through the same transactions → fraud_predictions → fraud_alerts tables as everything else — /officer/alerts and /client/alerts are just two different filtered views over one table, not separate systems.

3. Live smoke test (scripts/test_live.py)

Post-deploy (or post-restart) verification against a running instance of the API — local or behind the reverse proxy:

# against a local dev server
python -m scripts.test_live --base-url http://127.0.0.1:8811

# against the live deployment
python -m scripts.test_live --base-url https://bank-fraud.hdev.rw

What it does:

1. Checks /health.
2. Logs in as the seed officer, creates two clients, and submits three transactions across different simulated dates on client A's account (a legit payment, a legit partial transfer, and a full-balance drain) through the real /api/client/transactions endpoint — printing predicted vs. expected risk tier for each.
3. Confirms role separation live: the officer sees client A's alert in the cross-client queue; client A sees exactly their own alert; client B sees none of it; client B gets a 403 when hitting the officer-only client-detail route for client A.
4. Replays the full tests/fixtures/labeled_transactions.json fixture set through the officer-only /api/predict endpoint, printing predicted vs. expected risk tier per transaction and a final accuracy summary.

Exits 0 only if accuracy stays at or above 90% (the same bar test_labeled_fixtures.py enforces) and every role-separation check passed; exits 1 otherwise — safe to wire into a post-deploy CI/CD gate.

Note: this script creates two real client accounts + a few transactions/predictions/alerts against whatever database the target server is configured against — clean those up afterward if you're running it against a database other test suites also read from, e.g.:

from app.db.session import session_scope
from app.db.models import Transaction, FraudPrediction, FraudAlert, Client, User, Session as SessionModel

with session_scope() as db:
    db.query(FraudAlert).delete()
    db.query(FraudPrediction).delete()
    db.query(Transaction).delete()
    db.query(SessionModel).delete()
    db.query(Client).delete()
    db.query(User).filter(User.role == "client").delete()  # keeps the seed officer