title: Healthy database

expected_root_cause: Nothing is wrong. The correct answer is no finding.

naive_answer: find something anyway

query:
SELECT id, status, total_cents, created_at
    FROM orders
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

query:
SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

query:
SELECT id, user_id, total_cents, created_at
    FROM orders
    WHERE status = $1 AND created_at > now() - interval $2
    ORDER BY created_at DESC
    LIMIT $3

relname: events

relname: orders

relname: categories

relname: products

relname: users

state: idle

wait_event_type: Client

wait_event: ClientRead

query:
    SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

autovacuum

setting: on

unit: s

unit: kB


title: Missing index on a hot foreign key

expected_root_cause: order_items.product_id has no index; the product-lookup query falls back to a sequential scan over the whole table.

scenario_description: Drops idx_order_items_product_id and drives product-lookup traffic. The join seq-scans 2.5M rows on every call.

query:
SELECT o.id, o.created_at, oi.quantity
    FROM order_items oi
    JOIN orders o ON o.id = oi.order_id
    WHERE oi.product_id = $1
    ORDER BY o.created_at DESC
    LIMIT $2

query:
SELECT id, status, total_cents, created_at
    FROM orders
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

query:
SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

query:
SELECT id, user_id, total_cents, created_at
    FROM orders
    WHERE status = $1 AND created_at > now() - interval $2
    ORDER BY created_at DESC
    LIMIT $3

state: idle

wait_event_type: Client

wait_event: ClientRead

query:
    SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

category: indexing

fix_target: database

naive_answer: add an index


title: Plan regression from stale statistics

expected_root_cause: Statistics on orders are stale after a bulk load. The planner's row estimate for status = 'processing' is off by orders of magnitude, so it picks the wrong plan. Fix is ANALYZE orders, not a new index.

scenario_description: Inserts 300k orders with a brand-new status value and blocks autovacuum, so pg_statistic never learns the value exists.

query:
SELECT id, user_id, total_cents, created_at
    FROM orders
    WHERE status = $1 AND created_at > now() - interval $2
    ORDER BY created_at DESC
    LIMIT $3

query:
SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

query:
SELECT id, status, total_cents, created_at
    FROM orders
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

state: idle

wait_event_type: Client

wait_event: ClientRead

query:
    SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

category: statistics

naive_answer: add an index on orders.status


title: Table bloat — autovacuum disabled

expected_detector: bloat

expected_root_cause: autovacuum is disabled on events, so dead tuples from UPDATE churn are never reclaimed. Fix is re-enabling autovacuum and vacuuming.

scenario_description: Disables autovacuum on events and runs repeated mass UPDATEs, leaving several hundred thousand dead tuples that never get reclaimed.

query:
SELECT id, user_id, total_cents, created_at
    FROM orders
    WHERE status = $1 AND created_at > now() - interval $2
    ORDER BY created_at DESC
    LIMIT $3

query:
SELECT id, status, total_cents, created_at
    FROM orders
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

query:
SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

state: idle

wait_event_type: Client

wait_event: ClientRead

query:
    SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

category: vacuum

naive_answer: the table is just large


title: Lock contention — idle in transaction

expected_root_cause: A session is idle-in-transaction holding row locks on orders. Blocked writers are waiting on it. Fix targets the holder, not the waiters.

scenario_description: Leaves a session idle-in-transaction holding row locks on 1000 orders rows, plus three writers that block behind it.

query:
SELECT id, status, total_cents, created_at
    FROM orders
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 25

query:
SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT $2

query:
SELECT id, user_id, total_cents, created_at
    FROM orders
    WHERE status = $1 AND created_at > now() - interval '90 days'
    ORDER BY created_at DESC
    LIMIT 50

state: idle

wait_event_type: Client

wait_event: ClientRead

query:
    SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

state: idle in transaction

wait_event_type: Client

wait_event: ClientRead

query: UPDATE orders SET updated_at = now() WHERE id BETWEEN 1 AND 1000;

state: active

wait_event_type: Lock

wait_event: transactionid

query: SET application_name = 'dbra_blocked'; SELECT pg_sleep(2); UPDATE orders SET updated_at = now() WHERE id BETWEEN 1 AND 100;

state: active

wait_event_type: Lock

wait_event: transactionid

query: SET application_name = 'dbra_blocked'; SELECT pg_sleep(2); UPDATE orders SET updated_at = now() WHERE id BETWEEN 500 AND 600;

state: active

wait_event_type: Timeout

wait_event: PgSleep

query: SET application_name = 'dbra_blocked'; SELECT pg_sleep(2); UPDATE orders SET total_cents = total_cents WHERE id = 42;

category: locking

fix_target: session

naive_answer: kill the slow queries


title: ORM N+1 query pattern

expected_root_cause: Application-side N+1. order_items is queried once per order row instead of batched. High call count, low mean time, trivial rows per call. Fix is in the application, not the database.

scenario_description: Runs 20 pages x 50 rows of list-then-per-row-lookup traffic. No schema change: 1000 fast queries where 20 would do.

query: SELECT id, product_id, quantity FROM order_items WHERE order_id = $1

query:
SELECT id, status, total_cents, created_at
    FROM orders
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 25

query:
SELECT id, user_id, total_cents, created_at
    FROM orders
    WHERE status = $1 AND created_at > now() - interval '90 days'
    ORDER BY created_at DESC
    LIMIT 50

query:
SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

query:
SELECT id FROM orders
    WHERE status = $2
    ORDER BY created_at DESC
    LIMIT $1

state: idle

wait_event_type: Client

wait_event: ClientRead

query:
    SELECT event_type, created_at
    FROM events
    WHERE user_id = $1
    ORDER BY created_at DESC
    LIMIT 20

category: application

fix_target: application

naive_answer: nothing is slow, the database is healthy