knowledge_base/db_pool_runbook.md

DB Pool Exhaustion Runbook

Symptoms

• PostgreSQL connection pool at 20/20 active connections
• Lock contention detected on auth-db
• Cascading latency on auth-service and api-gateway

Diagnostic Steps

1. Query pool metric on auth-db to confirm exhaustion
2. Check lock_contention flag in pool status
3. Inspect auth-db logs for long-running query signatures

Remediation Options

Option A — kill long-running queries:

• remediation: kill_pid on auth-db
• Verify pool utilisation drops below 80%

Option B — release connections:

• remediation: update_config on auth-db with max_connections=40
• Verify lock_contention clears

Option C — scale upstream:

• remediation: scale_up on auth-service to reduce connection pressure

Key Metric

Pool utilisation must drop below 50% for full remediation credit. Lock contention must clear before submitting resolution.