knowledge_base/runbooks/cascading_errors.md

Runbook: Cascading Error Rate Increase

Overview

When error rates appear on multiple services simultaneously, always identify the origin service before attempting any remediation. Treating symptoms (downstream) before the root cause wastes time and can introduce additional instability.

Symptoms

• error_rate_pct elevated on 2+ services
• Errors appeared first (or are highest) on one specific service
• Log errors: "502 Bad Gateway", "upstream connection refused", "circuit breaker half-open"

Diagnosis Steps

1. Run diagnostic_query on ALL services (metric=error_rate_pct)
2. The service with the HIGHEST error rate is most likely the origin
3. Confirm with log_inspection — look for "502" or "connection refused" patterns
4. Check if downstream services show "cascaded_errors" in active_faults

Root Cause Identification

• The origin service will have both high error rate AND its own upstream errors in logs
• Downstream services will reference the upstream service name in their error logs

Remediation

• Fix origin service FIRST: rollback (if recent deploy) or restart_service
• Do NOT restart downstream services first — cascade will clear once origin is fixed
• Use circuit_breaker on downstream services only as a last resort to stop cascade spread

Quarantine Gate Notes

• restart_service on payment-service is HIGH RISK and will be blocked
• Use circuit_breaker on payment-service if it's downstream of the cascade

Verification

• After fixing origin: confirm its error_rate_pct drops to < 1%
• Downstream services should self-heal within 1-2 diagnostic cycles