| # Postmortem: [Incident Title] |
|
|
| - **Date**: YYYY-MM-DD |
| - **Severity**: SEV1 / SEV2 / SEV3 |
| - **Duration**: Xh Ym |
| - **Author**: @name |
| - **Status**: Draft / Final |
|
|
| ## Summary |
|
|
| One-paragraph summary of what happened. Read this first; if you only have 30 seconds, this is all you need. |
|
|
| ## Impact |
|
|
| - **Users affected**: N users / M% of traffic |
| - **Duration of user-visible impact**: Xh Ym |
| - **Revenue impact**: $X (estimated) |
| - **Data loss**: yes / no / partial |
|
|
| ## Timeline (all times UTC) |
|
|
| - **HH:MM** — first alert (Telegram / Sentry / user report) |
| - **HH:MM** — investigation began |
| - **HH:MM** — root cause identified |
| - **HH:MM** — mitigation applied (e.g. container restart, rollback) |
| - **HH:MM** — service recovered |
| - **HH:MM** — monitoring confirmed no recurrence |
|
|
| ## Root Cause |
|
|
| Technical explanation. No blame. Describe the technical chain that led to the incident. |
|
|
| ## Contributing Factors |
|
|
| - **Factor 1**: e.g. "alert threshold was set too high" |
| - **Factor 2**: e.g. "no runbook existed for this failure mode" |
| - **Factor 3**: e.g. "tests did not cover this code path" |
|
|
| ## What Went Well |
|
|
| - Alert fired within 60s of failure |
| - Rollback completed in 5min |
| - No data loss |
|
|
| ## What Went Poorly |
|
|
| - Alert was noisy (3 false positives before real one) |
| - Runbook was outdated |
| - Took 30min to identify root cause |
|
|
| ## Triggering Condition |
|
|
| What was the chain of events that led to the incident? Be specific. |
|
|
| ## Detection |
|
|
| How did we find out? Alert / user report / proactive check / etc. |
|
|
| ## Resolution |
|
|
| What did we do to fix it? |
|
|
| ## Action Items |
|
|
| - [ ] **Action 1**: [description] — owner: @name — due: YYYY-MM-DD |
| - [ ] **Action 2**: [description] — owner: @name — due: YYYY-MM-DD |
| - [ ] **Action 3**: [description] — owner: @name — due: YYYY-MM-DD |
|
|
| Action items must be: |
| - **Specific**: not "improve observability" — "add Prometheus alert for X at Y threshold" |
| - **Owned**: one named person, no "team" |
| - **Dated**: hard deadline, not "soon" |
|
|
| ## Lessons Learned |
|
|
| What would we do differently? Be honest. |
|
|
| ## References |
|
|
| - Slack thread: <link> |
| - Runbook: `docs/runbooks/X.md` |
| - Related ADRs: `docs/adr/NNNN-*.md` |
| - Code change: <commit hash or PR link> |
|
|
