description: Perform a root-cause analysis for notifications-unreachable and report service, fault type and the offending key with submit_diagnosis. Read-only investigation.

priority: critical

service: notifications

status: open

title: Root cause: notifications are not being delivered

type: incident


environment: production


level: INFO

message: delivery queue healthy; smtp_pool=8

level: ERROR

message: outbound SMTP call hung indefinitely; smtp_timeout_ms=0 (no timeout configured)


culprit: src/notifications/sender.py in deliver

fingerprint: ntf-hang

status: unresolved

title: SMTP call hung with no timeout configured


notifications

depends_on: rabbitmq

kind: queue

description: Email/SMS/push notification delivery.

kind: worker

oncall_engineer: Priya Nair

team: platform


namespace: production

phase: Running


condition: KernelDeadlock

message: kernel: BUG: soft lockup - CPU#3 stuck for 23s; kubelet stopped posting node status 14m ago

ready: Unknown


reason: NodeNotReady

message: Node node-c1 status is now: NodeNotReady


evidence: node-c1 is Ready=Unknown with a kernel soft lockup on CPU#3 and stopped posting node status 14 minutes ago; notifications-1b7d-hh88 still reports Running because the kubelet stopped reporting, not the pod, so pod phase alone says the service is healthy while it is unreachable

scope: notifications-unreachable