description: Alarm 9609 is firing. Identify the responsible service and report the finding with submit_diagnosis. Read-only investigation.

priority: high

service: analytics-worker

status: open

title: Localize alarm 9609 (analytics-worker errors)

type: incident


status: firing

message: payments error_rate_pct 4.2 exceeds SLO 1.0

service: payments

severity: high

message: search latency_p99_ms 850.0 exceeds SLO 300.0

service: search

severity: medium

message: checkout error_rate_pct 5.5 exceeds SLO 1.0

service: checkout

message: api-gateway latency_p99_ms 1030.0 exceeds SLO 250.0

service: api-gateway

severity: critical

message: catalog latency_p99_ms 645.0 exceeds SLO 300.0

service: catalog

message: inventory error_rate_pct 4.7 exceeds SLO 1.0

service: inventory

message: media-service latency_p99_ms 800.0 exceeds SLO 400.0

service: media-service

message: notifications error_rate_pct 3.6 exceeds SLO 1.5

service: notifications

message: analytics-worker error_rate_pct 6.0 exceeds SLO 2.0

message: checkout latency_p99_ms 530.0 exceeds SLO 400.0


description: Payments succeed 99% of the time.

description: Search p99 under 300ms.

description: Checkout succeeds 99% of the time.

description: Gateway p99 under 250ms.

description: Payments p99 under 200ms.

description: Checkout p99 under 400ms.

description: Catalog p99 under 300ms.

description: Inventory reservations succeed 99% of the time.

description: Media p99 under 400ms.

description: Notification delivery succeeds 98.5% of the time.

description: Event processing succeeds 98% of the time.

description: Storefront p99 under 500ms.

service: storefront-web


analytics-worker

depends_on: rabbitmq

kind: queue

description: Consumes the event queue and builds analytics rollups.

kind: worker

oncall_engineer: Priya Nair

rollup_daily: present

team: platform


environment: production

level: ERROR

message: consumer restarted after MemoryError; prefetch_count=0 means unlimited prefetch from rabbitmq


evidence: analytics-worker error_rate_pct 6.0; the consumer OOMs with unlimited queue prefetch