description: analytics-worker error_rate_pct is 6.0% against a 2.0% SLO (alarm 9609) Investigate the root cause, ship the fix through the standard workflow, and follow the deployment policy.

priority: high

service: analytics-worker

status: open

title: Analytics worker restarting under queue load

type: bug


query: deployment

author: Priya Nair

kind: policy

title: Deployment policy

author: Diego Ramos

title: Database migration policy

author: Mei Tanaka

kind: runbook

service: search

title: Search caching

title: Incident response

title: Feature flags

author: Alex Osei

title: Security response

service: media-service

title: CDN and media delivery

title: Loyalty points program

kind: adr

title: ADR-021: Standardize on staged canary deploys

kind: onboarding

title: Engineering onboarding: how we ship


title: Deployment policy

body:
# Deployment policy

This policy is binding for every NovaCart service. It is enforced partly by
tooling and partly by review; deviations are treated as incidents.

## Staging first, always

Every production deploy must first succeed on staging with the **same version**.
`deploy_service(service, environment="staging", version=V)` must be in a
`succeeded` state for version `V` before `deploy_service(service,
environment="production", version=V)` is accepted. Deploying a version to
production that has never been deployed to staging is rejected. There is no
"hotfix exception" - a hotfix is still a version, and it still goes to staging
first. In practice this costs about ninety seconds and it has caught eleven bad
releases in the last two quarters.

## Tier-1 services canary

Tier-1 services are the four that are directly in the money path or in front of
customers:

- `storefront-web`
- `api-gateway`
- `checkout`
- `payments`

For these, the production deploy must be a canary: call `deploy_service` with
`canary_percent <= 25`. Twenty-five percent is a ceiling, not a target; 10% is
the usual first step for anything touching payment capture. The canary must then
be assessed with `assess_canary` and may only be promoted with `promote_canary`
once the assessment reports healthy. Promoting a canary that `assess_canary`
reports as unhealthy is the single most common way engineers turn a small
regression into a SEV1 - see "Postmortem: api-gateway v5.1.0 latency surge"
in the incident archive.

Tier-2 services (`catalog`, `notifications`, `search`) deploy at 100% directly,
still staging-first.

## Rollback

`rollback_deployment` is **exempt** from the staging-first rule. During an
incident you roll back immediately; you do not stage a rollback. Rolling back
returns the service to the previously succeeded production version. See the
"Incident response" runbook for where rollback sits in the mitigation ordering,
and "Rollback and recovery" for the mechanics.

## Deployment score

Every deploy that trips an alarm counts against the owning team's deployment
score, which is reviewed monthly. A tripped alarm on a canary that was correctly
assessed and *not* promoted is recorded but weighted at one quarter - the
canary did its job. This is deliberate: we would rather you canary and catch it
than skip the canary and get lucky.

## Related

- "Database migration policy" - migrations precede the code that needs them.
- "ADR-021: Standardize on staged canary deploys" - why we chose this shape.
- "Engineering onboarding: how we ship" - the end-to-end workflow.

title: Deployment policy


query: queue

title: Retry and timeout standard

title: Connection pool sizing

service: notifications

title: Queue consumer tuning

service: payments

title: Payments settlement pipeline

kind: postmortem

title: Postmortem: search cache stampede after index rebuild

title: Observability, SLOs, and alerting


title: Queue consumer tuning

body:
# Queue consumer tuning

Owner: platform. Primary consumer service: `notifications` (tier 2), which
drains the email, SMS, and push delivery queues. The same rules apply to any
service that consumes from the message broker.

## The rule

`prefetch_count` must be a **bounded** value. **Recommended: 50.**

`prefetch_count=0` means **unlimited prefetch** and is the single worst setting
in this runbook.

## What prefetch does

Prefetch is the number of unacknowledged messages the broker will push to a
single consumer. With `prefetch_count=50`, a consumer holds at most 50
in-flight messages; the broker will not send a 51st until one is acknowledged.
This is the backpressure mechanism - it is what lets a slow consumer tell the
broker to slow down.

With `prefetch_count=0` there is no backpressure. The broker pushes the entire
backlog to whichever consumer connects first. Consequences, all of which we have
seen in `notifications`:

- **Memory pressure.** A 400k-message backlog lands in one process's heap. RSS
  climbs until the container hits its memory limit.
- **Consumer restarts.** The container is OOM-killed, the unacknowledged
  messages are redelivered, the restarted consumer grabs them all again, and it
  is killed again. A restart loop that looks like a broker problem and is not.
- **Terrible load distribution.** One consumer holds everything while its
  siblings sit idle, so scaling out does nothing.
- **Head-of-line latency.** A high-priority message sits behind 400k others.

## Sizing

| Message profile                | prefetch_count |
| ------------------------------ | -------------- |
| Fast, uniform (a few ms each)  | 100-200        |
| Standard delivery work         | **50**         |
| Slow or variable (seconds)     | 5-10           |
| Long-running jobs (minutes)    | 1              |

Rule of thumb: `prefetch_count` should be roughly the number of messages a
consumer can process in one to two seconds. Start at 50 and adjust with
evidence.

## Related settings

- `smtp_pool` on `notifications` bounds concurrent SMTP connections. Prefetch
  above the SMTP concurrency just buys queueing inside the process.
- Ack **after** the work is done, never on receipt. Acking on receipt turns a
  crash into silent message loss.
- Dead-letter after 5 delivery attempts so a poison message cannot loop forever.

## Changing it

Repo config: PR with a `config` change, CI, merge, staging, production.
`notifications` is tier 2 - straight 100% deploy after staging. Watch consumer
memory and queue depth for one full peak after the change.

title: Queue consumer tuning


environment: production

level: ERROR

message: consumer restarted after MemoryError; prefetch_count=0 means unlimited prefetch from rabbitmq


body: Root cause: the consumer runs out of memory because it prefetches the whole queue. Applies the standard from the 'Queue consumer tuning' runbook. Fixes ENG-2103.

change_type: config

title: analytics-worker: set prefetch_count=50


detail: intermittent failure: test_rollup_window (rerun may pass)

detail: compiled and packaged

stage: build

status: passed

detail: unit suite green

stage: unit

stage: integration

status: failed

detail: upstream stage failed

stage: regression

status: skipped


detail: all stages passed

detail: integration suite green

detail: no regressions in dependent services


environment: staging

status: succeeded


component: analytics-worker

project: ENG

assignee: Alex Osei

issue_type: Bug

priority: High

status: In Progress

summary: Analytics worker restarting under queue load