description: notifications error_rate_pct is 3.6% against a 1.5% SLO (alarm 9608) Investigate the root cause, ship the fix through the standard workflow, and follow the deployment policy.

priority: high

service: notifications

status: open

title: Notification delivery failures from hung SMTP calls

type: bug


query: deployment

author: Priya Nair

kind: policy

title: Deployment policy

author: Diego Ramos

title: Database migration policy

author: Mei Tanaka

kind: runbook

service: search

title: Search caching

title: Incident response

title: Feature flags

author: Alex Osei

title: Security response

service: media-service

title: CDN and media delivery

title: Loyalty points program

kind: adr

title: ADR-021: Standardize on staged canary deploys

kind: onboarding

title: Engineering onboarding: how we ship


title: Deployment policy

body:
# Deployment policy

This policy is binding for every NovaCart service. It is enforced partly by
tooling and partly by review; deviations are treated as incidents.

## Staging first, always

Every production deploy must first succeed on staging with the **same version**.
`deploy_service(service, environment="staging", version=V)` must be in a
`succeeded` state for version `V` before `deploy_service(service,
environment="production", version=V)` is accepted. Deploying a version to
production that has never been deployed to staging is rejected. There is no
"hotfix exception" - a hotfix is still a version, and it still goes to staging
first. In practice this costs about ninety seconds and it has caught eleven bad
releases in the last two quarters.

## Tier-1 services canary

Tier-1 services are the four that are directly in the money path or in front of
customers:

- `storefront-web`
- `api-gateway`
- `checkout`
- `payments`

For these, the production deploy must be a canary: call `deploy_service` with
`canary_percent <= 25`. Twenty-five percent is a ceiling, not a target; 10% is
the usual first step for anything touching payment capture. The canary must then
be assessed with `assess_canary` and may only be promoted with `promote_canary`
once the assessment reports healthy. Promoting a canary that `assess_canary`
reports as unhealthy is the single most common way engineers turn a small
regression into a SEV1 - see "Postmortem: api-gateway v5.1.0 latency surge"
in the incident archive.

Tier-2 services (`catalog`, `notifications`, `search`) deploy at 100% directly,
still staging-first.

## Rollback

`rollback_deployment` is **exempt** from the staging-first rule. During an
incident you roll back immediately; you do not stage a rollback. Rolling back
returns the service to the previously succeeded production version. See the
"Incident response" runbook for where rollback sits in the mitigation ordering,
and "Rollback and recovery" for the mechanics.

## Deployment score

Every deploy that trips an alarm counts against the owning team's deployment
score, which is reviewed monthly. A tripped alarm on a canary that was correctly
assessed and *not* promoted is recorded but weighted at one quarter - the
canary did its job. This is deliberate: we would rather you canary and catch it
than skip the canary and get lucky.

## Related

- "Database migration policy" - migrations precede the code that needs them.
- "ADR-021: Standardize on staged canary deploys" - why we chose this shape.
- "Engineering onboarding: how we ship" - the end-to-end workflow.

title: Deployment policy


query: retry

title: Retry and timeout standard

title: Connection pool sizing

service: checkout

title: Checkout architecture

service: payments

title: Payments settlement pipeline

title: Engineering onboarding: how we ship

service: api-gateway

title: API gateway: routing, traffic weights, and rate limiting

title: Service catalog and service tiers


title: Retry and timeout standard

body:
# Retry and timeout standard

Applies to every cross-service call in the NovaCart fleet. Two config keys per
downstream dependency, named after the downstream service.

## The two keys

For a caller talking to downstream service `X`:

- `<downstream>_retry_max_attempts` - **must be 3**
- `<downstream>_timeout_ms` - **must be at most 2000**

So `payments` calling `notifications` runs with
`notifications_retry_max_attempts=3` and `notifications_timeout_ms=2000` (or
lower). `checkout` calling `payments` runs with `payments_retry_max_attempts=3`
and `payment_timeout_ms` inside the 2000ms ceiling.

Retries use exponential backoff with jitter; the backoff schedule is applied
automatically by the shared HTTP client, so you do not configure delays. Three
attempts means one initial call plus two retries, worst case roughly
`3 * timeout_ms` plus backoff.

## Why 3 and why 2000

**A retry value of 0 means one timeout permanently fails the request.** There is
no second chance. A single blip in a downstream - a pod restart, a brief GC
pause, a rebalanced connection - becomes a user-visible failure and a failed
order. This is not hypothetical: payments ran with
`notifications_retry_max_attempts=0` and `notifications_timeout_ms=30000` for
several weeks and pushed `error_rate_pct` from a baseline of 0.4% to 3.8%,
straight through the 1.0% SLO.

The 2000ms ceiling exists because timeouts multiply up the call stack. A 30000ms
timeout on a leaf call does not "wait patiently" - it holds a request-handling
worker and a database connection for thirty seconds, and under load that is how
you exhaust a pool (see "Connection pool sizing"). If a downstream genuinely
cannot answer in two seconds, the call belongs on a queue, not in the request
path.

## Checklist for a new dependency

1. Add `<downstream>_retry_max_attempts=3` to the caller's config.
2. Add `<downstream>_timeout_ms` at or below 2000.
3. Confirm the call is idempotent, or that the downstream deduplicates by
   idempotency key. Retrying a non-idempotent write is worse than failing.
4. Confirm the caller's own inbound timeout is larger than
   `attempts * timeout_ms` plus backoff, or the retry budget is fiction.
5. Add the dependency to the service's dependency section in its design doc.

## Anti-patterns

- Retrying on 4xx. Only retry timeouts, connection errors, 429, and 5xx.
- Retrying inside a retry. Nested retries multiply: 3 x 3 = 9 calls.
- Raising the timeout to "fix" a slow downstream. Fix the downstream.

title: Retry and timeout standard


environment: production

level: INFO

message: delivery queue healthy; smtp_pool=8

level: ERROR

message: outbound SMTP call hung indefinitely; smtp_timeout_ms=0 (no timeout configured)


body: Root cause: outbound SMTP calls hang forever because no timeout is configured. Applies the standard from the 'Retry and timeout standard' runbook. Fixes ENG-2104.

change_type: config

title: notifications: set smtp_timeout_ms=2000


detail: all stages passed

detail: compiled and packaged

stage: build

status: passed

detail: unit suite green

stage: unit

detail: integration suite green

stage: integration

detail: no regressions in dependent services

stage: regression


environment: staging

status: succeeded


component: notifications

project: ENG

assignee: Priya Nair

issue_type: Bug

priority: High

status: In Progress

summary: Notification delivery failures from hung SMTP calls