---
title: Seeder auxiliary Redis writes crash on a single Upstash timeout
date: 2026-07-17
category: database-issues
module: scripts/_seed-utils.mjs
problem_type: database_issue
component: background_job
symptoms:
  - "`FATAL: The operation was aborted due to timeout` after a successful fetch in seed-gdelt-intel"
  - "Railway badge flips red with PUBLISH_TIMEOUT class; 3 crashes in 25-run window, 0 successes"
  - "Auxiliary `writeExtraKey` SETs or `extendExistingTtl` EXPIRE pipelines time out, taking the whole run down"
root_cause: missing_tooling
resolution_type: code_fix
severity: medium
tags:
  - redis
  - upstash
  - seeder
  - retry
  - seed-utils
  - afterpublish
  - timeout
---

# Seeder auxiliary Redis writes crash on a single Upstash timeout

## Problem

`seed-gdelt-intel` was crashing with `FATAL: The operation was aborted due to timeout` during the post-fetch Redis write phase. The upstream GDELT fetch had already succeeded and the canonical key publish (`atomicPublish`) already retried transient failures, but the auxiliary timeline-key writes and TTL extensions in `afterPublish` used one-shot `fetch()` calls. A single Upstash latency spike turned a transient blip into a full seeder crash and a Railway "Deploy Crashed!" email.

## Symptoms

- `FATAL: The operation was aborted due to timeout` appears after `Extended TTL on N key(s)` / `WARNING: N key(s) were expired/missing` logs.
- The seeder diagnostic classifies the service as `PUBLISH_TIMEOUT` with a warning severity.
- The crash recurs (3 in the inspected window) because every run re-rolls the same dice against Upstash tail latency.

## What Didn't Work

- **Retrying only the canonical publish.** `atomicPublish` already wrapped its staging/canonical SET/DEL in `withRetry`, but that only protects the canonical key. The `afterPublish` auxiliary writes (`writeExtraKey`, `extendExistingTtl`) were left single-shot.
- **Catching the timeout inside `extendExistingTtl`.** That helper already caught errors and returned `false`, but `writeExtraKey` threw on any non-ok response or abort, and neither helper retried — so a transient timeout still failed the run.

## Solution

Wrap both auxiliary Redis helpers in the same retry contract already used by `redisCommand` and `atomicPublish`:

- `writeExtraKey` (`scripts/_seed-utils.mjs:668`) now wraps its SET call in `withRetry` with 2 retries and a 1s base delay.
- `extendExistingTtl` (`scripts/_seed-utils.mjs:722`) now wraps its `/pipeline` call in `withRetry` with the same budget.
- Permanent 4xx errors are tagged `nonRetryable` so they fail fast.
- HTTP 429 errors honor the upstream `Retry-After` header.
- 5xx, timeouts, and network tears retry with exponential backoff.

The boolean contract of `extendExistingTtl` is preserved: it still returns `true` only when every `EXPIRE` returns `1`. A successful response with some `EXPIRE` no-ops (missing/expired keys) is a real data condition, not a transient error, so it returns `false` without burning retries.

Fixed in PR [#5364](https://github.com/koala73/worldmonitor/pull/5364).

## Why This Works

The root cause was not a bad source or bad data — it was a missing resilience layer on the auxiliary write path. Upstash REST is served over the public internet; a single stalled request or brief 503 is expected at scale. The canonical publish path already treated these as retryable; the auxiliary path did not. Adding retry makes the failure mode symmetric across all Redis writes in a seeder run.

## Prevention

- When adding a new Redis helper in `scripts/_seed-utils.mjs`, decide its retry contract up front. Helpers that write seeded data should default to `withRetry` unless the caller explicitly needs fail-fast semantics.
- Keep error tagging consistent with `redisCommand`:
  - `PERMANENT_4XX_STATUSES` → `err.nonRetryable = true`
  - `429` → parse `Retry-After` into `err.retryAfterMs`
  - everything else (5xx, timeout, network tear) → let `withRetry` back off
- Add a regression test that fails the first call and succeeds on retry for any new Redis write helper. The existing tests for `writeExtraKey` and `extendExistingTtl` now cover timeout, 503, 429, and permanent 401 paths.

## Related Issues

- `diagnose-railway-seeders` skill class `PUBLISH_TIMEOUT` — post-fetch Redis publish timed out.
- Memory: [[feedback_never_memorize_a_workaround_for_a_tool_bug_fix_the_tool]] — fix the shared helper rather than working around it in one seeder.