asterion/04-async-execution.md

# Async Execution — Design Specification

Status: **design approved; Phase 1 implementing as foundational substrate.**
Target rung: 2.5 (between projection completion and zoom-quantum).
Scope: one foundational pillar. Implementation proceeds phase-by-phase under separate directives.

## Framing (authoritative)

The pillar is **ownership of execution**, not "offline" as a feature.

Three layers, kept distinct:

- **Now Session** — the user-facing scope. The browser tab, the compose bar,
  the field as it reads during an interactive moment. UX lives here. In
  Phase 1 this is UNCHANGED: prompts still feel like "I asked here, it is
  working here, this thread is mine now."

- **Shadow job** — the execution substrate. A server-owned lifecycle record
  (`sa_job`) that tracks what the executor is actually doing. Introduced in
  Phase 1 as foundational infrastructure, with no UX surfacing. The seam
  that makes later decoupling possible without rewriting retries, state
  transitions, frontier selection, failure handling, or field updates.

- **Offline / Subconscious** — a future user-facing mode built on the
  shadow-job substrate. Marketing layer for background / recoverable / long-
  running work. NOT introduced by Phase 1. Introduced only when a later
  directive surfaces it.

The instruction the rest of this spec serves:

> Treat shadow jobs as foundational infrastructure introduced now, but keep
> their behavior scoped to the current Now Session until a later directive
> surfaces them as an explicit Offline/Subconscious capability.

---

## 0. Scope note

This spec describes **async prompt execution** as a first-class capability.

It does NOT cover:
- The client-state race bug-fix (a separate, independent commit — closes a symptom of the sync model; lands before or alongside phase 1 here).
- Implementation commits (each phase below becomes a directive; each directive becomes one or more commits).

It DOES cover:
- What a prompt job is, and how it differs from the current request.
- The canonical state machine.
- The storage model.
- The worker strategy (comparison + decision + rationale).
- The client interaction model (comparison + phased choice).
- A phased migration plan where each phase ships a working system.
- The failure / retry / recovery model.
- Explicit tradeoffs and risks.
- Measurable success criteria.

---

## 1. Motivation and Current Limitations

### Today

- `POST /sa-core/v1/prompt` is synchronous. The REST thread calls `SA_Executor::run`, which calls the adapter via `wp_remote_post`, which blocks the PHP request for up to 180 seconds (after the recent timeout raise).
- The browser tab owns the entire prompt lifecycle. The client's `fetch()` holds the connection; if the tab closes, the server continues running but the user loses all visibility into the outcome.
- Two prompts submitted in parallel are two parallel PHP workers, each doing a long cURL wait, with zero coordination, no priority, and no recovery if one of them crashes.
- The just-observed concurrent-submit race (two prompts attaching to the same parent) is a symptom: client-side `activeCtxRootId` is the only source of truth for "which node is the continuation root," so timing conflicts between user navigation and async resolution are inevitable.

### Why this is structural, not cosmetic

1. **Rung 5 (operator realism) is blocked.** Tests that simulate load, capacity planning, reliability SLAs, and meaningful metrics all require async job execution. Every one of these is meaningless against a sync model.
2. **Rung 4 (zoom-quantum engine) gets much harder on sync.** The engine wants cheap state-transition re-renders. Pushing state through long sync calls + `/mission/recent` refreshes introduces races we'd have to hunt repeatedly.
3. **Rungs 3 and 6 (self-hosting, carma-audit absorption) produce entity streams that shouldn't block HTTP.** Scanner runs, doc ingestion, code ingestion — none of these should tie up a web request.
4. **The tab is not a reliable process owner.** Real work — long code generations, multi-step agent chains, background analyses — must survive a closed tab.
5. **Browser-side state conflicts are fundamental with sync.** The concurrent-submit race is one; future race surfaces (multi-user live editing, real-time collaboration) are coming. Moving execution state authority to the server closes these by construction.

---

## 2. Execution Model

### What is a "prompt job"

A **prompt job** is the unit of work that executes the `SA_Executor` pipeline against a specific prompt entity. It is:

- A durable record with its own UUID identity (distinct from the prompt entity).
- A pointer to a persisted prompt entity (`prompt_id`).
- A member of the queue with a state, a priority, and a retry budget.
- Owned by the server, not the browser.
- Inspectable, retriable, cancellable, supersedable — as a first-class operation.

### How it differs from the current request model

| Concern | Today (sync) | Proposed (async) |
|---|---|---|
| Who owns the pipeline's lifecycle? | The REST thread | The worker layer |
| Where is "in-progress" state? | PHP memory + browser memory | `sa_job` DB row |
| Does closing the tab kill the work? | No (it continues) but the user loses visibility | No — and the user regains visibility on reload |
| Can two prompts run simultaneously? | Only as two parallel PHP workers with no coordination | Yes, first-class, priority-ordered |
| Can a failed execution be retried cleanly? | Via `retry_prompt` after the fact | Built into the state machine |
| Can the user inspect queued / running work? | No | Yes (CLI phase 2; UI later) |

### Canonical lifecycle

1. **Submit.** `POST /prompt` creates:
   - Prompt entity (as today, synchronously within the request — this is fast; the adapter call is what blocks).
   - `sa_job` row with `state='queued'`.
   - Returns **`202 Accepted`** with `prompt_id`, `job_id`, and initial state. Target: response within 50ms.
2. **Enqueue.** Client adds the prompt to its `entities[]` array with the job's state, renders a "queued" drop. Client begins polling (phase 2) or listening via SSE (phase 4).
3. **Claim.** A worker polls for the highest-priority queued job. Transactional update: `state='queued'` → `state='running'`, `worker_claim=<uuid>`, `claimed_at=now`.
4. **Run.** The worker invokes `SA_Executor::run_claimed(job_id)`. This is a refactored executor entry-point that operates on an already-persisted prompt + job, not a fresh submit.
5. **Complete.** On adapter success, the worker writes the response entity, the `flows-to` edge, the ledger row, the structural audit row (as today), and transitions the job to `state='succeeded'` with `response_id` set.
6. **Observe.** Client sees state transitions via polling or SSE, updates the drop's visual state class accordingly.
7. **Fail (if applicable).** On adapter error, the worker classifies the error, applies retry policy, and either re-queues the job with backoff OR marks it `state='failed'`.

---

## 3. State Machine

### States

| State | Meaning | Terminal? |
|---|---|---|
| `queued` | Created; waiting for a worker | No |
| `running` | Claimed by a worker; adapter call in flight | No |
| `succeeded` | Adapter returned OK; response entity persisted | Yes |
| `failed` | Retry budget exhausted or non-retryable error | Yes |
| `cancelled` | User or operator intervention | Yes |
| `superseded` | Replaced by a newer job (e.g., user retried) | Yes |

Once a job enters a terminal state, it does not change. A retry does **not** mutate an existing job — it creates a **new job** that references the old one via `supersedes`.

### Transitions

```
queued      → running      : worker successfully claims
queued      → cancelled    : user/operator action before claim
running     → succeeded    : adapter OK + response entity written
running     → queued       : transient failure; attempt < max_attempts; apply backoff
running     → failed       : retry budget exhausted OR non-retryable error
running     → failed       : stuck (started > 15 min ago, no heartbeat) via cleanup
running     → cancelled    : operator kill (response discarded even if it arrives later)
succeeded   → superseded   : user-initiated retry replaces result
failed      → superseded   : user-initiated retry
cancelled   → superseded   : user-initiated retry after cancel
```

### Heartbeat model

While `state='running'`, the worker MUST update `running_heartbeat_at` every 30 seconds. A separate cleanup task (run via system cron every 5 minutes) reclaims jobs where `running_heartbeat_at < now() - 5min` back to `state='queued'` (if `attempt < max_attempts`) or to `state='failed'` (else).

### Triggers

- `queued` ← `SA_Executor::enqueue` (called from REST handler).
- `running` ← worker process picks up a `queued` job and wins the claim race.
- `succeeded` ← worker completes execution and commits.
- `failed` ← worker exhausts retries OR cleanup declares job stuck.
- `cancelled` ← CLI `wp sa-core cancel-job <id>` OR admin action.
- `superseded` ← a new job is created referring to this one via `supersedes`.

---

## 4. Storage Model

### Decision: new `sa_job` table

Not overloading `sa_entity`.

Rationale:
- `sa_entity` represents authored knowledge: the user's prompt, the LLM's response, the derived context. It has one durable identity per artifact.
- `sa_job` represents execution: a prompt may have multiple execution attempts (retries), each with its own claim, timing, errors, heartbeats. Conflating these with entity state confuses semantics and crowds the entity row.
- Keeping them separate means: `sa_entity` stays durable knowledge; `sa_job` stays ephemeral execution records. Queries stay clean: "show me the user's recent prompts" is a `sa_entity` query; "show me what's queued" is a `sa_job` query.

### Schema

```sql
CREATE TABLE {prefix}sa_job (
    id                       CHAR(36) NOT NULL,
    prompt_id                CHAR(36) NOT NULL,

    state                    VARCHAR(16) NOT NULL DEFAULT 'queued',
        -- queued | running | succeeded | failed | cancelled | superseded

    priority                 TINYINT UNSIGNED NOT NULL DEFAULT 50,
        -- lower number = higher priority
        -- 0-10: urgent (interactive, current-tab)
        -- 20-40: normal user-initiated background
        -- 50: default
        -- 60-90: bulk / scheduled / idle-time
        -- 100: lowest (opportunistic)

    attempt                  SMALLINT UNSIGNED NOT NULL DEFAULT 1,
    max_attempts             SMALLINT UNSIGNED NOT NULL DEFAULT 3,

    execution_options        LONGTEXT NOT NULL,
        -- JSON: { model?, prefer_provider?, depth?, router_model?, ... }

    worker_claim             VARCHAR(64) NULL,
    claimed_at               DATETIME NULL,
    started_at               DATETIME NULL,
    running_heartbeat_at     DATETIME NULL,
    finished_at              DATETIME NULL,

    response_id              CHAR(36) NULL,
        -- set on succeeded; points to the response entity

    error_kind               VARCHAR(32) NULL,
        -- timeout | rate_limit | provider_5xx | provider_4xx
        -- | parse | auth | worker_crash | stuck | unknown

    error_message            TEXT NULL,

    supersedes               CHAR(36) NULL,
        -- if this job replaces a previous attempt

    org_id                   BIGINT UNSIGNED NOT NULL,
    created_by               BIGINT UNSIGNED NOT NULL,
    created_at               DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at               DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
                             ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (id),
    KEY idx_state_priority_created (state, priority, created_at),
    KEY idx_prompt                 (prompt_id),
    KEY idx_worker_claim           (worker_claim),
    KEY idx_org_user_state         (org_id, created_by, state),
    KEY idx_heartbeat              (state, running_heartbeat_at)
)
```

### Interaction with existing entity fields

| Field | Lives on | Set when | Changed by jobs? |
|---|---|---|---|
| `correlation_id` | `sa_entity` | At prompt creation time (unchanged) | No |
| `parent_id` | `sa_entity` | At prompt creation time (continuation semantics, unchanged) | No |
| `causation_id` | `sa_entity` | At prompt creation time (unchanged) | No |
| `source_refs` | `sa_entity` (response) | Written by worker at job completion | Indirectly (job runs the executor which sets it) |
| `truth_class` / `state_class` | `sa_entity` | Set at write time (unchanged) | No |

The `sa_job` table is orthogonal to the entity graph. Entities are durable structural records. Jobs are execution metadata.

### Retry and supersession

When a user retries a failed or succeeded prompt:
1. Create a **new** `sa_job` row with the same `prompt_id`, new UUID, `attempt=1`, `state=queued`, `supersedes=<old_job_id>`.
2. The old job stays in place (terminal state preserved).
3. If the retry path requires cleaning up a prior response entity (as the current `retry_prompt` does for failed responses), that happens in the worker that claims the new job, NOT at enqueue time.

This model means every execution attempt is recorded and queryable. "Show me the history of this prompt" becomes `SELECT * FROM sa_job WHERE prompt_id = X ORDER BY created_at`.

---

## 5. Worker Strategy

### Decision: custom worker, driven by WP-CLI + system cron

### Comparison

**Option A: WP-Cron alone** — not chosen.
- Pros: zero external dependencies.
- Cons: WP-Cron spawns only on page visits (unreliable for low-traffic sites); each spawn is bound by `max_execution_time`; no native priority; no concurrency management. The "real cron hitting `wp-cron.php`" workaround mitigates the first point but the others remain.

**Option B: Action Scheduler (the library)** — fallback / future option.
- Pros: mature, battle-tested by WooCommerce and Mailpoet, has built-in claim/retry/logs, comes with an admin UI, well-understood in the WP ecosystem.
- Cons: adds a dependency (~200KB library, or plugin requirement); its schema and patterns are WooCommerce-flavored; it wraps our job concept with its own scheduling layer that we'd then have to thread through.
- Worth reconsidering if we ever need to interoperate with a WooCommerce-running site that already uses AS.

**Option C: Custom worker** — chosen.
- Rationale: our `sa_job` table **is** our job system. Its state machine is specific to our semantics (prompt-centric, projection-aware, org-scoped). Adding Action Scheduler on top means we'd write adapters between our table and theirs. A custom worker is ~300 lines of code, directly aligned with our data model.
- Cons: we own the scheduler code. Edge cases (claim races, heartbeat recovery, backoff) must be solved by us. But: the surface is narrow, there's no distributed coordination to worry about (single-node plugin), and the patterns are well-known.

### Custom worker implementation

```
wp sa-core run-worker [--once] [--batch-size=N] [--priority-ceiling=N]
```

Behavior in one iteration:
1. Generate or resume a `worker_id` for this process (UUID).
2. Claim a batch of jobs:
   ```sql
   UPDATE sa_job
      SET state='running',
          worker_claim=<worker_id>,
          claimed_at=NOW(),
          started_at=NOW(),
          running_heartbeat_at=NOW()
    WHERE state='queued'
      AND priority <= <priority-ceiling>
    ORDER BY priority ASC, created_at ASC
    LIMIT <batch-size>
   ```
   The single-statement UPDATE is the claim atomic: whichever worker's UPDATE lands first, wins.
3. Re-query the claimed rows by `worker_claim=<worker_id>` to get the actual job ids.
4. For each claimed job:
   - Call `SA_Executor::run_claimed(job_id)`.
   - On success: set state='succeeded', response_id, finished_at.
   - On retryable failure: if attempt < max_attempts, set state='queued' with backoff delay (encoded via `created_at += backoff`); else state='failed'.
   - On non-retryable failure: state='failed' immediately.
5. Spawn a heartbeat thread/callback that updates `running_heartbeat_at` every 30s while the adapter is in flight.
6. Sleep 2 seconds. If `--once`, exit. Otherwise loop.

### Driving the worker

Three supported modes:

**Mode 1: Single-shot via system cron** (recommended for production)
```cron
* * * * * wp sa-core run-worker --once --batch-size=5
```
Every minute, claim up to 5 jobs, process them, exit. Reliable, bounded, manageable.

**Mode 2: Long-running service** (optional for high-volume)
```
wp sa-core run-worker
```
Starts a persistent worker. Polls continuously. Should be supervised (systemd / supervisor) for restarts on crash.

**Mode 3: WP-Cron via wp-cron.php** (default; fallback)
If a site has no system cron, register `sa-core-process-queue` as a WP-Cron event firing every minute. Relies on site traffic; acceptable for dev and low-traffic installs.

### Heartbeat and stuck-job cleanup

Separate command:
```
wp sa-core reclaim-stuck-jobs
```
System cron: every 5 minutes.

Behavior:
- Find jobs where `state='running'` AND `running_heartbeat_at < now() - 5 minutes`.
- For each:
  - If `attempt < max_attempts` AND `started_at > now() - 15 minutes`: reclaim to `state='queued'`, clear `worker_claim`.
  - Else: `state='failed'`, `error_kind='stuck'`.

### Double-execution safety

Risk: worker A claims a job, loses heartbeat, cleanup reclaims to queued, worker B claims and runs. Worker A's adapter call may still be in flight. Both workers could reach the "complete" step.

Mitigation: all state-finalization writes (`state='succeeded'`, `response_id`, etc.) are conditioned on `WHERE state='running' AND worker_claim=<my_worker_id>`. Whichever update lands second sees no matching row and aborts silently. The first worker's result wins; the other worker's result is discarded (wasted work, but no data corruption).

---

## 6. Client Interaction Model

### Decision: polling in phase 2, SSE in phase 4, WebSocket deferred

### Comparison

| Protocol | Pros | Cons | Chosen? |
|---|---|---|---|
| **Polling** | Simplest; works anywhere; zero new infra; trivial to implement | Latency up to poll interval; wastes bandwidth when idle (mitigated by only polling while in-flight jobs exist) | Phase 2 |
| **Server-Sent Events** | Efficient for state updates; built into browsers (`EventSource`); one-way but that's what we need | Requires long-lived HTTP connection; may conflict with PHP `max_execution_time` on some hosts; harder to debug | Phase 4 |
| **WebSocket** | Full-duplex; lowest latency; richest protocol | Needs separate infra (Node or Ratchet-in-PHP); significant deployment burden; overkill for our scale | Deferred |

### Polling protocol (phase 2)

**Submit flow:**

1. Client calls `POST /prompt` with prompt text + root_entity_id + etc.
2. Server returns `202` with `{ prompt_id, job_id, state: 'queued' }`.
3. Client:
   - Adds entity to `entities[]` with `state='queued'`, renders as queued drop.
   - Starts a poll loop if not already running.

**Poll loop:**

- Every 2 seconds, while at least one in-flight (`queued` or `running`) job exists:
  - Call `GET /sa-core/v1/mission/recent?since=<iso8601>`.
  - Server returns entities with any updates since the timestamp.
  - Client merges results by `prompt_id` (does not replace array wholesale).
  - If all known prompts are terminal (succeeded, failed, cancelled, superseded) or no in-flight jobs remain, stop polling.

**Server endpoint changes (phase 2):**

- `GET /mission/recent` accepts `since=<iso8601>`. Returns only prompts updated since that time (joins `sa_job.updated_at` into the "is this updated" test).
- Response includes an `as_of` timestamp the client uses as the next `since` value.

### Field state → visual class mapping

| Job state | CSS class on drop | Existing / new | Visual |
|---|---|---|---|
| `queued` | `is-queued` | **New** | Slow breath, paler halo; "waiting" reading |
| `running` | `is-pending` | Existing | Current pending animation (dashed rotating halo) |
| `succeeded` | (none) | Existing | Default settled drop |
| `failed` | `is-orphan` | Existing | Muted salmon halo + retry chip |
| `cancelled` | `is-cancelled` | **New** | Greyed-out, no retry chip |
| `superseded` | hidden OR `.is-superseded` | **New** | Typically hidden; shown faded if a debug toggle enabled |

Transitions:
- `queued → running`: swap `is-queued` for `is-pending`.
- `running → succeeded`: remove `is-pending`, merge real response data, `renderField()` for any layout adjustment.
- `running → failed`: swap `is-pending` for `is-orphan`.

### SSE protocol (phase 4, future)

- Client opens `EventSource('/sa-core/v1/mission/events')` instead of polling.
- Server streams events like `event: job.state_changed\ndata: {"prompt_id":"…","state":"succeeded",…}\n\n` whenever a job's state changes for this viewer.
- Client falls back to polling if the `EventSource` fails.

### WebSocket (deferred)

Adds separate infra (Node sidecar, Ratchet-in-PHP, or a managed service like Pusher). No concrete need yet. Revisit when multi-user real-time collaboration becomes a live concern.

---

## 7. Migration Plan

### Principle: each phase ships a working system

No phase leaves the codebase in a half-broken state. At every phase boundary, a user can submit a prompt and see a result.

### Phase 0 — Concurrent-submit race fix (prerequisite, separate scope)

- Single commit, fixes the browser-state race in the current sync model.
- Does NOT introduce async.
- Makes parallel-submit testing reliable, which phase 2 needs.

### Phase 1 — Schema + shadow jobs (1–2 commits, zero behavior change)

Goal: write `sa_job` rows alongside synchronous execution, so the data layer exists before the execution path changes.

Work:
- Migration: create `sa_job` table (schema in §4).
- `SA_Executor::run` remains synchronous. At the end, it writes a `sa_job` row with `state='succeeded'` (or `'failed'` if the adapter errored).
- No REST change. No client change. No worker yet.

Test: CLI proof submits a prompt, verifies an `sa_job` row with correct state exists and references the prompt.

Rollback: drop the table via migration down; no user impact.

### Phase 2 — Async execution path (3–4 commits)

Goal: move adapter invocation out of the REST handler.

Work:
- Refactor `SA_Executor` into two entry-points:
  - `SA_Executor::run` — retained for direct synchronous use (CLI proofs, back-compat).
  - `SA_Executor::enqueue($prompt_text, $options)` — writes prompt entity + queued job, returns ids. Fast.
  - `SA_Executor::run_claimed($job_id)` — worker's entry-point; runs the pipeline against an existing prompt + job.
- `POST /prompt`: call `enqueue`, return `202` with `{ prompt_id, job_id }`.
- Client `submitPrompt`: handle `202`, render queued drop, start poll loop.
- Server: `GET /mission/recent` accepts `since` param; returns updated rows.
- Add worker CLI: `wp sa-core run-worker`.
- Add heartbeat cleanup: `wp sa-core reclaim-stuck-jobs`.
- New CSS classes: `is-queued`, `is-cancelled`.

Test:
- CLI proof: submit a prompt, verify `state=queued`, run worker once, verify `state=running` → `state=succeeded`.
- Concurrent-submit proof: submit two prompts with different priorities (10 and 90), run worker, verify priority=10 runs first.
- Retry proof: submit, simulate transient failure (mock adapter), verify re-queue with backoff and eventual success.

Rollback: revert `POST /prompt` to call `run` synchronously. `sa_job` rows for already-queued jobs need manual cleanup (or a CLI command to replay them synchronously). Worker command stays but unused.

### Phase 3 — Priority queue (1–2 commits)

Goal: expose priority so active-thread prompts outrank bulk / background work.

Work:
- `SA_Executor::enqueue` accepts `priority` in options (default 50).
- Client: no UI change. Still submits at default. Future bulk operations (Scanner, Code Lens ingestion, etc.) pass higher priority values.
- Worker query already orders by priority — schema supports it from phase 1.

Test: submit 3 prompts with priorities {10, 50, 90}; verify worker claims in that order.

Rollback: trivial; remove priority from enqueue options, falls back to default.

### Phase 4 — SSE layer (2–3 commits)

Goal: eliminate polling overhead.

Work:
- Add `GET /sa-core/v1/mission/events` endpoint that streams SSE.
- Server-side: when a job's state changes, broadcast an SSE event to that viewer's stream.
- Client: detect `EventSource` support; subscribe if available; else continue polling.

Test: compare SSE event timing vs polling; verify identical state propagation; verify graceful fallback when connection drops.

Rollback: client falls back to polling automatically if SSE endpoint returns 404 or connection fails.

### Phase 5 — Hardening (2–4 commits)

Goal: operator visibility, intervention, metrics.

Work:
- `wp sa-core list-jobs [--state=] [--org=] [--since=]` — list jobs with filters.
- `wp sa-core job-details <job_id>` — full history for a job.
- `wp sa-core cancel-job <job_id>` — operator cancellation.
- `wp sa-core retry-job <job_id>` — explicit requeue.
- Optional: `wp sa-core queue-metrics` — count by state, average latency per org, error rate.

Test: CLI proofs for each command.

### Phase 6 — async-native features (optional, post-pillar)

Built on the same `sa_job` infrastructure, no further schema changes:

- Background bulk operations (Scanner, Code Lens, Doc Lens at scale).
- Scheduled prompts (run at 2am) via a `scheduled_for` column on `sa_job`.
- Multi-prompt chains (job B starts after job A succeeds).

Not part of the async pillar's completion criteria; listed for continuity.

### Test discipline at each phase

Every phase ships one or more CLI proofs in the `includes/cli/` pattern already established (geodesic-proof, promotion-proof, projection-proof, claim-continuation):

- `async-execution-proof`: state progression queued → running → succeeded against a synthetic prompt.
- `priority-queue-proof`: three synthetic prompts at different priorities; verify claim order.
- `retry-proof`: simulated transient adapter failure; verify re-queue and eventual success.
- `heartbeat-proof`: start a synthetic running job, sleep past heartbeat window, run reclaim, verify state.
- `concurrent-safety-proof`: simulate double-claim race; verify only one worker's writes take effect.

---

## 8. Failure and Retry Model

### Error classification

Every failure gets an `error_kind`:

| `error_kind` | Meaning | Retryable? | Backoff |
|---|---|---|---|
| `timeout` | cURL 28 or executor-level timeout | Yes | Exponential (5s, 25s, 125s + jitter) |
| `rate_limit` | Adapter HTTP 429 | Yes | Respect `Retry-After` header; else 60s min |
| `provider_5xx` | Adapter returned 5xx | Yes | Exponential |
| `provider_4xx` (non-429) | Adapter returned 4xx | No | N/A |
| `parse` | Response body couldn't be parsed | No | N/A |
| `auth` | Credentials invalid | No | N/A (needs operator) |
| `worker_crash` | Worker died mid-execution | Yes (via reclaim) | Immediate |
| `stuck` | Job has been running > 15 min | No | Terminal; mark failed |
| `unknown` | Catch-all for unclassified errors | Yes (conservative default) | Exponential |

### Retry rules

- Default `max_attempts = 3`. Configurable per-enqueue via options.
- Backoff stored as a delay on the re-queued job's `created_at`: `created_at = now() + backoff`. Worker query only claims jobs where `created_at <= now()`, so backoff is honored naturally.
- Backoff durations:
  - Attempt 2: 5s + random(0, 5s)
  - Attempt 3: 25s + random(0, 15s)
  - (After attempt 3 fails: terminal)

### Timeout handling

- **Adapter timeout (180s)**: cURL 28. `error_kind='timeout'`. Retryable.
- **PHP `max_execution_time` (300s)**: worker process killed. Heartbeat lost. Reclaimed by `reclaim-stuck-jobs`.
- **Hard job ceiling (15 minutes)**: a running job started > 15 min ago with no recent heartbeat is declared stuck. `error_kind='stuck'`. Terminal.

### Dead-letter handling

- Jobs that exhaust `max_attempts` go to `state='failed'` with `error_kind` set.
- They remain visible in the user's field as `is-orphan` drops (same treatment as today's orphans from sync-failure).
- User can retry via the retry chip (same UX as today). Retry creates a new `sa_job` row with `supersedes=<old_id>`.
- Operator CLI: `wp sa-core list-jobs --state=failed` surfaces recent failures.
- Operator can run `wp sa-core retry-job <id>` to force a retry without user interaction.

### Stuck-job recovery

- `wp sa-core reclaim-stuck-jobs` runs via system cron every 5 minutes.
- Query: `state='running' AND running_heartbeat_at < now() - INTERVAL 5 MINUTE`.
- For each:
  - If `attempt < max_attempts` AND `started_at > now() - 15min`: reclaim to `state='queued'`, clear `worker_claim`.
  - Else: `state='failed'`, `error_kind='stuck'`.

### Double-execution safety

Mitigation already detailed in §5: all finalization writes are conditioned on `WHERE state='running' AND worker_claim=<my_worker_id>`. Second-arriving writer sees no matching row and exits silently. First-arriving result wins.

Additional safeguard: response entity writes are keyed on a deterministic id derived from `(prompt_id, adapter_request_id)` when available, so duplicate writes collide on primary key and the second `INSERT` fails gracefully (caught by the finalization code).

---

## 9. Risks and Tradeoffs

### Risk: cron reliability on managed hosts

WP Engine (and many managed hosts) run cron via `wp-cron.php` spawned on page requests. On low-traffic sites this is unreliable.

**Mitigation.** Document that production deployments SHOULD use a real system cron hitting `wp sa-core run-worker --once`. The command is standalone (doesn't depend on wp-cron's spawning). An on-page-load spawn continues to work for dev and low-volume sites.

### Risk: worker compute resource pressure

A long-running worker process consumes PHP process slots. On shared hosting, this may conflict with web requests.

**Mitigation.** Default worker mode is `--once` (cron-driven, not service-style). Batch size is tunable. For high-volume users, the operator can tune cron frequency + batch size to stay within their host's limits.

### Risk: `sa_job` table growth

Every prompt creates ≥ 1 job. Retries create additional jobs per prompt. Over time, the table grows unbounded.

**Mitigation.** Optional archival policy (phase 5 or later): jobs in a terminal state older than 30 days get archived or removed. `wp sa-core archive-old-jobs --before=<date>`. Indexes keep queries fast regardless of row count at expected volumes; archival is an optimization, not a correctness concern.

### Risk: client polling overhead

Many users × many tabs × many in-flight jobs = many simultaneous polls.

**Mitigation.**
- Polls happen only while in-flight jobs exist; idle tabs don't poll.
- Poll interval is 2s; tunable.
- Phase 4 (SSE) eliminates this risk entirely for modern browsers.

### Risk: stale merges on client

If the client has an in-memory edit (user typing in compose) and a poll fetches fresh entity data, we must not discard user input.

**Mitigation.** Poll merges by `prompt_id` and only updates entity-state fields (response_id, state, context_ids, etc.). Compose-input state lives in the DOM, untouched by poll handling.

### Risk: concurrent-submit out-of-order display

Two prompts submitted close together may complete in the reverse order (e.g., P1 is slow, P2 is fast). The field would render P2 before P1.

**Mitigation.** Field display order is by `prompt.created_at`, not job completion order. Stable regardless of async resolution order. The visual `is-queued` vs `is-running` vs settled state makes the running state transparent to the user.

### Tradeoff: consistency window vs latency

Async means there is a window where client shows `state='queued'` while server has already transitioned to `state='running'`. Polling latency is up to 2s. SSE tightens this to sub-second.

**Accepted.** Up to 2s of client-side staleness is not a meaningful UX regression; phase 4 closes it.

### Tradeoff: simplicity vs ecosystem reuse

Custom worker vs Action Scheduler. Custom is simpler for our narrow scope but means we own the scheduler. AS is richer but adds a dependency and adapter layer.

**Accepted.** Custom for phase 2. If we ever need to coexist with an AS-using site, the `sa_job` table's facade can be driven by either our worker or an AS callback without changing job semantics.

### Risk: migrating users mid-transition

Between phase 1 (shadow jobs written) and phase 2 (async path switched on), a user's `sa_job` rows represent historical sync executions. When phase 2 lands, any prompt submitted mid-transition could have incomplete metadata.

**Mitigation.** Phase 2 rollout during a low-traffic window. On deploy, any `sa_job` rows with `state='queued'` from pre-phase-2 (there shouldn't be any, but defensively) are either processed by the first worker run or marked `state='failed'` with `error_kind='migration'` for manual review.

---

## 10. Success Criteria

The async pillar is complete (through phase 2) when:

1. `POST /prompt` returns within 100ms, 99th percentile, regardless of adapter latency.
2. Closing the browser tab during a prompt's execution does NOT cancel the job; reopening the site shows the completed result.
3. Two prompts submitted to different continuation parents from different tabs are processed correctly, with no race-induced data corruption.
4. A cURL timeout on the adapter causes an automatic retry with exponential backoff; a permanently-failing prompt surfaces as a retriable orphan after `max_attempts` exhaustion.
5. A crashed worker's job is reclaimed within 5 minutes and either completes on the next attempt or fails cleanly.
6. The CLI proofs described in §7's test discipline section all pass.
7. Existing `/mission` UI works throughout — no regressions on selection, trace, thread view, promote, verdict, continuation, retry, or content-collapse.
8. The `wp sa-core run-worker` command can be configured to run under standard system cron without supervisor-style infra.

The pillar is **complete through phase 4** when additionally:

9. SSE delivers state updates within 200ms of server-side transition.
10. Polling mode still works identically as a fallback when SSE is unavailable.

---

## 11. Out of Scope (deliberately deferred)

- **Multi-node distributed coordination.** Single-node assumption in claim locking. Revisit if horizontal scaling ever becomes a concern.
- **Priority inversion prevention.** A long-running low-priority job blocking urgent jobs is theoretically possible but rare in our expected volume. Observe first; fix if it ever occurs.
- **User-facing queue inspection UI.** CLI only through phase 5. A UI can be added later; `sa_job` data is already viewer-scoped via `org_id + created_by` indexing.
- **Scheduled prompts** (`scheduled_for`): phase 6 (optional).
- **Multi-prompt chains** (dependent jobs): phase 6 (optional).
- **Per-org quotas** (max N concurrent running jobs): defer until multi-tenant scaling matters.
- **Cross-process worker coordination** (multiple machines): defer until single-machine capacity is exhausted.
- **Real-time collaboration** (two users editing the same field simultaneously): defer to the ACL / perspective-projection pillar.

---

## 12. Appendix: Summary of Design Decisions

| Concern | Decision | Rationale |
|---|---|---|
| Storage | New `sa_job` table | Separate execution from knowledge; supports multiple attempts cleanly |
| Worker | Custom WP-CLI runner | Narrow scope; direct control; ~300 lines; no new dependency |
| Scheduling driver | System cron → `wp sa-core run-worker --once` | Reliable across hosts; standalone from wp-cron |
| State machine | 6 states + heartbeat + supersession | Covers all observed outcomes; no ambiguity |
| Client protocol | Polling (phase 2) → SSE (phase 4) | Ships working async in phase 2; optimizes in phase 4 |
| Retry | Exponential backoff, max_attempts=3 | Handles transient failures; bounded pessimism |
| Rollback | Phase-by-phase; each has a defined revert | Safe to implement incrementally |
| Migration | Shadow jobs first (phase 1) → switch path (phase 2) | Data layer proven before execution switch |

This spec is the contract. Implementation directives reference it by phase.