Spaces:

cmoss3
/

RcmEmailAutomation

Sleeping

App Files Files Community

RcmEmailAutomation / asterion /04-async-execution.md

NujacsaintS

Asterion Seed MarkDown files

0f71b56 3 months ago

preview code

Raw

History Blame Contribute Delete

36.4 kB

Async Execution — Design Specification

Status: design approved; Phase 1 implementing as foundational substrate. Target rung: 2.5 (between projection completion and zoom-quantum). Scope: one foundational pillar. Implementation proceeds phase-by-phase under separate directives.

Framing (authoritative)

The pillar is ownership of execution, not "offline" as a feature.

Three layers, kept distinct:

Now Session — the user-facing scope. The browser tab, the compose bar, the field as it reads during an interactive moment. UX lives here. In Phase 1 this is UNCHANGED: prompts still feel like "I asked here, it is working here, this thread is mine now."
Shadow job — the execution substrate. A server-owned lifecycle record (sa_job) that tracks what the executor is actually doing. Introduced in Phase 1 as foundational infrastructure, with no UX surfacing. The seam that makes later decoupling possible without rewriting retries, state transitions, frontier selection, failure handling, or field updates.
Offline / Subconscious — a future user-facing mode built on the shadow-job substrate. Marketing layer for background / recoverable / long- running work. NOT introduced by Phase 1. Introduced only when a later directive surfaces it.

The instruction the rest of this spec serves:

Treat shadow jobs as foundational infrastructure introduced now, but keep their behavior scoped to the current Now Session until a later directive surfaces them as an explicit Offline/Subconscious capability.

0. Scope note

This spec describes async prompt execution as a first-class capability.

It does NOT cover:

The client-state race bug-fix (a separate, independent commit — closes a symptom of the sync model; lands before or alongside phase 1 here).
Implementation commits (each phase below becomes a directive; each directive becomes one or more commits).

It DOES cover:

What a prompt job is, and how it differs from the current request.
The canonical state machine.
The storage model.
The worker strategy (comparison + decision + rationale).
The client interaction model (comparison + phased choice).
A phased migration plan where each phase ships a working system.
The failure / retry / recovery model.
Explicit tradeoffs and risks.
Measurable success criteria.

1. Motivation and Current Limitations

Today

POST /sa-core/v1/prompt is synchronous. The REST thread calls SA_Executor::run, which calls the adapter via wp_remote_post, which blocks the PHP request for up to 180 seconds (after the recent timeout raise).
The browser tab owns the entire prompt lifecycle. The client's fetch() holds the connection; if the tab closes, the server continues running but the user loses all visibility into the outcome.
Two prompts submitted in parallel are two parallel PHP workers, each doing a long cURL wait, with zero coordination, no priority, and no recovery if one of them crashes.
The just-observed concurrent-submit race (two prompts attaching to the same parent) is a symptom: client-side activeCtxRootId is the only source of truth for "which node is the continuation root," so timing conflicts between user navigation and async resolution are inevitable.

Why this is structural, not cosmetic

Rung 5 (operator realism) is blocked. Tests that simulate load, capacity planning, reliability SLAs, and meaningful metrics all require async job execution. Every one of these is meaningless against a sync model.
Rung 4 (zoom-quantum engine) gets much harder on sync. The engine wants cheap state-transition re-renders. Pushing state through long sync calls + /mission/recent refreshes introduces races we'd have to hunt repeatedly.
Rungs 3 and 6 (self-hosting, carma-audit absorption) produce entity streams that shouldn't block HTTP. Scanner runs, doc ingestion, code ingestion — none of these should tie up a web request.
The tab is not a reliable process owner. Real work — long code generations, multi-step agent chains, background analyses — must survive a closed tab.
Browser-side state conflicts are fundamental with sync. The concurrent-submit race is one; future race surfaces (multi-user live editing, real-time collaboration) are coming. Moving execution state authority to the server closes these by construction.

2. Execution Model

What is a "prompt job"

A prompt job is the unit of work that executes the SA_Executor pipeline against a specific prompt entity. It is:

A durable record with its own UUID identity (distinct from the prompt entity).
A pointer to a persisted prompt entity (prompt_id).
A member of the queue with a state, a priority, and a retry budget.
Owned by the server, not the browser.
Inspectable, retriable, cancellable, supersedable — as a first-class operation.

How it differs from the current request model

Concern	Today (sync)	Proposed (async)
Who owns the pipeline's lifecycle?	The REST thread	The worker layer
Where is "in-progress" state?	PHP memory + browser memory	`sa_job` DB row
Does closing the tab kill the work?	No (it continues) but the user loses visibility	No — and the user regains visibility on reload
Can two prompts run simultaneously?	Only as two parallel PHP workers with no coordination	Yes, first-class, priority-ordered
Can a failed execution be retried cleanly?	Via `retry_prompt` after the fact	Built into the state machine
Can the user inspect queued / running work?	No	Yes (CLI phase 2; UI later)

Canonical lifecycle

Submit. POST /prompt creates:
- Prompt entity (as today, synchronously within the request — this is fast; the adapter call is what blocks).
- sa_job row with state='queued'.
- Returns 202 Accepted with prompt_id, job_id, and initial state. Target: response within 50ms.
Enqueue. Client adds the prompt to its entities[] array with the job's state, renders a "queued" drop. Client begins polling (phase 2) or listening via SSE (phase 4).
Claim. A worker polls for the highest-priority queued job. Transactional update: state='queued' → state='running', worker_claim=<uuid>, claimed_at=now.
Run. The worker invokes SA_Executor::run_claimed(job_id). This is a refactored executor entry-point that operates on an already-persisted prompt + job, not a fresh submit.
Complete. On adapter success, the worker writes the response entity, the flows-to edge, the ledger row, the structural audit row (as today), and transitions the job to state='succeeded' with response_id set.
Observe. Client sees state transitions via polling or SSE, updates the drop's visual state class accordingly.
Fail (if applicable). On adapter error, the worker classifies the error, applies retry policy, and either re-queues the job with backoff OR marks it state='failed'.

3. State Machine

States

State	Meaning	Terminal?
`queued`	Created; waiting for a worker	No
`running`	Claimed by a worker; adapter call in flight	No
`succeeded`	Adapter returned OK; response entity persisted	Yes
`failed`	Retry budget exhausted or non-retryable error	Yes
`cancelled`	User or operator intervention	Yes
`superseded`	Replaced by a newer job (e.g., user retried)	Yes

Once a job enters a terminal state, it does not change. A retry does not mutate an existing job — it creates a new job that references the old one via supersedes.

Transitions

queued      → running      : worker successfully claims
queued      → cancelled    : user/operator action before claim
running     → succeeded    : adapter OK + response entity written
running     → queued       : transient failure; attempt < max_attempts; apply backoff
running     → failed       : retry budget exhausted OR non-retryable error
running     → failed       : stuck (started > 15 min ago, no heartbeat) via cleanup
running     → cancelled    : operator kill (response discarded even if it arrives later)
succeeded   → superseded   : user-initiated retry replaces result
failed      → superseded   : user-initiated retry
cancelled   → superseded   : user-initiated retry after cancel

Heartbeat model

While state='running', the worker MUST update running_heartbeat_at every 30 seconds. A separate cleanup task (run via system cron every 5 minutes) reclaims jobs where running_heartbeat_at < now() - 5min back to state='queued' (if attempt < max_attempts) or to state='failed' (else).

Triggers

queued ← SA_Executor::enqueue (called from REST handler).
running ← worker process picks up a queued job and wins the claim race.
succeeded ← worker completes execution and commits.
failed ← worker exhausts retries OR cleanup declares job stuck.
cancelled ← CLI wp sa-core cancel-job <id> OR admin action.
superseded ← a new job is created referring to this one via supersedes.

4. Storage Model

Decision: new `sa_job` table

Not overloading sa_entity.

Rationale:

sa_entity represents authored knowledge: the user's prompt, the LLM's response, the derived context. It has one durable identity per artifact.
sa_job represents execution: a prompt may have multiple execution attempts (retries), each with its own claim, timing, errors, heartbeats. Conflating these with entity state confuses semantics and crowds the entity row.
Keeping them separate means: sa_entity stays durable knowledge; sa_job stays ephemeral execution records. Queries stay clean: "show me the user's recent prompts" is a sa_entity query; "show me what's queued" is a sa_job query.

Schema

CREATE TABLE {prefix}sa_job (
    id                       CHAR(36) NOT NULL,
    prompt_id                CHAR(36) NOT NULL,

    state                    VARCHAR(16) NOT NULL DEFAULT 'queued',
        -- queued | running | succeeded | failed | cancelled | superseded

    priority                 TINYINT UNSIGNED NOT NULL DEFAULT 50,
        -- lower number = higher priority
        -- 0-10: urgent (interactive, current-tab)
        -- 20-40: normal user-initiated background
        -- 50: default
        -- 60-90: bulk / scheduled / idle-time
        -- 100: lowest (opportunistic)

    attempt                  SMALLINT UNSIGNED NOT NULL DEFAULT 1,
    max_attempts             SMALLINT UNSIGNED NOT NULL DEFAULT 3,

    execution_options        LONGTEXT NOT NULL,
        -- JSON: { model?, prefer_provider?, depth?, router_model?, ... }

    worker_claim             VARCHAR(64) NULL,
    claimed_at               DATETIME NULL,
    started_at               DATETIME NULL,
    running_heartbeat_at     DATETIME NULL,
    finished_at              DATETIME NULL,

    response_id              CHAR(36) NULL,
        -- set on succeeded; points to the response entity

    error_kind               VARCHAR(32) NULL,
        -- timeout | rate_limit | provider_5xx | provider_4xx
        -- | parse | auth | worker_crash | stuck | unknown

    error_message            TEXT NULL,

    supersedes               CHAR(36) NULL,
        -- if this job replaces a previous attempt

    org_id                   BIGINT UNSIGNED NOT NULL,
    created_by               BIGINT UNSIGNED NOT NULL,
    created_at               DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at               DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
                             ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (id),
    KEY idx_state_priority_created (state, priority, created_at),
    KEY idx_prompt                 (prompt_id),
    KEY idx_worker_claim           (worker_claim),
    KEY idx_org_user_state         (org_id, created_by, state),
    KEY idx_heartbeat              (state, running_heartbeat_at)
)

Interaction with existing entity fields

Field	Lives on	Set when	Changed by jobs?
`correlation_id`	`sa_entity`	At prompt creation time (unchanged)	No
`parent_id`	`sa_entity`	At prompt creation time (continuation semantics, unchanged)	No
`causation_id`	`sa_entity`	At prompt creation time (unchanged)	No
`source_refs`	`sa_entity` (response)	Written by worker at job completion	Indirectly (job runs the executor which sets it)
`truth_class` / `state_class`	`sa_entity`	Set at write time (unchanged)	No

The sa_job table is orthogonal to the entity graph. Entities are durable structural records. Jobs are execution metadata.

Retry and supersession

When a user retries a failed or succeeded prompt:

Create a new sa_job row with the same prompt_id, new UUID, attempt=1, state=queued, supersedes=<old_job_id>.
The old job stays in place (terminal state preserved).
If the retry path requires cleaning up a prior response entity (as the current retry_prompt does for failed responses), that happens in the worker that claims the new job, NOT at enqueue time.

This model means every execution attempt is recorded and queryable. "Show me the history of this prompt" becomes SELECT * FROM sa_job WHERE prompt_id = X ORDER BY created_at.

5. Worker Strategy

Decision: custom worker, driven by WP-CLI + system cron

Comparison

Option A: WP-Cron alone — not chosen.

Pros: zero external dependencies.
Cons: WP-Cron spawns only on page visits (unreliable for low-traffic sites); each spawn is bound by max_execution_time; no native priority; no concurrency management. The "real cron hitting wp-cron.php" workaround mitigates the first point but the others remain.

Option B: Action Scheduler (the library) — fallback / future option.

Pros: mature, battle-tested by WooCommerce and Mailpoet, has built-in claim/retry/logs, comes with an admin UI, well-understood in the WP ecosystem.
Cons: adds a dependency (~200KB library, or plugin requirement); its schema and patterns are WooCommerce-flavored; it wraps our job concept with its own scheduling layer that we'd then have to thread through.
Worth reconsidering if we ever need to interoperate with a WooCommerce-running site that already uses AS.

Option C: Custom worker — chosen.

Rationale: our sa_job table is our job system. Its state machine is specific to our semantics (prompt-centric, projection-aware, org-scoped). Adding Action Scheduler on top means we'd write adapters between our table and theirs. A custom worker is ~300 lines of code, directly aligned with our data model.
Cons: we own the scheduler code. Edge cases (claim races, heartbeat recovery, backoff) must be solved by us. But: the surface is narrow, there's no distributed coordination to worry about (single-node plugin), and the patterns are well-known.

Custom worker implementation

wp sa-core run-worker [--once] [--batch-size=N] [--priority-ceiling=N]

Behavior in one iteration:

Generate or resume a worker_id for this process (UUID).

Claim a batch of jobs:

UPDATE sa_job
   SET state='running',
       worker_claim=<worker_id>,
       claimed_at=NOW(),
       started_at=NOW(),
       running_heartbeat_at=NOW()
 WHERE state='queued'
   AND priority <= <priority-ceiling>
 ORDER BY priority ASC, created_at ASC
 LIMIT <batch-size>

The single-statement UPDATE is the claim atomic: whichever worker's UPDATE lands first, wins.

Re-query the claimed rows by worker_claim=<worker_id> to get the actual job ids.
For each claimed job:
- Call SA_Executor::run_claimed(job_id).
- On success: set state='succeeded', response_id, finished_at.
- On retryable failure: if attempt < max_attempts, set state='queued' with backoff delay (encoded via created_at += backoff); else state='failed'.
- On non-retryable failure: state='failed' immediately.
Spawn a heartbeat thread/callback that updates running_heartbeat_at every 30s while the adapter is in flight.
Sleep 2 seconds. If --once, exit. Otherwise loop.

Driving the worker

Three supported modes:

Mode 1: Single-shot via system cron (recommended for production)

* * * * * wp sa-core run-worker --once --batch-size=5

Every minute, claim up to 5 jobs, process them, exit. Reliable, bounded, manageable.

Mode 2: Long-running service (optional for high-volume)

wp sa-core run-worker

Starts a persistent worker. Polls continuously. Should be supervised (systemd / supervisor) for restarts on crash.

Mode 3: WP-Cron via wp-cron.php (default; fallback) If a site has no system cron, register sa-core-process-queue as a WP-Cron event firing every minute. Relies on site traffic; acceptable for dev and low-traffic installs.

Heartbeat and stuck-job cleanup

Separate command:

wp sa-core reclaim-stuck-jobs

System cron: every 5 minutes.

Behavior:

Find jobs where state='running' AND running_heartbeat_at < now() - 5 minutes.
For each:
- If attempt < max_attempts AND started_at > now() - 15 minutes: reclaim to state='queued', clear worker_claim.
- Else: state='failed', error_kind='stuck'.

Double-execution safety

Risk: worker A claims a job, loses heartbeat, cleanup reclaims to queued, worker B claims and runs. Worker A's adapter call may still be in flight. Both workers could reach the "complete" step.

Mitigation: all state-finalization writes (state='succeeded', response_id, etc.) are conditioned on WHERE state='running' AND worker_claim=<my_worker_id>. Whichever update lands second sees no matching row and aborts silently. The first worker's result wins; the other worker's result is discarded (wasted work, but no data corruption).

6. Client Interaction Model

Decision: polling in phase 2, SSE in phase 4, WebSocket deferred

Comparison

Protocol	Pros	Cons	Chosen?
Polling	Simplest; works anywhere; zero new infra; trivial to implement	Latency up to poll interval; wastes bandwidth when idle (mitigated by only polling while in-flight jobs exist)	Phase 2
Server-Sent Events	Efficient for state updates; built into browsers (`EventSource`); one-way but that's what we need	Requires long-lived HTTP connection; may conflict with PHP `max_execution_time` on some hosts; harder to debug	Phase 4
WebSocket	Full-duplex; lowest latency; richest protocol	Needs separate infra (Node or Ratchet-in-PHP); significant deployment burden; overkill for our scale	Deferred

Polling protocol (phase 2)

Submit flow:

Client calls POST /prompt with prompt text + root_entity_id + etc.
Server returns 202 with { prompt_id, job_id, state: 'queued' }.
Client:
- Adds entity to entities[] with state='queued', renders as queued drop.
- Starts a poll loop if not already running.

Poll loop:

Every 2 seconds, while at least one in-flight (queued or running) job exists:
- Call GET /sa-core/v1/mission/recent?since=<iso8601>.
- Server returns entities with any updates since the timestamp.
- Client merges results by prompt_id (does not replace array wholesale).
- If all known prompts are terminal (succeeded, failed, cancelled, superseded) or no in-flight jobs remain, stop polling.

Server endpoint changes (phase 2):

GET /mission/recent accepts since=<iso8601>. Returns only prompts updated since that time (joins sa_job.updated_at into the "is this updated" test).
Response includes an as_of timestamp the client uses as the next since value.

Field state → visual class mapping

Job state	CSS class on drop	Existing / new	Visual
`queued`	`is-queued`	New	Slow breath, paler halo; "waiting" reading
`running`	`is-pending`	Existing	Current pending animation (dashed rotating halo)
`succeeded`	(none)	Existing	Default settled drop
`failed`	`is-orphan`	Existing	Muted salmon halo + retry chip
`cancelled`	`is-cancelled`	New	Greyed-out, no retry chip
`superseded`	hidden OR `.is-superseded`	New	Typically hidden; shown faded if a debug toggle enabled

Transitions:

queued → running: swap is-queued for is-pending.
running → succeeded: remove is-pending, merge real response data, renderField() for any layout adjustment.
running → failed: swap is-pending for is-orphan.

SSE protocol (phase 4, future)

Client opens EventSource('/sa-core/v1/mission/events') instead of polling.
Server streams events like event: job.state_changed\ndata: {"prompt_id":"…","state":"succeeded",…}\n\n whenever a job's state changes for this viewer.
Client falls back to polling if the EventSource fails.

WebSocket (deferred)

Adds separate infra (Node sidecar, Ratchet-in-PHP, or a managed service like Pusher). No concrete need yet. Revisit when multi-user real-time collaboration becomes a live concern.

7. Migration Plan

Principle: each phase ships a working system

No phase leaves the codebase in a half-broken state. At every phase boundary, a user can submit a prompt and see a result.

Phase 0 — Concurrent-submit race fix (prerequisite, separate scope)

Single commit, fixes the browser-state race in the current sync model.
Does NOT introduce async.
Makes parallel-submit testing reliable, which phase 2 needs.

Phase 1 — Schema + shadow jobs (1–2 commits, zero behavior change)

Goal: write sa_job rows alongside synchronous execution, so the data layer exists before the execution path changes.

Work:

Migration: create sa_job table (schema in §4).
SA_Executor::run remains synchronous. At the end, it writes a sa_job row with state='succeeded' (or 'failed' if the adapter errored).
No REST change. No client change. No worker yet.

Test: CLI proof submits a prompt, verifies an sa_job row with correct state exists and references the prompt.

Rollback: drop the table via migration down; no user impact.

Phase 2 — Async execution path (3–4 commits)

Goal: move adapter invocation out of the REST handler.

Work:

Refactor SA_Executor into two entry-points:
- SA_Executor::run — retained for direct synchronous use (CLI proofs, back-compat).
- SA_Executor::enqueue($prompt_text, $options) — writes prompt entity + queued job, returns ids. Fast.
- SA_Executor::run_claimed($job_id) — worker's entry-point; runs the pipeline against an existing prompt + job.
POST /prompt: call enqueue, return 202 with { prompt_id, job_id }.
Client submitPrompt: handle 202, render queued drop, start poll loop.
Server: GET /mission/recent accepts since param; returns updated rows.
Add worker CLI: wp sa-core run-worker.
Add heartbeat cleanup: wp sa-core reclaim-stuck-jobs.
New CSS classes: is-queued, is-cancelled.

Test:

CLI proof: submit a prompt, verify state=queued, run worker once, verify state=running → state=succeeded.
Concurrent-submit proof: submit two prompts with different priorities (10 and 90), run worker, verify priority=10 runs first.
Retry proof: submit, simulate transient failure (mock adapter), verify re-queue with backoff and eventual success.

Rollback: revert POST /prompt to call run synchronously. sa_job rows for already-queued jobs need manual cleanup (or a CLI command to replay them synchronously). Worker command stays but unused.

Phase 3 — Priority queue (1–2 commits)

Goal: expose priority so active-thread prompts outrank bulk / background work.

Work:

SA_Executor::enqueue accepts priority in options (default 50).
Client: no UI change. Still submits at default. Future bulk operations (Scanner, Code Lens ingestion, etc.) pass higher priority values.
Worker query already orders by priority — schema supports it from phase 1.

Test: submit 3 prompts with priorities {10, 50, 90}; verify worker claims in that order.

Rollback: trivial; remove priority from enqueue options, falls back to default.

Phase 4 — SSE layer (2–3 commits)

Goal: eliminate polling overhead.

Work:

Add GET /sa-core/v1/mission/events endpoint that streams SSE.
Server-side: when a job's state changes, broadcast an SSE event to that viewer's stream.
Client: detect EventSource support; subscribe if available; else continue polling.

Test: compare SSE event timing vs polling; verify identical state propagation; verify graceful fallback when connection drops.

Rollback: client falls back to polling automatically if SSE endpoint returns 404 or connection fails.

Phase 5 — Hardening (2–4 commits)

Goal: operator visibility, intervention, metrics.

Work:

wp sa-core list-jobs [--state=] [--org=] [--since=] — list jobs with filters.
wp sa-core job-details <job_id> — full history for a job.
wp sa-core cancel-job <job_id> — operator cancellation.
wp sa-core retry-job <job_id> — explicit requeue.
Optional: wp sa-core queue-metrics — count by state, average latency per org, error rate.

Test: CLI proofs for each command.

Phase 6 — async-native features (optional, post-pillar)

Built on the same sa_job infrastructure, no further schema changes:

Background bulk operations (Scanner, Code Lens, Doc Lens at scale).
Scheduled prompts (run at 2am) via a scheduled_for column on sa_job.
Multi-prompt chains (job B starts after job A succeeds).

Not part of the async pillar's completion criteria; listed for continuity.

Test discipline at each phase

Every phase ships one or more CLI proofs in the includes/cli/ pattern already established (geodesic-proof, promotion-proof, projection-proof, claim-continuation):

async-execution-proof: state progression queued → running → succeeded against a synthetic prompt.
priority-queue-proof: three synthetic prompts at different priorities; verify claim order.
retry-proof: simulated transient adapter failure; verify re-queue and eventual success.
heartbeat-proof: start a synthetic running job, sleep past heartbeat window, run reclaim, verify state.
concurrent-safety-proof: simulate double-claim race; verify only one worker's writes take effect.

8. Failure and Retry Model

Error classification

Every failure gets an error_kind:

`error_kind`	Meaning	Retryable?	Backoff
`timeout`	cURL 28 or executor-level timeout	Yes	Exponential (5s, 25s, 125s + jitter)
`rate_limit`	Adapter HTTP 429	Yes	Respect `Retry-After` header; else 60s min
`provider_5xx`	Adapter returned 5xx	Yes	Exponential
`provider_4xx` (non-429)	Adapter returned 4xx	No	N/A
`parse`	Response body couldn't be parsed	No	N/A
`auth`	Credentials invalid	No	N/A (needs operator)
`worker_crash`	Worker died mid-execution	Yes (via reclaim)	Immediate
`stuck`	Job has been running > 15 min	No	Terminal; mark failed
`unknown`	Catch-all for unclassified errors	Yes (conservative default)	Exponential

Retry rules

Default max_attempts = 3. Configurable per-enqueue via options.
Backoff stored as a delay on the re-queued job's created_at: created_at = now() + backoff. Worker query only claims jobs where created_at <= now(), so backoff is honored naturally.
Backoff durations:
- Attempt 2: 5s + random(0, 5s)
- Attempt 3: 25s + random(0, 15s)
- (After attempt 3 fails: terminal)

Timeout handling

Adapter timeout (180s): cURL 28. error_kind='timeout'. Retryable.
PHP max_execution_time (300s): worker process killed. Heartbeat lost. Reclaimed by reclaim-stuck-jobs.
Hard job ceiling (15 minutes): a running job started > 15 min ago with no recent heartbeat is declared stuck. error_kind='stuck'. Terminal.

Dead-letter handling

Jobs that exhaust max_attempts go to state='failed' with error_kind set.
They remain visible in the user's field as is-orphan drops (same treatment as today's orphans from sync-failure).
User can retry via the retry chip (same UX as today). Retry creates a new sa_job row with supersedes=<old_id>.
Operator CLI: wp sa-core list-jobs --state=failed surfaces recent failures.
Operator can run wp sa-core retry-job <id> to force a retry without user interaction.

Stuck-job recovery

wp sa-core reclaim-stuck-jobs runs via system cron every 5 minutes.
Query: state='running' AND running_heartbeat_at < now() - INTERVAL 5 MINUTE.
For each:
- If attempt < max_attempts AND started_at > now() - 15min: reclaim to state='queued', clear worker_claim.
- Else: state='failed', error_kind='stuck'.

Double-execution safety

Mitigation already detailed in §5: all finalization writes are conditioned on WHERE state='running' AND worker_claim=<my_worker_id>. Second-arriving writer sees no matching row and exits silently. First-arriving result wins.

Additional safeguard: response entity writes are keyed on a deterministic id derived from (prompt_id, adapter_request_id) when available, so duplicate writes collide on primary key and the second INSERT fails gracefully (caught by the finalization code).

9. Risks and Tradeoffs

Risk: cron reliability on managed hosts

WP Engine (and many managed hosts) run cron via wp-cron.php spawned on page requests. On low-traffic sites this is unreliable.

Mitigation. Document that production deployments SHOULD use a real system cron hitting wp sa-core run-worker --once. The command is standalone (doesn't depend on wp-cron's spawning). An on-page-load spawn continues to work for dev and low-volume sites.

Risk: worker compute resource pressure

A long-running worker process consumes PHP process slots. On shared hosting, this may conflict with web requests.

Mitigation. Default worker mode is --once (cron-driven, not service-style). Batch size is tunable. For high-volume users, the operator can tune cron frequency + batch size to stay within their host's limits.

Risk: `sa_job` table growth

Every prompt creates ≥ 1 job. Retries create additional jobs per prompt. Over time, the table grows unbounded.

Mitigation. Optional archival policy (phase 5 or later): jobs in a terminal state older than 30 days get archived or removed. wp sa-core archive-old-jobs --before=<date>. Indexes keep queries fast regardless of row count at expected volumes; archival is an optimization, not a correctness concern.

Risk: client polling overhead

Many users × many tabs × many in-flight jobs = many simultaneous polls.

Mitigation.

Polls happen only while in-flight jobs exist; idle tabs don't poll.
Poll interval is 2s; tunable.
Phase 4 (SSE) eliminates this risk entirely for modern browsers.

Risk: stale merges on client

If the client has an in-memory edit (user typing in compose) and a poll fetches fresh entity data, we must not discard user input.

Mitigation. Poll merges by prompt_id and only updates entity-state fields (response_id, state, context_ids, etc.). Compose-input state lives in the DOM, untouched by poll handling.

Risk: concurrent-submit out-of-order display

Two prompts submitted close together may complete in the reverse order (e.g., P1 is slow, P2 is fast). The field would render P2 before P1.

Mitigation. Field display order is by prompt.created_at, not job completion order. Stable regardless of async resolution order. The visual is-queued vs is-running vs settled state makes the running state transparent to the user.

Tradeoff: consistency window vs latency

Async means there is a window where client shows state='queued' while server has already transitioned to state='running'. Polling latency is up to 2s. SSE tightens this to sub-second.

Accepted. Up to 2s of client-side staleness is not a meaningful UX regression; phase 4 closes it.

Tradeoff: simplicity vs ecosystem reuse

Custom worker vs Action Scheduler. Custom is simpler for our narrow scope but means we own the scheduler. AS is richer but adds a dependency and adapter layer.

Accepted. Custom for phase 2. If we ever need to coexist with an AS-using site, the sa_job table's facade can be driven by either our worker or an AS callback without changing job semantics.

Risk: migrating users mid-transition

Between phase 1 (shadow jobs written) and phase 2 (async path switched on), a user's sa_job rows represent historical sync executions. When phase 2 lands, any prompt submitted mid-transition could have incomplete metadata.

Mitigation. Phase 2 rollout during a low-traffic window. On deploy, any sa_job rows with state='queued' from pre-phase-2 (there shouldn't be any, but defensively) are either processed by the first worker run or marked state='failed' with error_kind='migration' for manual review.

10. Success Criteria

The async pillar is complete (through phase 2) when:

POST /prompt returns within 100ms, 99th percentile, regardless of adapter latency.
Closing the browser tab during a prompt's execution does NOT cancel the job; reopening the site shows the completed result.
Two prompts submitted to different continuation parents from different tabs are processed correctly, with no race-induced data corruption.
A cURL timeout on the adapter causes an automatic retry with exponential backoff; a permanently-failing prompt surfaces as a retriable orphan after max_attempts exhaustion.
A crashed worker's job is reclaimed within 5 minutes and either completes on the next attempt or fails cleanly.
The CLI proofs described in §7's test discipline section all pass.
Existing /mission UI works throughout — no regressions on selection, trace, thread view, promote, verdict, continuation, retry, or content-collapse.
The wp sa-core run-worker command can be configured to run under standard system cron without supervisor-style infra.

The pillar is complete through phase 4 when additionally:

SSE delivers state updates within 200ms of server-side transition.
Polling mode still works identically as a fallback when SSE is unavailable.

11. Out of Scope (deliberately deferred)

Multi-node distributed coordination. Single-node assumption in claim locking. Revisit if horizontal scaling ever becomes a concern.
Priority inversion prevention. A long-running low-priority job blocking urgent jobs is theoretically possible but rare in our expected volume. Observe first; fix if it ever occurs.
User-facing queue inspection UI. CLI only through phase 5. A UI can be added later; sa_job data is already viewer-scoped via org_id + created_by indexing.
Scheduled prompts (scheduled_for): phase 6 (optional).
Multi-prompt chains (dependent jobs): phase 6 (optional).
Per-org quotas (max N concurrent running jobs): defer until multi-tenant scaling matters.
Cross-process worker coordination (multiple machines): defer until single-machine capacity is exhausted.
Real-time collaboration (two users editing the same field simultaneously): defer to the ACL / perspective-projection pillar.

12. Appendix: Summary of Design Decisions

Concern	Decision	Rationale
Storage	New `sa_job` table	Separate execution from knowledge; supports multiple attempts cleanly
Worker	Custom WP-CLI runner	Narrow scope; direct control; ~300 lines; no new dependency
Scheduling driver	System cron → `wp sa-core run-worker --once`	Reliable across hosts; standalone from wp-cron
State machine	6 states + heartbeat + supersession	Covers all observed outcomes; no ambiguity
Client protocol	Polling (phase 2) → SSE (phase 4)	Ships working async in phase 2; optimizes in phase 4
Retry	Exponential backoff, max_attempts=3	Handles transient failures; bounded pessimism
Rollback	Phase-by-phase; each has a defined revert	Safe to implement incrementally
Migration	Shadow jobs first (phase 1) → switch path (phase 2)	Data layer proven before execution switch

This spec is the contract. Implementation directives reference it by phase.

Async Execution — Design Specification

Framing (authoritative)

0. Scope note

1. Motivation and Current Limitations

Today

Why this is structural, not cosmetic

2. Execution Model

What is a "prompt job"

How it differs from the current request model

Canonical lifecycle

3. State Machine

States

Transitions

Heartbeat model

Triggers

4. Storage Model

Decision: new sa_job table

Schema

Interaction with existing entity fields

Retry and supersession

5. Worker Strategy

Decision: custom worker, driven by WP-CLI + system cron

Comparison

Custom worker implementation

Driving the worker

Heartbeat and stuck-job cleanup

Double-execution safety

6. Client Interaction Model

Decision: polling in phase 2, SSE in phase 4, WebSocket deferred

Comparison

Polling protocol (phase 2)

Field state → visual class mapping

SSE protocol (phase 4, future)

WebSocket (deferred)

7. Migration Plan

Principle: each phase ships a working system

Phase 0 — Concurrent-submit race fix (prerequisite, separate scope)

Phase 1 — Schema + shadow jobs (1–2 commits, zero behavior change)

Phase 2 — Async execution path (3–4 commits)

Phase 3 — Priority queue (1–2 commits)

Phase 4 — SSE layer (2–3 commits)

Phase 5 — Hardening (2–4 commits)

Phase 6 — async-native features (optional, post-pillar)

Test discipline at each phase

8. Failure and Retry Model

Error classification

Retry rules

Timeout handling

Dead-letter handling

Stuck-job recovery

Double-execution safety

9. Risks and Tradeoffs

Risk: cron reliability on managed hosts

Risk: worker compute resource pressure

Risk: sa_job table growth

Risk: client polling overhead

Risk: stale merges on client

Risk: concurrent-submit out-of-order display

Tradeoff: consistency window vs latency

Tradeoff: simplicity vs ecosystem reuse

Risk: migrating users mid-transition

10. Success Criteria

11. Out of Scope (deliberately deferred)

12. Appendix: Summary of Design Decisions

Decision: new `sa_job` table

Risk: `sa_job` table growth