Make agent state durable without live bucket writes (#25)

* Checkpoint agent state off the bucket mount

* Fix state checkpoints and resumed input delivery

---------

Co-authored-by: Agent Manager <agents@agent-manager.local>

Dockerfile

@@ -17,7 +17,7 @@ FROM node:22-bookworm AS runtime
 # agents and humans reach for (jq/htop/sqlite3/editors/media, fonts so headless
 # Chromium screenshots don't render tofu).
 RUN apt-get update && apt-get install -y --no-install-recommends \
-      tmux git git-lfs ca-certificates curl python3 make g++ ripgrep bubblewrap rsync \
+      tmux git git-lfs ca-certificates curl python3 make g++ ripgrep bubblewrap rsync util-linux \
       jq htop lsof tree ncdu sqlite3 vim nano zip unzip file procps less \
       ffmpeg imagemagick fonts-liberation fonts-noto-color-emoji \
     && rm -rf /var/lib/apt/lists/* \

README.md

@@ -109,8 +109,11 @@ api.restart_space(space_id)
 
 Everything durable lives under `/data`: `sessions.json`, `groups.json`,
 `workspaces/<path>/` (agent working dirs + shared `skills/`), and each
-CLI's config/credentials/history (`HOME=/data/home`, plus `CLAUDE_CONFIG_DIR`
-and `CODEX_HOME` under `/data/state`).
+CLI's closed state checkpoints under `/data/state`. Active harness state lives
+on local POSIX storage and is restored/checkpointed by
+[`scripts/agent-state.sh`](scripts/agent-state.sh); SQLite harnesses use online
+database backups rather than copying live WAL files. See
+[`docs/agent-state-checkpoints.md`](docs/agent-state-checkpoints.md).
 
 ## Architecture
 

docs/agent-state-checkpoints.md

@@ -0,0 +1,132 @@
+# Agent state checkpoints
+
+## Problem
+
+The Space's `/data` volume is an hf-mount FUSE view over mutable object
+storage, not a POSIX disk. Its default streaming writer buffers an open file in
+memory and uploads it on close. That is a bad match for Codex, which holds one
+rollout open for an entire resumed session: an unclean process/container stop
+can discard every append since the previous open epoch closed.
+
+The old layout also copied opencode and Hermes SQLite databases, WALs, and SHMs
+with `rsync`. Those files can be observed at different transaction boundaries;
+having all three files is not proof that the copy is a consistent database.
+
+## Invariants
+
+1. An agent only mutates ordinary files on `$AM_LOCAL`.
+2. The bucket only receives writes from a short-lived checkpoint operation.
+3. A published SQLite checkpoint comes from SQLite's online backup API and
+   passes `PRAGMA quick_check` before upload.
+4. The previous durable checkpoint remains authoritative until its replacement
+   has been completely written and closed.
+5. A hot/dev restart never restores an older bucket copy over newer local
+   state (`rsync --update`).
+6. A normal shutdown stops the server and performs a final checkpoint. An
+   ungraceful stop loses at most the checkpoint interval (15 seconds by
+   default), not the lifetime of an open transcript.
+
+## Layout
+
+| Harness | Live state | Durable checkpoint | Adapter |
+| --- | --- | --- | --- |
+| Codex | `$AM_LOCAL/codex-home` | `/data/state/codex` | file tree; local SQLite cache excluded |
+| Claude Code | `$AM_LOCAL/agent-state/claude` | `/data/state/claude` | file tree |
+| Gemini CLI | `$AM_LOCAL/agent-state/gemini-home/.gemini` | `/data/state/gemini` | file tree |
+| OpenClaw | `$AM_LOCAL/oc-home/.openclaw` | `/data/state/openclaw-backup` | file tree |
+| opencode | `$AM_LOCAL/opencode-share` | `/data/state/opencode` | file tree plus online `opencode.db` backup |
+| Hermes | `$AM_LOCAL/hermes` | `/data/state/hermes` | file tree plus online `state.db` backup |
+
+Remote agents already use a good object-store pattern: one closed, immutable
+Markdown file per message. Shell sessions have no model transcript; workspace
+files remain durable but live process state and terminal scrollback are outside
+this checkpoint mechanism.
+
+## Lifecycle
+
+`scripts/agent-state.sh restore` runs after legacy-path migration and before the
+server starts. For file trees, durable files fill an empty local tree but do
+not overwrite newer local files left by an in-container dev restart. For
+SQLite harnesses, a valid existing local database is authoritative on a hot
+restart; otherwise a verified checkpoint database is preferred. The previous
+raw DB/WAL layout is accepted only as a one-release migration fallback. The
+server refuses to start if restore is incomplete, avoiding a new lineage from
+being written over partially restored state.
+
+While the server is running, one supervisor loop calls
+`scripts/agent-state.sh checkpoint` every
+`$AGENT_STATE_CHECKPOINT_SECONDS` (default 15). A local `flock` prevents a
+timer checkpoint from overlapping the shutdown checkpoint.
+
+Ordinary trees use rsync's temporary-destination/rename behavior. The FUSE
+writer therefore closes the replacement before it becomes the canonical
+object. A local per-adapter timestamp selects changed paths and feeds those
+paths to `rsync --files-from`; periodic checkpoints walk only the fast local
+tree and never enumerate the remote bucket tree. Files modified while a
+checkpoint is running are deliberately selected again on the next pass.
+When their database or WAL changed, opencode and Hermes run `.backup` to a
+local staging DB, validate that DB, then publish the closed result under
+`checkpoints/`; idle databases do not generate bucket writes.
+
+The first version does not propagate deletions from live trees. A stale durable
+file is safer than deleting history based on a transient or partially restored
+local view, but it can reappear after a fresh-container restore. Retention and
+garbage collection should be a separate, explicit operation with tombstones or
+a verified manifest rather than `rsync --delete` in the hot checkpoint loop.
+
+PID 1 remains a small shell supervisor instead of `exec`-ing Node. On
+`SIGTERM`, `SIGINT`, or `SIGHUP`, it forwards the signal to the server, waits
+up to one second for the held PTYs to stop and flush their local files, and
+takes a final checkpoint. The final checkpoint waits for any in-flight timer
+checkpoint's lock before reading the quiet state.
+
+## Migration and rollback
+
+- Codex's old `$CODEX_HOME/sessions -> /data/state/codex/sessions` symlink is
+  unlinked only at the known local path. Its durable target is never removed.
+- Existing Gemini state under `/data/home/.gemini` is copied into the new
+  durable checkpoint and retained as a rollback copy.
+- Existing real opencode/Hermes directories are copied to the durable store and
+  renamed to `*.pre-agent-state`; if that rollback name already exists, a
+  timestamped name is used. They are not deleted by this proposal.
+- Existing Codex SQLite remnants stay quarantined. They are caches; rollouts,
+  history, auth, and config are the restored source of truth.
+
+Before the first production deployment, any currently open Codex rollout must
+be copied to a new closed migration object and verified through the bucket API.
+Restarting first would repeat the loss mode this change is intended to fix.
+
+Rollback is configuration-only: point each harness back at its retained
+durable/legacy path. No migration step deletes the previous state.
+
+## Guarantees and remaining work
+
+This proposal bounds ungraceful loss to the checkpoint interval. It does not
+claim synchronous per-token durability. If that is required later, JSONL
+checkpoints can be replaced by immutable complete-line segments without
+changing the live layout or SQLite adapters.
+
+Gemini state becomes durable here, but Agent Manager still needs a separate
+conversation-identity change before it can safely resume an exact Gemini
+session in a folder shared by multiple Gemini panes. Using `--resume latest`
+without pinning would risk cross-session pickup, so this branch deliberately
+does not enable that shortcut.
+
+A future harness-independent input journal should write one immutable object
+per submitted prompt before delivery. That would preserve the operator's input
+even if a CLI fails before recording it, while transcript checkpoints remain
+the source for assistant/tool events.
+
+## Verification
+
+`server/state-checkpoint.test.mjs` exercises the failure boundaries without
+touching `/data`:
+
+- restores each file-backed harness to local storage;
+- preserves newer local state across a hot restart;
+- checkpoints a Codex rollout while another process holds its FD open;
+- kills that writer and reconstructs the rollout from the checkpoint;
+- snapshots committed opencode data while a WAL-mode writer remains alive;
+- validates opencode and Hermes checkpoint databases;
+- proves a corrupt live DB cannot replace the previous durable snapshot; and
+- restores SQLite without stale WAL/SHM companions.

docs/session-sharing.md

@@ -74,11 +74,12 @@ The **file vs database** split drives most of the design:
 - Codex is the same shape via `codexSessionId` + `codexRollout` (`runner.js:319-321`), and
   as of `1dfb753` **opencode too**, via `opencodeSessionId`. Three of the five harnesses now
   carry a per-session conversation pin that import can simply set.
-- opencode's and Hermes' SQLite live on **local disk via symlink** (also `1dfb753`), with a
-  durable copy synced to the bucket every 60s — because a synchronous read of a FUSE-backed
-  sqlite could stall the event loop and freeze the whole server. Extraction still goes
-  through `opencodeDbPath()`, so §6 is unaffected, but **never** read these DBs
-  synchronously on a request path.
+- opencode's and Hermes' SQLite live on **local disk via symlink**. Durable
+  checkpoints are made through SQLite's online backup API and validated before
+  upload; raw DB/WAL/SHM copies are not transactionally safe. This also keeps a
+  synchronous FUSE read from stalling the event loop and freezing the whole
+  server. Extraction still goes through `opencodeDbPath()`, so §6 is
+  unaffected, but **never** read these DBs synchronously on a request path.
 
 ## 3. Verified Hub behaviour
 

docs/trace-panel-spec.md

@@ -168,15 +168,17 @@ A session is a *query*, not a file.
   `type: 'text' | 'tool' | 'step-finish'`, plus `text`, `tool`, `state.input`, `state.output`
 - **Never copy or ship this database**: `account.access_token`, `account.refresh_token` and a
   `credential` table live in it. Select one conversation.
-- Live data is on **local disk via a symlink** with a durable copy synced to the bucket
-  (commit `1dfb753`), precisely because a synchronous read of a FUSE-backed sqlite froze the
-  whole server. Open **read-only**, and never on a hot path.
+- Live data is on **local disk via a symlink**. Durable state is a verified
+  SQLite online-backup checkpoint; copying a live DB/WAL/SHM set with `rsync`
+  is not transactionally safe. This layout also prevents a synchronous read of
+  FUSE-backed SQLite from freezing the whole server. Open **read-only**, and
+  never on a hot path.
 
 ### Hermes — **SQLite**, `~/.hermes/state.db`
 - `sessions(id, cwd, title, started_at, input_tokens, output_tokens, cache_read_tokens)`
 - `messages(id, session_id, role, content, timestamp, tool_name, token_count, active)` —
   flat `content`; a row with `tool_name` set is a tool interaction. `timestamp` is **seconds**
-  (multiply by 1000). Same FUSE/symlink note as opencode.
+  (multiply by 1000). Same local-live/online-checkpoint note as opencode.
 - Hermes has **no per-session pin** in Agent Manager, so it is attributed by `cwd`.
 - Alternative worth knowing: `hermes sessions export --format trace` emits Claude-Code JSONL
   specifically for the HF viewer, and `--redact` exists. We chose direct SQLite reads for one

entrypoint.sh

@@ -10,51 +10,12 @@ if ! mkdir -p "$DATA_DIR/workspaces" 2>/dev/null; then
 fi
 export DATA_DIR
 
-# Empty directories do NOT persist on the bucket — there is no backing object
-# key, so a dir created empty is gone after a restart, and anything pointing at
-# it (a symlink, a config path) breaks. `keepdir` occupies the path with a real
-# file so it survives. Use it for every bucket-backed dir created empty.
-#   Seen in the wild: $CODEX_DURABLE/sessions vanished, leaving
-#   $CODEX_HOME/sessions dangling -> codex "thread-store internal error:
-#   File exists (os error 17)" and every transcript lost.
-keepdir() {
-  for d in "$@"; do
-    mkdir -p "$d" 2>/dev/null || continue
-    [ -e "$d/.keep" ] || echo "keep marker: empty dirs are not persisted on the /data bucket" > "$d/.keep" 2>/dev/null || true
-  done
-}
-
-# Occupy a config path with a real file when nothing lives there. Two bugs need
-# this. (1) A tool that writes atomically (temp + rename) can leave the temp
-# behind and never land the target. (2) The bucket tree API then matches
-# `<path>` against the leftover `<path>.<uuid>.tmp` by RAW STRING PREFIX, and
-# hf-mount reads a non-empty listing as proof the path is a DIRECTORY — so the
-# missing file materializes as a phantom dir and readers die with EISDIR.
-# A real file short-circuits it: the HEAD succeeds, so the listing fallback that
-# synthesizes the directory never runs. See docs/fuse-phantom-directories.md.
-occupy_file() {
-  path="$1"; default="$2"
-  mkdir -p "$(dirname "$path")" 2>/dev/null || return 0
-  [ -d "$path" ] && rm -rf "$path" 2>/dev/null
-  [ -e "$path" ] || printf '%s\n' "$default" > "$path" 2>/dev/null || true
-}
-
-# Put HOME on the durable bucket so EVERY agent's logins/config persist across
-# restarts (gemini ~/.gemini, etc.). Agents whose SQLite state can't live on the
-# FUSE bucket (codex, openclaw, opencode, hermes) are relocated to local disk
-# below, each with its own durable copy on the bucket.
+# HOME remains the durable place for ordinary shell/user configuration. Agent
+# harness state is relocated to local disk below and checkpointed explicitly:
+# actively-mutated files and SQLite databases must not use the FUSE bucket as
+# their live filesystem.
 export HOME="$DATA_DIR/home"
 mkdir -p "$HOME"
-# Gemini writes its project registry atomically and the rename can fail on the
-# bucket, leaving projects.json.<uuid>.tmp orphans and no projects.json — after
-# which the prefix bug above turns projects.json into a directory and the CLI
-# dies with `EISDIR: illegal operation on a directory, read`. Keep a real file
-# there. Same class as the opencode.json guard in server/src/runner.js.
-occupy_file "$HOME/.gemini/projects.json" '{"projects": {}}'
-# Claude keeps its established dir (so existing logins keep working). Codex's
-# home moves to local disk below (its SQLite databases corrupt on the bucket).
-export CLAUDE_CONFIG_DIR="$DATA_DIR/state/claude"
-mkdir -p "$CLAUDE_CONFIG_DIR"
 
 # NOTE: every var exported below must be listed in NON_SECRET (server/src/
 # index.js) — this script runs after the build-time env snapshot, so anything
@@ -64,120 +25,118 @@ mkdir -p "$CLAUDE_CONFIG_DIR"
 # /data bucket — object storage is slow for many-small-files and can't mmap or
 # lock well, so running libraries from it is painful. These reinstall on demand.
 export AM_LOCAL="/home/node/local"
-if ! mkdir -p "$AM_LOCAL/bin" 2>/dev/null; then AM_LOCAL="$DATA_DIR/.local-cache"; mkdir -p "$AM_LOCAL/bin"; fi
+if ! mkdir -p "$AM_LOCAL/bin" 2>/dev/null; then
+  # Never fall back onto the bucket: agent state needs POSIX close/locking
+  # semantics even when the preferred local prefix is unavailable.
+  AM_LOCAL="/tmp/agent-manager-local"
+  mkdir -p "$AM_LOCAL/bin"
+fi
 export UV_CACHE_DIR="$AM_LOCAL/uv-cache"
 
-# Liveness handle for the state-sync loops below: this script ends with
-# `exec node …`, so $$ IS the app's pid. Each loop checks it and gives up once
-# its own app instance is gone, because a dev-mode app restart re-runs this
-# script inside the SAME container: the old loops survive (reparented to PID 1)
-# and the new run adds three more. Measured after three restart attempts: ten
-# loops alive, all waking every 60s to rsync the same bucket paths over each
-# other. Not exported — nothing downstream needs it (see NON_SECRET note above).
-AM_MAIN_PID=$$
-
-# Codex keeps a growing family of SQLite databases (logs_2, goals_1, memories_1
-# plus mmap'd -shm siblings) that corrupt on the FUSE bucket: SQLite needs real
-# locking/mmap. Same cure as OpenClaw — codex's HOME lives on LOCAL disk. The
-# heavyweight append-only rollouts stay on the bucket via one symlink (plain
-# files never corrupted there, and pinned rollout paths keep working); the
-# small durable state (auth, config, history) restores at boot and syncs back
-# every 60s; the SQLite caches are purely local, rebuilt from rollouts when
-# the disk resets.
-CODEX_DURABLE="$DATA_DIR/state/codex"
+# One state model for every harness:
+#   * live files are ordinary local POSIX files under $AM_LOCAL;
+#   * the bucket contains closed checkpoints only;
+#   * SQLite checkpoints use the online backup API rather than racing copies
+#     of a DB, WAL, and SHM.
+#
+# hf-mount's streaming writer buffers a long-lived append until close. Codex
+# keeps its rollout open for the life of a session, so the old sessions symlink
+# could lose the whole open epoch on a restart.
+export CODEX_DURABLE="$DATA_DIR/state/codex"
 export CODEX_HOME="$AM_LOCAL/codex-home"
-mkdir -p "$CODEX_HOME"
-# keepdir, not mkdir: `sessions` is the symlink target below, and an empty dir
-# on the bucket disappears — which is exactly how codex lost its thread store.
-keepdir "$CODEX_DURABLE/sessions" "$CODEX_DURABLE/db-backups"
-# quarantine sqlite remnants on the bucket (incl. the earlier symlink attempt)
-keepdir "$CODEX_DURABLE/db-backups/am-quarantine"
+export CLAUDE_DURABLE="$DATA_DIR/state/claude"
+export CLAUDE_CONFIG_DIR="$AM_LOCAL/agent-state/claude"
+export GEMINI_DURABLE="$DATA_DIR/state/gemini"
+export GEMINI_CLI_HOME="$AM_LOCAL/agent-state/gemini-home"
+export GEMINI_LIVE="$GEMINI_CLI_HOME/.gemini"
+export OPENCLAW_HOME="$AM_LOCAL/oc-home"
+export OPENCLAW_STATE_DIR="$OPENCLAW_HOME/.openclaw"
+export OPENCLAW_DURABLE="$DATA_DIR/state/openclaw-backup"
+export OPENCODE_LIVE="$AM_LOCAL/opencode-share"
+export OPENCODE_DURABLE="$DATA_DIR/state/opencode"
+export HERMES_LIVE="$AM_LOCAL/hermes"
+export HERMES_DURABLE="$DATA_DIR/state/hermes"
+
+mkdir -p "$CODEX_HOME" "$CODEX_DURABLE/sessions" "$CODEX_DURABLE/db-backups" \
+  "$CLAUDE_CONFIG_DIR" "$CLAUDE_DURABLE" "$GEMINI_LIVE" "$GEMINI_DURABLE" \
+  "$OPENCLAW_STATE_DIR" "$OPENCLAW_DURABLE" "$OPENCODE_LIVE" \
+  "$OPENCODE_DURABLE" "$HERMES_LIVE" "$HERMES_DURABLE"
+
+# Heal the old Codex layout on hot/dev restarts. Only unlink the known local
+# sessions symlink; never recursively remove its durable target.
+[ -L "$CODEX_HOME/sessions" ] && rm "$CODEX_HOME/sessions"
+mkdir -p "$CODEX_HOME/sessions"
+
+# Quarantine SQLite remnants from the old Codex bucket layout. Codex SQLite is
+# disposable cache state; transcripts and user history are checkpointed.
+mkdir -p "$CODEX_DURABLE/db-backups/am-quarantine"
 for f in "$CODEX_DURABLE"/logs_2.sqlite* "$CODEX_DURABLE"/goals_1.sqlite* "$CODEX_DURABLE"/memories_1.sqlite*; do
   [ -e "$f" ] || [ -L "$f" ] && mv "$f" "$CODEX_DURABLE/db-backups/am-quarantine/" 2>/dev/null || true
 done
-rsync -a --exclude 'sessions' --exclude '*.sqlite*' --exclude 'db-backups' \
-  --exclude 'cache' --exclude '.tmp' --exclude 'mcp-oauth-locks' \
-  "$CODEX_DURABLE/" "$CODEX_HOME/" 2>/dev/null || true
-ln -sfn "$CODEX_DURABLE/sessions" "$CODEX_HOME/sessions"
-( while :; do
-    sleep 60
-    kill -0 "$AM_MAIN_PID" 2>/dev/null || exit
-    rsync -a --exclude 'sessions' --exclude '*.sqlite*' --exclude 'db-backups' \
-      --exclude 'cache' --exclude '.tmp' --exclude 'mcp-oauth-locks' \
-      "$CODEX_HOME/" "$CODEX_DURABLE/" 2>/dev/null || true
-  done ) &
-export PIP_CACHE_DIR="$AM_LOCAL/pip-cache"
-export PYTHONPYCACHEPREFIX="$AM_LOCAL/pycache"
-export PYTHONUSERBASE="$AM_LOCAL/py"          # pip install --user → local, fast
-export NPM_CONFIG_PREFIX="$AM_LOCAL/npm"       # npm install -g → local, no root needed
-export PATH="$AM_LOCAL/py/bin:$AM_LOCAL/npm/bin:$AM_LOCAL/bin:$HOME/.local/bin:$PATH"
 
-# OpenClaw: its session engine fingerprints file metadata at nanosecond
-# precision and false-positives on the FUSE bucket ("session file changed while
-# embedded prompt lock was released"). Its state therefore lives on LOCAL disk,
-# with a durable copy on the bucket: restored on boot, synced back every 60s.
-# Worst case on an unclean stop: the last minute of chat history.
-# OpenClaw can't run its state on the FUSE bucket (its session fence
-# false-positives on unstable metadata) and it REJECTS symlinked paths (the
-# workspace boundary check). No symlinks, no env overrides — OpenClaw simply
-# gets its OWN HOME on local disk: a real, ordinary install from its point of
-# view. Durable copy on the bucket: restored on boot, synced back every 60s.
-# Worst case on an unclean stop: the last minute of claw state.
-export OPENCLAW_HOME="$AM_LOCAL/oc-home"                # runner launches openclaw with HOME=$OPENCLAW_HOME
-export OPENCLAW_STATE_DIR="$OPENCLAW_HOME/.openclaw"    # where the server finds its config/traces
-OC_BACKUP="$DATA_DIR/state/openclaw-backup"
-mkdir -p "$OPENCLAW_STATE_DIR" "$OC_BACKUP"
-# heal from the earlier symlink experiment
+# Preserve existing Gemini state while moving it out of durable HOME. The
+# legacy directory remains in place as a rollback copy.
+if [ -d "$HOME/.gemini" ] && [ ! -L "$HOME/.gemini" ]; then
+  if ! rsync -a --update "$HOME/.gemini/" "$GEMINI_DURABLE/"; then
+    echo "ERROR: could not migrate Gemini state; refusing to start with an empty live home" >&2
+    exit 1
+  fi
+fi
+
+# OpenClaw rejects a symlinked HOME/state path; migrate any state from the
+# earlier layouts into its durable checkpoint before restore.
 [ -L "$HOME/.openclaw" ] && rm "$HOME/.openclaw"
-# seed local state: backup (freshest) first, then legacy dirs fill gaps (--update: never clobber newer)
-[ -n "$(ls -A "$OC_BACKUP" 2>/dev/null)" ] && rsync -a "$OC_BACKUP/" "$OPENCLAW_STATE_DIR/" 2>/dev/null
 for legacy in "$HOME/.openclaw.pre-symlink" "$HOME/.openclaw"; do
   if [ -d "$legacy" ] && [ ! -L "$legacy" ]; then
-    rsync -a --update "$legacy/" "$OPENCLAW_STATE_DIR/" 2>/dev/null || true
+    if ! rsync -a --update "$legacy/" "$OPENCLAW_DURABLE/"; then
+      echo "ERROR: could not migrate OpenClaw state from $legacy" >&2
+      exit 1
+    fi
   fi
 done
-# small comforts in the private HOME (harmless if missing)
-cp "$HOME/.gitconfig" "$OPENCLAW_HOME/.gitconfig" 2>/dev/null || true
-( while :; do
-    sleep 60
-    kill -0 "$AM_MAIN_PID" 2>/dev/null || exit
-    rsync -a --delete "$OPENCLAW_STATE_DIR/" "$OC_BACKUP/" 2>/dev/null || true
-  done ) &
-
-# opencode + hermes keep their conversation history in SQLite (opencode at
-# ~/.local/share/opencode, hermes at ~/.hermes). SQLite on the FUSE bucket
-# corrupts, and worse: a SYNCHRONOUS read can STALL on FUSE and wedge the
-# server's event loop — the Overview reads these dbs on every poll, so one
-# stalled read takes the whole Space down. The live data therefore lives on
-# LOCAL disk, exposed at the well-known path via a symlink, with a durable copy
-# on the bucket: restored on boot, synced back every 60s. Unlike codex these
-# dbs ARE the source of truth (no rollout files to rebuild from), so the
-# sync-back INCLUDES the sqlite. Worst case on an unclean stop: the last minute
-# of chat history.
-OC_LIVE="$AM_LOCAL/opencode-share"; OC_DURABLE="$DATA_DIR/state/opencode"; OC_LINK="$HOME/.local/share/opencode"
-HERMES_LIVE="$AM_LOCAL/hermes"; HERMES_DURABLE="$DATA_DIR/state/hermes"; HERMES_LINK="$HOME/.hermes"
-mkdir -p "$OC_LIVE" "$HERMES_LIVE" "$(dirname "$OC_LINK")"
-keepdir "$OC_DURABLE" "$HERMES_DURABLE"
-# One-time migration: existing history is a REAL dir at the well-known path on
-# the bucket. Fold it into the durable store BEFORE the path becomes a symlink,
-# so no conversation is stranded on the bucket or lost.
-for pair in "$OC_LINK|$OC_DURABLE" "$HERMES_LINK|$HERMES_DURABLE"; do
+
+# opencode and Hermes expect their state at paths under HOME. The live targets
+# are local; legacy real directories are retained as rollback copies instead of
+# being deleted during migration.
+OPENCODE_LINK="$HOME/.local/share/opencode"
+HERMES_LINK="$HOME/.hermes"
+mkdir -p "$(dirname "$OPENCODE_LINK")"
+for pair in "$OPENCODE_LINK|$OPENCODE_DURABLE" "$HERMES_LINK|$HERMES_DURABLE"; do
   lnk="${pair%%|*}"; dur="${pair##*|}"
   if [ -e "$lnk" ] && [ ! -L "$lnk" ]; then
-    rsync -a "$lnk/" "$dur/" 2>/dev/null || true
-    rm -rf "$lnk" 2>/dev/null || true
+    if ! rsync -a --update "$lnk/" "$dur/"; then
+      echo "ERROR: could not migrate agent state from $lnk" >&2
+      exit 1
+    fi
+    backup="$lnk.pre-agent-state"
+    if [ -e "$backup" ] || [ -L "$backup" ]; then
+      backup="$backup.$(date -u +%Y%m%dT%H%M%SZ).$$"
+    fi
+    if ! mv "$lnk" "$backup"; then
+      echo "ERROR: could not retain rollback copy at $backup" >&2
+      exit 1
+    fi
   fi
 done
-rsync -a "$OC_DURABLE/" "$OC_LIVE/" 2>/dev/null || true       # restore durable → live (local disk is wiped each boot)
-rsync -a "$HERMES_DURABLE/" "$HERMES_LIVE/" 2>/dev/null || true
-ln -sfn "$OC_LIVE" "$OC_LINK"
+ln -sfn "$OPENCODE_LIVE" "$OPENCODE_LINK"
 ln -sfn "$HERMES_LIVE" "$HERMES_LINK"
-( while :; do
-    sleep 60
-    kill -0 "$AM_MAIN_PID" 2>/dev/null || exit
-    rsync -a "$OC_LIVE/" "$OC_DURABLE/" 2>/dev/null || true
-    rsync -a "$HERMES_LIVE/" "$HERMES_DURABLE/" 2>/dev/null || true
-  done ) &
+
+# Restore after all one-time migrations have populated the durable side. On a
+# hot/dev restart, --update preserves newer local state.
+AGENT_STATE_SCRIPT="${AGENT_STATE_SCRIPT:-/app/scripts/agent-state.sh}"
+export AGENT_STATE_SCRIPT
+if ! sh "$AGENT_STATE_SCRIPT" restore; then
+  echo "ERROR: agent-state restore was incomplete; refusing to start agents against partial state" >&2
+  exit 1
+fi
+
+# Small comforts in OpenClaw's private HOME (harmless if missing).
+cp "$HOME/.gitconfig" "$OPENCLAW_HOME/.gitconfig" 2>/dev/null || true
+export PIP_CACHE_DIR="$AM_LOCAL/pip-cache"
+export PYTHONPYCACHEPREFIX="$AM_LOCAL/pycache"
+export PYTHONUSERBASE="$AM_LOCAL/py"          # pip install --user → local, fast
+export NPM_CONFIG_PREFIX="$AM_LOCAL/npm"       # npm install -g → local, no root needed
+export PATH="$AM_LOCAL/py/bin:$AM_LOCAL/npm/bin:$AM_LOCAL/bin:$HOME/.local/bin:$PATH"
 
 # Durable, user-editable setup script. Runs on EVERY start (keep it idempotent);
 # seed a template on first boot.
@@ -221,4 +180,45 @@ else
 fi
 echo "[install.sh finished $(date -u) exit=$INSTALL_CODE]" >> "$DATA_DIR/install.log"
 
-exec node /app/server/src/index.js
+# Keep PID 1 as a tiny supervisor so normal Space/dev restarts receive a final
+# state checkpoint. The timer bounds loss on an ungraceful stop; the child is
+# stopped before the final checkpoint so SQLite and transcript state is quiet.
+AGENT_STATE_CHECKPOINT_SECONDS="${AGENT_STATE_CHECKPOINT_SECONDS:-15}"
+node /app/server/src/index.js &
+APP_PID=$!
+
+checkpoint_loop() {
+  while kill -0 "$APP_PID" 2>/dev/null; do
+    sleep "$AGENT_STATE_CHECKPOINT_SECONDS"
+    kill -0 "$APP_PID" 2>/dev/null || break
+    sh "$AGENT_STATE_SCRIPT" checkpoint \
+      || echo "WARN: periodic agent-state checkpoint failed"
+  done
+}
+checkpoint_loop &
+CHECKPOINT_PID=$!
+
+finish() {
+  code="${1:-0}"
+  kill "$CHECKPOINT_PID" 2>/dev/null || true
+  wait "$CHECKPOINT_PID" 2>/dev/null || true
+  # A timer checkpoint may still be finishing after its loop shell is stopped.
+  # checkpoint-final waits for that lock, then captures the quiet post-Node
+  # state instead of silently treating a busy lock as success.
+  sh "$AGENT_STATE_SCRIPT" checkpoint-final \
+    || echo "WARN: final agent-state checkpoint failed"
+  exit "$code"
+}
+
+shutdown() {
+  trap - TERM INT HUP
+  kill -TERM "$APP_PID" 2>/dev/null || true
+  wait "$APP_PID" 2>/dev/null
+  code=$?
+  finish "$code"
+}
+trap shutdown TERM INT HUP
+
+wait "$APP_PID"
+APP_CODE=$?
+finish "$APP_CODE"

scripts/agent-state.sh

@@ -0,0 +1,260 @@
+#!/bin/sh
+# Durable state bridge for the agent harnesses.
+#
+# Active state always lives on the container's POSIX filesystem. The mounted
+# bucket only receives short-lived, closed writes made by rsync (ordinary file
+# trees) or SQLite's online backup API (live databases). This is intentional:
+# hf-mount's streaming writer holds an open file in memory until close, while
+# copying a database and its WAL independently can produce a torn backup.
+
+set -u
+
+MODE="${1:-}"
+case "$MODE" in
+  restore|checkpoint|checkpoint-final) ;;
+  *) echo "usage: $0 restore|checkpoint|checkpoint-final" >&2; exit 2 ;;
+esac
+
+: "${DATA_DIR:?DATA_DIR is required}"
+: "${AM_LOCAL:?AM_LOCAL is required}"
+
+CODEX_HOME="${CODEX_HOME:-$AM_LOCAL/codex-home}"
+CODEX_DURABLE="${CODEX_DURABLE:-$DATA_DIR/state/codex}"
+CLAUDE_CONFIG_DIR="${CLAUDE_CONFIG_DIR:-$AM_LOCAL/agent-state/claude}"
+CLAUDE_DURABLE="${CLAUDE_DURABLE:-$DATA_DIR/state/claude}"
+GEMINI_CLI_HOME="${GEMINI_CLI_HOME:-$AM_LOCAL/agent-state/gemini-home}"
+GEMINI_LIVE="${GEMINI_LIVE:-$GEMINI_CLI_HOME/.gemini}"
+GEMINI_DURABLE="${GEMINI_DURABLE:-$DATA_DIR/state/gemini}"
+OPENCLAW_STATE_DIR="${OPENCLAW_STATE_DIR:-$AM_LOCAL/oc-home/.openclaw}"
+OPENCLAW_DURABLE="${OPENCLAW_DURABLE:-$DATA_DIR/state/openclaw-backup}"
+OPENCODE_LIVE="${OPENCODE_LIVE:-$AM_LOCAL/opencode-share}"
+OPENCODE_DURABLE="${OPENCODE_DURABLE:-$DATA_DIR/state/opencode}"
+HERMES_LIVE="${HERMES_LIVE:-$AM_LOCAL/hermes}"
+HERMES_DURABLE="${HERMES_DURABLE:-$DATA_DIR/state/hermes}"
+
+LOCK="$AM_LOCAL/agent-state-checkpoint.lock"
+mkdir -p "$AM_LOCAL"
+exec 9>"$LOCK"
+if [ "$MODE" = restore ] || [ "$MODE" = checkpoint-final ]; then
+  # A dev-mode restart may overlap the previous app's final checkpoint. Never
+  # restore an older durable view while that checkpoint is still being made.
+  flock 9
+else
+  # Timer and shutdown checkpoints can race; the one already holding the lock
+  # is sufficient, and a later timer/shutdown will catch subsequent writes.
+  flock -n 9 || exit 0
+fi
+
+# `find -newer` is a strict comparison. A file written in the same filesystem
+# timestamp tick as a checkpoint marker would otherwise be skipped forever.
+# Keep a small overlap at every successful boundary; an occasional repeat copy
+# is harmless, while a missed transcript/config is not.
+mark_checkpoint_floor() {
+  touch -d '2 seconds ago' "$1" 2>/dev/null || touch "$1"
+}
+
+restore_tree() {
+  key="$1" durable="$2" live="$3"; shift 3
+  mkdir -p "$durable" "$live"
+  stamp_dir="$AM_LOCAL/agent-state-stamps"
+  stamp="$stamp_dir/$key"
+  had_local=false
+  [ -n "$(find "$live" -type f -print -quit 2>/dev/null)" ] && had_local=true
+  # --update matters for hot/dev restarts: local disk survives those and can be
+  # newer than the last completed bucket checkpoint. A fresh container starts
+  # with an empty destination, so the same command performs a full restore.
+  if rsync -a --update "$@" "$durable/" "$live/"; then
+    mkdir -p "$stamp_dir"
+    if [ ! -e "$stamp" ]; then
+      if [ "$had_local" = true ]; then
+        # First deployment over an already-populated local tree: force one
+        # checkpoint so newer local files skipped by --update become durable.
+        touch -t 197001010000 "$stamp"
+      else
+        # Fresh container: every local byte came from this durable restore.
+        mark_checkpoint_floor "$stamp"
+      fi
+    fi
+  else
+    return 1
+  fi
+}
+
+checkpoint_tree() {
+  key="$1" live="$2" durable="$3"; shift 3
+  [ -d "$live" ] || return 0
+  stamp_dir="$AM_LOCAL/agent-state-stamps"
+  mkdir -p "$durable" "$stamp_dir"
+  stamp="$stamp_dir/$key"
+  if [ ! -e "$stamp" ]; then
+    touch -t 197001010000 "$stamp"
+  fi
+
+  # Mark the START of this checkpoint. Any file changed during/after its copy
+  # is newer than `next` and will therefore be selected again next time.
+  next="$stamp.next.$$"
+  list="$stamp.files.$$"
+  mark_checkpoint_floor "$next"
+  (cd "$live" && find . -type f -newer "$stamp" -print0) > "$list"
+
+  if [ -s "$list" ]; then
+    # --files-from means rsync walks only changed LOCAL paths. It does not scan
+    # the remote tree every 15 seconds — a critical property on the bucket
+    # mount. Destination temporaries close before rename, retaining the prior
+    # object if this process dies during transfer.
+    if ! rsync -a -r --from0 --files-from="$list" --delay-updates \
+      "$@" "$live/" "$durable/"; then
+      rm -f "$next" "$list"
+      return 1
+    fi
+  fi
+  mv "$next" "$stamp"
+  rm -f "$list"
+}
+
+restore_sqlite_tree() {
+  durable="$1" live="$2" db_name="$3" checkpoint_name="$4"
+  mkdir -p "$durable" "$live"
+
+  if ! restore_tree "$checkpoint_name-files" "$durable" "$live" \
+    --exclude 'checkpoints' --exclude '*.db' --exclude '*.db-*' \
+    --exclude '*.sqlite*' --exclude '*-wal' --exclude '*-shm'; then
+    return 1
+  fi
+
+  checkpoint="$durable/checkpoints/$checkpoint_name"
+  target="$live/$db_name"
+  if [ -s "$target" ] && [ "$(sqlite3 "$target" 'PRAGMA quick_check;' 2>/dev/null)" = ok ]; then
+    # Local disk survives in-container/dev restarts. Its database can be newer
+    # than the last checkpoint (including committed rows still in its WAL), so
+    # a valid live database is always the restore authority on a hot restart.
+    return 0
+  fi
+
+  # Retain an invalid local set for diagnosis, then recover from the last
+  # known-good checkpoint. These are explicit ephemeral paths, never bucket
+  # objects.
+  had_invalid=false
+  if [ -e "$target" ] || [ -e "$target-wal" ] || [ -e "$target-shm" ]; then
+    had_invalid=true
+    invalid_dir="$AM_LOCAL/agent-state-invalid/$checkpoint_name.$(date -u +%Y%m%dT%H%M%SZ).$$"
+    mkdir -p "$invalid_dir"
+    [ -e "$target" ] && mv "$target" "$invalid_dir/$db_name"
+    [ -e "$target-wal" ] && mv "$target-wal" "$invalid_dir/$db_name-wal"
+    [ -e "$target-shm" ] && mv "$target-shm" "$invalid_dir/$db_name-shm"
+  fi
+
+  if [ -s "$checkpoint" ]; then
+    # WAL/SHM are ephemeral coordination files, never part of a restored
+    # checkpoint. Removing these explicit local paths cannot touch bucket data.
+    rm -f "$target-wal" "$target-shm"
+    if ! rsync -a "$checkpoint" "$target"; then return 1; fi
+  elif [ -s "$durable/$db_name" ]; then
+    # One-release compatibility path for state written by the old raw-rsync
+    # mechanism. Copy the legacy DB and any WAL so SQLite can recover it; the
+    # first successful checkpoint replaces this path as the restore authority.
+    if ! rsync -a "$durable/$db_name" "$target"; then return 1; fi
+    if [ -f "$durable/$db_name-wal" ] && ! rsync -a "$durable/$db_name-wal" "$target-wal"; then return 1; fi
+    if [ -f "$durable/$db_name-shm" ] && ! rsync -a "$durable/$db_name-shm" "$target-shm"; then return 1; fi
+  fi
+
+  if [ "$had_invalid" = true ] && [ ! -s "$target" ]; then
+    echo "agent-state: invalid local SQLite state has no durable recovery for $target" >&2
+    return 1
+  fi
+  if [ -s "$target" ] && [ "$(sqlite3 "$target" 'PRAGMA quick_check;' 2>/dev/null)" != ok ]; then
+    echo "agent-state: restored SQLite state is invalid for $target" >&2
+    return 1
+  fi
+}
+
+checkpoint_sqlite_tree() {
+  live="$1" durable="$2" db_name="$3" checkpoint_name="$4"
+  source="$live/$db_name"
+
+  # A harness can write ordinary state before it creates its database (Hermes
+  # setup files are a real example). Always publish that file tree first. The
+  # database backup is optional until the database itself exists.
+  if ! checkpoint_tree "$checkpoint_name-files" "$live" "$durable" \
+    --exclude 'checkpoints' --exclude '*.db' --exclude '*.db-*' \
+    --exclude '*.sqlite*' --exclude '*-wal' --exclude '*-shm'; then
+    return 1
+  fi
+  [ -s "$source" ] || return 0
+
+  sqlite_stamp_dir="$AM_LOCAL/agent-state-stamps"
+  sqlite_stamp="$sqlite_stamp_dir/$checkpoint_name-sqlite"
+  mkdir -p "$sqlite_stamp_dir"
+  # Timestamp-only detection has an equal-tick hole, while deliberately
+  # overlapping the marker would rewrite an idle database on rapid successive
+  # checkpoints. Record the exact local DB/WAL metadata observed BEFORE the
+  # backup instead. A concurrent commit changes the next signature and is
+  # therefore picked up by the following checkpoint.
+  sqlite_next="$sqlite_stamp.next.$$"
+  {
+    stat -c 'db|%s|%y|%z' "$source"
+    if [ -e "$source-wal" ]; then
+      stat -c 'wal|%s|%y|%z' "$source-wal"
+    else
+      echo 'wal|absent'
+    fi
+  } > "$sqlite_next"
+  if cmp -s "$sqlite_next" "$sqlite_stamp"; then
+    rm -f "$sqlite_next"
+    return 0
+  fi
+
+  stage_dir="$AM_LOCAL/agent-state-snapshots"
+  mkdir -p "$stage_dir" "$durable/checkpoints"
+  staged="$stage_dir/$checkpoint_name.$$.tmp"
+  rm -f "$staged"
+  escaped=$(printf '%s' "$staged" | sed "s/'/''/g")
+
+  # .backup observes the main DB and WAL through one consistent SQLite read
+  # transaction. The staged file is ordinary local storage and is closed before
+  # rsync hands it to the bucket.
+  if ! sqlite3 "$source" ".timeout 5000" ".backup '$escaped'"; then
+    rm -f "$staged" "$sqlite_next"
+    return 1
+  fi
+  if [ "$(sqlite3 "$staged" 'PRAGMA quick_check;' 2>/dev/null)" != ok ]; then
+    echo "agent-state: refusing invalid SQLite checkpoint for $source" >&2
+    rm -f "$staged" "$sqlite_next"
+    return 1
+  fi
+  if ! rsync -a --delay-updates "$staged" "$durable/checkpoints/$checkpoint_name"; then
+    rm -f "$staged" "$sqlite_next"
+    return 1
+  fi
+  rm -f "$staged"
+  mv "$sqlite_next" "$sqlite_stamp"
+}
+
+failures=0
+
+if [ "$MODE" = restore ]; then
+  restore_tree codex "$CODEX_DURABLE" "$CODEX_HOME" \
+    --exclude '*.sqlite*' --exclude '*.db' --exclude '*.db-*' \
+    --exclude '*-wal' --exclude '*-shm' --exclude 'db-backups' \
+    --exclude 'cache' --exclude '.tmp' --exclude 'mcp-oauth-locks' || failures=$((failures + 1))
+  restore_tree claude "$CLAUDE_DURABLE" "$CLAUDE_CONFIG_DIR" || failures=$((failures + 1))
+  restore_tree gemini "$GEMINI_DURABLE" "$GEMINI_LIVE" || failures=$((failures + 1))
+  restore_tree openclaw "$OPENCLAW_DURABLE" "$OPENCLAW_STATE_DIR" || failures=$((failures + 1))
+  restore_sqlite_tree "$OPENCODE_DURABLE" "$OPENCODE_LIVE" opencode.db opencode.db || failures=$((failures + 1))
+  restore_sqlite_tree "$HERMES_DURABLE" "$HERMES_LIVE" state.db state.db || failures=$((failures + 1))
+else
+  checkpoint_tree codex "$CODEX_HOME" "$CODEX_DURABLE" \
+    --exclude '*.sqlite*' --exclude '*.db' --exclude '*.db-*' \
+    --exclude '*-wal' --exclude '*-shm' --exclude 'db-backups' \
+    --exclude 'cache' --exclude '.tmp' --exclude 'mcp-oauth-locks' || failures=$((failures + 1))
+  checkpoint_tree claude "$CLAUDE_CONFIG_DIR" "$CLAUDE_DURABLE" || failures=$((failures + 1))
+  checkpoint_tree gemini "$GEMINI_LIVE" "$GEMINI_DURABLE" || failures=$((failures + 1))
+  checkpoint_tree openclaw "$OPENCLAW_STATE_DIR" "$OPENCLAW_DURABLE" || failures=$((failures + 1))
+  checkpoint_sqlite_tree "$OPENCODE_LIVE" "$OPENCODE_DURABLE" opencode.db opencode.db || failures=$((failures + 1))
+  checkpoint_sqlite_tree "$HERMES_LIVE" "$HERMES_DURABLE" state.db state.db || failures=$((failures + 1))
+fi
+
+[ "$failures" -eq 0 ] || {
+  echo "agent-state: $MODE completed with $failures failed adapter(s)" >&2
+  exit 1
+}

server/migration.test.mjs

@@ -8,6 +8,41 @@ import path from 'node:path';
 import { WebSocket } from 'ws';
 
 const DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'am-migration-'));
+const BASHRC = path.join(DATA_DIR, 'test.bashrc');
+const BASH_ENV = path.join(DATA_DIR, 'test.bash-env');
+const BIN_DIR = path.join(DATA_DIR, 'bin');
+fs.mkdirSync(BIN_DIR);
+// runner launches each CLI through `bash -lc`, whose login profile replaces
+// PATH. BASH_ENV runs afterwards and keeps this fixture ahead of the real CLI.
+fs.writeFileSync(BASH_ENV, `export PATH="${BIN_DIR}:$PATH"\n`);
+const FAKE_OPENCODE = path.join(BIN_DIR, 'opencode');
+fs.writeFileSync(FAKE_OPENCODE, `#!/usr/bin/env bash
+printf 'OPENCODE-SCREEN-READY\\r\\n'
+saved=$(stty -g)
+stty -echo -icanon min 0 time 1
+end=$((SECONDS + 8))
+while [ "$SECONDS" -lt "$end" ]; do
+  IFS= read -r -n 4096 -t 1 _ || true
+done
+stty "$saved"
+printf 'OPENCODE-INPUT-READY\\r\\n'
+while IFS= read -r line; do
+  printf 'OPENCODE-EXECUTED:%s\\r\\n' "$line"
+done
+`);
+fs.chmodSync(FAKE_OPENCODE, 0o755);
+fs.writeFileSync(BASHRC, `
+if [ "$AM_NAME" = "input-readiness" ]; then
+  # Model a history-heavy TUI that drains startup keystrokes. The old fixed
+  # 3.5s delay delivered during this loop and lost the operator's prompt.
+  for i in 1 2 3 4 5; do
+    printf 'BOOT-FRAME-%s\\n' "$i"
+    IFS= read -r -t 1 _ || true
+  done
+  printf 'INPUT-READY\\n'
+fi
+PS1='test$ '
+`);
 
 const PORT = 7893;
 const CTRL = '\x00\x00AM:';
@@ -28,7 +63,10 @@ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
 const { SPACE_ID, AM_DISTRIBUTE_SKILLS, ...BASE_ENV } = process.env;
 
 const srv = spawn('node', ['src/index.js'], {
-  env: { ...BASE_ENV, PORT: String(PORT), DATA_DIR, AM_BASHRC: '/nonexistent', AM_ALLOW_MISSING_ORIGIN: '1' },
+  env: {
+    ...BASE_ENV, PATH: `${BIN_DIR}:${process.env.PATH}`, BASH_ENV,
+    PORT: String(PORT), DATA_DIR, AM_BASHRC: BASHRC, AM_ALLOW_MISSING_ORIGIN: '1',
+  },
   stdio: ['ignore', 'pipe', 'pipe'],
 });
 let bootLog = '';
@@ -197,6 +235,44 @@ try {
   check('stopped session reports stopped', after && after.state === 'stopped', after && after.state);
 
   await fetch(`${base}/api/sessions/${id}`, { method: 'DELETE' }).catch(() => {});
+
+  // --- waking a stopped pane waits for the TUI, not a fixed delay ----------
+  const delayed = await (await fetch(`${base}/api/sessions`, {
+    method: 'POST', headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({ cli: 'shell', name: 'input-readiness', path: '.' }),
+  })).json();
+  const delayedId = delayed.id;
+  const delivered = await fetch(`${base}/api/sessions/${delayedId}/input`, {
+    method: 'POST', headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({ text: "printf '%s-executed\\n' \"$AM_NAME\"" }),
+  });
+  await sleep(700);
+  const delayedTail = await (await fetch(`${base}/api/agents/${delayedId}/tail?lines=120`)).json();
+  check('stopped-session input waits through a draining startup TUI',
+    delivered.ok && (delayedTail.text || '').includes('INPUT-READY')
+      && (delayedTail.text || '').includes('input-readiness-executed'));
+  await fetch(`${base}/api/sessions/${delayedId}`, { method: 'DELETE' }).catch(() => {});
+
+  // OpenCode paints its stable welcome screen before its keyboard handler is
+  // always ready. The first typed attempt is silently drained here; delivery
+  // must wait for the composer to echo the real prompt before pressing Enter.
+  const openCode = await (await fetch(`${base}/api/sessions`, {
+    method: 'POST', headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({ cli: 'opencode', name: 'opencode-input-readiness', path: '.' }),
+  })).json();
+  const openCodeInput = 'opencode-delivery-survived';
+  const openCodeDelivered = await fetch(`${base}/api/sessions/${openCode.id}/input`, {
+    method: 'POST', headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({ text: openCodeInput }),
+  });
+  await sleep(700);
+  const openCodeTail = await (await fetch(`${base}/api/agents/${openCode.id}/tail?lines=120`)).json();
+  const executed = (openCodeTail.text || '').match(new RegExp(`OPENCODE-EXECUTED:${openCodeInput}`, 'g')) || [];
+  check('OpenCode delivery waits for the real input handler after stable paint',
+    openCodeDelivered.ok
+      && executed.length === 1,
+    `${openCodeDelivered.status} ${JSON.stringify(openCodeTail.text || '')}`);
+  await fetch(`${base}/api/sessions/${openCode.id}`, { method: 'DELETE' }).catch(() => {});
 } catch (err) {
   check('no exceptions', false, String(err && err.message ? err.message : err));
   console.log('--- server log tail ---\n' + bootLog.slice(-1200));

server/package.json

@@ -14,7 +14,7 @@
     "start": "node src/index.js",
     "dev": "node --watch src/index.js",
     "test:ui": "node terminal-ui.test.mjs",
-    "test": "node test/usage.test.mjs && node test/operations.test.mjs && node test/hidden.test.mjs && node test/slowfs.test.mjs && node test/spawn-group.test.mjs && node test/revive.test.mjs && node test/repin.test.mjs && node test/codex-repin.test.mjs && node test/opencode-resume.test.mjs && node test/terminal-modes.test.mjs && node test/trace-tail.test.mjs && node test/trace-window.test.mjs && node migration.test.mjs && node resize.test.mjs"
+    "test": "node state-checkpoint.test.mjs && node test/usage.test.mjs && node test/operations.test.mjs && node test/hidden.test.mjs && node test/slowfs.test.mjs && node test/spawn-group.test.mjs && node test/revive.test.mjs && node test/repin.test.mjs && node test/codex-repin.test.mjs && node test/opencode-resume.test.mjs && node test/terminal-modes.test.mjs && node test/trace-tail.test.mjs && node test/trace-window.test.mjs && node migration.test.mjs && node resize.test.mjs"
   },
   "engines": {
     "node": ">=20.19"

server/src/config.js

@@ -123,7 +123,7 @@ function isConfigured(id) {
         || fileOk(path.join(env.CODEX_HOME || path.join(home, '.codex'), 'auth.json'));
     case 'gemini':
       return hasEnv('GEMINI_API_KEY', 'GOOGLE_API_KEY')
-        || fileOk(path.join(home, '.gemini', 'oauth_creds.json'));
+        || fileOk(path.join(env.GEMINI_CLI_HOME || home, '.gemini', 'oauth_creds.json'));
     case 'opencode': {
       const xdgConfig = env.XDG_CONFIG_HOME || path.join(home, '.config');
       return hasEnv('ANTHROPIC_API_KEY', 'OPENAI_API_KEY', 'OPENROUTER_API_KEY')

server/src/index.js

@@ -19,7 +19,8 @@ import * as demo from './demo.js';
 import * as hidden from './hidden.js';
 import {
   attach, agentInfo, deriveState, stop, stopAll, ensureRunning, sendInput, isRunning,
-  capturePane, ghosttyReady, ghosttyError, installClaudeRepinHook, installOpencodeRepinPlugin,
+  waitForInputReady, capturePane, ghosttyReady, ghosttyError,
+  installClaudeRepinHook, installOpencodeRepinPlugin,
 } from './runner.js';
 
 // Control frames ride the terminal socket behind a leading NUL pair, which real
@@ -158,8 +159,14 @@ await Promise.race([
 // running instead of exiting.
 // No tmux to outlive us any more: kill the PTYs we hold on the way out so a
 // restart can't leave orphaned agents writing into the workspace.
+let shutdownStarted = false;
 for (const sig of ['SIGTERM', 'SIGINT']) {
-  process.on(sig, () => { try { stopAll(); } catch {} process.exit(0); });
+  process.on(sig, async () => {
+    if (shutdownStarted) return;
+    shutdownStarted = true;
+    try { await stopAll(); } catch {}
+    process.exit(0);
+  });
 }
 
 process.on('unhandledRejection', (e) => console.error('[unhandledRejection]', e));
@@ -314,8 +321,10 @@ async function deliver(session, text, from) {
     return false;
   }
   const started = ensureRunning(session);
-  if (started) await sleep(3500); // let the CLI boot before the keystrokes land
-  await sendInput(session.id, text);
+  if (started && !await waitForInputReady(session.id)) {
+    throw new Error('session did not become ready for input within 30 seconds — prompt was not sent');
+  }
+  await sendInput(session.id, text, { confirmEcho: started && session.cli === 'opencode' });
   return started;
 }
 
@@ -783,9 +792,14 @@ const BUILD_ENV_KEYS = (() => {
 // after the build-time snapshot, so its vars would otherwise be misdetected as
 // injected secrets. Keep this list in sync with entrypoint.sh.
 const NON_SECRET = new Set([
-  'HOME', 'CLAUDE_CONFIG_DIR', 'CODEX_HOME', 'NPM_CONFIG_PREFIX', 'PWD', 'OLDPWD', 'SHLVL', '_', 'HOSTNAME',
+  'HOME', 'CLAUDE_CONFIG_DIR', 'CLAUDE_DURABLE', 'CODEX_HOME', 'CODEX_DURABLE',
+  'GEMINI_CLI_HOME', 'GEMINI_LIVE', 'GEMINI_DURABLE',
+  'OPENCLAW_STATE_DIR', 'OPENCLAW_HOME', 'OPENCLAW_DURABLE',
+  'OPENCODE_LIVE', 'OPENCODE_DURABLE', 'HERMES_LIVE', 'HERMES_DURABLE',
+  'AGENT_STATE_SCRIPT', 'AGENT_STATE_CHECKPOINT_SECONDS', 'DATA_DIR',
+  'NPM_CONFIG_PREFIX', 'PWD', 'OLDPWD', 'SHLVL', '_', 'HOSTNAME',
   'ACCELERATOR', 'COMMIT_SHA', 'CPU_CORES', 'HF_DATASETS_TRUST_REMOTE_CODE', 'IMAGE_SHA', 'MEMORY', 'OMP_NUM_THREADS',
-  'UV_CACHE_DIR', 'PIP_CACHE_DIR', 'PYTHONPYCACHEPREFIX', 'PYTHONUSERBASE', 'OPENCLAW_STATE_DIR', 'OPENCLAW_HOME',
+  'UV_CACHE_DIR', 'PIP_CACHE_DIR', 'PYTHONPYCACHEPREFIX', 'PYTHONUSERBASE',
 ]);
 const NON_SECRET_PREFIX = ['SPACE_', 'KUBERNETES_', 'NVIDIA_', 'CUDA_', 'NV_', 'AM_'];
 function injectedEnvKeys() {
@@ -1385,10 +1399,14 @@ function skillTargetDirs() {
   const home = process.env.HOME || os.homedir();
   const claudeCfg = process.env.CLAUDE_CONFIG_DIR || path.join(home, '.claude');
   const dirs = [
-    path.join(home, '.agents', 'skills'),   // Codex, Gemini, opencode
+    path.join(home, '.agents', 'skills'),   // Codex and opencode
     path.join(claudeCfg, 'skills'),          // Claude Code
     path.join(home, '.hermes', 'skills'),    // Hermes
   ];
+  // GEMINI_CLI_HOME deliberately changes the home Gemini resolves its global
+  // .agents directory against; fan skills into that local/checkpointed home as
+  // well as the ordinary durable HOME.
+  if (process.env.GEMINI_CLI_HOME) dirs.push(path.join(process.env.GEMINI_CLI_HOME, '.agents', 'skills'));
   // OpenClaw runs with its own HOME (see entrypoint.sh) and reads managed
   // skills from ~/.agents/skills resolved against THAT home. Recreated on
   // every boot, so it needs no backup coverage.

server/src/runner.js

@@ -63,6 +63,13 @@ const bashLaunch = `exec bash --rcfile ${BASHRC} -i`;
 const BUSY_SECS = 4;
 // Re-rendering the grid to text on every chunk during a burst is wasteful.
 const SAMPLE_THROTTLE_MS = 250;
+// A prompt sent while a resumed TUI is still replaying its screen can be
+// discarded, or accept the text but swallow the Enter. Wait for the canonical
+// startup repaint and a genuinely quiet screen instead of guessing a boot
+// duration in the API layer.
+const INPUT_READY_QUIET_MS = Number(process.env.AM_INPUT_READY_QUIET_MS || BUSY_SECS * 1000);
+const INPUT_READY_TIMEOUT_MS = Number(process.env.AM_INPUT_READY_TIMEOUT_MS || 30000);
+const INPUT_ECHO_TIMEOUT_MS = Number(process.env.AM_INPUT_ECHO_TIMEOUT_MS || 20000);
 // Despite the Node wrapper's `scrollbackLimit` name, Ghostty's native option is
 // a byte budget. Passing a line count such as 20,000 retains only a small native
 // allocation (about 700 ordinary rows). Keep the unit explicit at our boundary.
@@ -1835,6 +1842,8 @@ export function ensureRunning(session, cols = 120, rows = 34) {
       console.error('[runner] history restore', error && error.message);
     }
   }
+  let resolveExit;
+  const exitPromise = new Promise((resolve) => { resolveExit = resolve; });
   const host = {
     id: session.id,
     runId,
@@ -1849,6 +1858,8 @@ export function ensureRunning(session, cols = 120, rows = 34) {
       .filter((line) => line.text && line.ansi).map((line) => [line.text, line.ansi])),
     terminalModes: createTerminalModeTracker(),
     startupHistory: captureResize ? persistedHistory : null,
+    exitPromise,
+    resolveExit,
     historyCheckpoint: null,
     traceHistoryPage: null,
     traceHistoryTimer: null,
@@ -1857,6 +1868,7 @@ export function ensureRunning(session, cols = 120, rows = 34) {
     gridTimer: null,
     startedAt: Date.now(),
     lastOutputAt: Date.now(),
+    outputSeq: 0,
     screenChangedAt: Date.now(),
     bells: 0,
   };
@@ -1882,6 +1894,7 @@ export function ensureRunning(session, cols = 120, rows = 34) {
 
   term.onData((chunk) => {
     host.lastOutputAt = Date.now();
+    host.outputSeq++;
     host.terminalModes.feed(chunk);
     if (host.traceHistoryTimer) {
       clearTimeout(host.traceHistoryTimer);
@@ -1941,6 +1954,7 @@ export function ensureRunning(session, cols = 120, rows = 34) {
     try { host.vt.dispose(); } catch {}
     for (const sub of host.subs) sub.onExit();
     host.subs.clear();
+    host.resolveExit();
   });
 
   hosts.set(session.id, host);
@@ -2028,7 +2042,7 @@ export function attach(session, cols, rows) {
 }
 
 /** Type a line into the session's terminal (works with no browser attached). */
-export async function sendInput(id, text) {
+export async function sendInput(id, text, { confirmEcho = false } = {}) {
   const host = hosts.get(id);
   if (!host) throw new Error('session is not running');
   // Multi-line prompts go in as a bracketed paste so the CLI's composer treats
@@ -2037,11 +2051,67 @@ export async function sendInput(id, text) {
   // The Enter must arrive as its OWN keypress: TUIs (codex) detect rapid input
   // bursts as a paste, and a CR inside the burst becomes a newline in the
   // composer instead of a submit. A short gap breaks the burst.
-  host.pty.write(payload);
+  if (confirmEcho) {
+    // OpenCode can finish its first stable paint several seconds before its
+    // input handler is installed. Probe with the real composer text, but do
+    // not press Enter until the TUI has painted that text back. Retrying only
+    // the unsubmitted composer is safe; Ctrl+U clears a late/partial attempt.
+    const expected = text.replace(/\s+/g, ' ').trim();
+    const probe = expected.slice(-Math.min(48, expected.length));
+    const deadline = Date.now() + INPUT_ECHO_TIMEOUT_MS;
+    let attempt = 0;
+    let echoed = false;
+    while (Date.now() < deadline && !echoed) {
+      if (hosts.get(id) !== host) throw new Error('session stopped while waiting for input acknowledgement');
+      if (attempt++) {
+        host.pty.write('\x15'); // clear any attempt accepted too late to paint
+        await new Promise((r) => setTimeout(r, 100));
+      }
+      const beforeOutput = host.outputSeq;
+      let beforeScreen = '';
+      try { beforeScreen = host.vt.getVisibleText(); } catch {}
+      host.pty.write(payload);
+      const attemptDeadline = Math.min(deadline, Date.now() + 1200);
+      while (Date.now() < attemptDeadline) {
+        if (hosts.get(id) !== host) throw new Error('session stopped while waiting for input acknowledgement');
+        let screen = '';
+        try { screen = host.vt.getVisibleText(); } catch {}
+        echoed = host.outputSeq > beforeOutput && screen !== beforeScreen
+          && screen.replace(/\s+/g, ' ').includes(probe);
+        if (echoed) break;
+        await new Promise((r) => setTimeout(r, 50));
+      }
+    }
+    if (!echoed) throw new Error('session did not acknowledge the input before the timeout — prompt was not submitted');
+  } else {
+    host.pty.write(payload);
+  }
   await new Promise((r) => setTimeout(r, 300));
   host.pty.write('\r');
 }
 
+/**
+ * Wait until a newly-started pane can safely receive its first input.
+ *
+ * Resumed primary-screen TUIs may paint a welcome frame, pause, then replay a
+ * large conversation. `resizeCapture` spans that whole transaction when a
+ * durable terminal seed exists. The quiet-window check covers fresh panes and
+ * older sessions without a terminal seed. On timeout callers get an explicit
+ * failure instead of silently losing the operator's prompt.
+ */
+export async function waitForInputReady(id, timeoutMs = INPUT_READY_TIMEOUT_MS) {
+  const deadline = Date.now() + Math.max(0, timeoutMs);
+  while (Date.now() < deadline) {
+    const host = hosts.get(id);
+    if (!host) throw new Error('session stopped while waiting for input readiness');
+    const lastActivity = Math.max(host.startedAt || 0, host.lastOutputAt || 0, host.screenChangedAt || 0);
+    if (!host.startupHistory && !host.resizeCapture
+      && Date.now() - lastActivity >= INPUT_READY_QUIET_MS) return true;
+    await new Promise((resolve) => setTimeout(resolve, 100));
+  }
+  return false;
+}
+
 /**
  * The session's rendered screen plus `lines` of scrollback above it — what a
  * human would see in the pane. Used by the agent API so one agent can watch
@@ -2075,8 +2145,19 @@ export function stop(id) {
  * Kill every session. Without tmux nothing outlives this process, so a clean
  * shutdown should not leave orphaned PTYs behind holding the workspace.
  */
-export function stopAll() {
-  for (const host of hosts.values()) {
+export async function stopAll(timeoutMs = 1000) {
+  const active = [...hosts.values()];
+  for (const host of active) {
     try { host.pty.kill(); } catch {}
   }
+  if (!active.length) return;
+
+  // Give each CLI a bounded chance to handle SIGHUP, close its transcript, and
+  // flush final local state before PID 1 takes the shutdown checkpoint.
+  let timer;
+  await Promise.race([
+    Promise.allSettled(active.map((host) => host.exitPromise)),
+    new Promise((resolve) => { timer = setTimeout(resolve, timeoutMs); }),
+  ]);
+  if (timer) clearTimeout(timer);
 }

server/state-checkpoint.test.mjs

@@ -0,0 +1,201 @@
+// Crash/recovery checks for scripts/agent-state.sh. Everything runs on local
+// temporary directories; no mounted bucket or real agent state is touched.
+import { spawn, execFileSync } from 'node:child_process';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const root = fs.mkdtempSync(path.join(os.tmpdir(), 'am-agent-state-'));
+const data = path.join(root, 'durable');
+const local = path.join(root, 'local');
+const script = path.resolve(here, '../scripts/agent-state.sh');
+
+const env = {
+  ...process.env,
+  DATA_DIR: data,
+  AM_LOCAL: local,
+  CODEX_HOME: path.join(local, 'codex'),
+  CODEX_DURABLE: path.join(data, 'state/codex'),
+  CLAUDE_CONFIG_DIR: path.join(local, 'claude'),
+  CLAUDE_DURABLE: path.join(data, 'state/claude'),
+  GEMINI_CLI_HOME: path.join(local, 'gemini-home'),
+  GEMINI_LIVE: path.join(local, 'gemini-home/.gemini'),
+  GEMINI_DURABLE: path.join(data, 'state/gemini'),
+  OPENCLAW_STATE_DIR: path.join(local, 'openclaw'),
+  OPENCLAW_DURABLE: path.join(data, 'state/openclaw'),
+  OPENCODE_LIVE: path.join(local, 'opencode'),
+  OPENCODE_DURABLE: path.join(data, 'state/opencode'),
+  HERMES_LIVE: path.join(local, 'hermes'),
+  HERMES_DURABLE: path.join(data, 'state/hermes'),
+};
+
+let failures = 0;
+const check = (name, ok, detail = '') => {
+  console.log(`${ok ? 'PASS' : 'FAIL'}  ${name}${detail ? `  ${detail}` : ''}`);
+  if (!ok) failures++;
+};
+const put = (file, body) => {
+  fs.mkdirSync(path.dirname(file), { recursive: true });
+  fs.writeFileSync(file, body);
+};
+const run = (mode, options = {}) => execFileSync('sh', [script, mode], {
+  env, encoding: 'utf8', stdio: options.stdio || 'pipe',
+});
+const sql = (db, statement) => execFileSync('sqlite3', [db, statement], { encoding: 'utf8' }).trim();
+const waitFor = (child, marker) => new Promise((resolve, reject) => {
+  let out = '';
+  const timer = setTimeout(() => reject(new Error(`timed out waiting for ${marker}: ${out}`)), 5000);
+  child.stdout.on('data', (chunk) => {
+    out += chunk;
+    if (out.includes(marker)) { clearTimeout(timer); resolve(); }
+  });
+  child.on('exit', (code) => { clearTimeout(timer); reject(new Error(`sqlite writer exited ${code}: ${out}`)); });
+});
+
+try {
+  execFileSync('sh', ['-n', script]);
+  check('checkpoint helper has valid POSIX shell syntax', true);
+
+  const codexRel = 'sessions/2026/08/04/rollout-test.jsonl';
+  const claudeRel = 'projects/work/session.jsonl';
+  const geminiRel = 'tmp/work/chats/session-test.jsonl';
+  const clawRel = 'agents/main/sessions/session.jsonl';
+  put(path.join(env.CODEX_DURABLE, codexRel), '{"turn":1}\n');
+  put(path.join(env.CLAUDE_DURABLE, claudeRel), '{"turn":1}\n');
+  put(path.join(env.GEMINI_DURABLE, geminiRel), '{"turn":1}\n');
+  put(path.join(env.OPENCLAW_DURABLE, clawRel), '{"turn":1}\n');
+
+  run('restore');
+  check('Codex rollout restores to a real local directory',
+    fs.readFileSync(path.join(env.CODEX_HOME, codexRel), 'utf8') === '{"turn":1}\n'
+      && !fs.lstatSync(path.join(env.CODEX_HOME, 'sessions')).isSymbolicLink());
+  check('Claude transcript restores locally',
+    fs.readFileSync(path.join(env.CLAUDE_CONFIG_DIR, claudeRel), 'utf8') === '{"turn":1}\n');
+  check('Gemini transcript restores under GEMINI_CLI_HOME',
+    fs.readFileSync(path.join(env.GEMINI_LIVE, geminiRel), 'utf8') === '{"turn":1}\n');
+  check('OpenClaw transcript restores locally',
+    fs.readFileSync(path.join(env.OPENCLAW_STATE_DIR, clawRel), 'utf8') === '{"turn":1}\n');
+
+  // Re-running entrypoint in the same container must not replace newer live
+  // state with an older bucket checkpoint.
+  const liveClaude = path.join(env.CLAUDE_CONFIG_DIR, claudeRel);
+  const durableClaude = path.join(env.CLAUDE_DURABLE, claudeRel);
+  put(durableClaude, '{"turn":"stale"}\n');
+  put(liveClaude, '{"turn":"new-local"}\n');
+  const now = Date.now() / 1000;
+  fs.utimesSync(durableClaude, now - 60, now - 60);
+  fs.utimesSync(liveClaude, now, now);
+  run('restore');
+  check('hot restart preserves newer local state',
+    fs.readFileSync(liveClaude, 'utf8') === '{"turn":"new-local"}\n');
+
+  // A first deployment can inherit a populated local tree but have no local
+  // timestamp yet. Restore must force that tree through one checkpoint instead
+  // of assuming every byte came from the durable side.
+  fs.rmSync(path.join(local, 'agent-state-stamps/claude'));
+  run('restore');
+  run('checkpoint');
+  check('first checkpoint captures pre-existing newer local state',
+    fs.readFileSync(durableClaude, 'utf8') === '{"turn":"new-local"}\n');
+
+  // SQLite-backed harnesses also have ordinary files (setup/auth/config)
+  // before their database is first created. Those files must not disappear
+  // merely because there is no database to snapshot yet.
+  const openPreDb = path.join(env.OPENCODE_LIVE, 'setup.json');
+  const hermesPreDb = path.join(env.HERMES_LIVE, 'config.yaml');
+  put(openPreDb, '{"configured":true}\n');
+  put(hermesPreDb, 'configured: true\n');
+  run('checkpoint');
+  check('opencode ordinary files checkpoint before opencode.db exists',
+    fs.readFileSync(path.join(env.OPENCODE_DURABLE, 'setup.json'), 'utf8') === '{"configured":true}\n');
+  check('Hermes ordinary files checkpoint before state.db exists',
+    fs.readFileSync(path.join(env.HERMES_DURABLE, 'config.yaml'), 'utf8') === 'configured: true\n');
+
+  // Reproduce the Codex access pattern: append while holding the canonical
+  // rollout FD open. Because the canonical file is now local, a different
+  // process can checkpoint the visible bytes without waiting for close.
+  const liveRollout = path.join(env.CODEX_HOME, codexRel);
+  const held = spawn(process.execPath, ['-e', `
+    const fs = require('fs');
+    const fd = fs.openSync(process.argv[1], 'a');
+    fs.writeSync(fd, '{"turn":2}\\n');
+    process.stdout.write('ready\\n');
+    setInterval(() => {}, 1000);
+  `, liveRollout], { stdio: ['ignore', 'pipe', 'pipe'] });
+  await waitFor(held, 'ready');
+  run('checkpoint');
+  check('open Codex rollout bytes reach a closed durable checkpoint',
+    fs.readFileSync(path.join(env.CODEX_DURABLE, codexRel), 'utf8').endsWith('{"turn":2}\n'));
+  held.kill('SIGKILL');
+
+  fs.rmSync(env.CODEX_HOME, { recursive: true, force: true });
+  run('restore');
+  check('rollout survives writer crash and fresh-local restore',
+    fs.readFileSync(path.join(env.CODEX_HOME, codexRel), 'utf8').endsWith('{"turn":2}\n'));
+
+  // Keep an opencode writer alive in WAL mode. A raw copy can observe the DB
+  // and WAL at different instants; SQLite .backup must still yield one valid
+  // database containing the committed row.
+  fs.mkdirSync(env.OPENCODE_LIVE, { recursive: true });
+  const openDb = path.join(env.OPENCODE_LIVE, 'opencode.db');
+  const writer = spawn('sqlite3', [openDb], { stdio: ['pipe', 'pipe', 'pipe'] });
+  writer.stdin.write('PRAGMA journal_mode=WAL;\n');
+  writer.stdin.write('PRAGMA wal_autocheckpoint=0;\n');
+  writer.stdin.write('CREATE TABLE message(id INTEGER PRIMARY KEY, body TEXT);\n');
+  writer.stdin.write("INSERT INTO message(body) VALUES ('survives');\n");
+  writer.stdin.write('.print writer-ready\n');
+  await waitFor(writer, 'writer-ready');
+  run('checkpoint');
+
+  const openCheckpoint = path.join(env.OPENCODE_DURABLE, 'checkpoints/opencode.db');
+  check('opencode online backup is structurally valid', sql(openCheckpoint, 'PRAGMA quick_check;') === 'ok');
+  check('opencode online backup includes committed WAL content',
+    sql(openCheckpoint, 'SELECT body FROM message;') === 'survives');
+  check('opencode checkpoint publishes no WAL/SHM companions',
+    !fs.existsSync(`${openCheckpoint}-wal`) && !fs.existsSync(`${openCheckpoint}-shm`));
+  const idleMarker = new Date('2000-01-01T00:00:00.000Z');
+  fs.utimesSync(openCheckpoint, idleMarker, idleMarker);
+  run('checkpoint');
+  check('idle SQLite database does not generate another durable write',
+    fs.statSync(openCheckpoint).mtimeMs === idleMarker.getTime());
+  const writerExited = new Promise((resolve) => writer.once('exit', resolve));
+  writer.stdin.end();
+  await writerExited;
+
+  sql(openDb, "INSERT INTO message(body) VALUES ('newer-local');");
+  run('restore');
+  check('hot restart does not replace a newer valid local SQLite database',
+    sql(openDb, 'SELECT count(*) FROM message;') === '2');
+
+  fs.mkdirSync(env.HERMES_LIVE, { recursive: true });
+  const hermesDb = path.join(env.HERMES_LIVE, 'state.db');
+  sql(hermesDb, 'CREATE TABLE messages(body TEXT); INSERT INTO messages VALUES (\'hello\');');
+  run('checkpoint');
+  const hermesCheckpoint = path.join(env.HERMES_DURABLE, 'checkpoints/state.db');
+  check('Hermes uses the same verified SQLite checkpoint adapter',
+    sql(hermesCheckpoint, 'SELECT body FROM messages;') === 'hello');
+
+  // A bad live DB must not replace the last known-good durable checkpoint.
+  const openRowsBeforeCorruption = sql(openCheckpoint, 'SELECT body FROM message ORDER BY id;');
+  fs.writeFileSync(openDb, 'not a database');
+  let badFailed = false;
+  try { run('checkpoint'); } catch { badFailed = true; }
+  check('invalid live SQLite makes the checkpoint fail closed', badFailed);
+  check('failed SQLite checkpoint leaves prior durable snapshot intact',
+    sql(openCheckpoint, 'SELECT body FROM message ORDER BY id;') === openRowsBeforeCorruption);
+
+  fs.rmSync(env.OPENCODE_LIVE, { recursive: true, force: true });
+  run('restore');
+  check('SQLite snapshot restores without stale WAL state',
+    sql(path.join(env.OPENCODE_LIVE, 'opencode.db'), 'SELECT body FROM message ORDER BY id;') === openRowsBeforeCorruption
+      && !fs.existsSync(path.join(env.OPENCODE_LIVE, 'opencode.db-wal')));
+} catch (error) {
+  check('no unexpected exception', false, error?.stack || String(error));
+} finally {
+  fs.rmSync(root, { recursive: true, force: true });
+}
+
+console.log(failures ? `\n${failures} FAILURE(S)` : '\nall agent-state checks passed');
+process.exit(failures ? 1 : 0);