docs/DEPLOYMENT_RUNBOOK.md

Deployment Runbook

Safety baseline

Hermes Futures Desk starts in Paper mode. Keep these values unchanged unless a protected operator gate has been approved:

TRADING_MODE=paper
HERMES_EXECUTION_ENABLED=false
HERMES_NONPAPER_EXECUTION_ENABLED=false

Never place credentials in source, the runbook, evidence files, container layers, screenshots, or general backups.

Supported build inputs

• Python 3.13.5
• Node 22.16.0
• upstream Hermes commit 8fc278207b0f5b25e567966f9615e1b1737f62af
• exact Python locks in requirements.lock and requirements-ci.lock
• deterministic dependency inventory in docs/dependency_inventory.json

Local verification

python scripts/generate_dependency_inventory.py --check
python scripts/verify_repository_metadata.py
python scripts/validate_startup_configuration.py
python scripts/check_migrations.py
python -m pytest hermes_overlay/tests -q
python scripts/validate_runbook_consistency.py

Static checks used by CI:

python -m compileall -q app.py hermes_overlay scripts
python -m ruff check app.py hermes_overlay scripts
bash -n scripts/entrypoint.sh
node --check scripts/dns-fix.cjs
find hermes_overlay -type f -name '*.js' -print0 | xargs -0 -r -n1 node --check

Container build

docker build \
  --pull \
  --build-arg HERMES_UPSTREAM_REF=8fc278207b0f5b25e567966f9615e1b1737f62af \
  --tag hermesface:local \
  .

docker run --rm --entrypoint cat hermesface:local /opt/hermes/HERMES_UPSTREAM_COMMIT

The build must fail when the upstream revision is unavailable, the checkout differs from the requested commit, required source markers are missing, dependency installation fails, or the overlay hash verification fails.

Startup sequence

entrypoint.sh validates configuration before restore or mutation, activates /opt/hermes/.venv, initializes persistent directories, and calls scripts/sync_hf.py. Critical router, authentication, tool-registration, CSP/CORS, public-path, and overlay integration failures terminate startup; they do not degrade to warning-and-continue behavior.

Database and migration procedure

SQLite is permitted only for isolated local Paper operation. Production-shaped deployments must configure PostgreSQL, while Testnet and Live execution remain disabled by independent gates.

TRADING_MODE=paper
HERMES_EXECUTION_ENABLED=false
HERMES_DATABASE_DIALECT=postgresql
HERMES_DATABASE_URL=postgresql://<injected-at-runtime>

Do not place the DSN in source, shell history, evidence, or screenshots. Supply it through the protected runtime secret store. The pinned runtime driver is psycopg[binary]==3.3.4.

Offline migration contract and isolated Paper verification:

python scripts/validate_startup_configuration.py
python scripts/validate_postgresql_migrations.py
python scripts/check_migrations.py
python -m pytest -q \
  hermes_overlay/tests/test_postgresql_migration_architecture.py \
  hermes_overlay/tests/test_domain_layer.py \
  hermes_overlay/tests/test_domain_durability_extended.py

Protected Desktop/CI PostgreSQL smoke check, using a disposable database and a DSN injected by the operator:

export HERMES_ENV=production
export TRADING_MODE=paper
export HERMES_EXECUTION_ENABLED=false
export HERMES_DATABASE_DIALECT=postgresql
export HERMES_DATABASE_URL="$PROTECTED_DISPOSABLE_POSTGRES_DSN"
python - <<'PY'
from trading.domain.schema import apply_migrations, validate_schema
ok, messages = apply_migrations(); print(ok, messages); assert ok
ok, messages = validate_schema(); print(ok, messages); assert ok
PY

The PostgreSQL runner obtains a session-level advisory migration lock, verifies immutable migration checksums, applies each migration in its own transaction, and releases the lock on success or failure. An unknown migration or checksum mismatch fails startup. Never edit an applied migration; add a new forward migration.

Backup before migration

For local Paper SQLite, stop writers or use SQLite's online backup API through the sanitized backup tooling. Do not copy a live WAL database with ordinary file copy. Verify the produced manifest and database integrity before retaining it.

For PostgreSQL, create a custom-format logical backup from a protected operator shell. Keep credentials outside command arguments and evidence:

pg_dump --format=custom --no-owner --no-privileges \
  --file hermes-pre-migration.dump "$HERMES_DATABASE_URL"
pg_restore --list hermes-pre-migration.dump > hermes-pre-migration.contents
sha256sum hermes-pre-migration.dump hermes-pre-migration.contents

Store the dump and checksum manifest in encrypted operator-controlled storage. General application backups must exclude session tokens, password hashes where policy requires, and credential material; database recovery archives are protected security artifacts, not routine evidence attachments.

Failure and forward recovery

1. Keep execution disabled and Paper-only.
2. Capture the sanitized error class, failed migration version, current application revision, and schema_migrations rows. Never capture the DSN.
3. Do not manually mark a failed migration as applied. PostgreSQL transactional DDL rolls back the failed migration; prior successful versions remain authoritative.
4. Correct the migration in a new unapplied revision, run the offline migration tests, and re-run against a disposable copy of the protected backup.
5. Apply forward to the target database, validate all required tables/constraints, then run the complete test and read-only runtime audit gates.
6. If data recovery is required, restore into a new database, validate checksums and schema, then switch the protected DSN. Do not overwrite the only recoverable database in place.

A source rollback is allowed only when its code supports the already-applied schema. Database down-migrations are not automatic and require a separately reviewed operator procedure.

Read-only deployed audit

export HERMES_ADMIN_PASSWORD='set-outside-source'
export HERMES_DASHBOARD_BASIC_AUTH_USERNAME='admin'
export HF_TOKEN='set-outside-source' # only for a private Space
python scripts/verify_futures_runtime.py \
  --base-url https://example.invalid \
  --symbol BTCUSDT \
  --analyze \
  --report .runtime_audit/futures_runtime_audit.json

The audit must not call Paper Execute. Verify authenticated status/symbol/position/market endpoints, canonical candle shape, source freshness, runtime hashes, disabled execution state, and explicit structured unavailability rather than fabricated data.

Rollback

1. Select the last verified immutable revision.
2. Revert the smallest faulty change.
3. Rebuild from the pinned inputs.
4. Re-run configuration, migration, complete tests, container scan, and read-only runtime audit.
5. Confirm Paper-only mode, kill switches, webhook-only Telegram behavior, and port 7860.

Retained evidence

Keep the source revision, build manifest, dependency inventory, migration result, complete test result, coverage result, secret/dependency/license/container scan results, runtime audit, and exact file hashes. Evidence containing secrets or session material is invalid.