docs/DEPLOYMENT_RUNBOOK.md

# Deployment Runbook

## Safety baseline

Hermes Futures Desk starts in Paper mode. Keep these values unchanged unless a protected operator gate has been approved:

```text
TRADING_MODE=paper
HERMES_EXECUTION_ENABLED=false
HERMES_NONPAPER_EXECUTION_ENABLED=false
```

Never place credentials in source, the runbook, evidence files, container layers, screenshots, or general backups.

## Supported build inputs

- Python `3.13.5`
- Node `22.16.0`
- upstream Hermes commit `8fc278207b0f5b25e567966f9615e1b1737f62af`
- exact Python locks in `requirements.lock` and `requirements-ci.lock`
- deterministic dependency inventory in `docs/dependency_inventory.json`

## Local verification

```bash
python scripts/generate_dependency_inventory.py --check
python scripts/verify_repository_metadata.py
python scripts/validate_startup_configuration.py
python scripts/check_migrations.py
python -m pytest hermes_overlay/tests -q
python scripts/validate_runbook_consistency.py
```

Static checks used by CI:

```bash
python -m compileall -q app.py hermes_overlay scripts
python -m ruff check app.py hermes_overlay scripts
bash -n scripts/entrypoint.sh
node --check scripts/dns-fix.cjs
find hermes_overlay -type f -name '*.js' -print0 | xargs -0 -r -n1 node --check
```

## Container build

```bash
docker build \
  --pull \
  --build-arg HERMES_UPSTREAM_REF=8fc278207b0f5b25e567966f9615e1b1737f62af \
  --tag hermesface:local \
  .

docker run --rm --entrypoint cat hermesface:local /opt/hermes/HERMES_UPSTREAM_COMMIT
```

The build must fail when the upstream revision is unavailable, the checkout differs from the requested commit, required source markers are missing, dependency installation fails, or the overlay hash verification fails.

## Startup sequence

`entrypoint.sh` validates configuration before restore or mutation, activates `/opt/hermes/.venv`, initializes persistent directories, and calls `scripts/sync_hf.py`. Critical router, authentication, tool-registration, CSP/CORS, public-path, and overlay integration failures terminate startup; they do not degrade to warning-and-continue behavior.

## Database and migration procedure

SQLite is permitted only for isolated local Paper operation. Production-shaped deployments must configure PostgreSQL, while Testnet and Live execution remain disabled by independent gates.

```text
TRADING_MODE=paper
HERMES_EXECUTION_ENABLED=false
HERMES_DATABASE_DIALECT=postgresql
HERMES_DATABASE_URL=postgresql://<injected-at-runtime>
```

Do not place the DSN in source, shell history, evidence, or screenshots. Supply it through the protected runtime secret store. The pinned runtime driver is `psycopg[binary]==3.3.4`.

Offline migration contract and isolated Paper verification:

```bash
python scripts/validate_startup_configuration.py
python scripts/validate_postgresql_migrations.py
python scripts/check_migrations.py
python -m pytest -q \
  hermes_overlay/tests/test_postgresql_migration_architecture.py \
  hermes_overlay/tests/test_domain_layer.py \
  hermes_overlay/tests/test_domain_durability_extended.py
```

Protected Desktop/CI PostgreSQL smoke check, using a disposable database and a DSN injected by the operator:

```bash
export HERMES_ENV=production
export TRADING_MODE=paper
export HERMES_EXECUTION_ENABLED=false
export HERMES_DATABASE_DIALECT=postgresql
export HERMES_DATABASE_URL="$PROTECTED_DISPOSABLE_POSTGRES_DSN"
python - <<'PY'
from trading.domain.schema import apply_migrations, validate_schema
ok, messages = apply_migrations(); print(ok, messages); assert ok
ok, messages = validate_schema(); print(ok, messages); assert ok
PY
```

The PostgreSQL runner obtains a session-level advisory migration lock, verifies immutable migration checksums, applies each migration in its own transaction, and releases the lock on success or failure. An unknown migration or checksum mismatch fails startup. Never edit an applied migration; add a new forward migration.

### Backup before migration

For local Paper SQLite, stop writers or use SQLite's online backup API through the sanitized backup tooling. Do not copy a live WAL database with ordinary file copy. Verify the produced manifest and database integrity before retaining it.

For PostgreSQL, create a custom-format logical backup from a protected operator shell. Keep credentials outside command arguments and evidence:

```bash
pg_dump --format=custom --no-owner --no-privileges \
  --file hermes-pre-migration.dump "$HERMES_DATABASE_URL"
pg_restore --list hermes-pre-migration.dump > hermes-pre-migration.contents
sha256sum hermes-pre-migration.dump hermes-pre-migration.contents
```

Store the dump and checksum manifest in encrypted operator-controlled storage. General application backups must exclude session tokens, password hashes where policy requires, and credential material; database recovery archives are protected security artifacts, not routine evidence attachments.

### Failure and forward recovery

1. Keep execution disabled and Paper-only.
2. Capture the sanitized error class, failed migration version, current application revision, and `schema_migrations` rows. Never capture the DSN.
3. Do not manually mark a failed migration as applied. PostgreSQL transactional DDL rolls back the failed migration; prior successful versions remain authoritative.
4. Correct the migration in a new unapplied revision, run the offline migration tests, and re-run against a disposable copy of the protected backup.
5. Apply forward to the target database, validate all required tables/constraints, then run the complete test and read-only runtime audit gates.
6. If data recovery is required, restore into a new database, validate checksums and schema, then switch the protected DSN. Do not overwrite the only recoverable database in place.

A source rollback is allowed only when its code supports the already-applied schema. Database down-migrations are not automatic and require a separately reviewed operator procedure.

## Read-only deployed audit

```bash
export HERMES_ADMIN_PASSWORD='set-outside-source'
export HERMES_DASHBOARD_BASIC_AUTH_USERNAME='admin'
export HF_TOKEN='set-outside-source' # only for a private Space
python scripts/verify_futures_runtime.py \
  --base-url https://example.invalid \
  --symbol BTCUSDT \
  --analyze \
  --report .runtime_audit/futures_runtime_audit.json
```

The audit must not call Paper Execute. Verify authenticated status/symbol/position/market endpoints, canonical candle shape, source freshness, runtime hashes, disabled execution state, and explicit structured unavailability rather than fabricated data.

## Rollback

1. Select the last verified immutable revision.
2. Revert the smallest faulty change.
3. Rebuild from the pinned inputs.
4. Re-run configuration, migration, complete tests, container scan, and read-only runtime audit.
5. Confirm Paper-only mode, kill switches, webhook-only Telegram behavior, and port `7860`.

## Retained evidence

Keep the source revision, build manifest, dependency inventory, migration result, complete test result, coverage result, secret/dependency/license/container scan results, runtime audit, and exact file hashes. Evidence containing secrets or session material is invalid.