docs/OPERATIONS_AND_TROUBLESHOOTING.md

# Operations and Troubleshooting

## Diagnostic order

1. Confirm Space is `RUNNING`.
2. Fetch `/futures` and inspect response/hash headers.
3. Check `/api/futures/status` with authentication.
4. Inspect runtime file status and datasource metadata.
5. Check Browser Console and Network.
6. Inspect market endpoint for one symbol/interval.
7. Compare DS4 raw/normalized fields and timestamps.
8. Check provider-specific diagnostics.
9. Run one analysis-only request.
10. Do not test Paper Execute during diagnosis.

## Common issues

### Dashboard returns 401

Likely causes:

- missing/incorrect `HERMES_ADMIN_PASSWORD`;
- wrong Basic username;
- browser session expired;
- reverse proxy did not preserve auth.

Actions:

- confirm `HERMES_ADMIN_USERNAME` and `HERMES_DASHBOARD_BASIC_AUTH_USERNAME` alignment;
- re-authenticate;
- verify `entrypoint.sh` logged successful auth configuration;
- never print the password in logs or reports.

### `/futures` returns 200 but old UI appears

Possible causes:

- stale installed overlay;
- wrong template path;
- restored old overlay taking precedence;
- CDN/browser cache;
- duplicate old page implementation.

Actions:

- compare `X-Hermes-Template-SHA256` with body hash;
- inspect `application.runtimeFiles` in status;
- confirm `/opt/hermesface_overlay` is preferred;
- confirm installed `/opt/hermes/tools/templates/...` matches manifest;
- hard reload only after server-side evidence is checked.

### Runtime status is `unknown`

`unknown` means evidence is absent, not that files match.

Actions:

- verify manifest path and permissions;
- verify overlay and runtime paths exist;
- check `HERMES_FUTURES_OVERLAY_MANIFEST`, `HERMES_OVERLAY_SOURCE`, and `HERMES_SYNC_SCRIPT` overrides;
- inspect overlay installation logs.

### Runtime status is `mismatch`

Actions:

- identify exact mismatched file in status payload;
- compare repository/image overlay and `/opt/hermes` file;
- verify `sync_hf.py` installed after persistence restore;
- rebuild/redeploy from a clean commit.

### Chart says market data unavailable

Check market endpoint payload:

- HTTP 503 and `API_UNAVAILABLE`: acquisition exception;
- `state=unavailable`: no real candles/current price;
- `state=stale`: provider timestamp invalid or stale;
- `state=partial`: freshness unknown or display fields incomplete.

Inspect:

```text
warnings
missingFields
analysisRequiredFieldsMissing
staleRequiredFields
sourceMetadata
technicalDiagnostics
```

Do not replace missing candles with mock data.

### Binance shows HTTP 451

This is an expected regional limitation in some Hugging Face regions.

Correct behavior:

```text
transportStatus=restricted
httpStatus=451
dataUsability=unavailable
reason=Regionally restricted
```

Do not use raw-IP or TLS-bypass workarounds. DS4 safety remains authoritative.

### KuCoin reports “Parameter 'from' must be milliseconds”

Verify DS4 request builder uses `build_kucoin_time_range()` and sends integer millisecond `from` and `to`. Check for upstream code that converts an already-millisecond value a second time.

### Market data is HTTP 200 but readiness is blocked

Transport and readiness are separate. Inspect:

- DS4 Futures verification;
- DS4 `noTradeGuard`;
- missing critical fields;
- non-fresh critical fields;
- merge rejection reasons.

A provider may be reachable while its data is unusable.

### `NO_TRADE` displayed before analysis

The initial state must be `NOT_ANALYZED`. Check frontend initialization and `/api/futures/status.analysisState`. A network error must be `ANALYSIS_FAILED` or `API_UNAVAILABLE`, not `NO_TRADE`.

### Signal score shows zero with no components

The UI must show `Unavailable`. Check whether `latestSignalScore` is `null` and whether signal components are empty. Do not coerce null to zero.

### Execute button is disabled

This is normally correct. Read the visible reason and inspect:

```text
latest plan exists
planId matches
symbol and risk match
plan not expired
LONG/SHORT decision
verified Futures
risk approved
noTradeGuard false
tradingReadiness ready
executable true
not already executed
```

### Telegram says Owner setup required

No configured/persisted owner exists. Use the one-time private `/claim <TELEGRAM_BOOTSTRAP_SECRET>` flow. Remove/rotate the bootstrap secret after claim. Do not expose owner ID in dashboard status.

### Telegram proactive alerts unavailable

Webhook responses can work without outbound connectivity, but proactive alerts need either:

- direct Telegram access with optional proxy; or
- configured relay URL and HMAC secret.

Keep polling disabled.

## Logs and artifacts

Useful logs:

```text
Space build log
entrypoint startup log
sync_hf overlay install/hash log
Hermes dashboard log under /opt/data/logs
sanitized runtime audit JSON
browser Console and Network export without credentials
```

Never attach raw cookies, Authorization headers, tokens, or unredacted provider payloads.