docs/OPERATIONS_AND_TROUBLESHOOTING.md

Operations and Troubleshooting

Diagnostic order

1. Confirm Space is RUNNING.
2. Fetch /futures and inspect response/hash headers.
3. Check /api/futures/status with authentication.
4. Inspect runtime file status and datasource metadata.
5. Check Browser Console and Network.
6. Inspect market endpoint for one symbol/interval.
7. Compare DS4 raw/normalized fields and timestamps.
8. Check provider-specific diagnostics.
9. Run one analysis-only request.
10. Do not test Paper Execute during diagnosis.

Common issues

Dashboard returns 401

Likely causes:

• missing/incorrect HERMES_ADMIN_PASSWORD;
• wrong Basic username;
• browser session expired;
• reverse proxy did not preserve auth.

Actions:

• confirm HERMES_ADMIN_USERNAME and HERMES_DASHBOARD_BASIC_AUTH_USERNAME alignment;
• re-authenticate;
• verify entrypoint.sh logged successful auth configuration;
• never print the password in logs or reports.

/futures returns 200 but old UI appears

Possible causes:

• stale installed overlay;
• wrong template path;
• restored old overlay taking precedence;
• CDN/browser cache;
• duplicate old page implementation.

Actions:

• compare X-Hermes-Template-SHA256 with body hash;
• inspect application.runtimeFiles in status;
• confirm /opt/hermesface_overlay is preferred;
• confirm installed /opt/hermes/tools/templates/... matches manifest;
• hard reload only after server-side evidence is checked.

Runtime status is unknown

unknown means evidence is absent, not that files match.

Actions:

• verify manifest path and permissions;
• verify overlay and runtime paths exist;
• check HERMES_FUTURES_OVERLAY_MANIFEST, HERMES_OVERLAY_SOURCE, and HERMES_SYNC_SCRIPT overrides;
• inspect overlay installation logs.

Runtime status is mismatch

Actions:

• identify exact mismatched file in status payload;
• compare repository/image overlay and /opt/hermes file;
• verify sync_hf.py installed after persistence restore;
• rebuild/redeploy from a clean commit.

Chart says market data unavailable

Check market endpoint payload:

• HTTP 503 and API_UNAVAILABLE: acquisition exception;
• state=unavailable: no real candles/current price;
• state=stale: provider timestamp invalid or stale;
• state=partial: freshness unknown or display fields incomplete.

Inspect:

warnings
missingFields
analysisRequiredFieldsMissing
staleRequiredFields
sourceMetadata
technicalDiagnostics

Do not replace missing candles with mock data.

Binance shows HTTP 451

This is an expected regional limitation in some Hugging Face regions.

Correct behavior:

transportStatus=restricted
httpStatus=451
dataUsability=unavailable
reason=Regionally restricted

Do not use raw-IP or TLS-bypass workarounds. DS4 safety remains authoritative.

KuCoin reports “Parameter 'from' must be milliseconds”

Verify DS4 request builder uses build_kucoin_time_range() and sends integer millisecond from and to. Check for upstream code that converts an already-millisecond value a second time.

Market data is HTTP 200 but readiness is blocked

Transport and readiness are separate. Inspect:

• DS4 Futures verification;
• DS4 noTradeGuard;
• missing critical fields;
• non-fresh critical fields;
• merge rejection reasons.

A provider may be reachable while its data is unusable.

NO_TRADE displayed before analysis

The initial state must be NOT_ANALYZED. Check frontend initialization and /api/futures/status.analysisState. A network error must be ANALYSIS_FAILED or API_UNAVAILABLE, not NO_TRADE.

Signal score shows zero with no components

The UI must show Unavailable. Check whether latestSignalScore is null and whether signal components are empty. Do not coerce null to zero.

Execute button is disabled

This is normally correct. Read the visible reason and inspect:

latest plan exists
planId matches
symbol and risk match
plan not expired
LONG/SHORT decision
verified Futures
risk approved
noTradeGuard false
tradingReadiness ready
executable true
not already executed

Telegram says Owner setup required

No configured/persisted owner exists. Use the one-time private /claim <TELEGRAM_BOOTSTRAP_SECRET> flow. Remove/rotate the bootstrap secret after claim. Do not expose owner ID in dashboard status.

Telegram proactive alerts unavailable

Webhook responses can work without outbound connectivity, but proactive alerts need either:

• direct Telegram access with optional proxy; or
• configured relay URL and HMAC secret.

Keep polling disabled.

Logs and artifacts

Useful logs:

Space build log
entrypoint startup log
sync_hf overlay install/hash log
Hermes dashboard log under /opt/data/logs
sanitized runtime audit JSON
browser Console and Network export without credentials

Never attach raw cookies, Authorization headers, tokens, or unredacted provider payloads.