docs/TESTING_AND_VERIFICATION.md

Testing and Verification

Complete release suite

python -m pytest hermes_overlay/tests -q \
  --cov=hermes_overlay \
  --cov=scripts \
  --cov-branch \
  --cov-report=term-missing \
  --cov-report=json:coverage.json \
  --cov-report=xml:coverage.xml
python scripts/check_critical_coverage.py coverage.json

A release result must identify the complete collected count. A selected subset may be reported only as a targeted subsystem check and never as the complete suite.

Static and metadata gates

python -m compileall -q app.py hermes_overlay scripts
python -m ruff check app.py hermes_overlay scripts
bash -n scripts/entrypoint.sh
node --check scripts/dns-fix.cjs
find hermes_overlay -type f -name '*.js' -print0 | xargs -0 -r -n1 node --check
python scripts/generate_dependency_inventory.py --check
python scripts/verify_repository_metadata.py

Safety, data, and failure gates

python scripts/ci_secret_scan.py
python scripts/check_dependency_licenses.py --declared-from requirements.lock requirements-ci.lock
python scripts/validate_startup_configuration.py
python scripts/check_migrations.py
python scripts/verify_required_failures.py
python scripts/verify_ci_failure_contract.py
python scripts/validate_runbook_consistency.py

CI additionally runs pip-audit, a clean exact-pinned container build, and a Trivy HIGH/CRITICAL container scan. Those checks require network and Docker-capable infrastructure and remain blocking when executed.

Targeted iteration policy

Run the smallest affected test module after each atomic subsystem change. Run broader affected groups after migrations, shared configuration, execution boundaries, or infrastructure contracts change. Run the complete suite exactly before release evidence generation and rerun it only when a subsequent change can affect results.

Required negative behavior

Tests must prove that:

• direct LLM execute, close, leverage, and mutation tools are absent or rejected;
• HERMES_EXECUTION_ENABLED=false blocks new entries;
• Testnet/Live modes remain unavailable without protected gates;
• invalid, zero, negative, NaN, infinite, stale, malformed, or ambiguous inputs fail closed;
• missing upstream integration markers stop build/startup;
• secret fixtures fail scanning;
• intentionally failing tests return a nonzero process result;
• migration failures roll back and do not mark versions applied;
• missing evidence, dependency drift, or runbook contradictions fail validation.

Runtime and browser verification

Perform authenticated read-only checks for status, symbols, positions, market data, analysis, health, evidence, metrics, source provenance, freshness, and runtime file hashes. Browser checks cover Console/Network errors, responsive rendering, chart controls, disabled execution reasons, and explicit unavailable states. Do not click Execute during release verification.

Evidence rules

Record exact command, UTC timestamp, exit code, complete test count, coverage report, environment limitations, and SHA-256 hashes of delivered files. Do not report CONSISTENT when the validator is bypassed or hashes are stale.