docs/monitoring.md

Reported health in an existing Prometheus workflow

Lumi Eggcracker operators with an already-supported installation can publish the existing read-only doctor report to an existing node_exporter textfile collector. This source-checkout integration adds no listener, daemon permission, package dependency, scheduler or installation step. It reports supervisor health, not measured containment effectiveness, protection, authenticated evidence or independent adoption. Native support and qualification boundaries are unchanged.

Collect once

Use Python 3.11 or newer from this reviewed source checkout, as an operator who already has permission to query Eggcracker. Prepare an existing absolute output directory owned by that operator and not writable by another user or group. Configure an existing same-UID node_exporter to read that directory. Then, from the checkout:

python3 scripts/export_monitoring_metrics.py --output /var/lib/eggcracker-metrics/health.prom

The example path is illustrative; choose an explicit neutral path you own. The collector does not create directories, grant access, install timers or change socket permissions. Node_exporter needs read/traverse permission; the output is created mode 0600, so this integration supports an existing same-UID exporter with directory traversal permission. A differently-owned exporter is outside this slice. Do not rely on chmod or ACL changes to an old output inode: atomic replacement creates a new mode-0600 file each time.

Exit 0 means a valid report was published (which may report unhealthy); exit 1 means the query failed or selected fields were unavailable and fresh query-invalid telemetry was published; exit 2 means publication failed. Errors are bounded and do not include raw response, exception, workload or path details. An existing output is replaced only if it is regular, single-link, operator-owned, not group/world writable and starts with the exact collector ownership marker. Never put that marker on an unrelated file. This is deliberate ownership recognition, not cryptographic provenance. Symlink/reparse targets and parents, unsafe parent ownership/permissions, and unrelated output are refused.

The output directory and ancestors must remain trusted and must not be concurrently mutated. System sticky temporary ancestors are allowed, but the output directory itself must be private. This is not protection against a hostile directory owner. Writes use an exclusive same-directory non-.prom temporary file then atomic replacement. A sibling .prom.lock is created exclusively before the query; overlap is refused, not queued. After an interrupted process, an operator must confirm no collector remains before removing its lock. Nothing automatically removes another invocation's lock. Failed replacement retains the prior metrics, whose age exposes publication failure; do not mistake that retained health for a fresh collection.

Wire into existing monitoring

Run the command every 60 seconds using your existing operator-controlled job runner. No scheduler is installed here. The unchanged client timeout is 30 seconds per operation, not a promised total deadline. Observe exit codes and arrange job duration supervision in your existing runner; the exclusive lock prevents overlap. Correct both collector and Prometheus clocks with your existing time synchronisation. The rules allow 60 seconds of future skew; larger skew is unknown freshness and suppresses health/query alerts.

Load eggcracker.rules.yml through your existing Prometheus rule_files configuration. Opt in each intended node_exporter scrape target with the target label eggcracker_monitor: "true". For example, merge these labels into your existing static target inventory:

static_configs:
  - targets: ["node-one:9100", "node-two:9100"]
    labels:
      eggcracker_monitor: "true"

Use unique (job, instance) identities per exporter and do not drop or rewrite these identity/opt-in labels on Eggcracker metrics. Keep expected targets in inventory even if a collector has never run: removing a target also removes the ability to alert on its absence. The rules use per-target up inventory, not global absent(), so one reporting host cannot hide another missing host. Exporter-down is a separate alert and suppresses Eggcracker missing/stale/health alerts. Missing output on a reachable exporter, stale output and query failure are distinct. A fresh, valid report of active incident lockdown has its own two-minute warning; clearing lockdown resolves it. This warning is also gated by the target opt-in label, a valid query and collection freshness/clock-skew checks. These rules do not monitor Prometheus's own availability.

Rules evaluate every 30 seconds and require two minutes of continuous condition. Collection is stale when older than 180 seconds, so stopping a fresh collector normally alerts after more than five minutes plus scrape/evaluation latency. Query-invalid, reported-unhealthy, missing and exporter-down conditions have the same two-minute pending period. Recovery clears the matching condition on the next relevant scrape/evaluation; no application restart is performed.

Fixed metric contract

All metrics are gauges without Eggcracker labels; timestamps are gauge values, never Prometheus sample timestamps. No names, PIDs, paths, arguments, environment values, catalogue data, event labels or cumulative detection counts are exported.

Metric prefix eggcracker_	Meaning
query_valid	1 only if all selected fields have recognised types/values; otherwise 0
collection_timestamp_seconds	Unix clock at the completed collection attempt
reported_ready	Exact boolean autonomous_discovery
discovery_healthy	Exact boolean discovery.healthy
receipt_storage_healthy	Exact boolean discovery.receipt_persistence_healthy
installation_healthy	1 for HEALTHY; 0 for DRIFT, RECOVERY_REQUIRED or NOT_INSTALLED
incident_lockdown	Exact boolean incidents.lockdown reported by the supervisor

The health and incident-lockdown gauges are absent when the query is invalid. Missing fields, unknown installation enums and nonboolean health or lockdown values make the query invalid; they are never exported as zero. Freshness checks gate health/query/lockdown alerts. The lockdown metric and warning report only the supervisor's local incident state; they do not establish containment effectiveness. This privacy contract covers Eggcracker-produced metrics, not the entire node_exporter endpoint: node_exporter's node_textfile_mtime_seconds includes a file-path label. Use neutral paths, restrict the existing endpoint appropriately and sanitise shared evidence.

Verification boundary

The unit journey uses the additive strict doctor-only length-framed client against a disposable unprivileged Unix socket, asserting only doctor with empty arguments. It is not a native daemon or containment test. The separate opt-in CI consumer test runs exact pinned test-only node_exporter on loopback with default collectors disabled and only textfile enabled, then real promtool evaluates these rules. Fixtures cover healthy, query loss, stopped/stale collection, recovery, missing output, two targets with one absent, exporter-down, clock skew, active and cleared incident lockdown, invalid/missing lockdown data, and alert pending delays. No user monitoring account, production endpoint, secret or additional proof run is needed. Test-only upstream licences/notices remain with downloaded archives.