| # DibaGuard — high availability (roadmap item 12) |
|
|
| > **Open-release copy** ([DibaAi/DibaGuard-Open](https://huggingface.co/DibaAi/DibaGuard-Open)). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (`http_v2`) were measured with that model, which is **not** part of the open release; the open release ships the licence-clean network models described in `CLEAN_MODELS.md`. Licence statements inside this document are superseded by `DATA_LICENSES.md`, `CLEAN_MODELS.md` and the repository's `LICENSE`, `LICENSE-WEIGHTS` and `NOTICE` files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited. |
|
|
| What survives a restart, what moves with a host, and how several replicas behind one load balancer stay one guard. |
| Everything is in `dibaguard/state.py`; tests in `tests/test_state.py`. |
|
|
| ## 1. The pieces |
|
|
| | piece | what it does | |
| |---|---| |
| | `snapshot(layer, dir)` / `restore(layer, dir)` | one versioned, integrity-checked `.npz` holding everything that matters operationally (below); atomic write (temp file + fsync + rename), keeps the newest K per replica; restore takes the newest *valid* one (a corrupted newest file falls back to the previous) | |
| | `Snapshotter` | periodic snapshots from the sidecar (`maybe()` every `every_s`), `restore_on_start()` before serving | |
| | `ReplicaSync` + `SharedState` | active-active: the shared enforcement view, operator lifts / allowlist, evidence federation; `FileSharedState` (a directory on shared storage, no locks) or `RedisSharedState` (optional, `pip install redis`) | |
| | `affinity(ip, replicas)` | the replica that owns a source: rendezvous hash of its /24 (/64 for IPv6) | |
| | `Housekeeper` / `from_env` | one background thread in the sidecar: sync, snapshots, optional `tick` | |
|
|
| ### What a snapshot holds |
|
|
| | part | content | depends on the model files? | |
| |---|---|---| |
| | `guard` | conformal calibration scores, rolling references (tier 1 / seq / HTTP, Mondrian service x age cells, per age), anomaly references, the immutable anchor, the re-calibration buffer (up to `recal_max_rows`), source and destination evidence (fast + slow CUSUM), entity risk, tier-0 stream counters per source / destination (with their remembered host:port keys), GRU states of warm sources, per-host HTTP site profiles + references, drift martingale, site pressure, global failure / scan rates, the alarm log | yes | |
| | `layer` | per-host HTTP bootstrap and re-profile buffers, the payload-provenance ring (web request -> DB query), playbook counts, posture findings | yes (with `guard`) | |
| | `db` | known application clients and query shapes per database, per-client volume references, the DbNet reference, decayed rows / bytes / auth-failure counters, operator-approved shapes, DB alarm log, MySQL thread map | yes (DbNet reference) | |
| | `enforcement` | block / challenge / rate-limit decisions with **absolute** expiry times, recent audit | no | |
| | `feedback` | operator verdicts, allowlist entries with their TTL | no | |
| | `rules` | the operator's rules (deduplicated against rules already loaded from the policy) | no | |
| | `allowlist` | the runtime allowlist (merged); the whole runtime policy with `policy_mode="replace"` | no | |
|
|
| Any other non-scalar attribute of the `Guard`, the `Layer` and the `DbGuard` — state added after this module was |
| written, e.g. the campaign trackers, the drift-guard events, the reference-hygiene counters, the layer's request-rate / |
| user-agent / SQL-context tables — is captured by a generic encoder: dicts (order kept), sets, tuples, deques, arrays, |
| per-key numeric tables, and instances of an explicit allowlist of DibaGuard classes (rebuilt with `__new__`, no |
| constructor, no import by name). Scalars are treated as configuration and are not restored. Anything the encoder |
| cannot represent is listed in `stats["not_snapshotted"]` of the snapshot result, and the round-trip test fails while |
| that list is not empty. |
|
|
| Bounded when huge: per-source tables keep every source in force, then up to `max_sources` (default 250 000) |
| others — at most half the riskiest (decayed risk, fast evidence, slow evidence, warm GRU sources), the rest the most |
| recently seen; `max_destinations` (100 000) likewise; logs keep their tail; `buffer_rows=N` trims the |
| re-calibration buffer (`0` drops it: the next epoch then re-calibrates from the anchor + new traffic only). |
|
|
| ### File format (schema 1) |
|
|
| One `.npz`, always loaded with `allow_pickle=False` (a pickled member is refused, never executed — tested): |
|
|
| * `manifest` — uint8 JSON: magic, schema, created, replica, host, **model fingerprint** (sha256 per component: |
| tier-1 trees, sequence net, HTTP model incl. its shipped site profile, DbNet; and an id over all), **code-layout id** |
| (feature names, horizons, memory, age edges, anomaly / destination feature indices), parts, stats, and the |
| **digest**: sha256 — or HMAC-SHA256 when `DIBAGUARD_SNAPSHOT_KEY` is set — over the manifest itself and every |
| other member (name, dtype, shape, bytes). |
| * `state` — uint8 JSON: the structure; numeric tables are referenced by member name. |
| * `keys` — uint8 JSON: one shared key table; every address / host / (host, port) is stored once and referenced by |
| int64 index. |
| * `a0 … aN` — numeric arrays only (float64 / float32 / int64 / int8). |
|
|
| Restore refuses: a bad digest (corrupted or tampered, including a tampered manifest), an unknown schema, a signed |
| snapshot without the key, an unsigned one when a key is configured, a member with an object dtype, a zip that |
| expands past 8 GB. A snapshot of **other model files or another feature layout** raises `FingerprintMismatch`; |
| `restore(..., force=True)` then brings back only `enforcement, feedback, rules, allowlist`. |
| `Snapshotter.restore_on_start()` does that by itself (`on_mismatch="independent"`), so blocks and operator |
| decisions survive a model upgrade while every reference is re-learned. Restore decodes everything before it |
| assigns anything: a failing restore leaves the layer as it was. |
|
|
| ## 2. Deployment patterns |
|
|
| ### 2.1 One node that restarts |
|
|
| ```sh |
| DIBAGUARD_SNAPSHOT_DIR=/var/lib/dibaguard/snap DIBAGUARD_SNAPSHOT_EVERY_S=60 \ |
| python -m dibaguard.llm http --models models --policy policy.json # (needs the hook in section 6) |
| ``` |
|
|
| On start: newest valid snapshot of this replica, expired decisions dropped, blocks put back into the host firewall |
| for their *remaining* time. On a clean stop `Housekeeper.stop()` writes a final snapshot (RPO 0 for a clean stop). |
|
|
| ### 2.2 Active-passive with snapshot shipping |
|
|
| ``` |
| VIP / DNS (keepalived, cloud LB health check) |
| | : |
| [active: proxy + sidecar] [passive: proxy + sidecar, not serving] |
| | snapshot every 30-60 s |
| +--> /var/lib/dibaguard/snap/ --rsync/scp every 30 s--> passive:/var/lib/dibaguard/snap/ |
| (or both mount one replicated volume) |
| ``` |
|
|
| * Ship only complete files: snapshots are renamed into place, so `rsync --include 'dibaguard-snap-*.npz' |
| --exclude '.*'` never copies a half-written one; the receiver verifies the digest anyway |
| (`state.verify(path)`). |
| * Promotion: start the passive's sidecar; `restore_on_start()` takes its own snapshots first and, when it has none, |
| the active's (`replica` filter `None`): a moved / promoted host restores the newest snapshot of any replica. |
| * Same model files on both nodes, or the promoted node restores only the model-independent parts. |
| * Fencing is the VIP manager's job (keepalived / the cloud LB); DibaGuard does not elect leaders. |
| * Set `DIBAGUARD_SNAPSHOT_KEY` on both nodes when the shipping path is not trusted. |
|
|
| ### 2.3 Active-active behind HAProxy / nginx with source affinity |
|
|
| Each replica is a fast path (`dibaguard-proxy` or `dibaguard-authz`) plus its own Python sidecar. The load balancer |
| sends one source to one replica (its evidence accumulates there); the shared view makes every replica enforce what |
| any replica decided, so a source that moves (failover, scale-out) is still blocked everywhere. |
|
|
| ``` |
| HAProxy / nginx (hash of the client /24 -> replica) |
| / | \ |
| [lb-a: proxy+sidecar] [lb-b: proxy+sidecar] [lb-c: proxy+sidecar] |
| \_______________ shared view ________________/ |
| /mnt/dibaguard (NFS/SMB/replicated volume) or Redis |
| ``` |
|
|
| HAProxy (2.6+, `balance hash`; older: `balance source` = per address, not per /24): |
|
|
| ``` |
| backend dibaguard |
| mode tcp # or http |
| balance hash src,ipmask(24,64) # one /24 (IPv6 /64) -> one replica |
| hash-type consistent # adding / removing a replica moves ~1/n of the sources |
| server lb-a 10.0.0.11:8080 check send-proxy-v2 |
| server lb-b 10.0.0.12:8080 check send-proxy-v2 |
| server lb-c 10.0.0.13:8080 check send-proxy-v2 |
| # replicas: dibaguard-proxy -proxy-protocol -trusted <haproxy CIDR> -challenge-secret <the same on every replica> |
| ``` |
|
|
| nginx (L7; `hash ... consistent` is ketama): |
|
|
| ``` |
| map $remote_addr $dg_net { # IPv4 /24; IPv6 hashes per address unless you add a /64 map |
| ~^(?<n>\d+\.\d+\.\d+)\.\d+$ $n; |
| default $remote_addr; |
| } |
| upstream dibaguard { |
| hash $dg_net consistent; |
| server 10.0.0.11:8080; server 10.0.0.12:8080; server 10.0.0.13:8080; |
| } |
| # proxy_set_header X-Forwarded-For $remote_addr; replicas: -trusted <nginx CIDR> |
| ``` |
|
|
| On every replica: |
|
|
| ```sh |
| DIBAGUARD_REPLICA=lb-a DIBAGUARD_SHARED_DIR=/mnt/dibaguard DIBAGUARD_SYNC_EVERY_S=5 \ |
| DIBAGUARD_SNAPSHOT_DIR=/var/lib/dibaguard/snap/lb-a DIBAGUARD_SNAPSHOT_EVERY_S=300 \ |
| python -m dibaguard.llm http --models models --policy policy.json |
| # or DIBAGUARD_REDIS_URL=redis://redis:6379/0 instead of DIBAGUARD_SHARED_DIR |
| ``` |
|
|
| Rules for active-active: |
|
|
| * **Unique replica ids.** `FileSharedState` refuses an id whose heartbeat is fresh from another host. |
| * **Same policy, rules and model files** everywhere (configuration management); rules and policy are not synced. |
| * **The real client address** must reach the replica (PROXY protocol or a trusted `X-Forwarded-For`); hashing the |
| LB's own address would put every source on one replica. |
| * **Shared `-challenge-secret`**, so a challenged client that moves keeps a valid cookie. |
| * **One HAProxy ACL syncer per HAProxy instance.** `dibaguard-authz -haproxy-socket` owns its ACL / map and deletes |
| every entry it did not push; with the shared view every replica's block set converges, but several syncers on one |
| HAProxy would fight during the sync period. Run the syncer from one replica (any: they all hold the cluster set). |
| * **Clocks within ~1 s (NTP)**: expiries are absolute and a lift cancels decisions published before it. |
| * `affinity()` is for routing DibaGuard does itself (a NetFlow / IPFIX fan-out to collector replicas, a log |
| shipper, a summary forwarder). Its hash differs from HAProxy's / nginx's: within one routing layer use one of |
| them consistently. Both are consistent hashes; `tests/test_state.py` checks that `affinity()` is stable, |
| independent of list order, balanced (each of 4 replicas 25 % +- 2 %), moves only the removed replica's sources, |
| moves ~1/5 to a fifth replica, and honours weights. |
|
|
| How the shared view behaves (`state.merge`): each replica publishes only what changed — new / escalated / extended |
| decisions with their absolute expiry, operator lifts (an entry gone before its expiry, or a `reject` / |
| `benign_source` verdict, which also resets that source's risk and evidence on every replica), allowlist entries. |
| For one source the strongest live action wins with the latest expiry of all live decisions (what one Enforcer would |
| have done had it seen every verdict); a lift cancels every decision published at or before it, on every replica, |
| and a later decision stands. Imported blocks are applied to the local host firewall too, so the proxy's fast path |
| (polling `guard_fast_state`) and HAProxy's ACL syncer (`dibaguard-authz -haproxy-socket`) enforce the cluster's |
| decisions. A replica that restarts empty, or joins later, converges on the view at its first sync. |
|
|
| Evidence federation: every replica publishes its top-K (1000) sources by decayed fast / slow evidence (those above |
| 10 % of the threshold). A source with evidence on two or more replicas is judged on the **sum** against the engine's |
| own conformal threshold (`alpha/4`, the same as the evidence tier); a hit is logged as a `federated` alarm and goes |
| through the policy like any verdict (a named family from the replicas' alarms -> its action; none -> `attack_generic` |
| -> monitor by default). A source seen on one replica only is left to that replica's engine. |
|
|
| `FileSharedState` layout (no locks; every replica appends / replaces only its own files): |
|
|
| ``` |
| <root>/enforce/<replica>.jsonl header {"gen"} + append-only records; compaction rewrites it (new gen, atomic rename) |
| <root>/evidence/<replica>.json latest evidence summary (atomic replace) |
| <root>/replicas/<replica>.json heartbeat (host, pid, time) |
| ``` |
|
|
| Readers keep a byte offset per journal and read only appended complete lines; a new generation means re-read. |
|
|
| ## 3. Measured RPO / RTO |
|
|
| Machine: Intel Core 12th gen (Family 6 Model 151), 8 logical CPUs, Windows 11, local NVMe; Python 3.13.3, numpy |
| 2.2.5. Real engine and Layer, test-fixture models (tier 1: 60 trees depth 4; GRU d=16, 2 layers; HTTP 4096 buckets). |
| Traffic: a site calibrated on 60 000 flows from 3 000 clients, then new sources with 4 flows each. Reproduce with |
| `PYTHONPATH=. python scripts/ha_bench.py` (ranges below = two runs). |
|
|
| | tracked sources | re-calibration buffer rows | warm GRU sources | snapshot size | engine lock held | compress + hash + write | restore (load, verify, decode, apply) | |
| |---:|---:|---:|---:|---:|---:|---:| |
| | 10 000 | 88 000 | ~1 000 | 8.2 MB (5.7 MB without buffer) | 0.05-0.09 s | 1.0-1.1 s | 0.20-0.24 s | |
| | 50 000 | 248 000 | ~6 800 | 19.5 MB (13.3 MB) | 0.29-0.38 s | 2.7-3.0 s | 0.63-0.76 s | |
| | 100 000 | 400 000 (cap) | ~14 000 | 30.7 MB (21.1 MB) | 0.59-0.73 s | 4.4-4.7 s | 1.13-1.34 s | |
|
|
| About 320 bytes per tracked source with the buffer at its cap, ~220 without. Only the "lock held" column stalls |
| scoring (live structures are copied; key encoding, compression, hashing and the write run after the lock). The GRU |
| part grows with the real model: `layers x d x 4` bytes per warm source. |
|
|
| Replica sync (`FileSharedState`, local disk, two replicas): |
|
|
| | decisions per replica | evidence sources per replica | first sync (imports all) | steady sync | journal | evidence summary | |
| |---:|---:|---:|---:|---:|---:| |
| | 1 000 | 10 000 | 0.03-0.05 s | 0.02 s | 98 kB | 39 kB | |
| | 10 000 | 100 000 | 0.16-0.40 s | 0.12-0.14 s | 0.99 MB | 40 kB | |
|
|
| (Sync holds the engine lock only while reading / changing the layer; journal and evidence I/O run outside it.) |
|
|
| What that means: |
|
|
| * **RPO, one node / active-passive:** the snapshot interval + ~5 s (write time at 100k sources). With 60 s |
| snapshots, at most ~65 s of evidence, risk and reference updates are lost on a crash; a clean stop loses nothing. |
| Enforcement decisions in force at the last snapshot come back with their original expiry. |
| * **RPO, active-active:** enforcement decisions, lifts and allowlist entries: one sync period (5 s default) — they |
| live in the shared view, not in one process. Per-source evidence of the crashed replica: its snapshot interval |
| (federation summaries of the other replicas still hold its top sources only for as long as they decay). |
| * **RTO:** process start + model load + restore (~1.1-1.3 s per 100k sources) + one firewall command per restored |
| block. In active-active the LB health check removes the dead replica (its sources re-hash to the others, which |
| already enforce its decisions); the restarted replica converges at its first sync. |
|
|
| ## 4. Failure modes |
|
|
| | failure | what happens | |
| |---|---| |
| | sidecar crash / kill -9 | restart restores the newest valid snapshot; decisions expired meanwhile are dropped, the rest are re-applied to the firewall for their remaining time | |
| | crash while writing a snapshot | the temp file (`.dibaguard-snap-*.tmp`) is never renamed; it is ignored, deleted after an hour; the previous snapshot stays valid | |
| | corrupted / truncated / tampered snapshot | digest (or zip CRC) mismatch -> `SnapshotError`; a directory restore falls back to the previous snapshot (`rep["skipped"]` says why) | |
| | model upgrade | fingerprint mismatch: `restore()` refuses, `restore_on_start()` restores enforcement / feedback / rules / allowlist and the engine re-calibrates from live traffic | |
| | code upgrade that changes the feature layout | layout-id mismatch: handled like a model change | |
| | host loss (active-passive) | promote the passive: it restores the last shipped snapshot (RPO = ship interval + snapshot interval) | |
| | replica loss (active-active) | LB re-hashes its sources; the others already enforce its decisions; their evidence restarts on the new owner, federation sums what the dead replica published until it decays | |
| | shared storage slow / unavailable | sync I/O runs outside the engine lock; a failed sync is reported by the Housekeeper and retried; each replica keeps enforcing its own decisions (degrades to independent replicas) | |
| | network partition | each side keeps its own decisions and converges when the partition heals; a lift made on one side reaches the other then (no quorum, no strong consistency) | |
| | clock skew | expiries are absolute: a replica whose clock is ahead lifts imported decisions early; a lift with a skewed time may cancel (or miss) a decision published within the skew; keep NTP | |
| | two hosts with one replica id | refused at start while the other's heartbeat is fresh (< 60 s) | |
| | LB scale-out / rehash | ~1/n of the sources move; their evidence restarts on the new owner, and is judged on its total through federation for as long as the old owner's summary holds it | |
| | Windows rename while a reader holds a file | retried; a compaction that still fails is skipped until the next round | |
| | redis down (RedisSharedState) | the sync raises, the Housekeeper records it and retries; replicas run independently meanwhile | |
|
|
| ## 5. What is NOT covered |
|
|
| * **No consensus.** The shared view is eventually consistent (one sync period); there is no leader election, quorum |
| or fencing. Active-passive failover needs an external VIP manager. |
| * **No confidentiality at rest.** HMAC gives integrity, not secrecy. A snapshot holds client addresses, request |
| texts (HTTP buffers, provenance ring), database query shapes and the alarm log: treat the snapshot directory and |
| the shared directory as sensitive (file permissions, encrypted volume). |
| * **Not synced between replicas:** rules, the policy, per-host HTTP profiles and references, the alarm log and |
| review queue (each replica reviews its own alarms; only the verdicts' lifts / allowlist propagate), the DB tier. |
| Rules and policy come from configuration management. |
| * **Federation covers source evidence only** (fast and slow CUSUM). Stream counters, GRU states, entity risk and the |
| destination-flood tier stay per replica; a summed evidence is an approximation (each replica floors its own CUSUM |
| at zero) and only the top-K sources per replica are exchanged. |
| * **Bounded snapshots** drop cold sources beyond `max_sources`; they start fresh after a restore. |
| * **Zero RPO** only for a clean stop; a crash loses what happened since the last snapshot (enforcement excepted in |
| active-active). |
| * **Tested on a local filesystem only** (this round); NFS / SMB semantics (close-to-open consistency, rename |
| atomicity) are relied upon but not exercised. The Redis backend is tested against an in-memory fake with the same |
| calls, not a server. |
| * The Go fast path still polls only its local sidecar; it learns other replicas' decisions through that sidecar |
| (one poll interval later). |
| * nftables: when an action's TTL is *extended*, the kernel set element keeps its original timeout (pre-existing |
| Enforcer behaviour); restored and imported blocks are added with their remaining time. |
|
|
| ## 6. Wiring (hook in the sidecar) |
|
|
| `state.py` needs no change to existing modules. To turn it on in the HTTP sidecar, `dibaguard/llm.py` needs: |
|
|
| ```python |
| # in main(), after the Dispatcher with a Layer exists and the server (make_server) is built: |
| from dibaguard import state |
| srv, cons = make_server(disp, a.host, a.port) |
| hk = state.from_env(disp.layer, cons.lock) if disp.layer is not None else None # restore on start + background thread |
| try: |
| srv.serve_forever() |
| finally: |
| if hk is not None: |
| hk.stop() # final snapshot |
| ``` |
|
|
| `from_env` reads `DIBAGUARD_SNAPSHOT_DIR`, `DIBAGUARD_SNAPSHOT_EVERY_S`, `DIBAGUARD_SNAPSHOT_KEEP`, |
| `DIBAGUARD_SNAPSHOT_KEY`, `DIBAGUARD_RESTORE`, `DIBAGUARD_SHARED_DIR` or `DIBAGUARD_REDIS_URL`, `DIBAGUARD_REPLICA`, |
| `DIBAGUARD_SYNC_EVERY_S`, `DIBAGUARD_TICK`; with none set it returns `None` and nothing changes. |
|
|
| `state.py` reads these existing private attributes, so a rename there must be mirrored in `state.py` (the |
| round-trip test's exact-state check fails if one is missed): `Guard._buffer`, `Guard._anchor`, `Guard._log()`, |
| `Layer._http_boot_by / _http_recent_by / _http_last_by / _http_last / _http_ring`, `Enforcer._record()`, |
| `Policy._allow`, `entity._Rec`, `database._MYSQL_THREADS`. A new piece of engine / layer state must be added to |
| `_guard_out` / `_guard_in` (or `_layer_*`, `_db_*`). |
|
|
