DibaGuard: public licence-clean release (Dibachain)

5d97275 verified 4 days ago

21.6 kB

DibaGuard — high availability (roadmap item 12)

Open-release copy (DibaAi/DibaGuard-Open). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (http_v2) were measured with that model, which is not part of the open release; the open release ships the licence-clean network models described in CLEAN_MODELS.md. Licence statements inside this document are superseded by DATA_LICENSES.md, CLEAN_MODELS.md and the repository's LICENSE, LICENSE-WEIGHTS and NOTICE files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.

What survives a restart, what moves with a host, and how several replicas behind one load balancer stay one guard. Everything is in dibaguard/state.py; tests in tests/test_state.py.

1. The pieces

piece	what it does
`snapshot(layer, dir)` / `restore(layer, dir)`	one versioned, integrity-checked `.npz` holding everything that matters operationally (below); atomic write (temp file + fsync + rename), keeps the newest K per replica; restore takes the newest valid one (a corrupted newest file falls back to the previous)
`Snapshotter`	periodic snapshots from the sidecar (`maybe()` every `every_s`), `restore_on_start()` before serving
`ReplicaSync` + `SharedState`	active-active: the shared enforcement view, operator lifts / allowlist, evidence federation; `FileSharedState` (a directory on shared storage, no locks) or `RedisSharedState` (optional, `pip install redis`)
`affinity(ip, replicas)`	the replica that owns a source: rendezvous hash of its /24 (/64 for IPv6)
`Housekeeper` / `from_env`	one background thread in the sidecar: sync, snapshots, optional `tick`

What a snapshot holds

part	content	depends on the model files?
`guard`	conformal calibration scores, rolling references (tier 1 / seq / HTTP, Mondrian service x age cells, per age), anomaly references, the immutable anchor, the re-calibration buffer (up to `recal_max_rows`), source and destination evidence (fast + slow CUSUM), entity risk, tier-0 stream counters per source / destination (with their remembered host:port keys), GRU states of warm sources, per-host HTTP site profiles + references, drift martingale, site pressure, global failure / scan rates, the alarm log	yes
`layer`	per-host HTTP bootstrap and re-profile buffers, the payload-provenance ring (web request -> DB query), playbook counts, posture findings	yes (with `guard`)
`db`	known application clients and query shapes per database, per-client volume references, the DbNet reference, decayed rows / bytes / auth-failure counters, operator-approved shapes, DB alarm log, MySQL thread map	yes (DbNet reference)
`enforcement`	block / challenge / rate-limit decisions with absolute expiry times, recent audit	no
`feedback`	operator verdicts, allowlist entries with their TTL	no
`rules`	the operator's rules (deduplicated against rules already loaded from the policy)	no
`allowlist`	the runtime allowlist (merged); the whole runtime policy with `policy_mode="replace"`	no

Any other non-scalar attribute of the Guard, the Layer and the DbGuard — state added after this module was written, e.g. the campaign trackers, the drift-guard events, the reference-hygiene counters, the layer's request-rate / user-agent / SQL-context tables — is captured by a generic encoder: dicts (order kept), sets, tuples, deques, arrays, per-key numeric tables, and instances of an explicit allowlist of DibaGuard classes (rebuilt with __new__, no constructor, no import by name). Scalars are treated as configuration and are not restored. Anything the encoder cannot represent is listed in stats["not_snapshotted"] of the snapshot result, and the round-trip test fails while that list is not empty.

Bounded when huge: per-source tables keep every source in force, then up to max_sources (default 250 000) others — at most half the riskiest (decayed risk, fast evidence, slow evidence, warm GRU sources), the rest the most recently seen; max_destinations (100 000) likewise; logs keep their tail; buffer_rows=N trims the re-calibration buffer (0 drops it: the next epoch then re-calibrates from the anchor + new traffic only).

File format (schema 1)

One .npz, always loaded with allow_pickle=False (a pickled member is refused, never executed — tested):

manifest — uint8 JSON: magic, schema, created, replica, host, model fingerprint (sha256 per component: tier-1 trees, sequence net, HTTP model incl. its shipped site profile, DbNet; and an id over all), code-layout id (feature names, horizons, memory, age edges, anomaly / destination feature indices), parts, stats, and the digest: sha256 — or HMAC-SHA256 when DIBAGUARD_SNAPSHOT_KEY is set — over the manifest itself and every other member (name, dtype, shape, bytes).
state — uint8 JSON: the structure; numeric tables are referenced by member name.
keys — uint8 JSON: one shared key table; every address / host / (host, port) is stored once and referenced by int64 index.
a0 … aN — numeric arrays only (float64 / float32 / int64 / int8).

Restore refuses: a bad digest (corrupted or tampered, including a tampered manifest), an unknown schema, a signed snapshot without the key, an unsigned one when a key is configured, a member with an object dtype, a zip that expands past 8 GB. A snapshot of other model files or another feature layout raises FingerprintMismatch; restore(..., force=True) then brings back only enforcement, feedback, rules, allowlist. Snapshotter.restore_on_start() does that by itself (on_mismatch="independent"), so blocks and operator decisions survive a model upgrade while every reference is re-learned. Restore decodes everything before it assigns anything: a failing restore leaves the layer as it was.

2. Deployment patterns

2.1 One node that restarts

DIBAGUARD_SNAPSHOT_DIR=/var/lib/dibaguard/snap DIBAGUARD_SNAPSHOT_EVERY_S=60 \
  python -m dibaguard.llm http --models models --policy policy.json      # (needs the hook in section 6)

On start: newest valid snapshot of this replica, expired decisions dropped, blocks put back into the host firewall for their remaining time. On a clean stop Housekeeper.stop() writes a final snapshot (RPO 0 for a clean stop).

2.2 Active-passive with snapshot shipping

             VIP / DNS (keepalived, cloud LB health check)
               |                          :
        [active: proxy + sidecar]   [passive: proxy + sidecar, not serving]
               |  snapshot every 30-60 s
               +--> /var/lib/dibaguard/snap/  --rsync/scp every 30 s-->  passive:/var/lib/dibaguard/snap/
                    (or both mount one replicated volume)

Ship only complete files: snapshots are renamed into place, so rsync --include 'dibaguard-snap-*.npz' --exclude '.*' never copies a half-written one; the receiver verifies the digest anyway (state.verify(path)).
Promotion: start the passive's sidecar; restore_on_start() takes its own snapshots first and, when it has none, the active's (replica filter None): a moved / promoted host restores the newest snapshot of any replica.
Same model files on both nodes, or the promoted node restores only the model-independent parts.
Fencing is the VIP manager's job (keepalived / the cloud LB); DibaGuard does not elect leaders.
Set DIBAGUARD_SNAPSHOT_KEY on both nodes when the shipping path is not trusted.

2.3 Active-active behind HAProxy / nginx with source affinity

Each replica is a fast path (dibaguard-proxy or dibaguard-authz) plus its own Python sidecar. The load balancer sends one source to one replica (its evidence accumulates there); the shared view makes every replica enforce what any replica decided, so a source that moves (failover, scale-out) is still blocked everywhere.

                  HAProxy / nginx  (hash of the client /24 -> replica)
                 /                 |                 \
   [lb-a: proxy+sidecar]  [lb-b: proxy+sidecar]  [lb-c: proxy+sidecar]
           \_______________ shared view ________________/
            /mnt/dibaguard (NFS/SMB/replicated volume)  or  Redis

HAProxy (2.6+, balance hash; older: balance source = per address, not per /24):

backend dibaguard
    mode tcp                                   # or http
    balance hash src,ipmask(24,64)             # one /24 (IPv6 /64) -> one replica
    hash-type consistent                       # adding / removing a replica moves ~1/n of the sources
    server lb-a 10.0.0.11:8080 check send-proxy-v2
    server lb-b 10.0.0.12:8080 check send-proxy-v2
    server lb-c 10.0.0.13:8080 check send-proxy-v2
# replicas: dibaguard-proxy -proxy-protocol -trusted <haproxy CIDR> -challenge-secret <the same on every replica>

nginx (L7; hash ... consistent is ketama):

map $remote_addr $dg_net {                     # IPv4 /24; IPv6 hashes per address unless you add a /64 map
    ~^(?<n>\d+\.\d+\.\d+)\.\d+$  $n;
    default                      $remote_addr;
}
upstream dibaguard {
    hash $dg_net consistent;
    server 10.0.0.11:8080; server 10.0.0.12:8080; server 10.0.0.13:8080;
}
# proxy_set_header X-Forwarded-For $remote_addr; replicas: -trusted <nginx CIDR>

On every replica:

DIBAGUARD_REPLICA=lb-a DIBAGUARD_SHARED_DIR=/mnt/dibaguard DIBAGUARD_SYNC_EVERY_S=5 \
DIBAGUARD_SNAPSHOT_DIR=/var/lib/dibaguard/snap/lb-a DIBAGUARD_SNAPSHOT_EVERY_S=300 \
  python -m dibaguard.llm http --models models --policy policy.json
# or DIBAGUARD_REDIS_URL=redis://redis:6379/0 instead of DIBAGUARD_SHARED_DIR

Rules for active-active:

Unique replica ids. FileSharedState refuses an id whose heartbeat is fresh from another host.
Same policy, rules and model files everywhere (configuration management); rules and policy are not synced.
The real client address must reach the replica (PROXY protocol or a trusted X-Forwarded-For); hashing the LB's own address would put every source on one replica.
Shared -challenge-secret, so a challenged client that moves keeps a valid cookie.
One HAProxy ACL syncer per HAProxy instance. dibaguard-authz -haproxy-socket owns its ACL / map and deletes every entry it did not push; with the shared view every replica's block set converges, but several syncers on one HAProxy would fight during the sync period. Run the syncer from one replica (any: they all hold the cluster set).
Clocks within ~1 s (NTP): expiries are absolute and a lift cancels decisions published before it.
affinity() is for routing DibaGuard does itself (a NetFlow / IPFIX fan-out to collector replicas, a log shipper, a summary forwarder). Its hash differs from HAProxy's / nginx's: within one routing layer use one of them consistently. Both are consistent hashes; tests/test_state.py checks that affinity() is stable, independent of list order, balanced (each of 4 replicas 25 % +- 2 %), moves only the removed replica's sources, moves ~1/5 to a fifth replica, and honours weights.

How the shared view behaves (state.merge): each replica publishes only what changed — new / escalated / extended decisions with their absolute expiry, operator lifts (an entry gone before its expiry, or a reject / benign_source verdict, which also resets that source's risk and evidence on every replica), allowlist entries. For one source the strongest live action wins with the latest expiry of all live decisions (what one Enforcer would have done had it seen every verdict); a lift cancels every decision published at or before it, on every replica, and a later decision stands. Imported blocks are applied to the local host firewall too, so the proxy's fast path (polling guard_fast_state) and HAProxy's ACL syncer (dibaguard-authz -haproxy-socket) enforce the cluster's decisions. A replica that restarts empty, or joins later, converges on the view at its first sync.

Evidence federation: every replica publishes its top-K (1000) sources by decayed fast / slow evidence (those above 10 % of the threshold). A source with evidence on two or more replicas is judged on the sum against the engine's own conformal threshold (alpha/4, the same as the evidence tier); a hit is logged as a federated alarm and goes through the policy like any verdict (a named family from the replicas' alarms -> its action; none -> attack_generic -> monitor by default). A source seen on one replica only is left to that replica's engine.

FileSharedState layout (no locks; every replica appends / replaces only its own files):

<root>/enforce/<replica>.jsonl    header {"gen"} + append-only records; compaction rewrites it (new gen, atomic rename)
<root>/evidence/<replica>.json    latest evidence summary (atomic replace)
<root>/replicas/<replica>.json    heartbeat (host, pid, time)

Readers keep a byte offset per journal and read only appended complete lines; a new generation means re-read.

3. Measured RPO / RTO

Machine: Intel Core 12th gen (Family 6 Model 151), 8 logical CPUs, Windows 11, local NVMe; Python 3.13.3, numpy 2.2.5. Real engine and Layer, test-fixture models (tier 1: 60 trees depth 4; GRU d=16, 2 layers; HTTP 4096 buckets). Traffic: a site calibrated on 60 000 flows from 3 000 clients, then new sources with 4 flows each. Reproduce with PYTHONPATH=. python scripts/ha_bench.py (ranges below = two runs).

tracked sources	re-calibration buffer rows	warm GRU sources	snapshot size	engine lock held	compress + hash + write	restore (load, verify, decode, apply)
10 000	88 000	~1 000	8.2 MB (5.7 MB without buffer)	0.05-0.09 s	1.0-1.1 s	0.20-0.24 s
50 000	248 000	~6 800	19.5 MB (13.3 MB)	0.29-0.38 s	2.7-3.0 s	0.63-0.76 s
100 000	400 000 (cap)	~14 000	30.7 MB (21.1 MB)	0.59-0.73 s	4.4-4.7 s	1.13-1.34 s

About 320 bytes per tracked source with the buffer at its cap, ~220 without. Only the "lock held" column stalls scoring (live structures are copied; key encoding, compression, hashing and the write run after the lock). The GRU part grows with the real model: layers x d x 4 bytes per warm source.

Replica sync (FileSharedState, local disk, two replicas):

decisions per replica	evidence sources per replica	first sync (imports all)	steady sync	journal	evidence summary
1 000	10 000	0.03-0.05 s	0.02 s	98 kB	39 kB
10 000	100 000	0.16-0.40 s	0.12-0.14 s	0.99 MB	40 kB

(Sync holds the engine lock only while reading / changing the layer; journal and evidence I/O run outside it.)

What that means:

RPO, one node / active-passive: the snapshot interval + ~5 s (write time at 100k sources). With 60 s snapshots, at most ~65 s of evidence, risk and reference updates are lost on a crash; a clean stop loses nothing. Enforcement decisions in force at the last snapshot come back with their original expiry.
RPO, active-active: enforcement decisions, lifts and allowlist entries: one sync period (5 s default) — they live in the shared view, not in one process. Per-source evidence of the crashed replica: its snapshot interval (federation summaries of the other replicas still hold its top sources only for as long as they decay).
RTO: process start + model load + restore (~1.1-1.3 s per 100k sources) + one firewall command per restored block. In active-active the LB health check removes the dead replica (its sources re-hash to the others, which already enforce its decisions); the restarted replica converges at its first sync.

4. Failure modes

failure	what happens
sidecar crash / kill -9	restart restores the newest valid snapshot; decisions expired meanwhile are dropped, the rest are re-applied to the firewall for their remaining time
crash while writing a snapshot	the temp file (`.dibaguard-snap-*.tmp`) is never renamed; it is ignored, deleted after an hour; the previous snapshot stays valid
corrupted / truncated / tampered snapshot	digest (or zip CRC) mismatch -> `SnapshotError`; a directory restore falls back to the previous snapshot (`rep["skipped"]` says why)
model upgrade	fingerprint mismatch: `restore()` refuses, `restore_on_start()` restores enforcement / feedback / rules / allowlist and the engine re-calibrates from live traffic
code upgrade that changes the feature layout	layout-id mismatch: handled like a model change
host loss (active-passive)	promote the passive: it restores the last shipped snapshot (RPO = ship interval + snapshot interval)
replica loss (active-active)	LB re-hashes its sources; the others already enforce its decisions; their evidence restarts on the new owner, federation sums what the dead replica published until it decays
shared storage slow / unavailable	sync I/O runs outside the engine lock; a failed sync is reported by the Housekeeper and retried; each replica keeps enforcing its own decisions (degrades to independent replicas)
network partition	each side keeps its own decisions and converges when the partition heals; a lift made on one side reaches the other then (no quorum, no strong consistency)
clock skew	expiries are absolute: a replica whose clock is ahead lifts imported decisions early; a lift with a skewed time may cancel (or miss) a decision published within the skew; keep NTP
two hosts with one replica id	refused at start while the other's heartbeat is fresh (< 60 s)
LB scale-out / rehash	~1/n of the sources move; their evidence restarts on the new owner, and is judged on its total through federation for as long as the old owner's summary holds it
Windows rename while a reader holds a file	retried; a compaction that still fails is skipped until the next round
redis down (RedisSharedState)	the sync raises, the Housekeeper records it and retries; replicas run independently meanwhile

5. What is NOT covered

No consensus. The shared view is eventually consistent (one sync period); there is no leader election, quorum or fencing. Active-passive failover needs an external VIP manager.
No confidentiality at rest. HMAC gives integrity, not secrecy. A snapshot holds client addresses, request texts (HTTP buffers, provenance ring), database query shapes and the alarm log: treat the snapshot directory and the shared directory as sensitive (file permissions, encrypted volume).
Not synced between replicas: rules, the policy, per-host HTTP profiles and references, the alarm log and review queue (each replica reviews its own alarms; only the verdicts' lifts / allowlist propagate), the DB tier. Rules and policy come from configuration management.
Federation covers source evidence only (fast and slow CUSUM). Stream counters, GRU states, entity risk and the destination-flood tier stay per replica; a summed evidence is an approximation (each replica floors its own CUSUM at zero) and only the top-K sources per replica are exchanged.
Bounded snapshots drop cold sources beyond max_sources; they start fresh after a restore.
Zero RPO only for a clean stop; a crash loses what happened since the last snapshot (enforcement excepted in active-active).
Tested on a local filesystem only (this round); NFS / SMB semantics (close-to-open consistency, rename atomicity) are relied upon but not exercised. The Redis backend is tested against an in-memory fake with the same calls, not a server.
The Go fast path still polls only its local sidecar; it learns other replicas' decisions through that sidecar (one poll interval later).
nftables: when an action's TTL is extended, the kernel set element keeps its original timeout (pre-existing Enforcer behaviour); restored and imported blocks are added with their remaining time.

6. Wiring (hook in the sidecar)

state.py needs no change to existing modules. To turn it on in the HTTP sidecar, dibaguard/llm.py needs:

# in main(), after the Dispatcher with a Layer exists and the server (make_server) is built:
from dibaguard import state
srv, cons = make_server(disp, a.host, a.port)
hk = state.from_env(disp.layer, cons.lock) if disp.layer is not None else None   # restore on start + background thread
try:
    srv.serve_forever()
finally:
    if hk is not None:
        hk.stop()                                                                     # final snapshot

from_env reads DIBAGUARD_SNAPSHOT_DIR, DIBAGUARD_SNAPSHOT_EVERY_S, DIBAGUARD_SNAPSHOT_KEEP, DIBAGUARD_SNAPSHOT_KEY, DIBAGUARD_RESTORE, DIBAGUARD_SHARED_DIR or DIBAGUARD_REDIS_URL, DIBAGUARD_REPLICA, DIBAGUARD_SYNC_EVERY_S, DIBAGUARD_TICK; with none set it returns None and nothing changes.

state.py reads these existing private attributes, so a rename there must be mirrored in state.py (the round-trip test's exact-state check fails if one is missed): Guard._buffer, Guard._anchor, Guard._log(), Layer._http_boot_by / _http_recent_by / _http_last_by / _http_last / _http_ring, Enforcer._record(), Policy._allow, entity._Rec, database._MYSQL_THREADS. A new piece of engine / layer state must be added to _guard_out / _guard_in (or _layer_*, _db_*).