DibaGuard: public licence-clean release (Dibachain)
5d97275 verified 4 days ago
23.6 kB
# DibaGuard — design (v0.1, 2026-10-30)

> **Open-release copy** ([DibaAi/DibaGuard-Open](https://huggingface.co/DibaAi/DibaGuard-Open)). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (`http_v2`) were measured with that model, which is **not** part of the open release; the open release ships the licence-clean network models described in `CLEAN_MODELS.md`. Licence statements inside this document are superseded by `DATA_LICENSES.md`, `CLEAN_MODELS.md` and the repository's `LICENSE`, `LICENSE-WEIGHTS` and `NOTICE` files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.

> **Historical note.** This is the v0.1 design document. Its statement that a licence-clean retrain could use the "MIT-licensed merged NetFlow v3" copy was wrong: that label was set by an uploader who is not the data owner (`DATA_LICENSES.md`, row 3). The licence-clean models were trained as described in `CLEAN_MODELS.md`.

A CPU-only attack-intensity model for network traffic and web requests, built on the DibaOne decision architecture
(typed calibrated outputs, tiered cascade with a router, adaptive memory with feedback). Working name: DibaGuard.

**Answer it gives, for every event and continuously for every entity and site:**

| level | output |
|---|---|
| event (one flow, one HTTP request, one packet burst) | `p_attack` (calibrated), `family` (open set: sqli, xss, traversal, cmdi, ssrf, ssti, xxe, jndi, scan, bruteforce, dos, botnet, exfil, …), `severity` 0–1, `score` 0–100, `evidence` (decoded payload span / features that drove it), `tier` and µs spent |
| entity (source IP, session, API key, JA3/UA fingerprint) | `risk` 0–100 with decay, `campaign` id when several entities share a fingerprint |
| site | `pressure` 0–100: how much of the current traffic is attack, by family |

## 1. Hard budgets (the product is the budget)

| budget | target, 1 vCPU, no GPU |
|---|---|
| model files, all tiers | < 15 MB (int8) |
| RAM, runtime + 1 M tracked entities | < 300 MB |
| HTTP throughput (tiers 0–1, batched) | ≥ 20 000 req/s; p99 < 1 ms incl. tier 2 |
| flow throughput (tier 1) | ≥ 200 000 flows/s |
| tier 2 share of traffic | ≤ 5 % (router-controlled) |
| dependencies at inference | numpy + onnxruntime (no torch) |

Every report states the machine, threads, batch size and the measured numbers. A run with no numbers is not a run.

## 2. Architecture (DibaOne heritage → what is new here)

```
             ┌──────── tier 0: stream state (O(1)/event) ─────────┐
 packets ──► │ flow aggregator (NetFlow v2-compatible features)   │
 NetFlow ──► │ per-entity counters: count-min sketch + EWMA       │──► features ─┐
 HTTP    ──► │ canonicaliser: multi-round decoding, normal form   │              │
             └────────────────────────────────────────────────────┘              ▼
             tier 1 (µs):  flows → gradient-boosted trees (ONNX/treelite)    calibrated p, family
                           HTTP  → hashed char/token n-grams + structure → linear (sparse)
                     router: confident? ──yes──► decision layer
                                         no (≈2–5 %)
             tier 2 (≈0.3 ms): tiny byte transformer (≈1–2 M params, int8), multi-task heads
                                         ▼
             decision layer: per-family temperature, Mondrian conformal thresholds (FPR ≤ α
             per site/endpoint), abstain = "review", entity risk accumulation, evidence
                                         ▼
             adaptive layer: site profile (template + parameter-type model), kNN memory,
             feedback (/v1/feedback) → instant correction without retraining
```

### Innovations (each one has a benchmark that can prove or kill it)

1. **Guaranteed false-alarm budget per deployment.** Thresholds are set by Mondrian conformal prediction on the
   site's own benign traffic, per endpoint group and per source type, so `FPR ≤ α` holds with a finite-sample
   guarantee instead of a vendor-chosen threshold. Conformal martingales over the same p-values detect drift and
   trigger recalibration. *Benchmark:* realised FPR vs α on held-out benign traffic of unseen sites.
2. **Compute-budgeted cascade.** The router (as in DibaOne X1) sends each event to the cheapest tier that is
   confident enough, under a per-request `max_us`; thresholds fitted on dev to minimise expected cost.
   *Benchmark:* accuracy vs µs/event curve; share of events per tier.
3. **Evasion-invariant training.** A deterministic canonicaliser (URL/HTML/unicode/hex/base64 decoding until a fixed
   point, SQL/JS comment and whitespace folding, case folding) plus a mutation engine (encodings, case, comments,
   synonyms, splitting) and a consistency loss: an attack and its obfuscated variants must score alike.
   *Benchmark:* detection on obfuscated attacks never seen in training (evasion suite).
4. **Self-learned positive model per site (zero-day).** A compact profile of each site's normal requests: path
   templates (Drain-style tree), parameter types (int / uuid / email / slug / free text / Persian text), length and
   charset distributions, in sketches of a few MB. Violations add an anomaly term to the supervised score, so
   attacks with no known signature still raise the score. *Benchmark:* leave-one-family-out recall.
5. **Attack intensity, not only labels.** Event probability × severity, accumulated per entity with decay and linked
   across entities by shared fingerprints (low-and-slow scans, credential stuffing from many IPs).
   *Benchmark:* time-to-detect and campaign recall on replayed multi-stage attacks.
6. **Persian-aware.** Benign hard negatives with Persian text in parameters (percent-encoded UTF-8, ZWNJ, Persian
   digits), Iranian site traffic (zanbil.ir logs) — a common false-positive source for Western WAF rules.
7. **Learning from analysts in seconds** (DibaOne adaptive memory): a marked false positive changes the next
   decision immediately (kNN override + gated adapter), and is kept for the next training round.
8. **Feeds anything:** raw packets (own aggregator), NetFlow/IPFIX exports from routers (no capture needed), access
   logs (nginx/apache/IIS), reverse-proxy mirror, or an inline SDK call.

## 3. Data (private R&D first; licences tracked per row, `meta.licence`)

| modality | sources (Kaggle-attached; nothing downloaded through the office network) |
|---|---|
| flows, NetFlow v2 view | NF-UQ-NIDS-v2 (UNSW-NB15, BoT-IoT, ToN-IoT, CSE-CIC-IDS2018), NF v3 sample |
| flows, CICFlowMeter view | CIC-IDS2017 & CSE-CIC-IDS2018 (DistriNet label-corrected), CIC-DDoS2019, CIC-IoT2023 |
| real traffic | LUFlow (university network, honeypot-labelled, to 2026) |
| packets | UNSW + CIC-IDS2017 labelled PCAP subset |
| HTTP | CSIC 2010, SQLi/XSS sets (incl. obfuscated), labelled access logs, attack-simulation lab, payload repos (PayloadsAllTheThings, SecLists — MIT; OWASP CRS regression tests — Apache-2.0) |
| benign HTTP | CSIC normal, zanbil.ir logs (CC0), requests generated from public OpenAPI specs (APIs.guru, CC0), Persian/code/SQL-looking benign text as hard negatives |

Rules: no IPs, timestamps or flow ids as features (shortcuts); dedup by feature hash before splitting; splits by
capture day / site where possible; licences per row (many NIDS sets are CC BY-NC-SA → private R&D; a commercial
release trains on the permissive subset or needs permission).

## 4. Evaluation (frozen test sets; one scorer)

* in-distribution, and **leave-one-dataset-out** (train on the others, test on the held-out dataset) — the honest
  generalisation number most papers omit;
* TPR at FPR = 1e-3 and 1e-4, AUROC, AUPRC, macro-F1 over families, ECE;
* evasion suite (obfuscated attacks), Persian benign hard-negative FPR, leave-one-family-out recall;
* µs/event, events/s, RAM, model MB on a stated CPU.

## 5. Plan

| step | where | status |
|---|---|---|
| S0 survey of every dataset (schemas, labels, counts) | Kaggle CPU, both accounts | running |
| S1 build `flows_v1` and `http_v1` corpora → `DibaAi/dibaguard-data-private` | Kaggle CPU | next |
| S2 tier 1 baselines (trees for flows, n-gram linear for HTTP) + leave-one-dataset-out | Kaggle CPU | |
| S3 tier 2 byte model + evasion training | Kaggle GPU (T4) | |
| S4 decision layer (calibration, conformal, entity risk) + runtime (numpy/onnxruntime) | local | |
| S5 site profile, feedback, private test Space, report | local + HF Space (private) | |

## 6. Deployment: one engine, many hosts (added 2026-10-30)

Goal: the strongest detector we can train, packaged so it drops in next to any load balancer or becomes one.

**Model contract (language-agnostic).** A tier-1 model is a flat file:
`feat[S] int16, border[S] float32, tree_tests[T,D] int16, leaves[T,2^D,C] float32|int8, bias[C], info json`
(`dibaguard/flows/trees.py`). Evaluation = S compares, T×D bit gathers, T leaf lookups, one sum. The HTTP tier-1
model adds `emb[B,E] int8 + scale` and the n-gram hash spec (`dibaguard/hashing.py`: FNV-64 rolling, splitmix mix,
orders 2–6). Both are ~100 lines in Go; a parity test (`tests/parity/*.jsonl` = inputs + expected raw scores,
bit-exact) is part of the release, so any port is proven, not trusted.

**Three independent layers, each replaceable:**
1. *Score* – stateless, pure function `record → raw score`. Library (Python/numpy, Go), or sidecar.
2. *Decide* – threshold (conformal, site- or shipped-calibrated), temperature, family head on alarms, entity risk,
   site pressure, drift martingale. Small mutable state (≈ a few MB), exportable/importable as JSON → can be shared
   across LB replicas or kept per replica.
3. *Adapt* – site profile (percentile grids; `siteprofile.py` compiles them into the tree borders, so the deployed
   model stays raw-feature, same size, same speed), feedback memory, periodic recalibration. Runs offline or in a
   control-plane process; ships new borders/thresholds to layer 1/2 atomically.

**Hosts.** (a) Sidecar: HAProxy SPOE / Envoy ext_proc / nginx `auth_request` → local UDS/HTTP, p50 < 100 µs.
(b) Native Go load balancer with the engine embedded (flows from conntrack/NetFlow mirror, HTTP from the proxy
itself) – score every request, rate-limit / challenge / block per entity risk and site pressure. (c) Library.
Decision outputs are the same objects everywhere: `{p, alarm, family, severity, score 0-100, entity_risk,
site_pressure, threshold_source}`.
**Test ladder (every release):** unit parity (Python↔Go bit-exact) → LODO cross-network numbers → site-calibrated
FPR actually ≤ α on held-out benign → evasion/obfuscation suites (HTTP) → load test (req/s, p99, RSS) → 24 h soak on
replayed real logs with drift alarms counted.
## 7. v2: the woven cascade (2026-10-30)
Why: v1 scores each flow / request alone. Cross-network evaluation with *source-capped* sampling (every source
contributes at most one 60k-flow run, so a few DoS bots cannot dominate) shows that single-record detectors - trees
or a GRU - transfer poorly; what a site can always have is its *own* normal traffic. v2 therefore weaves three
threads per source and lets the site's benign stream condition every one of them:
| layer | what | cost (Go, 1 core) | runs on |
|---|---|---|---|
| 0 entity state | decayed counters, novelty memory, gaps (36 features) | 275 ns | every flow / request |
| 1 trees | oblivious trees on layer-0 features (shipped prior: "what attacks look like") | 1.7 µs | every flow |
| 1' site anomaly | quantile p-values / isolation forest on layer-0 features, fitted on this site's benign only | ~1 µs | every flow |
| 2 sequence net | causal GRU per source over its event stream (+ optional site context FiLM) | 8.6 µs at d=24 (keep d ≤ 64) | warm sources only (tier-1 above the low-water mark) |
| HTTP | hashed n-gram bag + MLP conditioned on the site profile (HttpModel2); the request also enters layer 0 as an event | 8.9 µs | every request |
Decision: every thread yields a conformal p-value against the site's benign reference; the verdict is the minimum
(Bonferroni over threads), so the realised false-alarm rate stays at α whatever the mix. Family head on alarms
only; EntityRisk / SitePressure / drift as before.
Evaluation protocol (guard_jobs/seq_train.py): temporal + leave-one-network-out on source-capped corpora, each
run reporting flow AUROC, site-calibrated FPR/TPR, entity-minute AUROC and the intensity correlation for the GRU,
the trees, the two anomaly baselines and the fused verdicts. Honest result: single-record transfer is weak for
everyone; the fused, site-conditioned verdict is what ships.
Modern threat landscape (2025-26 vendor reports): L7 floods and bots (half of web traffic), API abuse, credential
stuffing, scanners, AI crawlers - i.e. *behaviour over time per client*, which is exactly what layers 0/2 and the
HTTP session stream measure. Injection payloads still matter and are covered by the n-gram model; the 2025
ModSecurity thirty-day archive (CC BY 4.0) is the modern attack-payload source.
LLM bridge (dibaguard/llm.py): five tools (status, score flows, score http, calibrate, report) as MCP over stdio,
OpenAI / Anthropic tool schemas and a stdlib HTTP server; the incident report (dibaguard/report.py) is a fixed JSON
schema rendered to English and Persian from the engine's own state - no model narrates numbers it did not compute.
## 8. v2.1: the real-world layer (2026-10-30, evening)
What was weak after v2 and what fixed it - every item measured, none tuned by hand:
| weakness | root cause | fix (dibaguard/engine.py) |
|---|---|---|
| per-flow detection at α = 1e-3 ≈ 0 on unseen networks | the flow is the wrong unit: an attacker sends hundreds of flows, a verdict per flow throws the sequence away | **per-source accumulated evidence** `E ← E·e^(-Δt/τ) + (−log p)` (τ = 120 s), judged per (source, minute) against the site's benign source-minute maxima (`decide.SourceEvidence`). LODO CIC-IDS2018: every attacking source caught, source-minute TPR 0.6 at FPR 2e-3 (was 0.0 per flow) |
| realised FPR above target when the traffic mix changes | one global reference; a new or bursty service moves every threshold | **Mondrian references by service × source age** (`ent.service_key`, `AGE_EDGES`): a client's first flows are judged only against other clients' first flows; a cell without reference **abstains** (p = 1) instead of guessing |
| false alarms on new clients (flash crowds) | births look extreme against steady-state sources | same age conditioning + abstention; destination-side features kept out of per-flow verdicts (they belong to site pressure) |
| FPR ratchet over hours (+1 %/h) | self-cleaning rolling references trimmed the benign tail; anomaly scores against a growing reference change scale | **epoch re-calibration**: every `recal_s` (30 min) all references are rebuilt at once from the last `recal_window_s` (2 h) of traffic, sources with evidence left out; the evidence reference is built out of sample (last quarter scored against the first three). Measured: 0.08 → 0.23 → 0.23 % per source-minute over three benign hours at α = 1e-3 (was 0.18 → 1.0 → 2.5 %) |
| wrong attack families | the GRU family head does not transfer | transparent **rules on the stream counters** (`ent.family_rules`): port walking → portscan, same (host, port) again and again → bruteforce, ≥ 10 flows/s at one service → dos/ddos, host walking with failures → recon |
| supervised prior scoring a SYN scan as benign | dataset artefacts in unconstrained trees | **monotone trees** on the 24 attack-direction features (more scanning can never look more benign) |
| unseen web application (WordPress) | vocabulary the model never saw | **site n-gram novelty** (`HttpModel2.novelty`): share of a request's n-grams absent from the site's own benign vocabulary, rarity-weighted - evaluated (WordPress AUROC 0.92 vs 0.61), **not yet part of the live verdict** |

False-alarm budget: α/2 to the evidence tier, α/(2·k) to each of the k per-flow tiers (Bonferroni), so the union stays at α.

What the engine learns on the server and what it does not: every reference, threshold, site profile and vocabulary is
re-learned from the site's own traffic every epoch (no labels, poisoning-resistant by excluding sources with evidence);
model weights (trees, GRU, HTTP MLP) are frozen - online weight updates without labels would be a poisoning surface.
The feedback loop (operator-confirmed alarms → periodic offline retraining) is the next step, not an online one.

## 9. v2.2: distributed floods, slow scans, feedback, rules, and the live lab (2026-10-31)

The static cross-dataset numbers hid live failures that only a running, multi-wave environment shows. A local
red-team/blue-team lab (scripts/redteam_lab.py: a self-owned localhost target, benign traffic, and the engine's own
playbook patterns replayed over several waves, scored against known ground truth) drove the next round of fixes:
| gap the live lab exposed | fix |
|---|---|
| distributed / volumetric floods (amplification, peer/sybil floods) have no single guilty source | **per-destination evidence tier**: a destination's (flows, no-reply, new-source) anomaly accumulated per destination-minute; a flow is blamed only if it is new to the destination *and* off-profile, so the server's established benign clients are spared. NTP amplification 1.0, peer flood 0.76-0.80 |
| slow scans (nmap -T1 over ~30 min) never accumulate at tau=120 s | **long-horizon source evidence** (tau=1800 s) in parallel, fed only by mildly-suspicious flows; once its reference is warm (~2 epochs) slow-scan detection is 0.95 |
| detection decayed wave-over-wave (partially-detected attacks seeped into the self-training) | **reference hygiene** (as of v2.5): a source whose alarms name an attack family, or whose alarms keep coming while it behaves like an attacker, is excluded from the reference entirely; a sporadic un-named alarm stays in (it is the benign tail - excluding it ratchets the FPR up); no source may write more than ~1% of a reference; plus an **immutable anchor** mixed into every epoch and a drift guard that asks who moved the tail. Destinations are not excluded. See docs/TECHNICAL.md section 4.3-4.4 |
| HTTP session attacks (credential stuffing, scrapers, API enumeration) under-detected | the request is also an event of the client's stream; with the burst preserved, stuffing/scraper/BOLA all reach 0.98-0.99 |
Live lab, 6 waves, shipped models, 3000-client synthetic site: HTTP attacks caught 100% (JNDI/WordPress via rules
1.0, credential stuffing 0.99, scraper 0.94, BOLA 0.98); flow attacks 77% averaged over cold+warm waves (NTP amp
1.0, rapid reset 0.95, SSH brute 0.75, peer flood 0.76, slow scan 0.16 cold -> 0.95 once warm); benign false-alarm
rate 2e-4 per (source, minute) for flows and 2e-4 for HTTP; blue-team feedback precision 0.95.
Operator layer:
  - **Rules** (dibaguard/shield/rules.py): plain-language rules in Persian or English are interpreted (a built-in
    bilingual parser, or an LLM returning the same JSON schema) into deterministic predicates and enforced from the
    first second - so there is no "unprotected while it learns" window. The interpretation is shown back for
    confirmation. An `allow` rule shields a source; the playbook's hard signatures (JNDI, CMS paths) also act at t=0.
  - **Feedback** (dibaguard/shield/feedback.py): the operator confirms / rejects / allowlists alarms from a
    dashboard, MCP or REST. Reject lifts the action and the source's traffic becomes benign reference immediately;
    confirm takes (and escalates) the action; every verdict is logged and exported as labelled data for the
    offline retraining job. So there is no online weight update (poisoning-safe), but the layer adapts at once to
    operator intent and accumulates a retraining set.
  - **Licence-clean retraining**: the network corpus is being rebuilt from the MIT-licensed merged NetFlow v3
    (guard_jobs/entity_build.py nfv3_parquet, corrupt row groups skipped) so the shipped prior can be trained
    without the CC BY-NC-SA sets.
## 10. v2.3: hard adversarial lab, zero-benign-block guarantee, graceful degradation (2026-10-31)

A hard-mode lab (scripts/redteam_hard.py) runs 8 waves in which the attacker *evolves* to evade (slower, more
source IPs, smaller payloads, UA mimicry) while the benign side is hostile (40% brand-new clients each wave, flash
crowds, Persian traffic, a newly-added service). It scores recall per family *and the benign BLOCK rate* (the real
harm), per wave, so the learning/degradation curve is visible.
What it forced, and the design principles that came out of it:
- **Evidence is a zero-mean CUSUM, not a sum.** Per-flow increment is `-log p - k` (k = the benign mean of the
  suspicious-flow -log p), floored at 0. A benign source drifts around 0 at *any* request rate, so a chatty client
  or a flash crowd no longer accumulates its way over the threshold; only a genuine excess builds up. Gated to
  off-profile flows (p < 0.5) on both the fast (tau=120 s) and slow (tau=1800 s) horizons.
- **First contact is never auto-blocked.** A source's first flows carry no behaviour to judge; behavioural tiers
  abstain for a newborn source unless the single flow is itself self-evident (p < alpha, e.g. a 1400-byte no-reply
  UDP to a victim) or the supervised prior fires hard. A soft newborn alarm becomes `block_ok = False` -> the layer
  issues a **challenge (HTTP) or monitor (flow)**, never a block. HTTP session alarms block only when the client's
  requests are largely failing (4xx/5xx): a user browsing many pages trips the path-diversity signal but gets 200s,
  so it is challenged; enumeration / credential stuffing / scanners fail requests and are blocked.

- **Result (8 waves, evolving attacker, 40% flash-crowd benign): zero benign auto-blocks on both flows and HTTP in
  every wave; feedback precision 1.0.** Recall: rapid reset / NTP amplification / SSH brute force ~1.0, peer flood
  0.94; the learning curve degrades *gracefully* only against the maximally-evading low-and-slow distributed cases
  (evolving scan 1.0 -> 0.16, distributed credential stuffing 0.98 -> 0.23 as it spreads over many client IPs).
  These are the known-hard, aggregate-only attacks; they are documented honestly and are the roadmap's global-rate
  tier, not a silent failure. Benign alarms that are not blocks (challenges/monitors) are expected under a flash
  crowd and are the standard "under attack" trade-off, tunable via policy.min_action_score.

- **Multilingual, by design.** Rule interpretation and the LLM bridge are language-agnostic: with an LLM wired in
  (the normal deployment of an LLM-facing product) a rule in any language is interpreted to the same JSON schema
  (rules.LLM_SCHEMA_PROMPT); the built-in parser is only the Persian/English offline fallback. The rendered
  report and dashboard are English + Persian.

What is honestly *not* done (roadmap, not claimed): a global/aggregate rate tier for fully-distributed low-volume
attacks; automatic database- and network-topology profiling beyond the posture audit's service recognition; and
online weight learning (kept offline on purpose, fed by the feedback-exported labels) - see section 8.