docs/DESIGN.md

DibaGuard — design (v0.1, 2026-10-30)

Open-release copy (DibaAi/DibaGuard-Open). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (http_v2) were measured with that model, which is not part of the open release; the open release ships the licence-clean network models described in CLEAN_MODELS.md. Licence statements inside this document are superseded by DATA_LICENSES.md, CLEAN_MODELS.md and the repository's LICENSE, LICENSE-WEIGHTS and NOTICE files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.

Historical note. This is the v0.1 design document. Its statement that a licence-clean retrain could use the "MIT-licensed merged NetFlow v3" copy was wrong: that label was set by an uploader who is not the data owner (DATA_LICENSES.md, row 3). The licence-clean models were trained as described in CLEAN_MODELS.md.

A CPU-only attack-intensity model for network traffic and web requests, built on the DibaOne decision architecture (typed calibrated outputs, tiered cascade with a router, adaptive memory with feedback). Working name: DibaGuard.

Answer it gives, for every event and continuously for every entity and site:

level	output
event (one flow, one HTTP request, one packet burst)	p_attack (calibrated), family (open set: sqli, xss, traversal, cmdi, ssrf, ssti, xxe, jndi, scan, bruteforce, dos, botnet, exfil, …), severity 0–1, score 0–100, evidence (decoded payload span / features that drove it), tier and µs spent
entity (source IP, session, API key, JA3/UA fingerprint)	risk 0–100 with decay, campaign id when several entities share a fingerprint
site	pressure 0–100: how much of the current traffic is attack, by family

1. Hard budgets (the product is the budget)

budget	target, 1 vCPU, no GPU
model files, all tiers	< 15 MB (int8)
RAM, runtime + 1 M tracked entities	< 300 MB
HTTP throughput (tiers 0–1, batched)	≥ 20 000 req/s; p99 < 1 ms incl. tier 2
flow throughput (tier 1)	≥ 200 000 flows/s
tier 2 share of traffic	≤ 5 % (router-controlled)
dependencies at inference	numpy + onnxruntime (no torch)

Every report states the machine, threads, batch size and the measured numbers. A run with no numbers is not a run.

2. Architecture (DibaOne heritage → what is new here)

             ┌──────── tier 0: stream state (O(1)/event) ─────────┐
 packets ──► │ flow aggregator (NetFlow v2-compatible features)   │
 NetFlow ──► │ per-entity counters: count-min sketch + EWMA       │──► features ─┐
 HTTP    ──► │ canonicaliser: multi-round decoding, normal form   │              │
             └────────────────────────────────────────────────────┘              ▼
             tier 1 (µs):  flows → gradient-boosted trees (ONNX/treelite)    calibrated p, family
                           HTTP  → hashed char/token n-grams + structure → linear (sparse)
                     router: confident? ──yes──► decision layer
                                         no (≈2–5 %)
             tier 2 (≈0.3 ms): tiny byte transformer (≈1–2 M params, int8), multi-task heads
                                         ▼
             decision layer: per-family temperature, Mondrian conformal thresholds (FPR ≤ α
             per site/endpoint), abstain = "review", entity risk accumulation, evidence
                                         ▼
             adaptive layer: site profile (template + parameter-type model), kNN memory,
             feedback (/v1/feedback) → instant correction without retraining

Innovations (each one has a benchmark that can prove or kill it)

1. Guaranteed false-alarm budget per deployment. Thresholds are set by Mondrian conformal prediction on the site's own benign traffic, per endpoint group and per source type, so FPR ≤ α holds with a finite-sample guarantee instead of a vendor-chosen threshold. Conformal martingales over the same p-values detect drift and trigger recalibration. Benchmark: realised FPR vs α on held-out benign traffic of unseen sites.
2. Compute-budgeted cascade. The router (as in DibaOne X1) sends each event to the cheapest tier that is confident enough, under a per-request max_us; thresholds fitted on dev to minimise expected cost. Benchmark: accuracy vs µs/event curve; share of events per tier.
3. Evasion-invariant training. A deterministic canonicaliser (URL/HTML/unicode/hex/base64 decoding until a fixed point, SQL/JS comment and whitespace folding, case folding) plus a mutation engine (encodings, case, comments, synonyms, splitting) and a consistency loss: an attack and its obfuscated variants must score alike. Benchmark: detection on obfuscated attacks never seen in training (evasion suite).
4. Self-learned positive model per site (zero-day). A compact profile of each site's normal requests: path templates (Drain-style tree), parameter types (int / uuid / email / slug / free text / Persian text), length and charset distributions, in sketches of a few MB. Violations add an anomaly term to the supervised score, so attacks with no known signature still raise the score. Benchmark: leave-one-family-out recall.
5. Attack intensity, not only labels. Event probability × severity, accumulated per entity with decay and linked across entities by shared fingerprints (low-and-slow scans, credential stuffing from many IPs). Benchmark: time-to-detect and campaign recall on replayed multi-stage attacks.
6. Persian-aware. Benign hard negatives with Persian text in parameters (percent-encoded UTF-8, ZWNJ, Persian digits), Iranian site traffic (zanbil.ir logs) — a common false-positive source for Western WAF rules.
7. Learning from analysts in seconds (DibaOne adaptive memory): a marked false positive changes the next decision immediately (kNN override + gated adapter), and is kept for the next training round.
8. Feeds anything: raw packets (own aggregator), NetFlow/IPFIX exports from routers (no capture needed), access logs (nginx/apache/IIS), reverse-proxy mirror, or an inline SDK call.

3. Data (private R&D first; licences tracked per row, meta.licence)

modality	sources (Kaggle-attached; nothing downloaded through the office network)
flows, NetFlow v2 view	NF-UQ-NIDS-v2 (UNSW-NB15, BoT-IoT, ToN-IoT, CSE-CIC-IDS2018), NF v3 sample
flows, CICFlowMeter view	CIC-IDS2017 & CSE-CIC-IDS2018 (DistriNet label-corrected), CIC-DDoS2019, CIC-IoT2023
real traffic	LUFlow (university network, honeypot-labelled, to 2026)
packets	UNSW + CIC-IDS2017 labelled PCAP subset
HTTP	CSIC 2010, SQLi/XSS sets (incl. obfuscated), labelled access logs, attack-simulation lab, payload repos (PayloadsAllTheThings, SecLists — MIT; OWASP CRS regression tests — Apache-2.0)
benign HTTP	CSIC normal, zanbil.ir logs (CC0), requests generated from public OpenAPI specs (APIs.guru, CC0), Persian/code/SQL-looking benign text as hard negatives

Rules: no IPs, timestamps or flow ids as features (shortcuts); dedup by feature hash before splitting; splits by capture day / site where possible; licences per row (many NIDS sets are CC BY-NC-SA → private R&D; a commercial release trains on the permissive subset or needs permission).

4. Evaluation (frozen test sets; one scorer)

• in-distribution, and leave-one-dataset-out (train on the others, test on the held-out dataset) — the honest generalisation number most papers omit;
• TPR at FPR = 1e-3 and 1e-4, AUROC, AUPRC, macro-F1 over families, ECE;
• evasion suite (obfuscated attacks), Persian benign hard-negative FPR, leave-one-family-out recall;
• µs/event, events/s, RAM, model MB on a stated CPU.

5. Plan

step	where	status
S0 survey of every dataset (schemas, labels, counts)	Kaggle CPU, both accounts	running
S1 build flows_v1 and http_v1 corpora → DibaAi/dibaguard-data-private	Kaggle CPU	next
S2 tier 1 baselines (trees for flows, n-gram linear for HTTP) + leave-one-dataset-out	Kaggle CPU
S3 tier 2 byte model + evasion training	Kaggle GPU (T4)
S4 decision layer (calibration, conformal, entity risk) + runtime (numpy/onnxruntime)	local
S5 site profile, feedback, private test Space, report	local + HF Space (private)

6. Deployment: one engine, many hosts (added 2026-10-30)

Goal: the strongest detector we can train, packaged so it drops in next to any load balancer or becomes one.

Model contract (language-agnostic). A tier-1 model is a flat file: feat[S] int16, border[S] float32, tree_tests[T,D] int16, leaves[T,2^D,C] float32|int8, bias[C], info json (dibaguard/flows/trees.py). Evaluation = S compares, T×D bit gathers, T leaf lookups, one sum. The HTTP tier-1 model adds emb[B,E] int8 + scale and the n-gram hash spec (dibaguard/hashing.py: FNV-64 rolling, splitmix mix, orders 2–6). Both are ~100 lines in Go; a parity test (tests/parity/*.jsonl = inputs + expected raw scores, bit-exact) is part of the release, so any port is proven, not trusted.

Three independent layers, each replaceable:

1. Score – stateless, pure function record → raw score. Library (Python/numpy, Go), or sidecar.
2. Decide – threshold (conformal, site- or shipped-calibrated), temperature, family head on alarms, entity risk, site pressure, drift martingale. Small mutable state (≈ a few MB), exportable/importable as JSON → can be shared across LB replicas or kept per replica.
3. Adapt – site profile (percentile grids; siteprofile.py compiles them into the tree borders, so the deployed model stays raw-feature, same size, same speed), feedback memory, periodic recalibration. Runs offline or in a control-plane process; ships new borders/thresholds to layer 1/2 atomically.

Hosts. (a) Sidecar: HAProxy SPOE / Envoy ext_proc / nginx auth_request → local UDS/HTTP, p50 < 100 µs. (b) Native Go load balancer with the engine embedded (flows from conntrack/NetFlow mirror, HTTP from the proxy itself) – score every request, rate-limit / challenge / block per entity risk and site pressure. (c) Library. Decision outputs are the same objects everywhere: {p, alarm, family, severity, score 0-100, entity_risk, site_pressure, threshold_source}.

Test ladder (every release): unit parity (Python↔Go bit-exact) → LODO cross-network numbers → site-calibrated FPR actually ≤ α on held-out benign → evasion/obfuscation suites (HTTP) → load test (req/s, p99, RSS) → 24 h soak on replayed real logs with drift alarms counted.

7. v2: the woven cascade (2026-10-30)

Why: v1 scores each flow / request alone. Cross-network evaluation with source-capped sampling (every source contributes at most one 60k-flow run, so a few DoS bots cannot dominate) shows that single-record detectors - trees or a GRU - transfer poorly; what a site can always have is its own normal traffic. v2 therefore weaves three threads per source and lets the site's benign stream condition every one of them:

layer	what	cost (Go, 1 core)	runs on
0 entity state	decayed counters, novelty memory, gaps (36 features)	275 ns	every flow / request
1 trees	oblivious trees on layer-0 features (shipped prior: "what attacks look like")	1.7 µs	every flow
1' site anomaly	quantile p-values / isolation forest on layer-0 features, fitted on this site's benign only	~1 µs	every flow
2 sequence net	causal GRU per source over its event stream (+ optional site context FiLM)	8.6 µs at d=24 (keep d ≤ 64)	warm sources only (tier-1 above the low-water mark)
HTTP	hashed n-gram bag + MLP conditioned on the site profile (HttpModel2); the request also enters layer 0 as an event	8.9 µs	every request

Decision: every thread yields a conformal p-value against the site's benign reference; the verdict is the minimum (Bonferroni over threads), so the realised false-alarm rate stays at α whatever the mix. Family head on alarms only; EntityRisk / SitePressure / drift as before.

Evaluation protocol (guard_jobs/seq_train.py): temporal + leave-one-network-out on source-capped corpora, each run reporting flow AUROC, site-calibrated FPR/TPR, entity-minute AUROC and the intensity correlation for the GRU, the trees, the two anomaly baselines and the fused verdicts. Honest result: single-record transfer is weak for everyone; the fused, site-conditioned verdict is what ships.

Modern threat landscape (2025-26 vendor reports): L7 floods and bots (half of web traffic), API abuse, credential stuffing, scanners, AI crawlers - i.e. behaviour over time per client, which is exactly what layers 0/2 and the HTTP session stream measure. Injection payloads still matter and are covered by the n-gram model; the 2025 ModSecurity thirty-day archive (CC BY 4.0) is the modern attack-payload source.

LLM bridge (dibaguard/llm.py): five tools (status, score flows, score http, calibrate, report) as MCP over stdio, OpenAI / Anthropic tool schemas and a stdlib HTTP server; the incident report (dibaguard/report.py) is a fixed JSON schema rendered to English and Persian from the engine's own state - no model narrates numbers it did not compute.

8. v2.1: the real-world layer (2026-10-30, evening)

What was weak after v2 and what fixed it - every item measured, none tuned by hand:

weakness	root cause	fix (dibaguard/engine.py)
per-flow detection at α = 1e-3 ≈ 0 on unseen networks	the flow is the wrong unit: an attacker sends hundreds of flows, a verdict per flow throws the sequence away	per-source accumulated evidence E ← E·e^(-Δt/τ) + (−log p) (τ = 120 s), judged per (source, minute) against the site's benign source-minute maxima (decide.SourceEvidence). LODO CIC-IDS2018: every attacking source caught, source-minute TPR 0.6 at FPR 2e-3 (was 0.0 per flow)
realised FPR above target when the traffic mix changes	one global reference; a new or bursty service moves every threshold	Mondrian references by service × source age (ent.service_key, AGE_EDGES): a client's first flows are judged only against other clients' first flows; a cell without reference abstains (p = 1) instead of guessing
false alarms on new clients (flash crowds)	births look extreme against steady-state sources	same age conditioning + abstention; destination-side features kept out of per-flow verdicts (they belong to site pressure)
FPR ratchet over hours (+1 %/h)	self-cleaning rolling references trimmed the benign tail; anomaly scores against a growing reference change scale	epoch re-calibration: every recal_s (30 min) all references are rebuilt at once from the last recal_window_s (2 h) of traffic, sources with evidence left out; the evidence reference is built out of sample (last quarter scored against the first three). Measured: 0.08 → 0.23 → 0.23 % per source-minute over three benign hours at α = 1e-3 (was 0.18 → 1.0 → 2.5 %)
wrong attack families	the GRU family head does not transfer	transparent rules on the stream counters (ent.family_rules): port walking → portscan, same (host, port) again and again → bruteforce, ≥ 10 flows/s at one service → dos/ddos, host walking with failures → recon
supervised prior scoring a SYN scan as benign	dataset artefacts in unconstrained trees	monotone trees on the 24 attack-direction features (more scanning can never look more benign)
unseen web application (WordPress)	vocabulary the model never saw	site n-gram novelty (HttpModel2.novelty): share of a request's n-grams absent from the site's own benign vocabulary, rarity-weighted - evaluated (WordPress AUROC 0.92 vs 0.61), not yet part of the live verdict

False-alarm budget: α/2 to the evidence tier, α/(2·k) to each of the k per-flow tiers (Bonferroni), so the union stays at α.

What the engine learns on the server and what it does not: every reference, threshold, site profile and vocabulary is re-learned from the site's own traffic every epoch (no labels, poisoning-resistant by excluding sources with evidence); model weights (trees, GRU, HTTP MLP) are frozen - online weight updates without labels would be a poisoning surface. The feedback loop (operator-confirmed alarms → periodic offline retraining) is the next step, not an online one.

9. v2.2: distributed floods, slow scans, feedback, rules, and the live lab (2026-10-31)

The static cross-dataset numbers hid live failures that only a running, multi-wave environment shows. A local red-team/blue-team lab (scripts/redteam_lab.py: a self-owned localhost target, benign traffic, and the engine's own playbook patterns replayed over several waves, scored against known ground truth) drove the next round of fixes:

gap the live lab exposed	fix
distributed / volumetric floods (amplification, peer/sybil floods) have no single guilty source	per-destination evidence tier: a destination's (flows, no-reply, new-source) anomaly accumulated per destination-minute; a flow is blamed only if it is new to the destination and off-profile, so the server's established benign clients are spared. NTP amplification 1.0, peer flood 0.76-0.80
slow scans (nmap -T1 over ~30 min) never accumulate at tau=120 s	long-horizon source evidence (tau=1800 s) in parallel, fed only by mildly-suspicious flows; once its reference is warm (~2 epochs) slow-scan detection is 0.95
detection decayed wave-over-wave (partially-detected attacks seeped into the self-training)	reference hygiene (as of v2.5): a source whose alarms name an attack family, or whose alarms keep coming while it behaves like an attacker, is excluded from the reference entirely; a sporadic un-named alarm stays in (it is the benign tail - excluding it ratchets the FPR up); no source may write more than ~1% of a reference; plus an immutable anchor mixed into every epoch and a drift guard that asks who moved the tail. Destinations are not excluded. See docs/TECHNICAL.md section 4.3-4.4
HTTP session attacks (credential stuffing, scrapers, API enumeration) under-detected	the request is also an event of the client's stream; with the burst preserved, stuffing/scraper/BOLA all reach 0.98-0.99

Live lab, 6 waves, shipped models, 3000-client synthetic site: HTTP attacks caught 100% (JNDI/WordPress via rules 1.0, credential stuffing 0.99, scraper 0.94, BOLA 0.98); flow attacks 77% averaged over cold+warm waves (NTP amp 1.0, rapid reset 0.95, SSH brute 0.75, peer flood 0.76, slow scan 0.16 cold -> 0.95 once warm); benign false-alarm rate 2e-4 per (source, minute) for flows and 2e-4 for HTTP; blue-team feedback precision 0.95.

Operator layer:

• Rules (dibaguard/shield/rules.py): plain-language rules in Persian or English are interpreted (a built-in bilingual parser, or an LLM returning the same JSON schema) into deterministic predicates and enforced from the first second - so there is no "unprotected while it learns" window. The interpretation is shown back for confirmation. An allow rule shields a source; the playbook's hard signatures (JNDI, CMS paths) also act at t=0.
• Feedback (dibaguard/shield/feedback.py): the operator confirms / rejects / allowlists alarms from a dashboard, MCP or REST. Reject lifts the action and the source's traffic becomes benign reference immediately; confirm takes (and escalates) the action; every verdict is logged and exported as labelled data for the offline retraining job. So there is no online weight update (poisoning-safe), but the layer adapts at once to operator intent and accumulates a retraining set.
• Licence-clean retraining: the network corpus is being rebuilt from the MIT-licensed merged NetFlow v3 (guard_jobs/entity_build.py nfv3_parquet, corrupt row groups skipped) so the shipped prior can be trained without the CC BY-NC-SA sets.

10. v2.3: hard adversarial lab, zero-benign-block guarantee, graceful degradation (2026-10-31)

A hard-mode lab (scripts/redteam_hard.py) runs 8 waves in which the attacker evolves to evade (slower, more source IPs, smaller payloads, UA mimicry) while the benign side is hostile (40% brand-new clients each wave, flash crowds, Persian traffic, a newly-added service). It scores recall per family and the benign BLOCK rate (the real harm), per wave, so the learning/degradation curve is visible.

What it forced, and the design principles that came out of it:

• Evidence is a zero-mean CUSUM, not a sum. Per-flow increment is -log p - k (k = the benign mean of the suspicious-flow -log p), floored at 0. A benign source drifts around 0 at any request rate, so a chatty client or a flash crowd no longer accumulates its way over the threshold; only a genuine excess builds up. Gated to off-profile flows (p < 0.5) on both the fast (tau=120 s) and slow (tau=1800 s) horizons.

• First contact is never auto-blocked. A source's first flows carry no behaviour to judge; behavioural tiers abstain for a newborn source unless the single flow is itself self-evident (p < alpha, e.g. a 1400-byte no-reply UDP to a victim) or the supervised prior fires hard. A soft newborn alarm becomes block_ok = False -> the layer issues a challenge (HTTP) or monitor (flow), never a block. HTTP session alarms block only when the client's requests are largely failing (4xx/5xx): a user browsing many pages trips the path-diversity signal but gets 200s, so it is challenged; enumeration / credential stuffing / scanners fail requests and are blocked.

• Result (8 waves, evolving attacker, 40% flash-crowd benign): zero benign auto-blocks on both flows and HTTP in every wave; feedback precision 1.0. Recall: rapid reset / NTP amplification / SSH brute force ~1.0, peer flood 0.94; the learning curve degrades gracefully only against the maximally-evading low-and-slow distributed cases (evolving scan 1.0 -> 0.16, distributed credential stuffing 0.98 -> 0.23 as it spreads over many client IPs). These are the known-hard, aggregate-only attacks; they are documented honestly and are the roadmap's global-rate tier, not a silent failure. Benign alarms that are not blocks (challenges/monitors) are expected under a flash crowd and are the standard "under attack" trade-off, tunable via policy.min_action_score.

• Multilingual, by design. Rule interpretation and the LLM bridge are language-agnostic: with an LLM wired in (the normal deployment of an LLM-facing product) a rule in any language is interpreted to the same JSON schema (rules.LLM_SCHEMA_PROMPT); the built-in parser is only the Persian/English offline fallback. The rendered report and dashboard are English + Persian.

What is honestly not done (roadmap, not claimed): a global/aggregate rate tier for fully-distributed low-volume attacks; automatic database- and network-topology profiling beyond the posture audit's service recognition; and online weight learning (kept offline on purpose, fed by the feedback-exported labels) - see section 8.