DibaAi/DibaGuard-Open

DibaGuard Open

A CPU-light attack-detection and protection layer for network flows, HTTP, JSON-RPC and databases, with calibrated false-alarm rates, explainable verdicts and a policy-driven enforcer. This is the public, licence-clean build. Code is Apache-2.0; the model files are CC BY-SA 4.0 (one training source requires ShareAlike). Built for the Build with Beam challenge; the companion Space is DibaAi/beam-dibaguard.

DibaGuard sits where a WAF, an IDS and a host firewall would sit: inline in front of an application, as an external decision service behind the load balancer you already run (nginx, Envoy, HAProxy, Traefik, Caddy), or beside a host reading flow and log exports. Every alarm comes with a controlled false-alarm rate (conformal calibration on the site's own traffic, default budget 10^-3 per source-minute), a named attack family, the evidence that produced it, and an action chosen by the operator's policy (report only, or enforce). Inference is numpy-only in Python, with a dependency-free Go fast path proven bit-exact against the Python one on parity fixtures.

What is in this repository, and what is not

path	content	licence
models/seq_v1/nf_entity_mono.npz	tier-1 monotone tree prior (24 behaviour counters, CatBoost oblivious trees evaluated in numpy)	CC BY-SA 4.0
models/seq_v1/seqnet.npz, seq_v1_info.json	tier-2 GRU sequence network (d = 64, 95,293 parameters)	CC BY-SA 4.0
models/entity_v1/nf_entity_detector.npz, lu_entity_detector.npz	fallback entity detectors (NetFlow view; LUFlow view)	CC BY-SA 4.0
models/dbnet.npz	DbNet, the database query network (own grammar-generated corpus, no third-party data)	CC BY-SA 4.0
models/**/*.bin, *.bin.json	the same weights as flat little-endian arrays for the Go runtime	CC BY-SA 4.0
runtime/python/dibaguard/	the engine, shield, HTTP edge, database tier, LLM bridge (MCP + REST) and operator console	Apache-2.0
runtime/go/	the Go fast path (dibaguard-authz, dibaguard-proxy) and its parity fixtures	Apache-2.0
beam/beam_client.py	the stdlib-only Beam client the Space uses (OpenAI-compatible endpoint; key from the environment)	Apache-2.0
docs/, reports/	technical documentation, independent validation report, licence audit, benchmark	Apache-2.0

Not in this repository: the HTTP request model (http_v2/http_tier1_v2.npz). The one in the private build is trained on data that is not cleared for redistribution (CSIC 2010, a labelled access log whose raw file the owner withholds, several payload lists of unknown licence; docs/DATA_LICENSES.md rows 12-19), and the licence-clean retrain collapses on applications it never saw (docs/CLEAN_MODELS.md section 0.4), so it is not shipped either. The engine supports this configuration natively (Guard.load with no http_v2 file; guard.http is None): the HTTP edge of this build runs the request signatures A01-A22 (JNDI, CMS exploit paths, SQL / NoSQL / command injection, traversal, XSS, SSTI / XXE / SSRF, sensitive-file probes, declared AI crawlers), the SQL-context analysis of every parameter (A22) with DbNet as a second opinion, and the behavioural session tier (every request is an event of its client's stream, so enumeration, brute force, scraping and scanners are caught as behaviour). What the HTTP model added on top in the private build is a per-request anomaly score for payloads the signatures do not name; this build does not have that.

The cascade

Every tier looks at a different scale, every score becomes a p-value against the site's own reference, and a tier alarms when its p-value is under its share of the false-alarm budget (docs/TECHNICAL.md, sections 2-7):

• Tier 0, entity state: 36 exponentially decayed behaviour counters per source over three horizons (10 s, 2 min, 30 min): no-reply, SYN-only, reset, new-port, new-host, repeat and short-flow shares, gaps, and three destination-side counters. Identity-free and scale-free, which is why the models transfer between networks.
• Tier 1, monotone tree prior: oblivious trees over 24 attack-direction counters with increasing constraints (more port walking or more failed connects can never look more benign), calibrated per service x source age (Mondrian conformal).
• Tier 1', site-only anomaly: no labels; Fisher's combination of upper-tail p-values of the shape features against the site's own traffic.
• Tier 2, GRU SeqNet: a causal GRU with one hidden state per source, FiLM-conditioned on a site context vector; runs on warm sources only.
• Tier 4, per-source evidence: fast and slow CUSUM over a source's stream, so weak signals accumulate and a first-contact source is never blocked on one event.
• Tier 5, destination flood and tier 6, campaign tracker: floods of new sources on one destination; distributed low-and-slow campaigns found through a fingerprint shared across thousands of sources.
• HTTP edge: one canonical request text on every path (Python and Go produce byte-identical text), signatures A01-A22, SQL-context analysis of parameters, DbNet second opinion, payload provenance between the database tier and the HTTP edge, session tier.
• Database tier: six engines' query / audit logs; DbNet + signatures + behaviour + provenance; an application server is never blocked at the network level, the web source behind the payload is acted on instead.
• Shield: policy (report | enforce), enforcer (nft / iptables / netsh / dry-run) with TTLs, posture audit, rules in any language, NAT / CGNAT / IPv6 identity, operator feedback that an attacker cannot poison, review queue, report.

Open build versus private build

The network models here are the licence-clean set of docs/CLEAN_MODELS.md, trained only on sources the audit marks GREEN (SIMARGL 2021 / 2022, APPRAISE, ACI-IoT-2023, CSE-CIC-IDS2018 from the publisher's AWS bucket, LUFlow from its author). Head-to-head with the private ("shipped") models on the same rows, conformal threshold at alpha = 10^-3, tier-1 prior, per flow:

scored network (neither model trained on it)	shipped AUROC	open (clean) AUROC
SIMARGL2022	0.968	0.999
APPRAISE	0.614	0.985
ACI-IoT-2023 (author release)	0.622	0.644
CSE-CIC-IDS2018, AWS, Tue 20-02	0.949	1.000

Per-source evidence on SIMARGL2021 (an unseen network for the clean run), judged per (source, minute): clean FPR 0.00081 with 8 / 8 attacking source-minutes caught, shipped FPR 0.0029 with 16 / 17. The GRU's leave-out AUROC on SIMARGL2021 is 0.991 (shipped 0.822); the LUFlow detector improves from 0.984 to 0.9998 after a loader clock fix.

Where the open models are weaker: on NF-UNSW-NB15-v3 (AUROC 0.958 -> 0.066) and NF-ToN-IoT-v3 (0.955 -> 0.879), which are the private model's own training networks (UNSW Canberra, academic-only terms) and have no openly licensed twin. Expect weaker tier-1 scores on IoT / UNSW-style traffic; the site-only anomaly tier and the evidence tiers do not depend on training data. The open build has no HTTP request model (above).

Results

Reproducible labs of the main repository (scripts/benchmark_all.py, three seeds, reports/benchmark.md). These numbers were measured on the private build with its full model set, including the HTTP model that is not in this repository; the two SQL-injection lines and the libinjection comparison are signature + SQL-context + DbNet results that involve no HTTP model. The open build has not yet been re-run through the labs (see Verified at build time for what was checked on this exact folder).

lab (3 seeds)	item	mean	min-max
evolving attacker, 12 waves	Log4Shell / SSH brute force / NTP amplification / credential stuffing / peer flood / rapid reset / WordPress exploit / AI scraper, final recall	1.000 each	1.0-1.0
evolving attacker, 12 waves	API enumeration, final recall	0.979	0.973-0.983
evolving attacker, 12 waves	benign sources blocked (flows / HTTP)	0 / 0	-
evolving attacker, 12 waves	benign alarm rate per source-minute (flows / HTTP)	0.0011 / 0.0010	budget 0.001
six-wave playbook lab	SSH brute force / HTTP/2 rapid reset / credential stuffing / slow scan / chain peer flood, recall	0.945 / 0.980 / 0.984 / 0.935 / 0.875	see report
databases, 6 engines, 12 scenarios	detected / correct response	1.000 / 1.000 (all twelve)	1.0-1.0
databases	benign alarms per million queries / app servers enforced	0 / 0	-
low-and-slow campaigns C1-C5 (150-3,000 sources)	member recall	1.000 / 0.959 / 0.953 / 1.000 / 1.000	detection delay 4.7 / 8.3 / 15.3 / 0.0 / 0.0 min
unfamiliar networks: hosting	flood on one tenant / slow distributed scan / SSH guessing	0.991 / 0.918 / 1.000	benign alarms 0.00096 per source-minute, 0 enforced
unfamiliar networks: office	SMB lateral movement / Kerberoasting / RDP spraying / LDAP enumeration	1.000 each	benign alarms 0.00014 per source-minute, 0 enforced
unfamiliar networks: OT	PLC subnet scan / Modbus writes / telnet-ssh guessing / MQTT flood	1.000 / 1.000 / 1.000 / 0.839	benign alarms 0.000012 per source-minute, 0 enforced
platforms (WordPress, Joomla, Drupal, JSON API, generic)	17 attack scenarios detected	0.91-1.00 (13 of 17 at 1.000)	0 benign sources blocked
chain RPC	eth_getLogs flood / debug_traceTransaction flood / mempool spam	1.000 / 1.000 / 0.967	-

Real data (same report): SQL injection in the 2025 ModSecurity blocked-request archive with a visible payload 1.000 [0.9977, 1.0] of 1,671 (libinjection on the same requests: 0.9856); all CRS-labelled SQLi rows 0.531 of 3,225; SR-BH 2020 WordPress SQLi 0.608 of 9,886; false alarms on 80,000 benign requests of a production shop log 0.00013 [0.00007, 0.00023]; DbNet + signatures on unseen query templates TPR 0.983 at FPR 0.0; DbNet alone on unseen injection techniques TPR 0.743.

Independent validation

A team that did not build DibaGuard treated every claim as unverified and measured it (docs/VALIDATION_REPORT.md). They reported 22 findings (bypasses of the canonicaliser, silent caps, identity tricks, reference poisoning, state attacks, harm to legitimate users, a global tier that flooded unfamiliar networks with alarms). All 22 were fixed and each reproduction is now a permanent regression test (section 8 of the report). After the fixes: canonicalisation fuzz of 3,000 cases, Python edge and Go path each block 99.3 % with 0 % diverging verdicts (72.5 % / 76.9 % and 32.8 % divergence before); 80,234 benign requests of a real shop log, 0 blocked on every path; harm lab (26 simulated hours, 3 seeds), users blocked per 1,000 per day 146 -> 0.0, partner API clients and the search crawler no longer challenged; OWASP CRS regression tests (4,138 attack / 1,054 negative), hard tiers 14.7 % of attack tests and 4.3 % of negatives (a lower bound: many CRS tests are fragments a context engine is not meant to flag).

Charts

Rendered by scripts/beam_charts.py from the result files (charts/).

How Beam is used

Detection never depends on a language model: every verdict above is deterministic and explainable. Beam (reflection/Beam-1.0-501B-A23B, through an OpenAI-compatible endpoint: Reflection's API or Hugging Face Inference Providers, beam/beam_client.py) is used where language is the interface:

• Teach with Beam. The operator writes a rule in any language ("block anyone brute forcing ssh over 30 attempts a minute", "challenge every user-agent containing bot", or the same in Persian, Turkish, Spanish). Beam is asked for one JSON object against the fixed rule schema of dibaguard/shield/rules.py (action, scope, conditions filled from the engine's own counters: requests per minute, new ports per minute, no-reply share, repeat share). The precise rule and Beam's reasoning are shown back to the operator, who confirms before it is installed (Layer.add_rules(texts, interpret_only=True) then add_rules(texts)); from then on it is enforced from the first second with no learning window. A built-in bilingual parser handles the common forms without a model.
• Analyst. For an incident, Beam receives the engine's evidence (dibaguard.report.REPORT_SCHEMA: families, per-source evidence, timeline, what the layer did and why, database remediation) and writes the explanation an on-call engineer needs, with a proposed action. The action sits behind an approval gate: nothing Beam proposes is applied until the operator approves it in the console, and Beam never sees raw credentials or the console token.

The Space (DibaAi/beam-dibaguard) demonstrates both on the labs' synthetic traffic.

Quick start

Python (numpy only)

pip install numpy
export PYTHONPATH=$PWD/runtime/python        # or: pip install -e runtime/python after adding a pyproject of your own

import numpy as np
from dibaguard.engine import Guard
from dibaguard.shield.layer import Layer

guard = Guard.load("models")                # tier-1 prior, GRU SeqNet; guard.http is None in this build
L = Layer.load("models", policy={"mode": "report"},            # report: verdicts and would-be actions, nothing enforced
               inputs={"http": True, "flows": True}, backend="dryrun")   # DbNet is picked up from models/dbnet.npz

# 1. calibrate on this site's own benign traffic (no attack labels needed)
L.calibrate_flows(benign_flows)             # dict of columns, see dibaguard.engine.FLOW_FIELDS
L.calibrate_http(["get /shop?item=12 | h:user-agent=mozilla/5.0", ...])   # canonical texts, dibaguard.http.parse
L.calibrate_db(benign_db_events)            # {t, engine, db, client, query, rows, status}, adapters in shield/database.py

# 2. score, in time order
out = L.flows(flows)                        # per-flow alarm / family / score + actions per source
h = L.http(texts, client=clients, host=hosts, status=statuses, t=times)   # playbook ids, families, actions
d = L.db(events)                            # verdicts with remediation; app servers are never blocked
print(L.report())                           # incident report (JSON); dibaguard.report.render_en for text

The sidecar serves the same engine over MCP and REST, with the operator console:

python -m dibaguard.llm http --models models --port 8787 --policy policy.json --backend dryrun   # REST + console
python -m dibaguard.llm mcp  --models models                                                      # MCP over stdio

Start in report mode with the dry-run backend; switch the policy to enforce only after reviewing the report (docs/INTEGRATIONS.md section 10).

Go fast path

cd runtime/go
go build -o dibaguard-authz ./cmd/dibaguard-authz      # stdlib only, Go 1.22
./dibaguard-authz -listen 127.0.0.1:9300 -models ../../models -sidecar http://127.0.0.1:8787 -mode report

dibaguard-authz answers nginx auth_request, Envoy ext_authz, Traefik ForwardAuth, Caddy forward_auth and HAProxy (docs/INTEGRATIONS.md). Without an HTTP model file it logs model unavailable once and runs on signatures and the sidecar's state, which is the intended configuration of this build. dibaguard-proxy (the inline reverse proxy) requires models/http_v2/http_tier1_v2.bin at start and therefore does not run against this model folder; use dibaguard-authz behind your load balancer, or the Python layer inline.

Data and licences

• Code: Apache-2.0 (LICENSE).
• Model files under models/: CC BY-SA 4.0 (LICENSE-WEIGHTS), because LUFlow (Mills et al., IEEE TNSM 2021), one training source, is CC BY-SA 4.0 and whether trained weights are Adapted Material is legally unsettled; this release applies ShareAlike rather than relying on the answer.
• Training data of the network models (NOTICE): RoEduNet-SIMARGL2021 and SIMARGL2022 (CC BY 4.0, Mihailescu et al., Sensors 2021); APPRAISE H2020 real labelled NetFlow (CC BY 4.0, ITTI); ACI-IoT-2023 (CC BY 4.0, Nack et al., MILCOM 2024, doi:10.21227/qacj-3x32); CSE-CIC-IDS2018 from the publisher's AWS bucket (Sharafaldin et al., ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018); LUFlow 2020 / 2021 (CC BY-SA 4.0, Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517). DbNet: own grammar-generated corpus. The same block is stored in each model's metadata.
• Audit: docs/DATA_LICENSES.md classifies every dataset the project ever touched (GREEN / AMBER / RED, with the owner's terms quoted); docs/CLEAN_MODELS.md records how the clean set was built, the licence gate in the training jobs, and every head-to-head number. Evaluation-only material (OWASP CRS tests, SR-BH 2020, the ModSecurity 2025 archive) was never trained on and nothing from it is bundled.

Limitations

• No HTTP request model in this build (above). Payloads the signatures and the SQL-context analysis do not name are caught only when their client's behaviour is; the benign alarm rate of the HTTP edge is correspondingly lower, and so is its recall on obfuscated XSS / command-injection variants.
• Benign alarm rate on public flow captures is above the budget when the scored period differs strongly from the calibration window (docs/VALIDATION_REPORT.md section 8.5, measured with the private models after the fixes): CSE-CIC-IDS2018 (AWS, DDoS day) 6.77 % of benign source-minutes alarmed (3.04 % without the two global tiers; 99.7 % of attack source-minutes caught); SIMARGL 2021 (a 2-million-flow scan) 24.07 % (1.03 % without the global tiers; 100 % caught); LUFlow 2021 temporal hold-out 1.12 % (10.3 % of reputation-labelled attack sources caught). The budget is 0.1 %; the labs meet it, these captures do not yet. A longer calibration window (a day rather than hours) is the first lever.
• IoT / UNSW-style networks: the open tier-1 prior is weak there (section above).
• Payload inspection needs clear text (TLS-terminating proxy or logs); encrypted traffic is covered for behaviour only.
• A slow distributed scan among internet background radiation is found by the campaign tier in 6-17 minutes.
• Python state costs about 3-3.6 KB per tracked source (design budget 300 B): size max_entities to RAM or put the Go tiers in front; re-calibration still runs on the hot path.
• Prose that has the exact shape of an injection (admin'#') is still blocked (2.09 % of technical docstring sentences with an apostrophe; 0 of 16 ordinary review sentences).
• Several uploader identities behind the CC BY sources (SIMARGL, LUFlow, ACI-IoT Kaggle accounts) are inferred from display names, not verified; the CIC redistribution wording should be confirmed by e-mail. Not legal advice.

Verified at build time

On this exact folder, in a fresh interpreter with PYTHONPATH=runtime/python: Guard.load("models") loaded the tier-1 prior and the GRU with guard.http is None; Layer.load picked up DbNet; after calibrating on 60,000 synthetic benign flows and 300 benign requests, a batch of 31,300 flows (synthetic SSH brute force from one source, a 400-probe slow scan from another, 30,000 benign flows from 3,000 clients): the brute-force source was reported as bruteforce and the scanner as portscan, both with would-be action block (report mode, nothing enforced); 57 of the 3,000 benign clients had 67 alarmed flows of their 30,000 (0.22 %; 19 block-safe, families ['attack_generic', 'dos'], would-be actions ['monitor']) during the attack minutes, on the test-suite's synthetic benign generator; HTTP: a JNDI request matched ['A08'] (family rce), a UNION SQL injection in a query parameter matched ['A22'] (family sqli), a benign shop request raised nothing; database tier: a UNION injection through the application's own connection alarmed as ['injection'], a benign query did not, and DROP DATABASE from an unknown public client alarmed as ['destructive', 'new_client']; the rule parser and both report renderers ran from the transformed sources.

python -m pytest -q tests/test_parity.py on the main tree (Python <-> Go parity fixtures, unchanged): not run (--skip-parity).

Model files

file	bytes
models/dbnet.bin	3,160,300
models/dbnet.bin.json	680
models/dbnet.npz	2,403,349
models/dbnet_eval.json	6,016
models/entity_v1/lu_entity_detector.bin	188,976
models/entity_v1/lu_entity_detector.bin.json	1,916
models/entity_v1/lu_entity_detector.npz	152,441
models/entity_v1/nf_entity_detector.bin	274,400
models/entity_v1/nf_entity_detector.bin.json	2,820
models/entity_v1/nf_entity_detector.npz	221,205
models/seq_v1/nf_entity_mono.bin	82,358
models/seq_v1/nf_entity_mono.bin.json	3,116
models/seq_v1/nf_entity_mono.npz	17,794
models/seq_v1/seq_v1_info.json	560
models/seq_v1/seqnet.bin	381,172
models/seq_v1/seqnet.bin.json	5,512
models/seq_v1/seqnet.npz	390,463

Documentation

docs/TECHNICAL.md (design, models, calibration, HTTP edge, database tier, shield, history, results, limitations), docs/VALIDATION_REPORT.md (independent validation and the remediation), docs/CLEAN_MODELS.md (licence-clean models, head-to-head), docs/DATA_LICENSES.md (licence audit), docs/INTEGRATIONS.md (nginx, Envoy, HAProxy, Traefik, Caddy, Kubernetes), docs/CONSOLE.md (operator console), docs/HA.md (state, restarts, replicas), docs/PERFORMANCE.md (throughput, latency, memory), docs/DESIGN.md (the original design note), reports/benchmark.md. The copies in this repository had their Persian sections removed and carry a note about what applies to the open build.

Citation

@software{dibaguard_open_2026,
  title   = {DibaGuard Open: a CPU-light, calibrated attack-detection and protection layer},
  author  = {Dibachain (DibaAi)},
  year    = {2026},
  url     = {https://huggingface.co/DibaAi/DibaGuard-Open},
  license = {Apache-2.0 (code), CC-BY-SA-4.0 (models)}
}

Built 2026-10-31 by scripts/make_open_release.py of the main repository.