DibaGuard — performance and scalability (measured 2026-10-31)
Open-release copy (DibaAi/DibaGuard-Open). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (
http_v2) were measured with that model, which is not part of the open release; the open release ships the licence-clean network models described inCLEAN_MODELS.md. Licence statements inside this document are superseded byDATA_LICENSES.md,CLEAN_MODELS.mdand the repository'sLICENSE,LICENSE-WEIGHTSandNOTICEfiles. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.
Roadmap items 9 (scalability) and 10 (runtime). Every number below was measured in this session on the hardware of
§2, with the shipped models (§1.3), on synthetic traffic only (§1.1). What was not measured is listed in §10.
Bugs and bottlenecks found are in §8 — none of them is fixed here (dibaguard/ and the Go sources belong to other work).
0. Summary
| question | answer (measured) |
|---|---|
| Does the Python engine run at 2M+ distinct sources? | Yes. 5,325,510 distinct sources over 8.2 simulated hours on a Kaggle CPU kernel, no crash, RSS flat at 2.8–3.0 GB (peak 3.4 GB) from 500k sources on: the per-source tables stop at their 500k LRU cap. |
| Where does it break? | With the caps removed, RSS grows 2,947 bytes per source (linear fit, 6,001,699 sources reached at 17.6 GB): the 30 GB kernel would run out at ≈ 10M sources (10.8M without the job's parent process). With the default caps it does not run out of memory — it forgets: 55% of the flows of returning sources found their record already evicted at 2M sources, 72% at 5M (§3.4). |
| Do the per-source tables stay bounded? | The four LRU tables do (500k entries each). The warm-source GRU table does not: 300k one-shot scan sources left 300,173 entries, all still there 30 minutes later (§3.7, B3). Per-record size is not bounded either: a scanner's record is 108 KB vs 1.9 KB (B9). |
| Detection at 2M / 5M sources | A fast SYN scan, an SSH brute force and a slow scan injected from new addresses: 100% / 100% / 100% of their flows alarmed at 1M, 2M and 5M sources, first flow alarmed in every case, families portscan / bruteforce / portscan (§3.5). |
| Does accuracy degrade? | Detection: no. Benign alarm rate: depends on the traffic, not on the source count. With a stationary population it was 0.76% of benign source-minutes at 1M sources (RUN4_SUMMARY), almost all from a site-wide tier with a fixed floor (B5). With a growing population it was 6.5–8.6%, from the campaign tier, which cannot recover from a benign level shift (B8). The engine's own target is alpha = 1e-3. |
| Memory pressure | A spoofed flood of 1.2M one-flow sources in 60 s (2.4x the tables) at 5.3M sources: no crash, tables stayed at 500k, 100% of the flood alarmed (portscan), and a slow scan and a slow SSH brute force running through it stayed 100% alarmed before and after — their records survived because they were active every few seconds. The flood's batches were 4x slower (B7). |
| Python flow throughput | 9.7k flows/s per core on the Kaggle Xeon (batch 5000, after a 400k-flow calibration), 6.6k flows/s per core sustained in the 5M-source run, 24.2k flows/s with 4 processes on 4 vCPUs. A fixed cost of ~20 ms per call (B1) makes small batches slow: 48 flows/s at batch 1, 3.2k at batch 100 (local; ~2x worse on the 05:13 tree, B1). |
| Python latency, batch of 5000 flows (Kaggle, 5M run) | p50 / p95 / p99 / p99.9 = 0.60 / 0.93 / 6.3 / 16.8 s per batch (119 / 185 / 1,252 / 3,362 µs per flow). The tail is the synchronous re-calibration: 13 batches, p50 15.3 s, max 17.2 s (B2); one took 17.5 minutes with the newest engine (B7). |
| Python HTTP / DB | 3.8–4.8k requests/s and 2.3–2.8k DB events/s per Kaggle core; per-request p50 200–255 µs in batches of 100–1000. |
| Go fast path per core (local i3-12100F) | tier 0+1 per flow p50 / p99 = 7.0 / 14.1 µs (134k flows/s); SeqNet step 34 / 64 µs; HTTP2 model 58 / 137 µs (32 / 75 µs with the site FiLM cached, B10); signatures 2.8 µs when the prefilter skips the regexes, 96 µs when it does not — and it does not for any browser User-Agent (B11). |
Against the budgets of DESIGN.md §1:
| budget (DESIGN.md §1) | target | measured | verdict |
|---|---|---|---|
| RAM, runtime + 1M tracked entities | < 300 MB | Python engine: 2,947 bytes per tracked source (Kaggle fit; 2.6–4.0 KB by tracemalloc) → ≈ 3 GB per 1M sources. Go state not measured. | not met by the Python engine (≈ 10x) |
| HTTP, tiers 0–1, batched | ≥ 20k req/s, p99 < 1 ms | Python: ≤ 4.8k req/s per Kaggle core. Go model: 15.3k/s per core, p99 137 µs; 28.9k/s per core with the FiLM cached; 48.7k/s with 4 workers. Go signatures + model: 6.6k/s per core, p99 351 µs. | Python not met; Go met with ≥ 2 cores (model alone, or 1 core after B10) |
| flows, tier 1 | ≥ 200k flows/s | Go tier 0+1: 133.6k/s per core, 463k/s with 4 workers. Python engine (all tiers): 6.6–9.7k/s per Kaggle core. | Go met with ≥ 2 cores; Python ≈ 20–30x below |
1. Method
1.1 Synthetic traffic (no datasets)
All generators are in guard_jobs/scale_bench.py, shared by the Kaggle job and scripts/bench_runtime.py.
- Growing population (runs 1–2, default): a flow comes from a never-seen source with probability q = 0.3
(one-off clients, NAT / mobile churn, internet background), otherwise from an existing source drawn as
id = floor(n * u**3): old sources are the heavy hitters, rank-frequency ~ rank^-0.67, flows per source ~ power law with exponent 2.5; the busiest source carries ~ n^(-1/3) of the returning flows (1% at 1M sources). This population keeps growing, so the number of distinct sources per 5-minute window also grows — a level shift some tiers react to (§3.5). - Stationary population (runs 3–4,
--zipf-a 0.8 --pop 8000000): a fixed population of 8M ids with Zipf(0.8) weights, every flow drawn i.i.d. (busiest source 0.86% of flows, top 100 7%): every window looks like every other, only the number of distinct sources seen so far grows (2M after ≈ 4.3M flows). - Both at a simulated 600 flows/s (2M distinct sources ≈ 3.1 h of simulated time in the growing population).
- Benign flows: each source has a home server (200 servers) and service (443 58%, 80 18%, 53/udp 10%, 8080, 22, 3306, 25, 123/udp) from a hash of its id and roams 15% of the time; log-normal sizes and durations; 0.5% failed connects (SYN, no reply).
- Attackers (the shapes of
tests/test_engine.py), from never-seen addresses, merged into an 1800 s window of the live stream when the population reaches 1M, 2M and 5M distinct sources: a fast SYN scan (3000 probes in 100 s), an SSH brute force (900 logins in 5 min), a slow scan (400 probes over 30 min). - HTTP: canonical request texts built with
dibaguard.http.parse.from_url(...).model_text()(products, Persian / English search, JSON cart API, orders API, static assets, logins with 15% 401s, blog; 2% 404s), clients from the growing population at 2000 requests/s; attackers: credential stuffing (401s), id enumeration (404s), payload requests (SQLi / XSS / traversal / JNDI / SSTI / cmd). - Database: 16 query shapes over PostgreSQL / MySQL / MongoDB / Redis / Elasticsearch with random literals, 20 application pods; calibration at 50 events/s, scoring at 2000 events/s; every 10k events an injection, an RCE, a privilege escalation, a destructive query, an auth-failure burst from a public address and a 2M-row read.
1.2 Timing and memory
- Python:
time.perf_counter()around each engine call (Guard.flows,Guard.http_requests,DbGuard.score). Per-item latency = call time / batch size (amortised). Percentiles are over calls; the number of calls is given — with fewer than ~1000 calls the P99.9 is effectively the maximum. BLAS is limited to one thread (threadpoolctl), so numbers are per core. Latency excludes queueing (the time a flow waits for its batch to fill), I/O and parsing; request canonicalisation is measured separately (§4). Throughput = items / summed engine time. - Go: every call timed on its own (
go/dibaguard/bench_latency_test.go). Go's monotonic clock on Windows advances in ~0.5 ms steps, sobench_latency_windows_test.goreads QueryPerformanceCounter instead; one read costs 100 ns and sits inside every measured call. - Memory: RSS from
/proc/self/status(Kaggle) or the Windows working set (local), peaks from VmHWM / PeakWorkingSetSize, bytes per structure fromtracemalloc(CPython object overhead included). The uncapped run ran in a separate interpreter, so its RSS is the engine's alone.
1.3 Code, models, runs
- Models: the shipped set from
D:/build-tmp/guard/dl/models, uploaded next to each run's code bundle (code/<run>/models/in the private results repo) and verified by sha256 in the job:seq_v1/nf_entity_mono.npz(a03fa0a7…, the tier-1 prior: 300 trees, depth 6, 24 features),seq_v1/seqnet.npz(4875b9c9…, 2 layers, d = 64),entity_v1/nf_entity_detector.npz(fea603ea…),http_v2/http_tier1_v2.npz(66ae80a4…, 262,144 buckets, E = 48, H = 128),dbnet.npz(4d953493…). Go used the flat exports of the same files (D:/build-tmp/guard/card_repo/models). dibaguard/engine.pyanddibaguard/campaign.pywere being edited by another agent throughout the session. Each Kaggle run executed the bundle taken at its launch (the report records a sha256 of every source file):
| run | launched (+03:30) | population | phases | engine notes |
|---|---|---|---|---|
| guard-scale-1 | 02:57 | growing | shard, http, db, capped to 5.3M, churn, dscan, uncapped to 6M | campaign tier (early version), no drift guard |
| guard-scale-2 | 03:36 | growing | capped to 2M | campaign tier, flood "piling" condition |
| guard-scale-3 | 04:01 | stationary | capped (stopped at 1.57M by its time budget) | + drift guard (B7) |
| guard-scale-4 | 04:48 | stationary | capped to 2M | RUN4_ENGINE |
Local runs used the working tree of 03:00–05:15. Every finding in §8 was re-checked against the working tree of 05:13 (line numbers are from that version).
2. Hardware
| Kaggle CPU kernel (all four runs) | local workstation | |
|---|---|---|
| CPU | Intel Xeon @ 2.20 GHz, 4 logical CPUs (affinity 4), AVX2, no AVX-512 | Intel Core i3-12100F, 4 cores / 8 threads, 3.3 GHz base |
| RAM | 32 GB (cgroup limit 30 GB), no swap | 16 GB; ~6 GB free during the runs (other agents' processes running) |
| OS / runtime | Linux 6.18, Python 3.13, numpy 2.1.3 | Windows 11, Python 3.13.3, numpy 2.2.5, Go 1.23.4 |
The local numbers come from a shared, busy workstation: treat them as ±15%.
3. Python engine: flows
3.1 Batch size (local, 150k-flow calibration, ~140k sources)
| batch | calls | flows/s | call p50 ms | p95 | p99 | p99.9 | per-flow p50 µs | per-flow p99 µs |
|---|---|---|---|---|---|---|---|---|
| 1 | 300 | 48 | 19.78 | 26.12 | 31.11 | 51.81 | 19,782 | 31,115 |
| 10 | 300 | 413 | 23.47 | 30.06 | 34.43 | 44.84 | 2,347 | 3,443 |
| 100 | 200 | 3,180 | 29.91 | 39.57 | 60.58 | 143.38 | 299 | 606 |
| 1000 | 60 | 15,180 | 64.16 | 80.09 | 92.00 | 99.59 | 64.2 | 92.0 |
| 5000 | 12 | 19,837 | 219.44 | 400.08 | 463.62 | 477.92 | 43.9 | 92.7 |
Every call costs ≈ 20 ms before the first flow is scored (B1). The engine is only efficient in batches of ≥ 1000 flows, so a flow's verdict waits for its batch: at 600 flows/s a 1000-flow batch is 1.7 s of traffic.
3.2 Where the time goes (cProfile, 25 calls of 1000 flows, local)
| function | self time | incl. callees |
|---|---|---|
Guard._remember (engine.py) — the per-call re-calibration buffer (B1) |
35.6% | 35.9% |
EntityState.update + _get + _remember (flows/entity.py) — tier 0 |
22.7% | 29.5% |
Guard._features (engine.py) |
1.9% | 33.1% |
Guard.flows body (numpy glue and per-flow Python loops) |
4.6% | — |
SourceEvidence.update x2 (decide.py) |
2.6% | 4.8% |
CampaignTracker.update (campaign.py) |
1.8% | 3.5% |
EntityRisk, SitePressure, DriftMartingale, ConformalFPR.pvalue, GlobalRate._bump |
4.4% | 8.0% |
Oblivious.raw (tier-1 trees, numpy) |
1.6% | 2.4% |
In batches where most flows alarm (a flood, a distributed scan) a different cost dominates: np.isin on string
arrays took 40% of the time (B7) — 936 ms per 5000-flow flood batch vs 362 ms for a benign one (local).
3.3 Throughput per core and with shards (Kaggle)
Each process runs its own Guard on its share of a 600 flows/s site (traffic split by source), calibrated on 400k
flows, then scores 200k flows in batches of 5000.
| processes | flows/s total | per process | flows per CPU-second | RSS per process |
|---|---|---|---|---|
| 1 | 9,654 | 9,654 | 9,674 | 1.32 GB |
| 2 | 18,566 | 9,283 | 9,389 | 1.32 GB |
| 4 | 24,240 | 6,060 | 6,275 | 1.34 GB |
Two processes scale linearly; four on 4 vCPUs (most likely 2 physical cores with hyper-threading) give 2.5x. A shard sees only its own sources' flows, so destination-side counters (the flood tier) see a fraction of each destination.
3.4 Scale: 2M and 5M distinct sources (Kaggle run 1, production settings)
Guard.load(models, alpha=1e-3, warmup_s=600) (max_entities 500k), calibrated on 900k flows (1500 s), then the
growing population in batches of 5000 until 5M distinct sources:
| distinct sources | simulated h | RSS MB | peak RSS MB | flows/s (last 40 batches) | per-source tables (state / evidence / risk) | GRU entries | returning flows whose record had been evicted |
|---|---|---|---|---|---|---|---|
| 500,786 | 0.77 | 2,754 | 3,221 | 8,416 | 500,000 / 408,369 / 408,369 | 103 | 0.0% |
| 1,001,054 | 1.55 | 2,790 | 3,221 | 6,711 | 500,000 / 500,000 / 500,000 | 158 | 33.9% |
| 2,000,475 | 3.09 | 2,821 | 3,221 | 7,536 | 500,000 / 500,000 / 500,000 | 197 | 55.1% |
| 3,000,288 | 4.63 | 2,836 | 3,332 | 7,820 | 500,000 / 500,000 / 500,000 | 199 | 63.8% |
| 4,000,286 | 6.18 | 2,850 | 3,332 | 7,714 | 500,000 / 500,000 / 500,000 | 186 | 68.9% |
| 5,001,219 | 7.72 | 2,821 | 3,332 | 7,331 | 500,000 / 500,000 / 500,000 | 168 | 72.4% |
End: 5,325,510 distinct sources, 8.2 simulated hours, 14.7M flows in the main loop (plus 2.2M in the injection windows), RSS 2,962 MB, peak 3,371 MB (the calibration's peak was 3,221 MB). Recalibration buffer 400k rows, alarm log at its 50k cap.
Latency over the 2,940 batches of 5000 flows (6,633 flows/s of engine time):
| p50 | p95 | p99 | p99.9 | max | |
|---|---|---|---|---|---|
| batch, all (ms) | 596 | 927 | 6,261 | 16,812 | 17,176 |
| batch, without the 13 re-calibration batches (ms) | 596 | 918 | 2,124 | 7,645 | 8,026 |
| the 13 re-calibration batches (ms) | 15,348 | 17,110 | — | — | 17,176 |
| per flow, all (µs) | 119 | 185 | 1,252 | 3,362 | 3,435 |
| per flow, without re-calibration batches (µs) | 119 | 184 | 425 | 1,529 | 1,605 |
The same phase with the later engines: run 2 (to 2M) 5,273 flows/s, per-flow p50 177 µs, 4 re-calibration batches of ~14.0 s; run 3 (stationary, to 1.57M) per-flow p50 145 µs, and its one re-calibration batch took 1,047,733 ms (B7). With the stationary population the evicted-record share was 17.8% at 1M and 27.8% at 1.5M sources (run 3).
The eviction column is the scale effect that matters: with more distinct sources than the 500k cap, a source that stays silent for longer than the tables' turnover time (at most ~46 minutes here: 500k entries / 180 new sources per second, less once returning sources churn too) has lost its counters, its evidence and its age; the engine then judges it as a first-contact source (newborn suppression applies again). Memory stays bounded; memory of who is who does not.
3.5 Detection and benign alarms at scale
Share of each attacker's flows that alarmed (dominant family), and benign alarm rates in the same 1800 s window:
| run (population) | distinct sources before | fast SYN scan | SSH brute force | slow scan | benign source-minutes alarmed | benign flows alarmed |
|---|---|---|---|---|---|---|
| 1 (growing) | 2,000,475 | 100.0% (portscan) | 100.0% (bruteforce) | 100.0% (portscan) | 7.33% | 9.69% |
| 1 (growing) | 5,001,219 | 100.0% (portscan) | 100.0% (bruteforce) | 100.0% (portscan) | 8.64% | 10.12% |
| 2 (growing) | 1,001,054 | 100.0% (portscan) | 100.0% (bruteforce) | 100.0% (portscan) | 6.46% | 9.03% |
| 2 (growing) | 2,001,312 | 100.0% (portscan) | 100.0% (bruteforce) | 100.0% (portscan) | 6.83% | 8.96% |
| 3 (stationary) | 1,001,062 | 100.0% (portscan) | 99.9% (bruteforce) | 100.0% (portscan) | 0.76% | 0.89% |
| RUN4_ROW |
Every attacker's first flow alarmed; the slow scan's record was still in the tables at the end of every window, the fast scan's and the brute force's had already been evicted (they had stopped ~25 minutes earlier).
Where the benign alarms come from (alarmed benign flows by tier in the window):
- Growing population, runs 1–2: tier 6 (campaign) ≈ 49–52k, tier 4 (evidence) ≈ 40–50k, tier 5 (site-wide scan floor) ≈ 3.9–4.4k, tiers 1–3 ≈ 1–5k. The campaign tier counts distinct failing sources per 5-minute window per fingerprint; a growing population raises that count organically, the CUSUM fires, and because active windows never enter the history it never comes back (B8; reproduced in isolation, §8). From 400k sources on, tier 6 alone flagged ~5% of all benign flows in every 100k-source segment.
- Stationary population, run 3: tier 5 5,263, tier 4 4,281, tier 3 100, tier 1 11 — no tier 6. The global scan tier flags ~96% of the benign failed connects at this site rate (B5); without it the benign alarm rate would be ≈ 0.2% of source-minutes. Segment rates in the main loop: 0.52–0.57% of benign flows from 600k to 1M sources, 0.92% in the 1.5M segment. RUN4_BENIGN
- Local attribution at 600 flows/s (162k sources): the run-1 engine alarmed 6.0% of benign flows (77% of them tier 6);
the 03:22 working tree 1.4% of flows / 0.67% of source-minutes — tier 5 on 96% of the failed connects, and sources
with ≥ 50 flows in the window (12.6% of the traffic) alarmed on 7.7% of their flows (named
bruteforce: one heavy client hitting its server:443 several times a second) vs 0.48% for everyone else.
3.6 Memory pressure: a spoofed flood flushes the per-source tables (Kaggle run 1, at 5.3M sources)
Two 1800 s windows, each with a slow scan (300 probes, ~1 per 6 s) and a slow SSH brute force (600 logins, ~1 per 3 s) from new addresses; in the second, a spoofed SYN flood of 1,200,000 one-flow sources at one server between 600 s and 660 s — 2.4x the tables' capacity in one minute.
| control (no flood) | with the flood | |
|---|---|---|
| slow scan alarmed before / after the flood | 100% / 100% | 100% / 100% |
| SSH brute force alarmed before / after | 100% / 100% | 100% / 100% |
| attackers' records right after the flood | — | both present: age 117 / 224 flows, evidence 98.9 / 90.2 |
| flood flows alarmed | — | 100% (family portscan) |
| per-source tables | 500,000 | 500,000 throughout |
| batch latency p50 / p99 / max (5000 flows) | 0.74 / 1.43 / 18.1 s | 3.04 / 10.87 / 14.98 s |
| window wall time | 191 s | 1,036 s |
The LRU forgets the least recently seen sources: 1.2M new keys in 60 s push out anything not seen in the previous ≈ 25 s (500k / 1.2M x 60 s — arithmetic, not counted per source). Attackers active every few seconds keep their state; an attacker probing less often than that, and every quiet benign client, is forgotten by such a flood.
3.7 Distributed scan: the warm-source GRU table (Kaggle run 1)
300,000 new sources x 6 SYN probes each within 10 minutes, inside the normal stream:
| value | |
|---|---|
| scan flows alarmed | 100% (attack_generic 1,788,641, portscan 11,359) |
| GRU entries before / after the scan / after 30 more minutes of normal traffic | 169 / 300,173 / 300,174 |
| scan sources still holding a GRU state after those 30 minutes | 299,999 |
| size of one entry (tracemalloc) | 717 B → ≈ 215 MB for these 300k |
| batch latency p50 / p99 (5000 flows, mostly alarms) | 3.8 s / 11.0 s |
RSS moved only 3,437 → 3,487 MB because the phase freed other memory at the same time; the entry count is the measurement. Locally: 4,000 scan sources → 4,000 entries, all still held after 100k more flows.
3.8 Per-source memory and where the RAM runs out
Bytes per tracked source (local, tracemalloc, CPython 3.13, 64-bit):
| structure | bytes per source |
|---|---|
EntityState record, a one-flow source |
1,938 |
EntityState record, 8 flows to 8 (host, port) keys |
3,387 |
EntityState record, a scanner whose 512-key (host, port) memory is full |
107,940 |
SourceEvidence entry (two per source: fast and slow horizon) |
225 |
EntityRisk entry |
205 |
| warm-source GRU entry (2 x 64 float32 state, tuple, dict slot) | 717 |
| engine total: one-flow / 8-flow / worst-case scanner | 2,592 / 4,042 / 109,312 |
Uncapped (Kaggle run 1, max_entities = 10**9, q = 0.5, a clean interpreter):
| distinct sources | RSS MB | MemAvailable MB | flows/s (recent) |
|---|---|---|---|
| 432,067 | 1,515 | 27,609 | 9,829 |
| 931,879 | 3,072 | 26,039 | 8,380 |
| 1,930,275 | 5,844 | 23,272 | 6,453 |
| 2,931,652 | 8,844 | 20,252 | 6,934 |
| 3,930,928 | 11,427 | 17,661 | 10,034 |
| 4,931,869 | 14,034 | 15,067 | 9,910 |
| 6,001,699 | 17,617 (peak) | 11,663 | 9,803 |
Linear fit: 2,947 bytes per source, intercept 396 MB. It stopped at its 6M target, not at a limit; at that slope the kernel's 30 GB cgroup limit is reached at ≈ 10.8M sources (≈ 10.0M with the parent process's 2.2 GB beside it). Engine time: 7,871 flows/s, per-flow p50 / p99 / p99.9 = 96 / 653 / 6,070 µs. No eviction, so no state loss — but also no bound: B9's worst case (a scanner's 512-key record) is 37x the measured average.
3.9 The re-calibration stall
Every recal_s (1800 s) of stream time, the call that crosses the boundary rebuilds every reference from the buffer
(≤ 400k rows) plus the 80k-row anchor, synchronously, inside Guard.flows (B2):
| where | stall |
|---|---|
local, recalibrate_now() on 365k buffered rows (03:xx working tree) |
6.4 s |
| Kaggle run 1 (13 stalls) | p50 15.3 s, p95 17.1 s, max 17.2 s |
| Kaggle run 2 (4 stalls) | p50 14.0 s, max 14.1 s |
| Kaggle run 3 (drift guard present, 1 stall) | 1,047.7 s (B7) |
| RUN4_RECAL |
4. Python engine: HTTP requests
Per call of Guard.http_requests with clients, status and time (model + session stream + site-wide failure tier),
one host:
| batch | Kaggle req/s | Kaggle call p50 / p99 / p99.9 ms | Kaggle per-request p50 µs | local req/s | local per-request p50 µs |
|---|---|---|---|---|---|
| 1 | 958 | 1.02 / 1.45 / 1.98 | 1,023 | 2,397 | 400 |
| 10 | 3,132 | 3.07 / 5.36 / 8.87 | 307 | 7,455 | 126 |
| 100 | 4,796 | 19.96 / 31.14 / 145.34 | 200 | 7,409 | 130 |
| 1000 | 3,788 | 254.8 / 521.3 / 749.5 | 255 | 5,810 | 169 |
| 5000 | 2,405 | 2,086.8 / 2,526.7 / 2,548.8 | 417 | — | — |
Throughput falls above ~100 requests per call (B12). Model only (no client stream), batch 1000: 5,107 req/s
(Kaggle). Request canonicalisation (parse.from_url(...).model_text()) adds 13 µs p50 / 26 µs p99 per request (local).
Hosts per batch (2000 requests per call, no clients; each host is scored with its own profile):
| distinct hosts in the batch | Kaggle req/s | Kaggle call p50 ms | local req/s |
|---|---|---|---|
| 1 | 2,915 | 733 | 4,875 |
| 10 | 5,269 | 384 | 7,226 |
| 100 | 5,100 | 389 | 9,451 |
| 1000 | 2,197 | 908 | 4,850 |
Small per-host groups are cheap (smaller bags), but at 1000 hosts the per-host Python grouping and one model call per host cost 2.3x (B4).
Volume (Kaggle): 300,000 requests from 90,015 distinct clients at a simulated 2000 req/s, batches of 1000:
3,934 req/s, per call p50 / p95 / p99 / p99.9 = 246 / 275 / 360 / 848 ms; RSS 698 → 707 MB from 60k to 90k clients.
Session attackers among them: credential stuffing (118 requests) 100% alarmed, 92% block-safe, family bruteforce;
id enumeration (238) 97.9% alarmed and block-safe, family scan; payload requests (119) 100% alarmed (family
bruteforce 83, webattack 36 — the session verdict names the client before its payload). 1.57% of the benign
requests alarmed: a local replay shows all of them are challenges (none block-safe) on failed requests — 404
pages and 401 logins flagged by the site-wide failure tier, whose fixed floor normal failures exceed at 2000 req/s (B5).
5. Database tier (DbGuard.score)
| batch | Kaggle events/s with DbNet | per-event p50 µs | without DbNet | per-event p50 µs | local with / without DbNet (events/s) |
|---|---|---|---|---|---|
| 1 | 1,137 | 872 | 3,206 | 306 | 2,719 / 6,703 |
| 10 | 2,249 | 440 | 3,435 | 290 | 4,804 / 6,672 |
| 100 | 2,731 | 360 | 3,423 | 286 | 5,276 / 6,802 |
| 1000 | 2,447 | 406 | 2,846 | 360 | 4,656 / 6,646 |
Volume (Kaggle, 200k events, batches of 1000): 2,293 events/s with DbNet (call p99 495 ms), 2,765 without. Locally
classify() (the signature regexes) costs 99 µs p50 / 270 µs p99 per query and DbNet.score 49 µs per query in
batches of 500. Injected attacks: privilege 20/20, injection 60/60, destructive 20/20, RCE 40/40, exfiltration 20/20;
auth-failure events named bruteforce 232/240 (the first failures of the first burst are below the floor of 8). 1.7%
of the other events alarmed — a local replay shows every one is a later, normal query of the pod that performed the
injected 2M-row read: its decayed row counter (tau 300 s) stays over threshold for ~19 minutes, and all its queries are
reported as exfil meanwhile.
Peer-volume path (an over-threshold read compares the client with every tracked client, B6):
| (db, client) keys tracked | Kaggle ms per over-threshold event | local |
|---|---|---|
| 1,000 | 0.44 | 0.15 |
| 10,000 | 3.29 | 0.90 |
| 100,000 | 26.8 | 8.3 |
6. Go fast path (local i3-12100F, shipped models, 200,000 timed calls per row)
Run of 05:20 against sigs.go of 04:03 (an earlier run against the 01:36 file gave signatures p50 78.6 µs,
everything else within 5%). Corpus: 20,000 canonical requests — 75% with browser User-Agents, 25% with an API-client UA without ;; ~10%
attacks (SQLi, XSS, traversal, JNDI, cmd injection, SSTI, NoSQLi, scanner paths, SSRF), ~5% benign that look
suspicious (quotes, &, "select" in prose). Latency in µs per call; workers = GOMAXPROCS (-cpu 1,4,8).
| path | workers | calls/s | p50 | p95 | p99 | p99.9 |
|---|---|---|---|---|---|---|
MatchSigs (signatures) |
1 | 12,285 | 89.3 | 171.2 | 201.8 | 264.1 |
| 4 | 35,466 | 107.9 | 196.4 | 222.0 | 275.1 | |
| 8 | 43,067 | 161.1 | 201.7 | 226.6 | 11,833.8 | |
HTTPModel2.Logits (shipped http_v2) |
1 | 15,287 | 58.0 | 108.8 | 137.4 | 209.9 |
| 4 | 48,726 | 76.1 | 111.4 | 131.6 | 234.2 | |
| 8 | 58,184 | 94.4 | 112.9 | 148.2 | 8,958.5 | |
| the same logit with the site FiLM cached (checked equal) | 1 | 28,906 | 32.1 | 62.9 | 75.1 | 130.9 |
| 4 | 91,255 | 36.5 | 68.3 | 76.9 | 130.8 | |
| 8 | 101,151 | 56.5 | 71.9 | 83.0 | 1,010.9 | |
request path: MatchSigs, then the model if allowed |
1 | 6,638 | 151.8 | 298.2 | 351.3 | 442.4 |
| 4 | 21,473 | 169.0 | 307.0 | 341.8 | 436.4 | |
| 8 | 29,263 | 255.3 | 315.6 | 401.2 | 18,629.2 | |
SeqNet.Step (shipped seq_v1: d = 64, 2 layers) |
1 | 26,355 | 33.7 | 55.9 | 63.5 | 140.1 |
| 4 | 79,092 | 55.3 | 62.7 | 66.3 | 107.1 | |
| 8 | 101,357 | 56.5 | 62.9 | 66.8 | 563.9 | |
tier 0+1 per flow: EntityState.Update + the monotone trees |
1 | 133,605 | 7.0 | 11.4 | 14.1 | 32.7 |
| 4 | 463,238 | 7.1 | 12.7 | 18.5 | 78.8 | |
| 8 | 587,136 | 7.1 | 12.1 | 16.6 | 77.3 |
Signatures by outcome (1 worker): benign requests that pass the prefilter (23.8% of the benign — the API-client UA)
p50 / p99 = 2.8 / 7.4 µs; benign requests that reach the regexes 96.3 / 208.4 µs; attacks 61.0 / 172.5 µs (B11).
Attack requests are cheaper than benign ones on the request path (72 vs 156 µs p50): a signature hit skips the
model. With 8 workers on 4 physical cores the p99.9 is scheduler noise (other processes were running), not code.
In the tier 0+1 benchmark each worker owns one EntityState shard (max 500k sources; 315k distinct sources in the
1M-flow pool), as a deployment would shard by source.
Existing package benchmarks (go test -bench . -benchmem; test fixtures = small random models: HTTP E = 16 / H = 24,
SeqNet d = 24 — not the shipped sizes): Decide benign 24.5 µs, Persian 27.7 µs, SQLi 89.8 µs, Log4Shell 96.4 µs,
blocked source 20 ns; ObliviousRaw 1.8 µs; HTTPLogits 8.1 µs; EntityUpdate 312 ns; SeqNetStep 8.8 µs;
HTTP2Logits 10.0 µs; MatchSigs 2.1 µs (a UA without ;); MatchSigsAttack 41.6 µs. With the shipped models the model
steps are 3.7x (SeqNet) and 5.8x (HTTP2) slower than the fixtures suggest, and DESIGN.md's per-layer timings
(8.6 µs GRU at d = 24, 8.9 µs HTTP) are fixture-size numbers.
7. Behaviour under memory pressure — what holds and what does not
- Holds: RSS is bounded by the caps (2.8–3.4 GB at 500k sources, 5.3M sources seen); no crash or MemoryError in any run; the alarm log (50k), the re-calibration buffer (400k rows), the rolling references (200k rows), the conformal lists (200k), the campaign tracker (50k keys x 20k) and the site-wide rate tables (200k keys) are all bounded; attackers active within the LRU horizon keep their state through a 2.4x-capacity flood; detection of the injected attackers did not change between 1M and 5M sources.
- Does not hold: the warm-source GRU table grows without bound with one-shot warm sources (B3); a record's size is not bounded by the cap (B9: up to 108 KB for a scanner, so 500k scanner records would need ~54 GB); beyond the cap the engine forgets returning sources (34–72% of returning flows found no record between 1M and 5M sources in the growing population) and re-treats them as first contact; alarm-heavy batches (floods, distributed scans) are 3–6x slower (B7), i.e. the engine is slowest exactly when it is attacked; re-calibration stalls every 30 minutes of stream time (B2).
8. Bottlenecks and bugs found
Line numbers: working tree of 2026-10-31 05:13 (+03:30). Not fixed here.
| id | where | what (measured) | cause | proposed fix |
|---|---|---|---|---|
| B1 | dibaguard/engine.py:259-290 Guard._remember (concatenate at 267, rng.choice at 272, per-source cap 279-290) |
~20–25 ms fixed per flows() call locally, 36% of flow time at batch 1000; batch 1 → 48 flows/s. The 05:13 version added a per-source cap that runs np.unique(b["src"].astype(str), ...) over the whole buffer on every call (187 ms alone at 400k rows): measured again on that tree, 49 ms per call at batch 1 (20 flows/s), 1.7k flows/s at batch 100, 9.3k at batch 1000 (95k-row buffer) |
every call concatenates the whole re-calibration buffer (≤ 400k rows x 9 columns, two of them object arrays), then draws choice(idx, 400_000, replace=False), sorts it and copies every column through a mask — and now also sorts its source column |
preallocated ring buffer per column with a write cursor; thin to the random subset, and apply the per-source cap, only when a re-calibration runs (or keep per-source counts incrementally) — makes the per-call cost O(batch) |
| B2 | engine.py:378-386 _maybe_recalibrate → _calibrate_refs; per-row Python loops in _calibrate_evidence (444) and _calibrate_dst_evidence (498) |
6.4 s locally, ≈ 13–17 s on Kaggle (p50 15.3 s over 13 stalls in run 1, 14.0 s over 4 in run 2), once per 1800 s of stream time, inside the caller's flows() call; it is the whole p99 / p99.9 of batch latency |
every reference is rebuilt synchronously from ≤ 480k rows; the evidence references replay SourceEvidence.update row by row |
build the new references off the hot path (a thread / process on a snapshot of the buffer, then swap); vectorise the evidence replay (sort by source, decayed cumulative sums in numpy) |
| B3 | engine.py:126 self.gru, _warm_mask (474-490), cold pop (577) |
300k one-shot scan sources → 300,173 entries, 299,999 still held 30 min later (Kaggle); 717 B each | an entry is removed only when its source sends warm_flows (64) more flows; a source that warms and never returns stays forever; max_entities does not apply |
make it an OrderedDict LRU capped at max_entities and drop entries idle longer than the warm horizon (e.g. at each re-calibration) |
| B4 | engine.py:796-797 http_requests per-host loop |
2000 requests over 1000 hosts: 908 ms vs 384–389 ms over 10–100 hosts (Kaggle) | for h in set(hosts_in): idx = [i for i in range(n) if hosts_in[i] == h] is O(requests x hosts), plus one model call per host |
group in one pass (np.unique(..., return_inverse=True) or a dict of index lists); embed the whole batch once, apply each host's profile (distance + FiLM) per group |
| B5 | engine.py:109 global_floor = 10.0, engine.py:111 global_scan_floor = 60.0, used at 677 and 877 |
at 300–600 flows/s, 96–97% of benign failed connects alarmed (tier 5); at 2000 req/s, 1.57% of benign requests challenged | absolute floors on decayed site-wide counts (tau 120 s): the scan tier fires above 0.5 failed connects/s site-wide (any site above ~100 flows/s with 0.5% failures), the HTTP tier above ~5 failures/min on one path. GlobalRate.calibrate() (decide.py:358) exists but is never used for the scan tier, and the HTTP tier's floor ignores its reference |
derive both floors from the site's own benign window (the per-minute peak quantile at the alpha budget, which GlobalRate.calibrate already computes), or scale them with the site's rate |
| B6 | dibaguard/shield/database.py:365-372 DbGuard._peer_volume |
0.44 / 3.3 / 26.8 ms per over-threshold event with 1k / 10k / 100k (db, client) keys tracked (Kaggle) | scans every tracked key of the volume table (up to 200k) for each event to find the db's known peers | iterate self.known_clients[db] (tens of app pods) and look each up; also cache the median in _vol_thr (353, 363), recomputed per event for clients without history |
| B7 | engine.py:730 np.isin(src, list(bad_src)); engine.py:352 (drift guard) np.isin(<buffer src>, list(self._anchor_src)) |
730: 5000-flow batches where every flow is a named alarm take 3.0 s (flood) / 3.8 s (distributed scan) vs 0.6–0.9 s (Kaggle); 352: one re-calibration took 1,047,733 ms (17.5 min) on Kaggle (run 3) | numpy's isin on object (string) arrays falls back to an O(n x m) loop: isin(5000, 5000) = 278 ms, isin(400k, 5000) = 21 s locally; the anchor holds ≈ 55–66k distinct sources and the buffer ≈ 480k rows |
set membership: S = set(...); np.fromiter((s in S for s in arr), bool, len(arr)) — 24 ms for 400k rows locally — or integer source ids (factorise once) |
| B8 | dibaguard/campaign.py:99-122 CampaignTracker._close; young spike path 160-164 |
growing population: tier 6 flagged ≈ 5% of all benign flows from 400k sources on (Kaggle runs 1–2, ~50k per 1800 s window); isolated trace with the 05:13 file: auth/3306 active with S = 655 after 1.5 h, 6.5% of flows flagged as members; stationary population: zero | windows are added to a fingerprint's history only while it is not active, and a young fingerprint fired by the spike path has no own history, so the CUSUM never runs for it: once active, a benign level shift (a growing audience, a diurnal ramp, a new client population) keeps it active forever and S grows without bound. (The worse variant — every busy fingerprint active from the first window after a short calibration — reproduced on the 04:01 file; the 04:32 file fixed it via _young_bar.) |
a recovery path: admit active windows to the history at reduced weight, or re-baseline after N active windows when the excess is broad (many sources, low concentration — the drift guard's broad-vs-concentrated test); cap S; deactivate a young-fired key when its count falls back under the bar |
| B9 | dibaguard/flows/entity.py:30 MEMORY = 512; module docstring lines 10-11 |
2,947 B per source on average for the whole engine (Kaggle fit) but 108 KB for a scanner's record (tracemalloc); EntityState alone at the 500k cap is 1.0–1.7 GB for ordinary sources (1.9–3.4 KB each), the whole engine plateaus at 2.8–3.4 GB RSS, vs "a few hundred MB at most" in the docstring; 34–72% of returning flows lost their record between 1M and 5M sources |
the LRU caps the number of records, not their size (each holds up to 512 (host, port) keys in an OrderedDict plus a hosts dict); state is Python objects (~2 KB of overhead per record) |
a global key budget, or a small per-source key memory (e.g. 64 hashed keys in a fixed array) with a distinct-count sketch for the new-port / new-host shares; size max_entities to RAM at ~3 KB per source; report the eviction rate as a metric |
| B10 | go/dibaguard/http2.go:152-162 HTTPModel2.Logits |
58.0 → 32.1 µs p50 per request (single core) when the FiLM is computed once (outputs checked equal); 45% of the model's per-request cost | the site FiLM (tanh(ctx · Wc + bc), H x 2H = 32,768 multiply-adds and 256 tanh with the shipped H = 128) depends only on the site profile but is recomputed for every request |
compute gamma / beta when the Site is set (LoadHTTP2, SiteProfile) and keep them on Site |
| B11 | go/dibaguard/sigs.go:50 triggerBytes (prefilter in MatchSigs, 63-78) |
benign requests with a browser User-Agent: 96.3 µs p50 instead of 2.8 µs; only 23.8% of the benign corpus skipped the regexes | ; is a trigger byte and the canonical text includes the User-Agent, whose browser forms always contain ; ("Windows NT 10.0; Win64; x64", "iPhone; CPU …", "Linux; Android …"); the package's own BenchmarkMatchSigs uses a UA without ; |
prefilter per field: ;, ` |
| B12 | dibaguard/hashing.py ngram_buckets / bag_mean (via HttpModel.embed) |
per-request cost rises with batch size: 200 µs at 100, 255 µs at 1000, 417 µs at 5000 (Kaggle) | inferred, not profiled: the batch's n-gram index arrays and the gathered [n-grams x 48] embedding rows outgrow the cache |
chunk inside embed (~100–200 requests per chunk) |
Status after the fixes of 2026-10-31 (afternoon; the numbers above are the measurements that found them).
| id | status | what changed |
|---|---|---|
| B1 | fixed | Guard._remember queues the batch (O(batch)); the window cut, subsample and per-source cap run in _flush_buffer every 50k rows, before a re-calibration and before a snapshot |
| B2 | open | re-calibration still runs synchronously (6–17 s once per 30 min of stream time); move it off the hot path |
| B3 | fixed | the warm-source GRU table is an LRU capped at max_entities // 5 (100k at the default) |
| B4 | fixed | requests are grouped by Host in one pass |
| B5 | fixed | GlobalRate (scan tier and per-path HTTP tier) fires above floor + 4 x the key's own 1-hour baseline (seeded from the calibration window for the scan tier); while hot the baseline learns at 0.1 of the rate |
| B6 | fixed | _peer_volume looks up the database's known clients only |
| B7 | fixed | hash-set membership (engine._member) instead of np.isin on object arrays |
| B8 | fixed | the campaign CUSUM is capped at 2h; a fingerprint fired without its own history ends when its windows fall under half the young bar; one active window in four joins its own history, so a lasting benign level shift is adopted within hours |
| B9 | open (documented) | per-record size is still unbounded by key count; size max_entities to RAM (~3–3.6 KB per source) |
| B10 | open | the site FiLM is still computed per request in HTTPModel2.Logits (Go) |
| B11 | fixed | ;, ` |
| B12 | open | chunk HttpModel.embed for batches over ~200 requests |
Behaviours that are not bugs but matter for operators: after an exfiltration alarm the client's decayed volume keeps
every later query of it alarmed as exfil for ~19 minutes (§5); the HTTP site-wide tier challenges (never blocks).
9. Recommendations
- B7 first (one line each): the drift guard's
np.isinturns every re-calibration into a 17-minute stall on the current engine, and the per-batchisinmakes the engine 3–6x slower during floods and distributed scans. - B1 + B2: get the Python engine's per-call cost to O(batch). A ring buffer for the re-calibration rows and an off-the-hot-path re-calibration remove the ~20 ms per call and the 13–17 s stalls; small batches (10–100 flows, tens of milliseconds of traffic) then become viable, which is what bounds a flow's time-to-verdict.
- Bound everything per source: B3 and B9. Cap the GRU table, give records a fixed size, and size
max_entitiesto RAM with the measured ~3 KB per source (2M sources ≈ 6 GB + 0.4 GB; 500k ≈ 1.5 GB + references). Report the eviction rate: at 2M sources with a 500k cap, more than half of returning traffic is judged without its history. - Fix the two biggest benign-alarm sources before calibrating anything else: B8 (campaign lock-in) and B5 (fixed site-wide floors). On stationary traffic they leave ≈ 0.2% of benign source-minutes; with organic growth, B8 alone flags ~5% of all flows.
- Throughput planning: the Python engine does 6.6–9.7k flows/s per Kaggle core at batch 5000; 2 shards scale linearly. For more, put the Go tier 0+1 (134k flows/s per core, p99 14 µs) in front and send only warm / flagged sources to the Python engine — the cascade the design describes. Sharding by source needs a merged view for the destination-side tiers.
- Go: B10 and B11 together would bring the request path from 152 µs p50 to roughly 35 µs per core (estimate from the measured parts: 2.8 µs signatures + 32.1 µs model; not measured as a whole). Add a shipped-model, browser-UA case to the package benchmarks so the numbers in DESIGN.md track the real models.
- HTTP / DB: batch HTTP requests at ~100–200 per call (B12, B4 for multi-site layers); fix B6 before a database with thousands of clients meets its first dump.
10. Not measured
- The Go engine's memory per source and its behaviour at millions of sources (only per-call latency); the Go
EntityState(map +container/listLRU per table) was not run at scale. cmd/dibaguard-proxy/dibaguard-authzend to end (network, TLS, HTTP parsing, the sidecar round trip) — being refactored by another agent; only the library calls were timed.Decider.Decidewas timed only by the package's existing fixture benchmarks.- Queueing latency (time to fill a batch) in a live deployment; the Python engine fed by a real NetFlow / packet collector.
- Real traffic: every input here is synthetic, and benign alarm rates are those of these generators, not of a real site. Detection quality beyond the injected attackers (no labelled corpus was used).
- The Kaggle runs measured four different engine versions (§1.3); per-version speed differences between them were not isolated. GPU, ARM and any machine other than the two in §2; more than 4 processes.
- RSS cost of the GRU entries at scale (counted, not isolated in RSS); per-source eviction counts during the flood.
11. Reproduce
# Kaggle (free CPU kernel): the full scale run (~2.3 h), and the stationary-population variant
python kag.py run --job scale_bench --run-name guard-scale-1 --cpu --hours 3
python kag.py run --job scale_bench --run-name guard-scale-4 --cpu --hours 2.6 -- --phases capped --zipf-a 0.8 \
--pop 8000000 --max-sources 2000000 --inject-at 2000000 --reserve-h 0.1 --mark-every 250000
python kag.py status --run-name guard-scale-1 --result # report: bench/<run>.json in the private results repo
# (the job pulls the shipped models from code/<run>/models/ in the results repo, else models/)
# local Python micro-benchmarks (< 1 GB RSS, ~6 min) -> D:/build-tmp/guard/bench/runtime.json
python scripts/bench_runtime.py
# Go (shipped flat models under $DIBAGUARD_MODELS, default D:/build-tmp/guard/card_repo/models)
cd go/dibaguard && go test -run XXX -bench Latency -cpu 1,4,8 -benchtime 200000x -v .
Raw results: D:/build-tmp/guard/bench/runtime.json (local Python), D:/build-tmp/guard/bench/guard-scale-{1,2,3,4}.json
(Kaggle), D:/build-tmp/guard/bench/go_latency.txt and go_existing.txt (Go).