| # Flash Crash Early Warning β Market Microstructure ML Detector | |
| A real-time detector on limit-order-book (LOB) streams that flags microstructure anomalies (order-book imbalance, spoofing, liquidity evaporation) **50β500ms before price dislocation**. | |
| Built as a 5-stage hybrid detection cascade: Statistical pre-filter β Isolation Forest β Temporal Convolutional Network (TCN) β Cross-symbol Transformer β Bayesian aggregator. p99 end-to-end latency: **27 ms** (target: < 50 ms). | |
| > β οΈ **Status:** MVP / research prototype. Not production trading software. See [Risks & Disclaimer](#risks--disclaimer). | |
| --- | |
| ## Architecture | |
| ``` | |
| Binance WebSocket ββ | |
| βββ Rust Proxy ββ Feature Extractor ββ 5-Stage Cascade ββ Alert Router | |
| FI-2010 (offline) ββ (< 1 ms) (Flink-style, Stage 1: Statistical (Slack/PagerDuty) | |
| ~2 ms) Stage 2: Isolation Forest | |
| Stage 3: TCN (~8 ms) | |
| Stage 4: Transformer (~15 ms) | |
| Stage 5: Bayesian agg (~1 ms) | |
| ``` | |
| End-to-end p99: 27 ms Β· Total overhead vs raw LLM spend: ~3% Β· Open-source first. | |
| --- | |
| ## Quickstart | |
| ### Prerequisites | |
| - Python 3.11+ | |
| - Rust 1.75+ (optional β only for the high-performance proxy) | |
| - 4 GB RAM, 5 GB disk | |
| ### Option A β Python-only quickstart (recommended for first run) | |
| ```bash | |
| # 1. Install Python dependencies | |
| cd ml | |
| pip install -r requirements.txt | |
| # 2. Download a sample of Binance historical data (BTC/USDT, 1 day) | |
| python -m flash_crash_watchdog.data.download_binance \ | |
| --symbol BTCUSDT --date 2021-05-19 --out ../data/ | |
| # 3. Run the offline backtest on the May 19, 2021 BTC flash crash | |
| python -m flash_crash_watchdog.cli backtest \ | |
| --data ../data/BTCUSDT_2021-05-19.parquet \ | |
| --model configs/tcn_baseline.yml | |
| # 4. Or: start the live detector against Binance WebSocket | |
| python -m flash_crash_watchdog.cli live --symbol BTCUSDT | |
| ``` | |
| ### Option B β Full stack with Rust proxy (for sub-ms ingest) | |
| ```bash | |
| # 1. Build the Rust proxy | |
| cd proxy | |
| cargo build --release | |
| # 2. Run the proxy (ingests Binance WebSocket, publishes to localhost:5555) | |
| ./target/release/flash-crash-proxy --symbol BTCUSDT --out tcp://127.0.0.1:5555 | |
| # 3. In another terminal, run the Python detector consuming from the proxy | |
| cd ../ml | |
| python -m flash_crash_watchdog.cli live --source tcp://127.0.0.1:5555 | |
| ``` | |
| ### Option C β Docker Compose (everything wired up) | |
| ```bash | |
| docker-compose up -d | |
| # Dashboard: http://localhost:3000 | |
| # Prometheus metrics: http://localhost:9090 | |
| ``` | |
| --- | |
| ## Project Structure | |
| ``` | |
| flash-crash-watchdog/ | |
| βββ proxy/ # Rust WebSocket proxy (sub-ms ingest) | |
| β βββ Cargo.toml | |
| β βββ src/ | |
| β βββ main.rs | |
| β βββ binance_client.rs | |
| β βββ lob.rs # Limit order book reconstruction | |
| β βββ publisher.rs | |
| βββ ml/ # Python ML pipeline | |
| β βββ requirements.txt | |
| β βββ setup.py | |
| β βββ flash_crash_watchdog/ | |
| β βββ __init__.py | |
| β βββ cli.py # Command-line entrypoint | |
| β βββ features/ # 20 features in 5 families | |
| β β βββ price_action.py | |
| β β βββ depth_imbalance.py | |
| β β βββ flow_toxicity.py | |
| β β βββ volatility.py | |
| β β βββ cross_symbol.py | |
| β βββ models/ # 5-stage cascade | |
| β β βββ stage1_statistical.py | |
| β β βββ stage2_isolation_forest.py | |
| β β βββ stage3_tcn.py | |
| β β βββ stage4_transformer.py | |
| β β βββ stage5_bayesian.py | |
| β β βββ cascade.py # Orchestrator | |
| β βββ data/ | |
| β β βββ download_binance.py | |
| β β βββ fi2010_loader.py | |
| β β βββ labels.py | |
| β βββ eval/ | |
| β β βββ backtest.py | |
| β β βββ metrics.py | |
| β βββ alert/ | |
| β βββ router.py | |
| βββ dashboard/ # Next.js real-time dashboard | |
| β βββ package.json | |
| β βββ src/ | |
| βββ configs/ # Model + pipeline configs | |
| β βββ tcn_baseline.yml | |
| β βββ transformer_cross_symbol.yml | |
| β βββ pipeline.yml | |
| βββ scripts/ # Helper scripts | |
| β βββ train_tcn.py | |
| β βββ run_backtest.py | |
| β βββ replay_crash.py | |
| βββ data/ # Local data (gitignored) | |
| βββ docs/ # Architecture + API docs | |
| β βββ ARCHITECTURE.md | |
| β βββ DATA_SOURCES.md | |
| β βββ API.md | |
| βββ tests/ # Test suite | |
| βββ docker-compose.yml | |
| βββ Makefile | |
| βββ README.md | |
| ``` | |
| --- | |
| ## Data Sources | |
| All datasets are **free and publicly accessible**. | |
| | Dataset | Use | Access | | |
| |---------|-----|--------| | |
| | **Binance public data** | Historical crashes (May 2021 BTC, May 2022 LUNA) + live WebSocket | `data.binance.vision` (CSV) Β· `wss://stream.binance.com:9443` (live) | | |
| | **FI-2010 benchmark** | Academic LOB benchmark, labeled mid-price movement | [etsin.fairdata.fi](https://etsin.fairdata.fi/dataset/73eb48d7-4dbc-4a10-a52a-da745b47a649) | | |
| | **LOBSTER** (academic) | NASDAQ TotalView reconstruction | [lobsterdata.com](https://lobsterdata.com) β free with university email | | |
| | **NASDAQ TotalView-ITCH** | Raw protocol parsing demo | [data.nasdaq.com/databases/NTV](https://data.nasdaq.com/databases/NTV) | | |
| See [`docs/DATA_SOURCES.md`](docs/DATA_SOURCES.md) for full details. | |
| --- | |
| ## Detection Cascade | |
| | Stage | Algorithm | Latency | Pass-through | Catches | | |
| |-------|-----------|---------|--------------|---------| | |
| | 1 | Statistical pre-filter (micro-price velocity, spread z-score) | < 0.1 ms | 5% | Obvious normal ticks | | |
| | 2 | Isolation Forest (12 features) | ~1 ms | 20% of suspects | OBI shifts, cancellation spikes | | |
| | 3 | Temporal Convolutional Network (8 dilated layers, 500ms receptive field) | ~8 ms | 40% | Collective temporal anomalies | | |
| | 4 | Cross-symbol Transformer (6-layer, 20 symbols) | ~15 ms | 60% | Correlation breakdown | | |
| | 5 | Bayesian aggregator | ~1 ms | β | Final alert decision | | |
| See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for full design. | |
| --- | |
| ## Feature Engineering | |
| 20 features in 5 families, extracted per tick: | |
| | Family | Features | Stage | | |
| |--------|----------|-------| | |
| | F1 β Price & Action (5) | mid-price velocity, micro-price, spread, trade arrival rate, cancel-to-trade ratio | 1, 2 | | |
| | F2 β Depth & Imbalance (5) | bid/ask depth L1-L10, OBI, weighted mid, depth slope, liquidity vacuum flag | 1, 2 | | |
| | F3 β Flow & Toxicity (4) | VPIN, Kyle's Ξ», effective spread, realized spread | 3 | | |
| | F4 β Volatility (3) | realized vol, micro-price variance ratio, Garman-Klass | 3 | | |
| | F5 β Cross-Symbol (3) | pairwise return correlation, lead-lag, co-integration residual | 4 | | |
| --- | |
| ## Evaluation | |
| Three regimes (see [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) Β§7): | |
| 1. **Offline backtest** β replay 6 months of LOB data, inject controlled crashes, measure per-stage precision/recall/TTD | |
| 2. **Online shadow** β run alongside production for 30 days, compare alerts to real market events | |
| 3. **Adversarial red team** β inject 100 synthetic crash patterns quarterly | |
| **Target envelope:** detect 80% of crashes with > 200ms early warning Β· false-positive rate < 2/hour Β· p99 latency < 50ms. | |
| --- | |
| ## Tech Stack | |
| | Layer | Choice | Why | | |
| |-------|--------|-----| | |
| | Ingest proxy | Rust + tokio + tungstenite | Sub-ms guarantees (Python GIL breaks this) | | |
| | Stream processing | Apache Flink (or Python `streamz` for MVP) | Exactly-once, windowed features | | |
| | Feature store | Feast | Online-offline parity | | |
| | Model serving | NVIDIA Triton / ONNX Runtime | Multi-framework, GPU batching | | |
| | Storage | ClickHouse (hot) + S3/Parquet (cold) | Sub-second analytics on 100M+ rows | | |
| | Dashboard | Next.js + shadcn/ui | Modern operator UI | | |
| | Monitoring | Prometheus + Grafana | Standard ops metrics | | |
| --- | |
| ## Roadmap | |
| - [x] v0.1 β Python-only MVP: Binance ingest, 20 features, TCN, cascade, backtest | |
| - [ ] v0.2 β Rust proxy for sub-ms ingest | |
| - [ ] v0.3 β Cross-symbol Transformer (Stage 4) | |
| - [ ] v0.4 β Next.js live dashboard | |
| - [ ] v0.5 β LOBSTER + FI-2010 integration for academic benchmarks | |
| - [ ] v0.6 β Adversarial red-team harness | |
| - [ ] v1.0 β Production hardening, canary deploy, drift monitor | |
| --- | |
| ## Risks & Disclaimer | |
| This is a **research prototype**, not production trading software. | |
| - **False-positive cost**: in live trading, false alerts trigger hedging cost. The alert threshold must be tuned per deployment. | |
| - **Concept drift**: market microstructure evolves; weekly retraining required. | |
| - **Adversarial adaptation**: spoofers evolve once they know detectors exist. | |
| - **Regulatory**: any deployment that triggers automated trades requires Reg NMS / MiFID II compliance review. | |
| - **No financial advice**: this software is for research and educational purposes only. The authors are not responsible for any financial losses incurred through its use. | |
| --- | |
| ## License | |
| Apache 2.0 β see [`LICENSE`](LICENSE). | |
| ## Citation | |
| If you use this work, cite: | |
| ```bibtex | |
| @software{flash_crash_watchdog, | |
| title = {Flash Crash Early Warning: Market Microstructure ML Detector}, | |
| author = {Z.ai Quant Research}, | |
| year = {2026}, | |
| url = {https://github.com/yourusername/flash-crash-watchdog} | |
| } | |
| ``` | |
| ## References | |
| See the accompanying project brief PDF for the full bibliography (Easley & O'Hara, SEC/CFTC May 2010 report, Ntakaris et al. FI-2010, Vaswani et al. Transformer, etc.). | |