File size: 3,194 Bytes
2bbc43c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | # Data Sources
All datasets are **free and publicly accessible**.
## Primary Datasets
### 1. Binance Public Data ⭐⭐⭐⭐⭐
- **What**: Free historical trade + order-book data for every Binance-listed symbol
- **Granularity**: Trade-by-trade CSVs, daily/monthly aggregates, depth snapshots
- **Real-time**: Free WebSocket (`wss://stream.binance.com:9443`)
- **Coverage**: All Binance symbols, 2017–present
- **Cost**: Free, no auth for public data
- **URLs**:
- Historical: https://data.binance.vision
- Live: `wss://stream.binance.com:9443/ws`
- Docs: https://github.com/binance/binance-public-data
- **Use for**: 80% of the demo. Live stream + historical crash windows (May 2021 BTC, May 2022 LUNA)
### 2. FI-2010 Benchmark ⭐⭐⭐⭐⭐
- **What**: First publicly available HFT LOB benchmark
- **Granularity**: ~10ms ticks, 10 levels of depth
- **Coverage**: 5 Finnish stocks, 10 days
- **Labels**: Pre-annotated mid-price movement (up/down/stationary, 3 horizons)
- **Cost**: Free
- **URL**: https://etsin.fairdata.fi/dataset/73eb48d7-4dbc-4a10-a52a-da745b47a649
- **Paper**: Ntakaris et al. (2018), Journal of Forecasting
- **Use for**: Academic benchmark, training, comparing to published baselines
### 3. LOBSTER ⭐⭐⭐⭐
- **What**: Reconstructed NASDAQ TotalView-ITCH limit order books
- **Granularity**: Nanosecond, every order event
- **Coverage**: Any NASDAQ stock, 2007–present
- **Cost**: Free for academic use (university email); paid for commercial
- **URL**: https://lobsterdata.com
- **Use for**: Equity-market credibility, May 6, 2010 flash crash reconstruction
### 4. NASDAQ TotalView-ITCH Raw ⭐⭐⭐⭐
- **What**: Raw NASDAQ direct-feed protocol
- **Spec**: Public PDF (nasdaqtrader.com, NQTVITCH 5.0)
- **Parsers**: Open-source (Python, Julia, Rust)
- **Sample**: Free at https://data.nasdaq.com/databases/NTV
- **Cost**: Full historical is paid; samples free
- **Use for**: Low-latency parsing demo
### 5. Tardis.dev ⭐⭐⭐
- **What**: Commercial crypto tick data (L2/L3 snapshots)
- **Coverage**: All major crypto exchanges
- **Cost**: Paid (limited free tier)
- **URL**: https://tardis.dev
- **Use for**: LUNA crash L3 data
## Labeled Crash Windows
### May 6, 2010 — US Equities Flash Crash
- Dow dropped 998.5 points (9.2%) in 36 minutes
- $1 trillion wiped, recovered in 20 minutes
- Source: LOBSTER reconstruction
- SEC/CFTC report: https://www.sec.gov/files/marketevents-report.pdf
### May 19, 2021 — Bitcoin Flash Crash
- BTC -30% in hours
- $8B liquidations
- Binance outage during crash
- Source: Binance public data (data.binance.vision)
### May 2022 — LUNA / UST Death Spiral
- LUNA -99.9% in 48 hours
- $40B wiped
- UST stablecoin depeg
- Source: Binance public data + Tardis.dev
## Data Pipeline
```
Ingest → Normalize → Features → Detect → Alert
│ │ │ │ │
│ │ │ │ └── Slack/PagerDuty
│ │ │ └── 5-stage cascade (27ms p99)
│ │ └── Flink (20 features, 2ms)
│ └── Rust proxy (sub-ms)
└── Binance WebSocket / CSVs
```
|