docs/DATA_SOURCES.md

# Data Sources

All datasets are **free and publicly accessible**.

## Primary Datasets

### 1. Binance Public Data ⭐⭐⭐⭐⭐
- **What**: Free historical trade + order-book data for every Binance-listed symbol
- **Granularity**: Trade-by-trade CSVs, daily/monthly aggregates, depth snapshots
- **Real-time**: Free WebSocket (`wss://stream.binance.com:9443`)
- **Coverage**: All Binance symbols, 2017–present
- **Cost**: Free, no auth for public data
- **URLs**:
  - Historical: https://data.binance.vision
  - Live: `wss://stream.binance.com:9443/ws`
  - Docs: https://github.com/binance/binance-public-data
- **Use for**: 80% of the demo. Live stream + historical crash windows (May 2021 BTC, May 2022 LUNA)

### 2. FI-2010 Benchmark ⭐⭐⭐⭐⭐
- **What**: First publicly available HFT LOB benchmark
- **Granularity**: ~10ms ticks, 10 levels of depth
- **Coverage**: 5 Finnish stocks, 10 days
- **Labels**: Pre-annotated mid-price movement (up/down/stationary, 3 horizons)
- **Cost**: Free
- **URL**: https://etsin.fairdata.fi/dataset/73eb48d7-4dbc-4a10-a52a-da745b47a649
- **Paper**: Ntakaris et al. (2018), Journal of Forecasting
- **Use for**: Academic benchmark, training, comparing to published baselines

### 3. LOBSTER ⭐⭐⭐⭐
- **What**: Reconstructed NASDAQ TotalView-ITCH limit order books
- **Granularity**: Nanosecond, every order event
- **Coverage**: Any NASDAQ stock, 2007–present
- **Cost**: Free for academic use (university email); paid for commercial
- **URL**: https://lobsterdata.com
- **Use for**: Equity-market credibility, May 6, 2010 flash crash reconstruction

### 4. NASDAQ TotalView-ITCH Raw ⭐⭐⭐⭐
- **What**: Raw NASDAQ direct-feed protocol
- **Spec**: Public PDF (nasdaqtrader.com, NQTVITCH 5.0)
- **Parsers**: Open-source (Python, Julia, Rust)
- **Sample**: Free at https://data.nasdaq.com/databases/NTV
- **Cost**: Full historical is paid; samples free
- **Use for**: Low-latency parsing demo

### 5. Tardis.dev ⭐⭐⭐
- **What**: Commercial crypto tick data (L2/L3 snapshots)
- **Coverage**: All major crypto exchanges
- **Cost**: Paid (limited free tier)
- **URL**: https://tardis.dev
- **Use for**: LUNA crash L3 data

## Labeled Crash Windows

### May 6, 2010 — US Equities Flash Crash
- Dow dropped 998.5 points (9.2%) in 36 minutes
- $1 trillion wiped, recovered in 20 minutes
- Source: LOBSTER reconstruction
- SEC/CFTC report: https://www.sec.gov/files/marketevents-report.pdf

### May 19, 2021 — Bitcoin Flash Crash
- BTC -30% in hours
- $8B liquidations
- Binance outage during crash
- Source: Binance public data (data.binance.vision)

### May 2022 — LUNA / UST Death Spiral
- LUNA -99.9% in 48 hours
- $40B wiped
- UST stablecoin depeg
- Source: Binance public data + Tardis.dev

## Data Pipeline

```
Ingest → Normalize → Features → Detect → Alert
  │         │          │          │        │
  │         │          │          │        └── Slack/PagerDuty
  │         │          │          └── 5-stage cascade (27ms p99)
  │         │          └── Flink (20 features, 2ms)
  │         └── Rust proxy (sub-ms)
  └── Binance WebSocket / CSVs
```