| --- |
| title: US Flow Scanner |
| emoji: "📈" |
| colorFrom: blue |
| colorTo: green |
| sdk: gradio |
| sdk_version: 5.13.0 |
| app_file: app.py |
| pinned: false |
| license: mit |
| --- |
| |
| # US Stock Institutional Flow Scanner |
|
|
| A Gradio web app that scans the US common-stock universe for stocks showing |
| institutional-style buying or selling pressure. Designed to run on |
| **Hugging Face Spaces** (free CPU) or locally. |
|
|
| > **Disclaimer:** this is a public-data proxy built from price/volume only. |
| > It is *not* actual 13F / block-trade / dark-pool data. Not financial advice. |
|
|
| ## What it does |
|
|
| For each of ~5000+ US-listed common stocks, the app computes **9 factors** |
| (5 daily flow proxies + 4 "real" institutional-flow signals) and combines |
| them into a single **-100 to +100 institutional-flow score**: |
|
|
| | Factor | What it measures | |
| |----------------------|---------------------------------------------------------------| |
| | **CMF(20)** | Chaikin Money Flow over 20 days | |
| | **OBV slope** | 20-day linear regression of On-Balance Volume, normalized | |
| | **Big-bar ratio** | Unusual-volume + wide-range bars today vs 20-day baseline | |
| | **VWAP dev** | Close vs 20-day volume-weighted average price | |
| | **RVOL signed** | Relative volume, sign-flipped by CMF direction | |
| | **L2 imbalance** | Top-of-book depth + large resting orders (with spoofing discount) | |
| | **Unusual options** | Vol/OI z-score across 20 days, weighted by moneyness | |
| | **Block aggression** | Net buy/sell bias in ≥10k-share trades (Lee-Ready tick rule) | |
| | **Buy persistence** | Buy-ratio average over last 1 hour of 5-min bars | |
|
|
| Factors are z-scored cross-sectionally (median + MAD robust normalization) |
| so the most-extreme name in the universe gets ±100. |
|
|
| The last 4 factors come from a pluggable `FactorDataSource` (see |
| [Data sources](#data-sources) below). By default they are computed from |
| synthetic stub data committed to the repo, so the app works on |
| Hugging Face Spaces with **no live market feed**. |
|
|
| **Rating thresholds** (tweakable in the UI): |
| - `≥ +50` → **Strong Buy** |
| - `+20 … +49` → **Buy** |
| - `-19 … +19` → **Neutral** |
| - `-49 … -20` → **Sell** |
| - `≤ -50` → **Strong Sell** |
|
|
| The default weights are |
| `0.20 / 0.15 / 0.10 / 0.10 / 0.05` for the daily flow factors and |
| `0.15 / 0.10 / 0.10 / 0.05` for the institutional factors. |
| The 5 daily sliders are exposed in the sidebar accordion; the 4 |
| institutional factors always run at their default weights. |
|
|
| ## Project layout |
|
|
| ``` |
| us-flow-scanner/ |
| ├── app.py # Gradio UI |
| ├── scanner/ |
| │ ├── __init__.py |
| │ ├── universe.py # Load & filter ~5000 US tickers |
| │ ├── data_fetcher.py # yfinance + Polygon fallback + cache |
| │ ├── flow_algo.py # 5 daily-flow factors |
| │ ├── l2_factor.py # Level-2 large-resting-order factor |
| │ ├── options_factor.py # Unusual options activity |
| │ ├── tick_factor.py # Lee-Ready tick + size buckets |
| │ ├── intraday_factor.py # Intraday VWAP + buy persistence |
| │ ├── factor_sources.py # StubDataSource (default) + FutuDataSource (live) |
| │ ├── scorer.py # Cross-sectional z-score + composite |
| │ ├── persistence.py # Optional HF Dataset round-trip |
| │ ├── history.py # Per-scan snapshots + delta vs prior |
| │ ├── watchlist.py # User watchlist (parse + persist) |
| │ ├── performance.py # IC evaluation + self-tuning weights |
| │ └── paths.py # Centralised, env-var-overridable paths |
| ├── data/ |
| │ ├── us_tickers.csv # ~5260 US common stocks (committed) |
| │ └── stubs/ # 30-ticker synthetic L2/options/ticks/intraday |
| ├── tests/ # pytest suite (isolated disk paths) |
| ├── requirements.txt |
| ├── requirements-dev.txt |
| ├── README.md |
| └── .gitignore |
| ``` |
|
|
| ## Run locally |
|
|
| ```bash |
| pip install -r requirements.txt |
| python app.py |
| ``` |
|
|
| Then open http://localhost:7860. |
|
|
| ## Data sources |
|
|
| The four institutional-flow factors (`l2_imbalance`, `unusual_options`, |
| `block_aggression`, `buy_persistence`) read from a pluggable |
| `FactorDataSource` (`scanner/factor_sources.py`). |
|
|
| | Source | When to use | Config env var | |
| |---------------------|----------------------------------------------------------|----------------------------| |
| | `StubDataSource` | **Default.** Reads `data/stubs/` synthetic data | `FSCANNER_DATA_SOURCE` unset or `=stub` | |
| | `FutuDataSource` | Live Level-2 / options / tick data via local Futu OpenD | `FSCANNER_DATA_SOURCE=futu` + `FUTU_OPEND_HOST` + `FUTU_OPEND_PORT` | |
|
|
| The Space uses `StubDataSource` so it never depends on a live feed. |
| To regenerate the stub data: |
|
|
| ```bash |
| python data/stubs/_build_stubs.py |
| ``` |
|
|
| ## Tests |
|
|
| ```bash |
| pip install -r requirements-dev.txt |
| pytest -q |
| ``` |
|
|
| The suite isolates every persistent path via an autouse fixture, so it |
| runs in a sandboxed `tmp_path` and never touches your real cache or |
| watchlist. Tests cover factor math, scoring, history snapshots, |
| watchlist parsing, the sector cache, and the auto-tuning loop (the |
| optimizer reliably discovers which factor is the true signal in a |
| synthetic dataset). |
|
|
| ## Deploy to Hugging Face Spaces |
|
|
| ### One-time setup |
|
|
| ```bash |
| pip install "huggingface_hub[cli]" |
| huggingface-cli login |
| |
| # Create a new Space (Gradio SDK) |
| huggingface-cli repo create us-flow-scanner --type space --space-sdk gradio |
| cd .. |
| git clone https://huggingface.co/spaces/<your-username>/us-flow-scanner |
| cp -r us-flow-scanner/* us-flow-scanner/.gitignore us-flow-scanner/ |
| cd us-flow-scanner |
| git add . |
| git commit -m "Initial commit" |
| git push |
| ``` |
|
|
| The Space will install dependencies and start the app. The public URL is |
| `https://huggingface.co/spaces/<your-username>/us-flow-scanner`. |
|
|
| ### Optional: persistent cache |
|
|
| The OHLCV cache is stored under `/tmp`, which is wiped on container restart. |
| For persistence across restarts: |
|
|
| 1. Create a private HF Dataset, e.g. `us-flow-cache`. |
| 2. In your Space's **Settings → Variables and secrets**, add: |
| - `FSCANNER_CACHE_REPO` = `<your-username>/us-flow-cache` |
| - `HF_TOKEN` = (a write-enabled token, also added to your dataset repo) |
| 3. The app will push the cache after each scan and pull it on startup. |
|
|
| ### Optional: Polygon fallback |
|
|
| If you hit yfinance rate limits, set: |
| - `POLYGON_API_KEY` = (your polygon.io API key) |
|
|
| in Space secrets. Tickers that fail on yfinance are retried with Polygon. |
|
|
| ## Performance & limitations |
|
|
| - **Full universe scan** of ~5000 liquid names: ~3–6 min on a free HF Space |
| CPU. Use **Filtered** mode (default) to keep it under ~2 min. |
| - **Filtered** default: `price ≥ $5` and `20-day ADV ≥ $5M`, which trims |
| the universe to ~1500–3000 liquid names. |
| - yfinance is an unofficial scraper and may be flaky. The app retries with |
| backoff and falls back to Polygon (if configured) for individual tickers. |
| - The algorithm is a **proxy** for institutional flow based on public |
| OHLCV. It does not see 13F filings, dark-pool prints, or order-book |
| imbalance. |
| - Free HF Spaces may sleep after ~48 h of inactivity; the app restarts on |
| the next visit. |
|
|
| ## Self-tuning weights |
|
|
| After every scan, the app: |
| 1. Persists the full results as a timestamped snapshot in `history/`. |
| 2. Computes the realised forward return for every ticker over the next |
| `HORIZON_DAYS` (default 5) trading days using the OHLCV cache. |
| 3. Aggregates a **Spearman IC** per snapshot for the current weights. |
| 4. Runs a small Dirichlet random search (200 candidates + 5 structured |
| seeds) over alternative weight vectors and picks the one with the |
| highest mean IC. |
| 5. If the new vector beats the baseline IC by ≥ 0.005, it is persisted |
| to `learned.json` and a row is appended to `perf.parquet`; otherwise |
| the default weights are kept. |
| 6. The "Use auto-tuned weights" checkbox in the Performance tab tells |
| the next scan to use the learned vector instead of the slider |
| values, so you can A/B test the improvement. |
|
|
| Tuning is intentionally lightweight (seconds, not minutes) so it can |
| run in a background thread after every scan. It needs at least 5 |
| snapshots with valid forward data and at least 30 tickers per snapshot |
| to fire. |
|
|
| ## Tickers list |
|
|
| `data/us_tickers.csv` contains ~5260 tickers filtered from Nasdaq Trader's |
| official daily files (`nasdaqlisted.txt` + `otherlisted.txt`): |
| - ETFs, warrants, units, rights, preferreds, structured products excluded |
| - Test issues and distressed financial-status tickers excluded |
| - Tickers must be ≤5 chars, no `.`/`-`/space (i.e. no class-B or dual-class |
| weirdness) |
|
|
| To refresh the list: |
|
|
| ```powershell |
| # From the data/ directory |
| Invoke-WebRequest -Uri "https://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt" -OutFile nasdaqlisted_raw.txt |
| Invoke-WebRequest -Uri "https://www.nasdaqtrader.com/dynamic/SymDir/otherlisted.txt" -OutFile otherlisted_raw.txt |
| # ... (use the same PowerShell filter as in earlier dev work) |
| ``` |
|
|
| ## License |
|
|
| MIT. |
|
|
