README.md

metadata

title: US Flow Scanner
emoji: 📈
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.13.0
app_file: app.py
pinned: false
license: mit

US Stock Institutional Flow Scanner

A Gradio web app that scans the US common-stock universe for stocks showing institutional-style buying or selling pressure. Designed to run on Hugging Face Spaces (free CPU) or locally.

Disclaimer: this is a public-data proxy built from price/volume only. It is not actual 13F / block-trade / dark-pool data. Not financial advice.

What it does

For each of ~5000+ US-listed common stocks, the app computes 9 factors (5 daily flow proxies + 4 "real" institutional-flow signals) and combines them into a single -100 to +100 institutional-flow score:

Factor	What it measures
CMF(20)	Chaikin Money Flow over 20 days
OBV slope	20-day linear regression of On-Balance Volume, normalized
Big-bar ratio	Unusual-volume + wide-range bars today vs 20-day baseline
VWAP dev	Close vs 20-day volume-weighted average price
RVOL signed	Relative volume, sign-flipped by CMF direction
L2 imbalance	Top-of-book depth + large resting orders (with spoofing discount)
Unusual options	Vol/OI z-score across 20 days, weighted by moneyness
Block aggression	Net buy/sell bias in ≥10k-share trades (Lee-Ready tick rule)
Buy persistence	Buy-ratio average over last 1 hour of 5-min bars

Factors are z-scored cross-sectionally (median + MAD robust normalization) so the most-extreme name in the universe gets ±100.

The last 4 factors come from a pluggable FactorDataSource (see Data sources below). By default they are computed from synthetic stub data committed to the repo, so the app works on Hugging Face Spaces with no live market feed.

Rating thresholds (tweakable in the UI):

• ≥ +50 → Strong Buy
• +20 … +49 → Buy
• -19 … +19 → Neutral
• -49 … -20 → Sell
• ≤ -50 → Strong Sell

The default weights are 0.20 / 0.15 / 0.10 / 0.10 / 0.05 for the daily flow factors and 0.15 / 0.10 / 0.10 / 0.05 for the institutional factors. The 5 daily sliders are exposed in the sidebar accordion; the 4 institutional factors always run at their default weights.

Project layout

us-flow-scanner/
├── app.py                       # Gradio UI
├── scanner/
│   ├── __init__.py
│   ├── universe.py              # Load & filter ~5000 US tickers
│   ├── data_fetcher.py          # yfinance + Polygon fallback + cache
│   ├── flow_algo.py             # 5 daily-flow factors
│   ├── l2_factor.py             # Level-2 large-resting-order factor
│   ├── options_factor.py        # Unusual options activity
│   ├── tick_factor.py           # Lee-Ready tick + size buckets
│   ├── intraday_factor.py       # Intraday VWAP + buy persistence
│   ├── factor_sources.py        # StubDataSource (default) + FutuDataSource (live)
│   ├── scorer.py                # Cross-sectional z-score + composite
│   ├── persistence.py           # Optional HF Dataset round-trip
│   ├── history.py               # Per-scan snapshots + delta vs prior
│   ├── watchlist.py             # User watchlist (parse + persist)
│   ├── performance.py           # IC evaluation + self-tuning weights
│   └── paths.py                 # Centralised, env-var-overridable paths
├── data/
│   ├── us_tickers.csv           # ~5260 US common stocks (committed)
│   └── stubs/                   # 30-ticker synthetic L2/options/ticks/intraday
├── tests/                       # pytest suite (isolated disk paths)
├── requirements.txt
├── requirements-dev.txt
├── README.md
└── .gitignore

Run locally

pip install -r requirements.txt
python app.py

Then open http://localhost:7860.

Data sources

The four institutional-flow factors (l2_imbalance, unusual_options, block_aggression, buy_persistence) read from a pluggable FactorDataSource (scanner/factor_sources.py).

Source	When to use	Config env var
StubDataSource	Default. Reads data/stubs/ synthetic data	FSCANNER_DATA_SOURCE unset or =stub
FutuDataSource	Live Level-2 / options / tick data via local Futu OpenD	FSCANNER_DATA_SOURCE=futu + FUTU_OPEND_HOST + FUTU_OPEND_PORT

The Space uses StubDataSource so it never depends on a live feed. To regenerate the stub data:

python data/stubs/_build_stubs.py

Tests

pip install -r requirements-dev.txt
pytest -q

The suite isolates every persistent path via an autouse fixture, so it runs in a sandboxed tmp_path and never touches your real cache or watchlist. Tests cover factor math, scoring, history snapshots, watchlist parsing, the sector cache, and the auto-tuning loop (the optimizer reliably discovers which factor is the true signal in a synthetic dataset).

Deploy to Hugging Face Spaces

One-time setup

pip install "huggingface_hub[cli]"
huggingface-cli login

# Create a new Space (Gradio SDK)
huggingface-cli repo create us-flow-scanner --type space --space-sdk gradio
cd ..
git clone https://huggingface.co/spaces/<your-username>/us-flow-scanner
cp -r us-flow-scanner/* us-flow-scanner/.gitignore us-flow-scanner/
cd us-flow-scanner
git add .
git commit -m "Initial commit"
git push

The Space will install dependencies and start the app. The public URL is https://huggingface.co/spaces/<your-username>/us-flow-scanner.

Optional: persistent cache

The OHLCV cache is stored under /tmp, which is wiped on container restart. For persistence across restarts:

1. Create a private HF Dataset, e.g. us-flow-cache.
2. In your Space's Settings → Variables and secrets, add:
   • FSCANNER_CACHE_REPO = <your-username>/us-flow-cache
   • HF_TOKEN = (a write-enabled token, also added to your dataset repo)
3. The app will push the cache after each scan and pull it on startup.

Optional: Polygon fallback

If you hit yfinance rate limits, set:

• POLYGON_API_KEY = (your polygon.io API key)

in Space secrets. Tickers that fail on yfinance are retried with Polygon.

Performance & limitations

• Full universe scan of ~5000 liquid names: ~3–6 min on a free HF Space CPU. Use Filtered mode (default) to keep it under ~2 min.
• Filtered default: price ≥ $5 and 20-day ADV ≥ $5M, which trims the universe to ~1500–3000 liquid names.
• yfinance is an unofficial scraper and may be flaky. The app retries with backoff and falls back to Polygon (if configured) for individual tickers.
• The algorithm is a proxy for institutional flow based on public OHLCV. It does not see 13F filings, dark-pool prints, or order-book imbalance.
• Free HF Spaces may sleep after ~48 h of inactivity; the app restarts on the next visit.

Self-tuning weights

After every scan, the app:

1. Persists the full results as a timestamped snapshot in history/.
2. Computes the realised forward return for every ticker over the next HORIZON_DAYS (default 5) trading days using the OHLCV cache.
3. Aggregates a Spearman IC per snapshot for the current weights.
4. Runs a small Dirichlet random search (200 candidates + 5 structured seeds) over alternative weight vectors and picks the one with the highest mean IC.
5. If the new vector beats the baseline IC by ≥ 0.005, it is persisted to learned.json and a row is appended to perf.parquet; otherwise the default weights are kept.
6. The "Use auto-tuned weights" checkbox in the Performance tab tells the next scan to use the learned vector instead of the slider values, so you can A/B test the improvement.

Tuning is intentionally lightweight (seconds, not minutes) so it can run in a background thread after every scan. It needs at least 5 snapshots with valid forward data and at least 30 tickers per snapshot to fire.

Tickers list

data/us_tickers.csv contains ~5260 tickers filtered from Nasdaq Trader's official daily files (nasdaqlisted.txt + otherlisted.txt):

• ETFs, warrants, units, rights, preferreds, structured products excluded
• Test issues and distressed financial-status tickers excluded
• Tickers must be ≤5 chars, no ./-/space (i.e. no class-B or dual-class weirdness)

To refresh the list:

# From the data/ directory
Invoke-WebRequest -Uri "https://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt" -OutFile nasdaqlisted_raw.txt
Invoke-WebRequest -Uri "https://www.nasdaqtrader.com/dynamic/SymDir/otherlisted.txt" -OutFile otherlisted_raw.txt
# ... (use the same PowerShell filter as in earlier dev work)

License

MIT.