---
title: US Flow Scanner
emoji: "📈"
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.13.0
app_file: app.py
pinned: false
license: mit
---

# US Stock Institutional Flow Scanner

A Gradio web app that scans the US common-stock universe for stocks showing
institutional-style buying or selling pressure.  Designed to run on
**Hugging Face Spaces** (free CPU) or locally.

> **Disclaimer:** this is a public-data proxy built from price/volume only.
> It is *not* actual 13F / block-trade / dark-pool data.  Not financial advice.

## What it does

For each of ~5000+ US-listed common stocks, the app computes **9 factors**
(5 daily flow proxies + 4 "real" institutional-flow signals) and combines
them into a single **-100 to +100 institutional-flow score**:

| Factor               | What it measures                                              |
|----------------------|---------------------------------------------------------------|
| **CMF(20)**          | Chaikin Money Flow over 20 days                               |
| **OBV slope**        | 20-day linear regression of On-Balance Volume, normalized     |
| **Big-bar ratio**    | Unusual-volume + wide-range bars today vs 20-day baseline     |
| **VWAP dev**         | Close vs 20-day volume-weighted average price                 |
| **RVOL signed**      | Relative volume, sign-flipped by CMF direction                |
| **L2 imbalance**     | Top-of-book depth + large resting orders (with spoofing discount) |
| **Unusual options**  | Vol/OI z-score across 20 days, weighted by moneyness          |
| **Block aggression** | Net buy/sell bias in ≥10k-share trades (Lee-Ready tick rule)  |
| **Buy persistence**  | Buy-ratio average over last 1 hour of 5-min bars              |

Factors are z-scored cross-sectionally (median + MAD robust normalization)
so the most-extreme name in the universe gets ±100.

The last 4 factors come from a pluggable `FactorDataSource` (see
[Data sources](#data-sources) below).  By default they are computed from
synthetic stub data committed to the repo, so the app works on
Hugging Face Spaces with **no live market feed**.

**Rating thresholds** (tweakable in the UI):
- `≥ +50` → **Strong Buy**
- `+20 … +49` → **Buy**
- `-19 … +19` → **Neutral**
- `-49 … -20` → **Sell**
- `≤ -50` → **Strong Sell**

The default weights are
`0.20 / 0.15 / 0.10 / 0.10 / 0.05` for the daily flow factors and
`0.15 / 0.10 / 0.10 / 0.05` for the institutional factors.
The 5 daily sliders are exposed in the sidebar accordion; the 4
institutional factors always run at their default weights.

## Project layout

```
us-flow-scanner/
├── app.py                       # Gradio UI
├── scanner/
│   ├── __init__.py
│   ├── universe.py              # Load & filter ~5000 US tickers
│   ├── data_fetcher.py          # yfinance + Polygon fallback + cache
│   ├── flow_algo.py             # 5 daily-flow factors
│   ├── l2_factor.py             # Level-2 large-resting-order factor
│   ├── options_factor.py        # Unusual options activity
│   ├── tick_factor.py           # Lee-Ready tick + size buckets
│   ├── intraday_factor.py       # Intraday VWAP + buy persistence
│   ├── factor_sources.py        # StubDataSource (default) + FutuDataSource (live)
│   ├── scorer.py                # Cross-sectional z-score + composite
│   ├── persistence.py           # Optional HF Dataset round-trip
│   ├── history.py               # Per-scan snapshots + delta vs prior
│   ├── watchlist.py             # User watchlist (parse + persist)
│   ├── performance.py           # IC evaluation + self-tuning weights
│   └── paths.py                 # Centralised, env-var-overridable paths
├── data/
│   ├── us_tickers.csv           # ~5260 US common stocks (committed)
│   └── stubs/                   # 30-ticker synthetic L2/options/ticks/intraday
├── tests/                       # pytest suite (isolated disk paths)
├── requirements.txt
├── requirements-dev.txt
├── README.md
└── .gitignore
```

## Run locally

```bash
pip install -r requirements.txt
python app.py
```

Then open http://localhost:7860.

## Data sources

The four institutional-flow factors (`l2_imbalance`, `unusual_options`,
`block_aggression`, `buy_persistence`) read from a pluggable
`FactorDataSource` (`scanner/factor_sources.py`).

| Source              | When to use                                              | Config env var             |
|---------------------|----------------------------------------------------------|----------------------------|
| `StubDataSource`    | **Default.** Reads `data/stubs/` synthetic data          | `FSCANNER_DATA_SOURCE` unset or `=stub` |
| `FutuDataSource`    | Live Level-2 / options / tick data via local Futu OpenD  | `FSCANNER_DATA_SOURCE=futu` + `FUTU_OPEND_HOST` + `FUTU_OPEND_PORT` |

The Space uses `StubDataSource` so it never depends on a live feed.
To regenerate the stub data:

```bash
python data/stubs/_build_stubs.py
```

## Tests

```bash
pip install -r requirements-dev.txt
pytest -q
```

The suite isolates every persistent path via an autouse fixture, so it
runs in a sandboxed `tmp_path` and never touches your real cache or
watchlist.  Tests cover factor math, scoring, history snapshots,
watchlist parsing, the sector cache, and the auto-tuning loop (the
optimizer reliably discovers which factor is the true signal in a
synthetic dataset).

## Deploy to Hugging Face Spaces

### One-time setup

```bash
pip install "huggingface_hub[cli]"
huggingface-cli login

# Create a new Space (Gradio SDK)
huggingface-cli repo create us-flow-scanner --type space --space-sdk gradio
cd ..
git clone https://huggingface.co/spaces/<your-username>/us-flow-scanner
cp -r us-flow-scanner/* us-flow-scanner/.gitignore us-flow-scanner/
cd us-flow-scanner
git add .
git commit -m "Initial commit"
git push
```

The Space will install dependencies and start the app.  The public URL is
`https://huggingface.co/spaces/<your-username>/us-flow-scanner`.

### Optional: persistent cache

The OHLCV cache is stored under `/tmp`, which is wiped on container restart.
For persistence across restarts:

1. Create a private HF Dataset, e.g. `us-flow-cache`.
2. In your Space's **Settings → Variables and secrets**, add:
   - `FSCANNER_CACHE_REPO` = `<your-username>/us-flow-cache`
   - `HF_TOKEN` = (a write-enabled token, also added to your dataset repo)
3. The app will push the cache after each scan and pull it on startup.

### Optional: Polygon fallback

If you hit yfinance rate limits, set:
- `POLYGON_API_KEY` = (your polygon.io API key)

in Space secrets.  Tickers that fail on yfinance are retried with Polygon.

## Performance & limitations

- **Full universe scan** of ~5000 liquid names: ~3–6 min on a free HF Space
  CPU.  Use **Filtered** mode (default) to keep it under ~2 min.
- **Filtered** default: `price ≥ $5` and `20-day ADV ≥ $5M`, which trims
  the universe to ~1500–3000 liquid names.
- yfinance is an unofficial scraper and may be flaky.  The app retries with
  backoff and falls back to Polygon (if configured) for individual tickers.
- The algorithm is a **proxy** for institutional flow based on public
  OHLCV.  It does not see 13F filings, dark-pool prints, or order-book
  imbalance.
- Free HF Spaces may sleep after ~48 h of inactivity; the app restarts on
  the next visit.

## Self-tuning weights

After every scan, the app:
1. Persists the full results as a timestamped snapshot in `history/`.
2. Computes the realised forward return for every ticker over the next
   `HORIZON_DAYS` (default 5) trading days using the OHLCV cache.
3. Aggregates a **Spearman IC** per snapshot for the current weights.
4. Runs a small Dirichlet random search (200 candidates + 5 structured
   seeds) over alternative weight vectors and picks the one with the
   highest mean IC.
5. If the new vector beats the baseline IC by ≥ 0.005, it is persisted
   to `learned.json` and a row is appended to `perf.parquet`; otherwise
   the default weights are kept.
6. The "Use auto-tuned weights" checkbox in the Performance tab tells
   the next scan to use the learned vector instead of the slider
   values, so you can A/B test the improvement.

Tuning is intentionally lightweight (seconds, not minutes) so it can
run in a background thread after every scan.  It needs at least 5
snapshots with valid forward data and at least 30 tickers per snapshot
to fire.

## Tickers list

`data/us_tickers.csv` contains ~5260 tickers filtered from Nasdaq Trader's
official daily files (`nasdaqlisted.txt` + `otherlisted.txt`):
- ETFs, warrants, units, rights, preferreds, structured products excluded
- Test issues and distressed financial-status tickers excluded
- Tickers must be ≤5 chars, no `.`/`-`/space (i.e. no class-B or dual-class
  weirdness)

To refresh the list:

```powershell
# From the data/ directory
Invoke-WebRequest -Uri "https://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt" -OutFile nasdaqlisted_raw.txt
Invoke-WebRequest -Uri "https://www.nasdaqtrader.com/dynamic/SymDir/otherlisted.txt" -OutFile otherlisted_raw.txt
# ... (use the same PowerShell filter as in earlier dev work)
```

## License

MIT.