agents.md

agents.md

Quick reference for AI agents working on the US Stock Institutional Flow Scanner (a Gradio + yfinance web app). Optimised for HF Spaces deployment, runs locally with python app.py.

What this app does

Cross-sectional scan of ~5,000 US common stocks. For each ticker it computes 5 factors and combines them into a -100 … +100 "institutional flow" score:

Factor	Implementation
cmf	Chaikin Money Flow (20-day)
obv_slope	Linear regression of On-Balance Volume (20d)
big_bar_ratio	Unusual-volume + wide-range bars vs 20d
vwap_dev	Close vs rolling 20-day VWAP
rvol_signed	RVOL with sign flipped by CMF direction
l2_imbalance	Level-2 large-resting-order factor
unusual_options	Vol/OI z-score across 20 days (moneyness-weighted)
block_aggression	Net buy/sell in ≥10k-share trades (Lee-Ready)
buy_persistence	Buy-ratio average over last 1h of 5-min bars

Factors are z-scored cross-sectionally with robust_zscore (median + MAD, clipped to ±5). Final score scaled to ±100.

Quick start

pip install -r requirements.txt
python app.py          # http://localhost:7860

Repo layout

app.py                # Gradio UI; defines `demo` at module level
scanner/
  paths.py            # All write paths (env-overridable)
  universe.py         # ~5,260 US tickers + sector cache
  data_fetcher.py     # yfinance batched + Polygon fallback + parquet cache
  flow_algo.py        # 5 daily-flow factors (compute_factors)
  l2_factor.py        # Level-2 large-resting-order factor
  options_factor.py   # Unusual options activity (vol/OI z-score)
  tick_factor.py      # Lee-Ready tick + size buckets
  intraday_factor.py  # Intraday VWAP + buy persistence
  factor_sources.py   # StubDataSource (default) + FutuDataSource (live)
  scorer.py           # robust_zscore, score_factors, top_n, DEFAULT_WEIGHTS
  history.py          # Per-scan snapshots + with_delta
  watchlist.py        # User watchlist (parse + persist, ≤100 tickers)
  performance.py      # IC evaluation + self-tuning weights
  persistence.py      # Optional HF Dataset round-trip
data/
  us_tickers.csv      # Canonical ticker list, ~5,260 rows
  stubs/              # 30-ticker synthetic L2/options/ticks/intraday data
tests/                # pytest, isolated disk paths via autouse fixture

Environment variables (all optional)

Var	Default	Purpose
FSCANNER_TMP_DIR	/tmp (HF) or tempfile.gettempdir()	Base tmp dir
FSCANNER_CACHE_PATH	<tmp>/us_flow_scanner_ohlcv.parquet	OHLCV cache
FSCANNER_HISTORY_DIR	<tmp>/us_flow_scanner_history	Snapshot dir
FSCANNER_SECTOR_CACHE	<tmp>/us_flow_scanner_sectors.parquet	Sector disk cache
FSCANNER_WATCHLIST	<tmp>/us_flow_scanner_watchlist.json	Watchlist path
FSCANNER_LEARNED_WEIGHTS	<tmp>/us_flow_scanner_learned_weights.json	Auto-tuned weights
FSCANNER_PERFORMANCE_LOG	<tmp>/us_flow_scanner_performance.parquet	Tuning history log
FSCANNER_CACHE_REPO	(unset)	HF Dataset for cache persistence
HF_TOKEN	(unset)	Required when FSCANNER_CACHE_REPO is set
POLYGON_API_KEY	(unset)	Polygon fallback for yfinance failures
PORT	7860	Gradio server port
FSCANNER_DATA_SOURCE	stub	stub (synthetic) or futu (live OpenD) for the 4 institutional factors
FUTU_OPEND_HOST	127.0.0.1	Futu OpenD host (only when FSCANNER_DATA_SOURCE=futu)
FUTU_OPEND_PORT	11111	Futu OpenD port

Tabs in the UI

1. All results — full scored table + CSV download
2. Top 20 buys / sells — short-list
3. Watchlist — tickers that bypass liquidity + sector filters
4. Sector breakdown — mean score per sector (uses disk-cached sectors)
5. Per-stock detail — candlestick + volume + CMF(20) chart
6. History — list of saved snapshots (timestamped parquets)
7. Performance / auto-tune — IC chart, learned-vs-default table, "Re-tune now" + "Apply learned weights to sliders"

Self-tuning algorithm (the "auto_improve" loop)

After every scan (app.py::_start_auto_improve), in a background thread:

1. Load all saved snapshots from HISTORY_DIR.
2. For each snapshot, compute the realised forward return for every ticker over the next HORIZON_DAYS (default 5) trading days using the OHLCV cache.
3. Aggregate per-snapshot Spearman IC for the current weights.
4. Run a small Dirichlet random search (200 candidates + 5 structured seeds: current, uniform, one-hot-heavy per factor) over alternative weight vectors; pick the one with the highest mean IC.
5. If the new vector beats the baseline IC by ≥ IC_IMPROVEMENT_THRESHOLD (default 0.005), persist to LEARNED_WEIGHTS_PATH and append a row to PERFORMANCE_LOG_PATH. Otherwise keep the default weights.
6. The "Use auto-tuned weights" checkbox tells the next scan to use the learned vector instead of the slider values.

Gating: needs ≥ MIN_SNAPSHOTS_FOR_TUNING (5) snapshots and ≥ MIN_VALID_TICKERS_PER_SNAPSHOT (30) tickers per snapshot.

Constraints agents should respect

• Read-only ticker list: data/us_tickers.csv is curated; don't regenerate it casually.
• No new top-level deps without checking requirements.txt first. Plotly, gradio, yfinance, pandas, numpy, scipy, huggingface_hub, tqdm, requests, pyarrow are already there.
• Cache paths are module-level in scanner/paths.py. Consumer modules read paths.X (not from .paths import X) so tests can monkeypatch without reloads.
• No hardcoded /tmp in source — use paths._base_tmp() / tempfile.gettempdir() fallback so it works on Windows.
• No from .paths import X — always from . import paths; paths.X.
• Free HF Spaces CPU is the deployment target; avoid heavy dependencies and big in-memory dataframes (full-universe scan is ~3–6 min; filtered ~1–2 min).

Common tasks for agents

Task	Where to look
Add a new daily factor	scanner/flow_algo.py::compute_factors (returns FactorSet dataclass) → scanner/scorer.py (FACTOR_KEYS + DEFAULT_WEIGHTS → also update sliders in app.py)
Add a new institutional factor (L2 / options / ticks / intraday)	New module in scanner/, must implement the FactorDataSource Protocol methods. Then add a name to FACTOR_KEYS + DEFAULT_WEIGHTS in scanner/scorer.py and wire the call in app.py::_do_scan (see the extra_factors dict).
Change the data source	scanner/factor_sources.py — set FSCANNER_DATA_SOURCE=futu for live Futu OpenD, or use the default StubDataSource
Change score scale / rating buckets	scanner/scorer.py::score_factors, _rating_for
Add a new tab	app.py::build_ui — see existing tabs for pattern; add to scan_outputs list in same order as returned by run_scan
Change auto-tune aggressiveness	scanner/performance.py constants: HORIZON_DAYS, MIN_SNAPSHOTS_FOR_TUNING, N_RANDOM_CANDIDATES, IC_IMPROVEMENT_THRESHOLD
Add an env-var path	scanner/paths.py — append new const with os.environ.get("FSCANNER_…", …)
Add a test	Drop tests/test_*.py; conftest autouse fixture isolates disk paths via tmp_path

Testing

pip install -r requirements-dev.txt
pytest -q

The isolated_paths fixture monkeypatches every path attribute in scanner.paths to a fresh tmp_path per test, so the suite is sandboxed and parallel-safe.

Deploy

See DEPLOY.md for the full Hugging Face Spaces walkthrough. Short version: git init && git add . && git commit -m "init" && git remote add space https://huggingface.co/spaces/<you>/us-flow-scanner && git push space main. Free CPU tier is sufficient.

License

MIT.