# agents.md Quick reference for AI agents working on the **US Stock Institutional Flow Scanner** (a Gradio + yfinance web app). Optimised for HF Spaces deployment, runs locally with `python app.py`. ## What this app does Cross-sectional scan of ~5,000 US common stocks. For each ticker it computes 5 factors and combines them into a `-100 … +100` "institutional flow" score: | Factor | Implementation | |-----------------|-----------------------------------------------| | `cmf` | Chaikin Money Flow (20-day) | | `obv_slope` | Linear regression of On-Balance Volume (20d) | | `big_bar_ratio` | Unusual-volume + wide-range bars vs 20d | | `vwap_dev` | Close vs rolling 20-day VWAP | | `rvol_signed` | RVOL with sign flipped by CMF direction | | `l2_imbalance` | Level-2 large-resting-order factor | | `unusual_options` | Vol/OI z-score across 20 days (moneyness-weighted) | | `block_aggression` | Net buy/sell in ≥10k-share trades (Lee-Ready) | | `buy_persistence` | Buy-ratio average over last 1h of 5-min bars | Factors are z-scored cross-sectionally with `robust_zscore` (median + MAD, clipped to ±5). Final score scaled to `±100`. ## Quick start ```bash pip install -r requirements.txt python app.py # http://localhost:7860 ``` ## Repo layout ``` app.py # Gradio UI; defines `demo` at module level scanner/ paths.py # All write paths (env-overridable) universe.py # ~5,260 US tickers + sector cache data_fetcher.py # yfinance batched + Polygon fallback + parquet cache flow_algo.py # 5 daily-flow factors (compute_factors) l2_factor.py # Level-2 large-resting-order factor options_factor.py # Unusual options activity (vol/OI z-score) tick_factor.py # Lee-Ready tick + size buckets intraday_factor.py # Intraday VWAP + buy persistence factor_sources.py # StubDataSource (default) + FutuDataSource (live) scorer.py # robust_zscore, score_factors, top_n, DEFAULT_WEIGHTS history.py # Per-scan snapshots + with_delta watchlist.py # User watchlist (parse + persist, ≤100 tickers) performance.py # IC evaluation + self-tuning weights persistence.py # Optional HF Dataset round-trip data/ us_tickers.csv # Canonical ticker list, ~5,260 rows stubs/ # 30-ticker synthetic L2/options/ticks/intraday data tests/ # pytest, isolated disk paths via autouse fixture ``` ## Environment variables (all optional) | Var | Default | Purpose | |---------------------------|------------------------------------------|---------| | `FSCANNER_TMP_DIR` | `/tmp` (HF) or `tempfile.gettempdir()` | Base tmp dir | | `FSCANNER_CACHE_PATH` | `/us_flow_scanner_ohlcv.parquet` | OHLCV cache | | `FSCANNER_HISTORY_DIR` | `/us_flow_scanner_history` | Snapshot dir | | `FSCANNER_SECTOR_CACHE` | `/us_flow_scanner_sectors.parquet` | Sector disk cache | | `FSCANNER_WATCHLIST` | `/us_flow_scanner_watchlist.json` | Watchlist path | | `FSCANNER_LEARNED_WEIGHTS`| `/us_flow_scanner_learned_weights.json` | Auto-tuned weights | | `FSCANNER_PERFORMANCE_LOG`| `/us_flow_scanner_performance.parquet` | Tuning history log | | `FSCANNER_CACHE_REPO` | *(unset)* | HF Dataset for cache persistence | | `HF_TOKEN` | *(unset)* | Required when `FSCANNER_CACHE_REPO` is set | | `POLYGON_API_KEY` | *(unset)* | Polygon fallback for yfinance failures | | `PORT` | `7860` | Gradio server port | | `FSCANNER_DATA_SOURCE` | `stub` | `stub` (synthetic) or `futu` (live OpenD) for the 4 institutional factors | | `FUTU_OPEND_HOST` | `127.0.0.1` | Futu OpenD host (only when `FSCANNER_DATA_SOURCE=futu`) | | `FUTU_OPEND_PORT` | `11111` | Futu OpenD port | ## Tabs in the UI 1. **All results** — full scored table + CSV download 2. **Top 20 buys / sells** — short-list 3. **Watchlist** — tickers that bypass liquidity + sector filters 4. **Sector breakdown** — mean score per sector (uses disk-cached sectors) 5. **Per-stock detail** — candlestick + volume + CMF(20) chart 6. **History** — list of saved snapshots (timestamped parquets) 7. **Performance / auto-tune** — IC chart, learned-vs-default table, "Re-tune now" + "Apply learned weights to sliders" ## Self-tuning algorithm (the "auto_improve" loop) After **every** scan (`app.py::_start_auto_improve`), in a background thread: 1. Load all saved snapshots from `HISTORY_DIR`. 2. For each snapshot, compute the realised forward return for every ticker over the next `HORIZON_DAYS` (default 5) trading days using the OHLCV cache. 3. Aggregate per-snapshot **Spearman IC** for the *current* weights. 4. Run a small **Dirichlet random search** (200 candidates + 5 structured seeds: current, uniform, one-hot-heavy per factor) over alternative weight vectors; pick the one with the highest mean IC. 5. If the new vector beats the baseline IC by ≥ `IC_IMPROVEMENT_THRESHOLD` (default 0.005), persist to `LEARNED_WEIGHTS_PATH` and append a row to `PERFORMANCE_LOG_PATH`. Otherwise keep the default weights. 6. The "Use auto-tuned weights" checkbox tells the next scan to use the learned vector instead of the slider values. Gating: needs ≥ `MIN_SNAPSHOTS_FOR_TUNING` (5) snapshots and ≥ `MIN_VALID_TICKERS_PER_SNAPSHOT` (30) tickers per snapshot. ## Constraints agents should respect - **Read-only ticker list**: `data/us_tickers.csv` is curated; don't regenerate it casually. - **No new top-level deps** without checking `requirements.txt` first. Plotly, gradio, yfinance, pandas, numpy, scipy, huggingface_hub, tqdm, requests, pyarrow are already there. - **Cache paths are module-level** in `scanner/paths.py`. Consumer modules read `paths.X` (not `from .paths import X`) so tests can monkeypatch without reloads. - **No hardcoded `/tmp`** in source — use `paths._base_tmp()` / `tempfile.gettempdir()` fallback so it works on Windows. - **No `from .paths import X`** — always `from . import paths; paths.X`. - **Free HF Spaces CPU** is the deployment target; avoid heavy dependencies and big in-memory dataframes (full-universe scan is ~3–6 min; filtered ~1–2 min). ## Common tasks for agents | Task | Where to look | |-------------------------------------|---------------| | Add a new **daily** factor | `scanner/flow_algo.py::compute_factors` (returns `FactorSet` dataclass) → `scanner/scorer.py` (`FACTOR_KEYS` + `DEFAULT_WEIGHTS` → also update sliders in `app.py`) | | Add a new **institutional** factor (L2 / options / ticks / intraday) | New module in `scanner/`, must implement the `FactorDataSource` Protocol methods. Then add a name to `FACTOR_KEYS` + `DEFAULT_WEIGHTS` in `scanner/scorer.py` and wire the call in `app.py::_do_scan` (see the `extra_factors` dict). | | Change the data source | `scanner/factor_sources.py` — set `FSCANNER_DATA_SOURCE=futu` for live Futu OpenD, or use the default `StubDataSource` | | Change score scale / rating buckets | `scanner/scorer.py::score_factors`, `_rating_for` | | Add a new tab | `app.py::build_ui` — see existing tabs for pattern; add to `scan_outputs` list in same order as returned by `run_scan` | | Change auto-tune aggressiveness | `scanner/performance.py` constants: `HORIZON_DAYS`, `MIN_SNAPSHOTS_FOR_TUNING`, `N_RANDOM_CANDIDATES`, `IC_IMPROVEMENT_THRESHOLD` | | Add an env-var path | `scanner/paths.py` — append new const with `os.environ.get("FSCANNER_…", …)` | | Add a test | Drop `tests/test_*.py`; conftest autouse fixture isolates disk paths via `tmp_path` | ## Testing ```bash pip install -r requirements-dev.txt pytest -q ``` The `isolated_paths` fixture monkeypatches every path attribute in `scanner.paths` to a fresh `tmp_path` per test, so the suite is sandboxed and parallel-safe. ## Deploy See `DEPLOY.md` for the full Hugging Face Spaces walkthrough. Short version: `git init && git add . && git commit -m "init" && git remote add space https://huggingface.co/spaces//us-flow-scanner && git push space main`. Free CPU tier is sufficient. ## License MIT.