agents.md

# agents.md

Quick reference for AI agents working on the **US Stock Institutional Flow
Scanner** (a Gradio + yfinance web app).  Optimised for HF Spaces deployment,
runs locally with `python app.py`.

## What this app does

Cross-sectional scan of ~5,000 US common stocks.  For each ticker it
computes 5 factors and combines them into a `-100 … +100` "institutional
flow" score:

| Factor          | Implementation                                |
|-----------------|-----------------------------------------------|
| `cmf`           | Chaikin Money Flow (20-day)                   |
| `obv_slope`     | Linear regression of On-Balance Volume (20d)  |
| `big_bar_ratio` | Unusual-volume + wide-range bars vs 20d       |
| `vwap_dev`      | Close vs rolling 20-day VWAP                  |
| `rvol_signed`   | RVOL with sign flipped by CMF direction       |
| `l2_imbalance`  | Level-2 large-resting-order factor             |
| `unusual_options` | Vol/OI z-score across 20 days (moneyness-weighted) |
| `block_aggression` | Net buy/sell in ≥10k-share trades (Lee-Ready)  |
| `buy_persistence` | Buy-ratio average over last 1h of 5-min bars   |

Factors are z-scored cross-sectionally with `robust_zscore` (median + MAD,
clipped to ±5).  Final score scaled to `±100`.

## Quick start

```bash
pip install -r requirements.txt
python app.py          # http://localhost:7860
```

## Repo layout

```
app.py                # Gradio UI; defines `demo` at module level
scanner/
  paths.py            # All write paths (env-overridable)
  universe.py         # ~5,260 US tickers + sector cache
  data_fetcher.py     # yfinance batched + Polygon fallback + parquet cache
  flow_algo.py        # 5 daily-flow factors (compute_factors)
  l2_factor.py        # Level-2 large-resting-order factor
  options_factor.py   # Unusual options activity (vol/OI z-score)
  tick_factor.py      # Lee-Ready tick + size buckets
  intraday_factor.py  # Intraday VWAP + buy persistence
  factor_sources.py   # StubDataSource (default) + FutuDataSource (live)
  scorer.py           # robust_zscore, score_factors, top_n, DEFAULT_WEIGHTS
  history.py          # Per-scan snapshots + with_delta
  watchlist.py        # User watchlist (parse + persist, ≤100 tickers)
  performance.py      # IC evaluation + self-tuning weights
  persistence.py      # Optional HF Dataset round-trip
data/
  us_tickers.csv      # Canonical ticker list, ~5,260 rows
  stubs/              # 30-ticker synthetic L2/options/ticks/intraday data
tests/                # pytest, isolated disk paths via autouse fixture
```

## Environment variables (all optional)

| Var                       | Default                                  | Purpose |
|---------------------------|------------------------------------------|---------|
| `FSCANNER_TMP_DIR`        | `/tmp` (HF) or `tempfile.gettempdir()`   | Base tmp dir |
| `FSCANNER_CACHE_PATH`     | `<tmp>/us_flow_scanner_ohlcv.parquet`    | OHLCV cache |
| `FSCANNER_HISTORY_DIR`    | `<tmp>/us_flow_scanner_history`          | Snapshot dir |
| `FSCANNER_SECTOR_CACHE`   | `<tmp>/us_flow_scanner_sectors.parquet`  | Sector disk cache |
| `FSCANNER_WATCHLIST`      | `<tmp>/us_flow_scanner_watchlist.json`   | Watchlist path |
| `FSCANNER_LEARNED_WEIGHTS`| `<tmp>/us_flow_scanner_learned_weights.json` | Auto-tuned weights |
| `FSCANNER_PERFORMANCE_LOG`| `<tmp>/us_flow_scanner_performance.parquet` | Tuning history log |
| `FSCANNER_CACHE_REPO`     | *(unset)*                                | HF Dataset for cache persistence |
| `HF_TOKEN`                | *(unset)*                                | Required when `FSCANNER_CACHE_REPO` is set |
| `POLYGON_API_KEY`         | *(unset)*                                | Polygon fallback for yfinance failures |
| `PORT`                    | `7860`                                    | Gradio server port |
| `FSCANNER_DATA_SOURCE`    | `stub`                                    | `stub` (synthetic) or `futu` (live OpenD) for the 4 institutional factors |
| `FUTU_OPEND_HOST`         | `127.0.0.1`                                | Futu OpenD host (only when `FSCANNER_DATA_SOURCE=futu`) |
| `FUTU_OPEND_PORT`         | `11111`                                    | Futu OpenD port |

## Tabs in the UI

1. **All results** — full scored table + CSV download
2. **Top 20 buys / sells** — short-list
3. **Watchlist** — tickers that bypass liquidity + sector filters
4. **Sector breakdown** — mean score per sector (uses disk-cached sectors)
5. **Per-stock detail** — candlestick + volume + CMF(20) chart
6. **History** — list of saved snapshots (timestamped parquets)
7. **Performance / auto-tune** — IC chart, learned-vs-default table, "Re-tune now" + "Apply learned weights to sliders"

## Self-tuning algorithm (the "auto_improve" loop)

After **every** scan (`app.py::_start_auto_improve`), in a background thread:

1. Load all saved snapshots from `HISTORY_DIR`.
2. For each snapshot, compute the realised forward return for every
   ticker over the next `HORIZON_DAYS` (default 5) trading days using
   the OHLCV cache.
3. Aggregate per-snapshot **Spearman IC** for the *current* weights.
4. Run a small **Dirichlet random search** (200 candidates + 5
   structured seeds: current, uniform, one-hot-heavy per factor) over
   alternative weight vectors; pick the one with the highest mean IC.
5. If the new vector beats the baseline IC by ≥ `IC_IMPROVEMENT_THRESHOLD`
   (default 0.005), persist to `LEARNED_WEIGHTS_PATH` and append a row
   to `PERFORMANCE_LOG_PATH`.  Otherwise keep the default weights.
6. The "Use auto-tuned weights" checkbox tells the next scan to use the
   learned vector instead of the slider values.

Gating: needs ≥ `MIN_SNAPSHOTS_FOR_TUNING` (5) snapshots and
≥ `MIN_VALID_TICKERS_PER_SNAPSHOT` (30) tickers per snapshot.

## Constraints agents should respect

- **Read-only ticker list**: `data/us_tickers.csv` is curated; don't
  regenerate it casually.
- **No new top-level deps** without checking `requirements.txt` first.
  Plotly, gradio, yfinance, pandas, numpy, scipy, huggingface_hub, tqdm,
  requests, pyarrow are already there.
- **Cache paths are module-level** in `scanner/paths.py`.  Consumer
  modules read `paths.X` (not `from .paths import X`) so tests can
  monkeypatch without reloads.
- **No hardcoded `/tmp`** in source — use `paths._base_tmp()` /
  `tempfile.gettempdir()` fallback so it works on Windows.
- **No `from .paths import X`** — always `from . import paths; paths.X`.
- **Free HF Spaces CPU** is the deployment target; avoid heavy
  dependencies and big in-memory dataframes (full-universe scan is
  ~3–6 min; filtered ~1–2 min).

## Common tasks for agents

| Task                                | Where to look |
|-------------------------------------|---------------|
| Add a new **daily** factor          | `scanner/flow_algo.py::compute_factors` (returns `FactorSet` dataclass) → `scanner/scorer.py` (`FACTOR_KEYS` + `DEFAULT_WEIGHTS` → also update sliders in `app.py`) |
| Add a new **institutional** factor (L2 / options / ticks / intraday) | New module in `scanner/`, must implement the `FactorDataSource` Protocol methods. Then add a name to `FACTOR_KEYS` + `DEFAULT_WEIGHTS` in `scanner/scorer.py` and wire the call in `app.py::_do_scan` (see the `extra_factors` dict). |
| Change the data source             | `scanner/factor_sources.py` — set `FSCANNER_DATA_SOURCE=futu` for live Futu OpenD, or use the default `StubDataSource` |
| Change score scale / rating buckets | `scanner/scorer.py::score_factors`, `_rating_for` |
| Add a new tab                       | `app.py::build_ui` — see existing tabs for pattern; add to `scan_outputs` list in same order as returned by `run_scan` |
| Change auto-tune aggressiveness     | `scanner/performance.py` constants: `HORIZON_DAYS`, `MIN_SNAPSHOTS_FOR_TUNING`, `N_RANDOM_CANDIDATES`, `IC_IMPROVEMENT_THRESHOLD` |
| Add an env-var path                 | `scanner/paths.py` — append new const with `os.environ.get("FSCANNER_…", …)` |
| Add a test                          | Drop `tests/test_*.py`; conftest autouse fixture isolates disk paths via `tmp_path` |

## Testing

```bash
pip install -r requirements-dev.txt
pytest -q
```

The `isolated_paths` fixture monkeypatches every path attribute in
`scanner.paths` to a fresh `tmp_path` per test, so the suite is
sandboxed and parallel-safe.

## Deploy

See `DEPLOY.md` for the full Hugging Face Spaces walkthrough.  Short
version: `git init && git add . && git commit -m "init" && git remote
add space https://huggingface.co/spaces/<you>/us-flow-scanner && git
push space main`.  Free CPU tier is sufficient.

## License

MIT.