HANDOFF.md

# Stock Predictor — Codex Handoff

## Project in one line
Taiwan stock ML predictor (FastAPI + React/Vite). Runs locally on a Mac mini, also deployed on HuggingFace Space. Mac computes predictions; HF Space reads them from a shared HF Dataset queue.

---

## Repo layout
```
stock-predictor/
├── main.py                     # FastAPI app entry, lifespan warmup
├── routers/
│   ├── stock.py                # /api/stock/* endpoints
│   └── line_webhook.py         # LINE Bot webhook
├── services/
│   ├── predictor_service.py    # get_prediction(), caches, quick estimate
│   ├── hf_queue_service.py     # HF Dataset queue read/write
│   ├── backtest_service.py     # walk-forward backtest
│   └── news_service.py         # Gemini news overlay
├── models/
│   └── predictor.py            # StockPredictor, FEATURE_COLUMNS, _build_features
├── data/
│   ├── fetcher.py              # yfinance + TWSE history
│   ├── institutional_flow.py   # 三大法人
│   ├── margin_flow.py          # 融資融券
│   ├── monthly_revenue.py      # 月營收
│   ├── ptt_sentiment.py        # PTT [標的] sentiment (currently returns zeros)
│   └── ...
├── indicators/
│   └── technical.py            # add_all_indicators(), add_cross_asset_tw()
├── scripts/
│   ├── hf_queue_worker.py      # Mac launchd job: poll queue, push predictions
│   ├── backtest_3way.py        # baseline backtest runner
│   ├── backtest_c*.py          # per-experiment backtest scripts (C1–C12)
│   └── improvement_harness.py  # runs backtest on 5 stocks, checks thresholds
├── frontend/
│   └── src/
│       └── components/         # React components (StockPage, RecommendPage, etc.)
├── static/                     # built frontend (served by FastAPI)
└── docs/
    └── improvements_log.md     # history of model experiments
```

---

## Deployment

### Mac local (port 7861)
- Managed by launchd: `com.dennis.stock-predictor`
- Restart: `launchctl stop com.dennis.stock-predictor && launchctl start com.dennis.stock-predictor`
- Logs: `logs/server.log`

### HF Space
- URL: `https://dennischan0909-dockerspace.hf.space/`
- Rebuilds automatically on push to `main` branch
- HF Dataset queue: `DennisChan0909/stock-predictor-queue` (stores predictions as JSON)
- Current runbook: `docs/huggingface_integration.md`
- HF is the lightweight serving layer.  Mac remains the durable ML/precompute
  worker; HF returns `quick_rules` while waiting for Mac queue results.

### Mac worker (launchd, every 5 min)
- `scripts/hf_queue_worker.py` — computes predictions for WATCHLIST (20 stocks) + any pending queue items, pushes to HF Dataset
- Commands:
  - `./venv/bin/python scripts/hf_queue_worker.py status` — show queue
  - `./venv/bin/python scripts/hf_queue_worker.py push 2330 0050` — force-push specific stocks

### Frontend build + deploy
```bash
cd frontend && npm run build
cp frontend/dist/index.html static/index.html
cp frontend/dist/assets/* static/assets/
git add -u && git commit -m "..." && git push origin main
```

---

## HF Space prediction flow (as of 2026-05-12)

1. `/api/stock/<code>/predict` called
2. `get_prediction()` checks in-memory pred cache (30 min TTL on HF)
3. **Cache miss**: checks HF Dataset queue synchronously (`get_cached_result`)
   - Queue hit → return Mac ML result immediately, cache it for 30 min
   - Queue miss → call `enqueue()` + fall through to quick estimate
4. Quick estimate: rule-based signal from technical indicators (<100ms), NOT cached on HF Space
5. Background ML training is **skipped** on HF Space (Mac worker handles it)
6. Startup warmup (`_warmup_from_queue` in `main.py`) pre-loads all "done" queue results before accepting requests

---

## Model

### Stack
RF + XGB + LGBM ensemble (CatBoost compiled but optional). 29 features after SHAP pruning.

### Labels
Triple barrier (C3): upper/lower barrier = close × (1 ± vol_20d × 1.0), vertical = 5 trading days. Label = first barrier hit.

### Current baseline (5-stock average, 24-month walk-forward)
- `dir_accuracy ≈ 42%`
- `up_precision ≈ 53%`
- `FEATURE_COLUMNS`: 29 features in `models/predictor.py:153`

### Meta-label filter (C4)
`meta_filtered` field in prediction output. Secondary RF trained on (features + buy_prob → correct?). Threshold 0.50.

---

## Experiments history (`docs/improvements_log.md`)

| ID | What | Result |
|----|------|--------|
| C1 | fracdiff price features | FAILED |
| **C3** | **Triple barrier labels** | **PASSED — dir_acc +9.8pp** |
| **C4** | **Meta-label filter** | **PASSED — up_prec +4.1pp** |
| C5 | SHAP pruning (removed 11 low-signal features) | PASSED (+1pp), now in FEATURE_COLUMNS |
| C6 | HMM regime detection | FAILED (<1.5pp) |
| C2 | PTT sentiment (ptt_sentiment_1d, _5d_ma) | FAILED — data pipeline returns 0 rows for all stocks |
| C7 | Calendar features + 36m lookback | FAILED — up_prec dropped -5.8pp |
| C8 | Optuna RF tuning | FAILED — params already near-optimal |
| C9 | Multi-stock training universe | FAILED |
| C10 | Asymmetric triple barrier grid | FAILED |
| C11 | Full ensemble walk-forward | FAILED |
| C12 | 1-day label horizon | FAILED catastrophically |

All C-experiments are self-contained in `scripts/backtest_c*.py`. New experiments should follow the same pattern: run `scripts/improvement_harness.py` to validate before touching `models/predictor.py`.

---

## Known issues / pending work

### Hermes / agent-team integration audit (2026-05-21)
Hermes added standalone agent-team utilities: Antigravity notes, W&B logging,
data quality validation, and Telegram notification wrappers. These are
intended to stay outside the core predictor path unless separately promoted.

Hermes hierarchy entry point: `docs/hermes/README.md`. Read
`docs/hermes/rules.md` before changing Hermes utilities; it records that generic
Hermes helper cleanup must not modify predictor/recommendation/precompute
logic.

Codex audit fixes applied:
- `scripts/telegram_bot_wrapper.py`: fixed `.env.telegram` parsing, and made
  CLI arguments mode-specific so `--update`, `--validation`, and experiment
  modes can run without a dummy `--message`.
- `scripts/test_agent_integration.py`: added missing `argparse` import and
  avoided importing Python's built-in `antigravity` easter-egg module. The
  check now uses the CLI/config path instead.
- `.gitignore`: ignore local `.env.telegram` and `.env.wandb` files; keep
  `.env.telegram.example` as the shareable template.

Verification:
```bash
./venv/bin/python -m py_compile scripts/telegram_bot_wrapper.py scripts/send_experiment_notification.py scripts/data_quality_validator.py scripts/test_agent_integration.py scripts/wandb_experiment_tracker.py
./venv/bin/python scripts/telegram_bot_wrapper.py --message "Hermes side-effect smoke" --success
./venv/bin/python scripts/telegram_bot_wrapper.py --update --step 1 --total 2
./venv/bin/python scripts/data_quality_validator.py --check features
./venv/bin/python scripts/test_agent_integration.py --quiet
```

Residual risk: `hermes_tools` is not importable in the normal venv, so Telegram
wrapper uses fallback console mode unless Hermes injects that module at runtime.
Do not wire these utilities into production precompute or recommendation
publishing until a real Telegram send and no-lookahead validation gate pass.

### PTT sentiment (data/ptt_sentiment.py)
`add_ptt_sentiment()` exists but returns 0 rows for all stocks. PTT scraper (`scripts/survey_ptt.py`) exists. The merge pipeline needs debugging — likely a date alignment or cache miss issue. Features `ptt_sentiment_1d`, `ptt_sentiment_5d_ma` are NOT in `FEATURE_COLUMNS` yet.

### Improve harness threshold
Current pass gate: `mean(dir_accuracy) > 31.5%` — this was set before C3/C4/C5 raised the baseline to ~42%. Should be updated to `> 43.5%` and `up_precision > 54%` for any new experiment to be meaningful.

### Market Scan (RecommendPage)
Custom stock input added (2026-05-12). Backend `/api/stock/recommend` accepts `?watchlist=2330,0050` query param. State and handlers are wired.

---

## Key env vars
```
HF_TOKEN (or .hf_token file)       — HF Hub write access
SPACE_ID / HF_SPACE_ID / SPACE_HOST — set automatically on HF, detect HF vs Mac
WATCHLIST_STOCKS                   — override default 20-stock watchlist in worker
LIGHTWEIGHT_MODE=1                 — use 12-month training window (HF)
LINE_CHANNEL_SECRET / ACCESS_TOKEN — LINE Bot
```

---

## How to run a new backtest experiment

```bash
# 1. Write scripts/backtest_cN.py following backtest_c3.py as template
# 2. Run improvement_harness.py which tests on 5 stocks:
./venv/bin/python scripts/improvement_harness.py --script scripts/backtest_cN.py

# Pass gate: mean dir_accuracy > 43.5% AND up_precision > 54%
# If PASSED: update FEATURE_COLUMNS in models/predictor.py, add row to docs/improvements_log.md
# If FAILED: do NOT touch models/predictor.py, log result only
```

---

## Quick test commands
```bash
# Predict
curl http://localhost:7861/api/stock/2330/predict | python3 -m json.tool | head -20

# Backtest
curl "http://localhost:7861/api/stock/2330/backtest?months=6" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['stats'])"

# Queue status
./venv/bin/python scripts/hf_queue_worker.py status

# Force-push to queue
./venv/bin/python scripts/hf_queue_worker.py push 2330 0050

# Run frontend dev server
cd frontend && npm run dev
```