ARCHITECTURE.md

# Architecture & Component Reference

A precise map of every component, how it is called, what it returns,
and how to replace it in isolation. Read this before making changes.

---

## 1. System Overview

```
Browser / Vercel (xenex-ai.io)
        │
        ├── GET /quick/AAPL  ──────────────────────────────────────────────────────┐
        │   (via Next.js rewrite → HF Space)                                       │
        │                                                                          ▼
        │                                                              HF Space (FastAPI)
        │                                                                  backend/main.py
        │                                                                          │
        └── EventSource /stream/AAPL  ──────────────────────────────────────────► │
            (direct browser → HF, bypasses Vercel 30s timeout)                    │
                                                                                   │
                                                      ┌────────────────────────────┘
                                                      │
                                              orchestrator.py
                                                      │
                                    ┌─────────────────┼──────────────────┐
                                    │                 │                  │
                                 Phase 1           Phase 2           Phase 3
                               DataAgent     Tech + Fund          Report + Risk
                              (sequential)     (parallel)         (sequential)
```

---

## 2. Entry Points

### `GET /quick/{ticker}` — Fast Signal (no LLM)

**File:** [backend/main.py](backend/main.py) → `quick_signal()`

**Calls:** `tools/tool_signal_fusion.py` → `fuse_signals(ticker)`

**Returns:**
```json
{
  "ticker": "AAPL",
  "recommendation": "HOLD",
  "confidence": 46,
  "final_score": 0.028,
  "current_price": 260.86,
  "breakdown": { "ml": {...}, "technical": {...}, "fundamental": {...}, "sentiment": {...} },
  "risk_flags": [],
  "price_series": {...},
  "ma20": ..., "ma50": ..., "bb_upper": ..., "bb_lower": ...
}
```

**Cache:** 15 minutes in-memory (`_QUICK_CACHE`). TTL = `_QUICK_TTL`.

**To replace:** Swap `fuse_signals()` in `tool_signal_fusion.py`. The response shape must keep `recommendation`, `confidence`, `current_price`, `breakdown`.

---

### `GET /stream/{ticker}` — Deep 6-Agent Analysis (SSE)

**File:** [backend/main.py](backend/main.py) → `stream_analysis()`

**Calls:** `orchestrator.py` → `run_analysis_stream(ticker, user_id)`

**Returns:** Server-Sent Events stream. Each event:
```json
{"agent": "DataAgent", "content": "...", "done": false}
```
Final event:
```json
{"agent": "__complete__", "content": "<full report>", "recommendation": "BUY", "confidence": 72, "done": true}
```
Followed by `data: [DONE]`.

**Cache:** 4 hours in-memory (`_STREAM_CACHE`). Cached runs replay instantly — no LLM called.

**Rate limit:** 1 real LLM run per IP per 20 min (`_IP_RATE_WINDOW`). Bypass with `?bypass=<DEV_BYPASS_KEY>`.

**To replace:** Swap `run_analysis_stream()` in `orchestrator.py`. Keep the SSE event shape.

---

### `GET /portfolio` — All Holdings Quick Signals

**File:** [backend/main.py](backend/main.py) → `get_portfolio()`

**Calls:** `fuse_signals()` for each ticker in `config/portfolio.py` via `ThreadPoolExecutor(max_workers=4)`.

**To replace:** Change holdings in `config/portfolio.py`. Signal logic unchanged.

---

### `GET /health` — Dependency Health Check

**File:** [backend/main.py](backend/main.py) → `health_check()`

**Tests:** NewsAPI reachable, ML model file loadable, `fuse_signals()` end-to-end.

**Does NOT test:** Groq key validity, LLM reachability. ← Known gap.

**Cache:** 30 minutes in-memory (`_HEALTH_CACHE`).

---

## 3. Orchestrator — 3-Phase Pipeline

**File:** [orchestrator.py](orchestrator.py)

Entry: `run_analysis_stream(ticker, user_id)` → calls `_run_swarm_stream()`.

```
Phase 1 ─ DataAgent           _run_data_phase()
Phase 2 ─ Tech + Fund         _run_analysts_parallel()     asyncio.gather()
Phase 3 ─ Report + Risk       _run_report_phase()          sequential
```

### Phase 1 — Data Collection

```python
agent = build_data_agent(ctx, memory_context)
agent.run_stream(task=...)
```

DataAgent calls all 8 tools sequentially (LLM decides order from prompt).
Output: `data_summary` string passed to Phase 2.

### Phase 2 — Parallel Analysis

```python
tech_resp, fund_resp = await asyncio.gather(
    tech_agent.on_messages(msg, token),
    fund_agent.on_messages(msg, token),
)
```

Both analysts receive `data_summary` as input. No tools — reasoning only.
Output: `tech_out`, `fund_out` strings passed to Phase 3.

### Phase 3 — Report + Risk (sequential, no Swarm)

```python
report_resp = await report_agent.on_messages([...combined...], token)
risk_resp   = await risk_agent.on_messages([...combined + report...], token)
```

**Why no Swarm here:** Groq rejects AutoGen's handoff tool JSON schema
(`required` present but `properties` missing). Running sequentially via
`on_messages()` avoids generating handoff tools entirely.

---

## 4. Agents

All agents are AutoGen `AssistantAgent`. Built by factory functions in `agents/`.

| Agent | File | Model tier | Tools | Handoffs | Role |
|---|---|---|---|---|---|
| DataAgent | `agents/data_agent.py` | `data` (cheap) | 8 tools | none | Fetch all data |
| TechnicalAnalyst | `agents/technical_agent.py` | `reasoning` | none | none | Technical JSON |
| FundamentalAnalyst | `agents/fundamental_agent.py` | `reasoning` | none | none | Fundamental JSON |
| ReportWriter | `agents/report_agent.py` | `analysis` | none | none | Full markdown report |
| RiskAgent | `agents/risk_agent.py` | `reasoning` | none | none | Risk review |
| PortfolioAgent | `agents/portfolio_agent.py` | `reasoning` | none | RiskAgent | Optional — disabled |

**Why handoffs are all empty:**
Groq's API rejects AutoGen's auto-generated handoff tool schemas. All routing
is done by the orchestrator directly. Never add `handoffs=[...]` unless
you confirm the LLM provider accepts the schema.

**To replace an agent:** Edit the `build_*` factory in its file. Change the
`system_message` for behaviour, `model_client` for the LLM tier, `tools`
for capabilities. The orchestrator does not need to change.

---

## 5. LLM Model Tiers

**File:** [config/settings.py](config/settings.py) → `get_model_client(task)`

| Task | Tier | Default model | Used by |
|---|---|---|---|
| `"data"` | cheap / fast | `llama-3.1-8b-instant` | DataAgent |
| `"reasoning"` | mid | `llama-3.3-70b-versatile` | TechAnalyst, FundAnalyst, RiskAgent |
| `"analysis"` | best prose | `llama-3.3-70b-versatile` | ReportWriter |

**Priority:** Groq (primary) → Gemini (fallback).

**Key bug fixed:** `GROQ_API_KEY` must be `.strip()`-ed — HF Spaces adds a
trailing `\n` to secrets, which makes HTTP headers illegal. Already fixed in
`config/settings.py` at all 3 read sites.

**To swap provider:** Replace `_analysis_client()`, `_reasoning_client()`,
`_cheap_client()` in `config/settings.py`. Keep returning an
`OpenAIChatCompletionClient`-compatible object.

**To swap model:** Set env vars: `ANALYSIS_MODEL`, `REASONING_MODEL`, `DATA_MODEL`.

---

## 6. Data Tools

All tools live in `tools/`. Called by DataAgent (via LLM tool use) and
`fuse_signals()` (direct Python calls).

| Tool | File | Data source | What it returns |
|---|---|---|---|
| `get_price_history(ticker)` | `tool_price.py` | yfinance via `tool_finnhub.get_history()` | Price series, MAs, Bollinger, RSI, MACD |
| `get_financials(ticker)` | `tool_financial.py` | Finnhub `/stock/metric` + `/profile2` | P/E, margins, debt, growth |
| `get_news_sentiment(ticker)` | `tool_news.py` | Finnhub `/company-news` + FinBERT | Sentiment score, headline count |
| `get_earnings_calendar(ticker)` | `tool_earnings.py` | yfinance | Next earnings date, EPS estimates |
| `compute_signal_ensemble(ticker)` | `tool_signal.py` | yfinance via `tool_finnhub.get_history()` | 6-indicator vote (RSI, MACD, BB, etc.) |
| `forecast_price(ticker)` | `tool_forecast.py` | yfinance via `tool_finnhub.get_history()` | Prophet 30-day forecast + confidence interval |
| `predict_signal(ticker)` | `tool_ml_signal.py` | yfinance via `tool_finnhub.get_history()` | ML BUY/HOLD/SELL + probabilities |
| `fuse_signals(ticker)` | `tool_signal_fusion.py` | All of the above | Weighted fusion score, final recommendation |

### Data Source Split (important)

| Data type | Source | Why |
|---|---|---|
| OHLCV price history | **yfinance** | Finnhub free tier returns 403 on `/stock/candle` |
| Current price (live) | **Finnhub** `/quote` | Faster, more reliable for single tick |
| Company profile / sector | **Finnhub** `/profile2` | Free tier ✅ |
| News articles | **Finnhub** `/company-news` | Free tier ✅ |
| Financial metrics | **Finnhub** `/stock/metric` | Free tier ✅ |
| Sentiment scoring | **FinBERT** (local model) | ProsusAI/finbert, loaded once on startup |

**`tool_finnhub.get_history()`** is a wrapper that calls yfinance internally.
All other tools call `tool_finnhub.get_history()` — so replacing yfinance
means changing only this one function.

---

## 7. ML Pipeline

| File | Purpose |
|---|---|
| `tools/tool_ml_train.py` | Training — fetches 5y data, builds features, trains GBM+RF+LR stack |
| `tools/tool_ml_features.py` | Feature engineering (`_build_features`) and file path constants |
| `tools/tool_ml_signal.py` | Live prediction — loads model, builds features, returns signal |

### Models

| File | What |
|---|---|
| `data/signal_model_stock.pkl` | Stacked classifier (GBM + RF + LR) for individual stocks |
| `data/signal_scaler_stock.pkl` | StandardScaler fitted on training data |
| `data/signal_selector_stock.pkl` | Feature selector (drops low-signal features) |
| `data/signal_model_etf.pkl` | Separate pipeline for ETFs |

### Key constraint

`_build_features()` computes `dist_52w_high` / `dist_52w_low` which need 252 trading days.
**Always fetch `period="2y"` for prediction** — `13mo` is not enough (returns empty features).
Training uses 5y so it's fine.

### Training trigger

`docker-entrypoint.sh` trains models at container startup if pkl files are absent.
HF free tier has no persistent disk — models train on every cold start (~3 min).

**To eliminate training delay:** Pre-train locally, push pkl files to a HF Dataset,
load from there at startup. See "Known Issues" below.

---

## 8. Memory

**File:** [memory/memory_manager.py](memory/memory_manager.py)

- Local: SQLite fallback
- Production: Supabase (set `SUPABASE_URL` + `SUPABASE_KEY`)
- Stores completed analyses per user, used to build `memory_context` string injected into agent prompts

---

## 9. Environment Variables

| Variable | Where used | Required |
|---|---|---|
| `GROQ_API_KEY` | `config/settings.py` — all LLM clients | Yes |
| `GOOGLE_API_KEY` | `config/settings.py` — Gemini fallback | No |
| `FINNHUB_API_KEY` | `tools/tool_finnhub.py` | Yes |
| `NEWS_API_KEY` | `tools/tool_news.py` | Optional (Finnhub news used if missing) |
| `DEMO_MODE` | `backend/main.py` | No (default false) |
| `DEV_BYPASS_KEY` | `backend/main.py` | No (skips IP rate limit if set) |
| `SUPABASE_URL` + `SUPABASE_KEY` | `memory/` | No (SQLite fallback) |
| `NEXT_PUBLIC_BACKEND_URL` | `xenex-ai` Next.js frontend | Yes (Vercel) |
| `MODAL_BACKEND_URL` | `xenex-ai` API routes + admin | Yes (Vercel) |

**HF Spaces secret gotcha:** HF appends `\n` to secrets. All env reads in
`config/settings.py` call `.strip()` to prevent illegal HTTP headers.
Apply `.strip()` everywhere you read an API key from env.

---

## 10. Frontend → Backend Call Map

| What | Frontend code | Next.js layer | HF endpoint |
|---|---|---|---|
| Quick signal | `StockAnalysis.jsx:70` `fetch('/quick/AAPL')` | Rewrite → `NEXT_PUBLIC_BACKEND_URL/quick/AAPL` | `/quick/{ticker}` |
| Deep stream | `AgentStream.jsx:123` `new EventSource(BACKEND + '/stream/AAPL')` | Direct (bypasses Vercel — 30s timeout) | `/stream/{ticker}` |
| Portfolio | Admin page | Next.js proxy | `/portfolio` |
| Health | Admin page | `app/api/admin/health/route.ts` → `MODAL_BACKEND_URL/health` | `/health` |

**Two env var problem:** `NEXT_PUBLIC_BACKEND_URL` (rewrites + AgentStream)
and `MODAL_BACKEND_URL` (API routes + admin) both point to the same HF URL.
If one is wrong, different features fail silently.

---

## 11. Known Issues & Decisions Log

| Issue | Root cause | Fix applied | Date |
|---|---|---|---|
| `NameError: run_analysis_stream` | Import missing in `backend/main.py` | Added import | 2026-04-06 |
| Groq `Illegal header value` | `\n` in HF Spaces secret | `.strip()` on all key reads | 2026-04-06 |
| Groq 400 bad handoff schema | AutoGen generates `required` with no `properties` | Removed all handoffs; Phase 3 runs sequentially | 2026-04-06 |
| `Could not compute features` | `predict_signal` used `13mo` history — not enough for 52w features | Changed to `2y` | 2026-04-06 |
| `No data found for ticker` | `tool_finnhub.get_history()` called Finnhub `/stock/candle` — 403 on free tier | Replaced with yfinance | 2026-04-06 |
| HF proxy 404 after restart | ML training takes 3 min; HF proxy times out before uvicorn starts | Known — train models at build time or use HF Dataset for persistence | Open |
| Rate limit blocks testing | `_IP_LAST_RUN` is in-memory, resets on restart | Added `DEV_BYPASS_KEY` query param | 2026-04-06 |

---

## 12. How to Test Before Deploying

```bash
# Level 1 — tool unit tests only (~30s, no server)
./test_local.sh 1

# Level 2 — tool tests + all endpoints (~60s)
./test_local.sh 2

# Level 3 — full LLM pipeline end-to-end (~2-3 min)
./test_local.sh 3
```

Requires `.env.local` with `GROQ_API_KEY`, `FINNHUB_API_KEY`, `DEV_BYPASS_KEY`.

**Rule:** Always pass Level 2 before pushing to HF.

---

## 13. How to Deploy

```bash
# 1. Test locally
./test_local.sh 2

# 2. Commit to cost-opt
git add <files> && git commit -m "..."

# 3. Rebuild hf-deploy (orphan branch — no binary history)
git checkout hf-deploy
git checkout cost-opt -- .   # or cherry-pick specific commits
git add -A && git commit -m "Deploy: ..."
git push hf hf-deploy:main --force

# 4. Switch back
git checkout cost-opt
```

**Why orphan branch:** HF rejects pushes containing binary files (pkl).
`hf-deploy` is a clean branch with no pkl history.

---

## 14. Replacing Components in Isolation

| Want to replace | Change only | Contract to preserve |
|---|---|---|
| LLM provider (Groq → OpenAI) | `config/settings.py` — `_*_client()` functions | Return `OpenAIChatCompletionClient`-compatible object |
| LLM model | Set `ANALYSIS_MODEL` / `REASONING_MODEL` / `DATA_MODEL` env vars | None |
| Agent behaviour | `agents/<agent>.py` system_message | Keep agent name, do not add handoffs |
| Price history source | `tools/tool_finnhub.py` → `get_history()` | Return `pd.DataFrame` with `Open, High, Low, Close, Volume` index DatetimeIndex tz-naive |
| News source | `tools/tool_news.py` | Return `{"score": float, "label": str, "articles": [...]}` |
| ML model architecture | `tools/tool_ml_train.py` + `tool_ml_features.py` | Saved as joblib, loaded in `tool_ml_signal.py` via `STOCK_MODEL_PATH` |
| Signal fusion weights | `tools/tool_signal_fusion.py` → `BASE_WEIGHTS` | Must sum to 1.0; keys: ml, technical, fundamental, sentiment |
| Memory backend | `memory/backends.py` | Implement `save()`, `get_history()`, `get_all_history()` |
| Rate limiting | `backend/main.py` → `_IP_LAST_RUN` block | Return 429 with `{"detail": "Rate limit exceeded", "retry_after": N}` |