# Blum AI Financial Intelligence Roadmap

This roadmap is the execution plan for turning Blum into a credible open-source AI financial intelligence platform. Each phase should ship with working code, API coverage, UI coverage, documentation and explicit limitations.

## Current Increment - Live Intelligence Runtime

Shipped in the current architecture:

- FastAPI startup launches a background APScheduler worker.
- Startup pipeline ingests public news first, then historical OHLCV, signals and ETF trends.
- Public news ingestion combines publisher RSS feeds, thematic Google News RSS queries and asset-specific public web-search RSS queries.
- `/news/live`, `/sentiment/market` and `/pipeline/status` expose live news, market sentiment and worker state.
- Dashboard polls live endpoints every 30 seconds and surfaces news tape, sentiment mix, source/model state and signal readiness.
- Price pipeline remains real-data-only through yfinance, Yahoo Chart API and Stooq; no synthetic OHLCV fallback is allowed.

## Phase 0 - Stabilize Docker Space Deployment

Goal: make the Hugging Face Docker Space build and boot reliably.

Deliverables:

- Confirm Docker build succeeds on Hugging Face Spaces.
- Verify FastAPI serves the exported Next.js frontend on port `7860`.
- Verify embedded PostgreSQL starts when `DATABASE_URL` is not provided.
- Verify `/health`, `/docs`, `/assets` and `/dashboard/overview`.
- Add lightweight demo mode if model dependencies exceed Space resources.
- Add clear build/runtime troubleshooting notes.

Exit criteria:

- Public Space loads without manual intervention.
- API docs are reachable.
- Seed asset universe is visible.
- No Gradio remnants remain in runtime metadata.

## Phase 1 - Data Ingestion Reliability

Goal: make prices, assets and news ingestion reliable enough for a serious demo.

Deliverables:

- Harden yfinance provider with retries, partial failures and provider status.
- Keep real-data-only behavior: no synthetic OHLCV fallback in production or demo paths.
- Maintain a public provider chain for prices: yfinance, Yahoo Chart API and Stooq.
- Add incremental OHLCV updates instead of replacing recent rows blindly.
- Improve RSS source health diagnostics.
- Add article text extraction fallback with `newspaper3k` and BeautifulSoup.
- Add stronger duplicate detection across titles, URLs and canonical keys.
- Add source reliability and stale-data scoring.
- Add ingestion audit logs.

Exit criteria:

- `/market/update` reports per-ticker success/failure.
- `/news/update` reports source-level diagnostics.
- Duplicate articles are materially reduced.
- Failed feeds do not break the pipeline.

## Phase 2 - AI Model Productionization

Goal: make AI model usage explicit, measurable and robust under limited compute.

Deliverables:

- Add model availability endpoint.
- Add lazy loading and memory-aware fallback for FinBERT, embeddings and LLM.
- Persist model run metadata for every sentiment and AI insight.
- Add prompt templates with strict evidence-only formatting.
- Add deterministic fallback explanations for low-resource mode.
- Add batch sentiment processing for article updates.
- Add configurable model names through Space variables.

Exit criteria:

- `/ai/explain/{ticker}` returns structured JSON with models used.
- FinBERT is the primary sentiment engine when available.
- VADER is clearly labeled as baseline/fallback.
- No explanation claims facts absent from retrieved evidence.

## Phase 3 - Semantic Intelligence Layer

Goal: turn news embeddings into useful narrative intelligence, not just search.

Deliverables:

- Persist FAISS indexes by namespace or rebuild them efficiently at startup.
- Add semantic cluster snapshots in `ThemeCluster`.
- Link themes to assets, sectors, ETFs and macro drivers.
- Add theme trend over time.
- Add narrative intensity, recurrence and polarization metrics.
- Add `/themes/{label}` detail endpoint.
- Add UI for theme drill-down and related assets.

Exit criteria:

- Theme Explorer shows real cluster metadata.
- Semantic search returns relevant articles with similarity scores.
- Each high-scoring signal can cite related semantic themes.

## Phase 4 - Signal Engine Upgrade

Goal: make the Blum Intelligence Score more defensible and auditable.

Deliverables:

- Split score modules into separate files: momentum, trend, risk, technicals, sentiment, semantics, ETF and anomaly.
- Add score versioning.
- Store full score inputs and normalized factors.
- Add factor weights in config.
- Add confidence score distinct from signal score.
- Add signal lifecycle states: new, confirmed, faded, invalidated.
- Add price/sentiment divergence logic with explicit thresholds.

Exit criteria:

- Every score is reproducible from stored inputs.
- UI can show why each score changed.
- `/signals/{ticker}` includes score version and factor weights.

## Phase 5 - ETF And Sector Intelligence

Goal: make ETF confirmation a first-class intelligence layer.

Deliverables:

- Map stocks to confirming ETFs and sector proxies.
- Add stock/ETF correlation analysis.
- Add ETF rotation rankings by sector and theme.
- Add ETF confirmation score into every asset signal.
- Add sector heatmap and rotation charts.
- Add ETF vs benchmark comparison views.

Exit criteria:

- ETF Radar identifies rotation leaders.
- Asset Detail shows confirming or contradicting ETFs.
- Signal explanations reference ETF confirmation when available.

## Phase 6 - Backtesting And Validation

Goal: validate historical signal behavior without implying prediction or advice.

Deliverables:

- Add walk-forward validation by score threshold and classification.
- Add benchmark-relative forward returns.
- Add false positive analysis by signal type.
- Add max adverse/favorable excursion distributions.
- Add result storage by run configuration.
- Add backtest charts: equity curve, drawdown, forward return distribution.
- Add methodology caveats in UI and API.

Exit criteria:

- Backtest page can compare classifications historically.
- API reports hit rate, average forward return and benchmark-relative behavior.
- Disclaimer is visible on every validation output.

## Phase 7 - Frontend Intelligence UX

Goal: elevate the UI from technical demo to portfolio-grade case study.

Deliverables:

- Add skeleton loading and refined error states on every page.
- Add richer Plotly charts: RSI, MACD, Bollinger Bands, sentiment timeline, score history and correlation heatmap.
- Add sortable/filterable tables.
- Add advanced Signal Lab filters.
- Add exportable signal results.
- Add responsive refinements for tablet/mobile.
- Add real screenshots generated from a running Space build to README.

Exit criteria:

- Dashboard immediately communicates what to watch and why.
- Asset Detail explains price, sentiment, risk, news and ETF confirmation in one flow.
- UI remains dense but readable.

## Phase 8 - Provider Architecture

Goal: make the platform extensible beyond yfinance and RSS.

Deliverables:

- Define provider interfaces for market data, news, filings, transcripts, estimates and ownership.
- Add provider registry.
- Add mock provider for tests.
- Add optional adapters for filings and public company IR pages.
- Prepare connectors for future licensed data sources.

Exit criteria:

- New providers can be added without changing signal code.
- Provider status appears in API and UI.

## Phase 9 - Testing, Observability And Quality

Goal: make the codebase credible as an open-source engineering project.

Deliverables:

- Add backend unit tests for indicators, scoring, ingestion and API.
- Add frontend component tests where practical.
- Add smoke tests for Docker startup.
- Add structured logging.
- Add error telemetry fields in API responses.
- Add CI-ready commands in README.

Exit criteria:

- Core scoring and ingestion logic are covered by tests.
- Docker smoke test catches startup regressions.
- Contributors can run checks locally.

## Phase 10 - Open Source Polish

Goal: make the repository compelling as a public case study.

Deliverables:

- Add architecture diagram.
- Add screenshots.
- Add contribution guide.
- Add issue templates.
- Add model/data limitation section.
- Add changelog.
- Add clear demo setup instructions for low-resource and full-AI modes.

Exit criteria:

- A reviewer can understand the project, run it, inspect the architecture and contribute without needing private context.

## Working Rule

Every roadmap step should ship as a complete increment:

- backend logic;
- API endpoint or schema update when needed;
- frontend visibility when user-facing;
- documentation update;
- verification command;
- explicit limitation or disclaimer when relevant.

All implementation work must follow [`ENGINEERING_STANDARDS.md`](ENGINEERING_STANDARDS.md): no placeholders, no fabricated data, no synthetic market-data fallback, evidence-bound AI, efficient provider calls and explicit verification.