ROADMAP.md

Blum AI Financial Intelligence Roadmap

This roadmap is the execution plan for turning Blum into a credible open-source AI financial intelligence platform. Each phase should ship with working code, API coverage, UI coverage, documentation and explicit limitations.

Current Increment - Live Intelligence Runtime

Shipped in the current architecture:

• FastAPI startup launches a background APScheduler worker.
• Startup pipeline ingests public news first, then historical OHLCV, signals and ETF trends.
• Public news ingestion combines publisher RSS feeds, thematic Google News RSS queries and asset-specific public web-search RSS queries.
• /news/live, /sentiment/market and /pipeline/status expose live news, market sentiment and worker state.
• Dashboard polls live endpoints every 30 seconds and surfaces news tape, sentiment mix, source/model state and signal readiness.
• Price pipeline remains real-data-only through yfinance, Yahoo Chart API and Stooq; no synthetic OHLCV fallback is allowed.

Phase 0 - Stabilize Docker Space Deployment

Goal: make the Hugging Face Docker Space build and boot reliably.

Deliverables:

• Confirm Docker build succeeds on Hugging Face Spaces.
• Verify FastAPI serves the exported Next.js frontend on port 7860.
• Verify embedded PostgreSQL starts when DATABASE_URL is not provided.
• Verify /health, /docs, /assets and /dashboard/overview.
• Add lightweight demo mode if model dependencies exceed Space resources.
• Add clear build/runtime troubleshooting notes.

Exit criteria:

• Public Space loads without manual intervention.
• API docs are reachable.
• Seed asset universe is visible.
• No Gradio remnants remain in runtime metadata.

Phase 1 - Data Ingestion Reliability

Goal: make prices, assets and news ingestion reliable enough for a serious demo.

Deliverables:

• Harden yfinance provider with retries, partial failures and provider status.
• Keep real-data-only behavior: no synthetic OHLCV fallback in production or demo paths.
• Maintain a public provider chain for prices: yfinance, Yahoo Chart API and Stooq.
• Add incremental OHLCV updates instead of replacing recent rows blindly.
• Improve RSS source health diagnostics.
• Add article text extraction fallback with newspaper3k and BeautifulSoup.
• Add stronger duplicate detection across titles, URLs and canonical keys.
• Add source reliability and stale-data scoring.
• Add ingestion audit logs.

Exit criteria:

• /market/update reports per-ticker success/failure.
• /news/update reports source-level diagnostics.
• Duplicate articles are materially reduced.
• Failed feeds do not break the pipeline.

Phase 2 - AI Model Productionization

Goal: make AI model usage explicit, measurable and robust under limited compute.

Deliverables:

• Add model availability endpoint.
• Add lazy loading and memory-aware fallback for FinBERT, embeddings and LLM.
• Persist model run metadata for every sentiment and AI insight.
• Add prompt templates with strict evidence-only formatting.
• Add deterministic fallback explanations for low-resource mode.
• Add batch sentiment processing for article updates.
• Add configurable model names through Space variables.

Exit criteria:

• /ai/explain/{ticker} returns structured JSON with models used.
• FinBERT is the primary sentiment engine when available.
• VADER is clearly labeled as baseline/fallback.
• No explanation claims facts absent from retrieved evidence.

Phase 3 - Semantic Intelligence Layer

Goal: turn news embeddings into useful narrative intelligence, not just search.

Deliverables:

• Persist FAISS indexes by namespace or rebuild them efficiently at startup.
• Add semantic cluster snapshots in ThemeCluster.
• Link themes to assets, sectors, ETFs and macro drivers.
• Add theme trend over time.
• Add narrative intensity, recurrence and polarization metrics.
• Add /themes/{label} detail endpoint.
• Add UI for theme drill-down and related assets.

Exit criteria:

• Theme Explorer shows real cluster metadata.
• Semantic search returns relevant articles with similarity scores.
• Each high-scoring signal can cite related semantic themes.

Phase 4 - Signal Engine Upgrade

Goal: make the Blum Intelligence Score more defensible and auditable.

Deliverables:

• Split score modules into separate files: momentum, trend, risk, technicals, sentiment, semantics, ETF and anomaly.
• Add score versioning.
• Store full score inputs and normalized factors.
• Add factor weights in config.
• Add confidence score distinct from signal score.
• Add signal lifecycle states: new, confirmed, faded, invalidated.
• Add price/sentiment divergence logic with explicit thresholds.

Exit criteria:

• Every score is reproducible from stored inputs.
• UI can show why each score changed.
• /signals/{ticker} includes score version and factor weights.

Phase 5 - ETF And Sector Intelligence

Goal: make ETF confirmation a first-class intelligence layer.

Deliverables:

• Map stocks to confirming ETFs and sector proxies.
• Add stock/ETF correlation analysis.
• Add ETF rotation rankings by sector and theme.
• Add ETF confirmation score into every asset signal.
• Add sector heatmap and rotation charts.
• Add ETF vs benchmark comparison views.

Exit criteria:

• ETF Radar identifies rotation leaders.
• Asset Detail shows confirming or contradicting ETFs.
• Signal explanations reference ETF confirmation when available.

Phase 6 - Backtesting And Validation

Goal: validate historical signal behavior without implying prediction or advice.

Deliverables:

• Add walk-forward validation by score threshold and classification.
• Add benchmark-relative forward returns.
• Add false positive analysis by signal type.
• Add max adverse/favorable excursion distributions.
• Add result storage by run configuration.
• Add backtest charts: equity curve, drawdown, forward return distribution.
• Add methodology caveats in UI and API.

Exit criteria:

• Backtest page can compare classifications historically.
• API reports hit rate, average forward return and benchmark-relative behavior.
• Disclaimer is visible on every validation output.

Phase 7 - Frontend Intelligence UX

Goal: elevate the UI from technical demo to portfolio-grade case study.

Deliverables:

• Add skeleton loading and refined error states on every page.
• Add richer Plotly charts: RSI, MACD, Bollinger Bands, sentiment timeline, score history and correlation heatmap.
• Add sortable/filterable tables.
• Add advanced Signal Lab filters.
• Add exportable signal results.
• Add responsive refinements for tablet/mobile.
• Add real screenshots generated from a running Space build to README.

Exit criteria:

• Dashboard immediately communicates what to watch and why.
• Asset Detail explains price, sentiment, risk, news and ETF confirmation in one flow.
• UI remains dense but readable.

Phase 8 - Provider Architecture

Goal: make the platform extensible beyond yfinance and RSS.

Deliverables:

• Define provider interfaces for market data, news, filings, transcripts, estimates and ownership.
• Add provider registry.
• Add mock provider for tests.
• Add optional adapters for filings and public company IR pages.
• Prepare connectors for future licensed data sources.

Exit criteria:

• New providers can be added without changing signal code.
• Provider status appears in API and UI.

Phase 9 - Testing, Observability And Quality

Goal: make the codebase credible as an open-source engineering project.

Deliverables:

• Add backend unit tests for indicators, scoring, ingestion and API.
• Add frontend component tests where practical.
• Add smoke tests for Docker startup.
• Add structured logging.
• Add error telemetry fields in API responses.
• Add CI-ready commands in README.

Exit criteria:

• Core scoring and ingestion logic are covered by tests.
• Docker smoke test catches startup regressions.
• Contributors can run checks locally.

Phase 10 - Open Source Polish

Goal: make the repository compelling as a public case study.

Deliverables:

• Add architecture diagram.
• Add screenshots.
• Add contribution guide.
• Add issue templates.
• Add model/data limitation section.
• Add changelog.
• Add clear demo setup instructions for low-resource and full-AI modes.

Exit criteria:

• A reviewer can understand the project, run it, inspect the architecture and contribute without needing private context.

Working Rule

Every roadmap step should ship as a complete increment:

• backend logic;
• API endpoint or schema update when needed;
• frontend visibility when user-facing;
• documentation update;
• verification command;
• explicit limitation or disclaimer when relevant.

All implementation work must follow ENGINEERING_STANDARDS.md: no placeholders, no fabricated data, no synthetic market-data fallback, evidence-bound AI, efficient provider calls and explicit verification.