PIPELINE.md

Quantitative Portfolio Engine: Full Pipeline Architecture

This document provides a comprehensive, analytical breakdown of the system pipeline. It covers the complete journey from user input on the Web UI, through the AI and Data processing layers, down to the core Mathematical Convex Optimization, and back to the HTML reporting stage.

1. Web Application & AI Orchestration Pipeline

1.1 Web UI (index.html & app.js)

The frontend is a vanilla Javascript application focused on performance and dynamic state management.

• Form State Management: Captures tickers, capital, risk_input, base_currency ($, €, £, ¥, CHF), model selections, and flags like allow_shorting.
• AI Strategy Generator (generateAIStrategy): A conversational interface (NOVA) that translates natural language into a JSON configuration for the engine. It parses outputs like base_currency: 'EUR' and seamlessly injects them into the UI dropdowns.
• WebSocket / Server-Sent Events (SSE): Streaming interface for real-time portfolio generation logs (e.g., fetching data, optimizing weights).
• Vision Integration: Allows uploading screenshots of charts or macro conditions. The image is Base64 encoded and sent to the multimodal AI endpoint for contextual strategy configuration.

1.2 Backend API Routing (app.py)

Built on FastAPI, the backend acts as the central conductor.

• Authentication (access_manager.py): Utilizes X-Access-Key headers to validate users, enforce rate limits, and map permissions (e.g., MASTER_KEY grants up to 8000 context tokens and circumvents UI restrictions).
• /api/generate: Intercepts the UI request, initiates a background trace (BACKGROUND_TASKS), and executes core_engine.run_engine. It implements a robust circuit breaker: if deployed on a Serverless host (like Render), it intelligently proxies the heavy mathematical computation to a Hugging Face Hub inference container.
• /api/chat (AI Engine Room): Integrates with HuggingFace Hub Serverless Inference. It dynamically injects current macro conditions (^TNX, ^VIX) and system time into the AI's system prompt to maintain accurate contextual awareness of the market.

2. Data Ingestion & Transformation

2.1 Fetching & Alignment (data.py & data_repository.py)

• Primary Source: yfinance fetches daily adjusted close prices.
• Circuit Breaker: If yfinance times out or is rate-limited, the system falls back to pandas_datareader via Stooq.
• Currency Normalization: The guess_currency function determines the base denomination of each asset (e.g., .AT for EUR, .L for GBP). The Engine dynamically fetches necessary FX pairs (e.g., EURUSD=X or GBPEUR=X) to geometrically align all asset prices into the user-selected base_currency.
• Missing Value Imputation: Trailing NaNs are forward-filled, and leading NaNs are backward-filled to prevent mismatched tensor shapes during covariance calculations.

3. Mathematical Risk & Optimization Pipeline

flowchart TD
    %% Define external data sources
    db[(PostgreSQL / SQLite)]
    yfinance[("Yahoo Finance (API)")]
    
    %% Ingestion Layer
    subgraph Data Layer ["Data Ingestion & Pre-Processing"]
        yfinance --> |"Raw Pricing + FX Pairs"| df_fetch(fetch_data)
        df_fetch --> cleaning[Currency Normalization & Imputation]
        cleaning --> returns[Log/Linear Daily Returns]
    end

    %% Modeling Layer
    subgraph Quant Models ["Risk & Return Modeling"]
        returns --> ewma[Trailing EWMA Covariance]
        returns --> garch[GARCH Volatility Regime]
        returns --> ff[Fama-French Factor Betas]
        returns --> hmm[HMM Regime Detection]
        ewma --> rmt[RMT Noise Filtering]
    end

    %% Optimization Layer
    subgraph Solver Engine ["Convex Optimization (cvxpy)"]
        direction TB
        rmt --> cov[Clean Covariance Matrix]
        garch --> cov
        cov --> cvx_setup[Build CVX Objective]
        ff --> expected_rets[Calculate Expected Returns]
        expected_rets --> cvx_setup
        cvx_setup --> constraints[Apply Bounds, Sectors, Turnover, CVaR Limits]
        constraints --> cvx_solve[Solve SCS / ECOS]
        cvx_solve --> target_weights[Target Asset Weights]
    end

    %% Returns to Pipeline
    returns --> Solver Engine

3.1 Regime Detection & Covariance Filtering (models.py)

• HMM Regime Detection: A Gaussian Hidden Markov Model detects market states (e.g., "Crash / High Volatility" vs "Bull Market").
• EWMA Covariance: Instead of a static sample covariance, the engine utilizes an Exponentially Weighted Moving Average (EWMA) covariance matrix.
• Random Matrix Theory (RMT): Eigenvalue clipping (Marchenko-Pastur distribution) filters out statistical noise from the covariance matrix, producing a mathematically robust target for the solver.

3.2 Target Returns (Black-Litterman & Fama-French)

• Black-Litterman: Blends subjective views with market equilibrium priors using a Bayesian framework, outputting an adjusted posterior expected return vector.
• Fama-French: Regresses asset returns against factors (Mkt-RF, SMB, HML) to decompose expected returns based on historical factor loadings.

3.3 Convex Optimization (solver.py)

Powered by cvxpy and solved via SCS/ECOS.

• Objective Function: Maximize Sharpe (Mean-Variance), Minimize CVaR, or Equal Risk Contribution (ERC).
• Constraints:
  • Risk Limits (CVaR): Hard constraints ensuring the 95% Conditional Value at Risk does not breach user tolerances.
  • Bounds: Bounds weights dynamically based on the allow_shorting flag.
  • Turnover: Minimizes unnecessary trading friction relative to the current portfolio state.

4. Execution, Simulation & HTML Reporting

4.1 Post-Trade Simulation (report_data.py)

• HIFO Tax Optimization: If enabled, highest-cost lots are liquidated first to defer capital gains tax, directly modeling the tax drag on the equity curve.
• Almgren-Chriss Market Impact: Calculates slippage based on trading volume, executing a penalty on high-turnover trades in illiquid assets.
• Monte Carlo & Deflated Sharpe: Stresses the resulting portfolio with 10,000 MC paths and applies a Deflated Sharpe Ratio to check against overfitting and multiple testing bias.

4.2 Report Generation

The engine formats the results using isolated HTML builders (e.g., html_performance.py, html_risk.py) which are then synthesized into portfolio_report.html. This output contains all Chart.js structures, mathematical badges, and the interactive UI components ready for consumption.