# Pipeline Architecture ## Abstract The Portfolio Engine implements a multi-stage pipeline that transforms raw market data into optimised portfolio allocations, validated through rigorous out-of-sample econometric testing, and exported as interactive reports. This document describes the full execution flow, the data structures that mediate inter-stage communication, the mathematical validation framework, and the report generation subsystem. It serves as the architectural reference for understanding how the engine's components compose into a coherent analytical system. --- ## 1. Pipeline Overview The engine is orchestrated by the `PortfolioPipeline` class in `core_engine.py`, which implements a four-stage execution model: ``` ┌────────────────────────────────────────────────────────────────────────┐ │ Pipeline Stages │ │ │ │ ┌───────────┐ ┌────────────────┐ ┌───────────┐ ┌───────────┐ │ │ │ Stage 1 │──▶│ Stage 2 │──▶│ Stage 3 │──▶│ Stage 4 │ │ │ │ load_data()│ │run_validation()│ │ optimize() │ │ reports() │ │ │ └───────────┘ └────────────────┘ └───────────┘ └───────────┘ │ │ │ │ Data Fetch Walk-Forward CV Full-Sample HTML + CSV │ │ Regime Detect Econometric Tests Optimisation PDF Export │ │ Risk Aversion DM / Christoffersen Sensitivity Serve │ │ Adjustment PSR / DSR Stress Test │ └────────────────────────────────────────────────────────────────────────┘ ``` ### Entry Point ```python def run_engine(overrides=None): pipeline = PortfolioPipeline(overrides=overrides) pipeline.load_data() val_bundle = pipeline.run_validation() opt_bundle = pipeline.optimize() pipeline.generate_reports(val_bundle, opt_bundle) ``` The `overrides` dictionary enables headless execution from the API layer (`api.py`), test harnesses, or scheduled batch jobs, bypassing the interactive CLI wizard. --- ## 2. Stage 1 — Data Loading (`load_data`) ### 2.1 Data Sources | Source | Target | Module | |---------------------------|-----------------------|---------------------| | Yahoo Finance / DB Cache | Daily OHLCV prices | `data.py` | | Kenneth French Library | Fama-French factors | `data.py` | | FRED / ^TNX proxy | Risk-free rate series | `data.py` | | PostgreSQL / SQLite | Cached price data | `database.py` | ### 2.2 Data Validation - **Minimum History:** Assets must have ≥ 2× `trading_days_per_year` (default: 504 business days) of return history to be included. Assets with insufficient history are silently dropped. - **Missing Data:** Returns DataFrames are constructed via `pd.DataFrame.dropna()`, ensuring a common date index across all assets. - **Frequency Conversion:** When `return_frequency = 'monthly'`, daily returns are geometrically compounded to monthly via `build_monthly_returns()`. ### 2.3 Regime Detection If `hmm_regime = True` (default), the engine fits a Hidden Markov Model to benchmark returns via `regime_detection.detect_volatility_regime()`. The detected regime (Bull, Normal, Crash) informs: - Dynamic risk aversion adjustment (Stage 2 and 3). - PID volatility target in the research cybernetic ensemble. - Report visualisation annotations. ### 2.4 Dynamic Risk Aversion If `dynamic_risk = True` (default), the VIX level is used to adjust the user's stated risk aversion via `regime_detection.dynamic_risk_aversion()`. This implements a counter-cyclical risk management policy: risk aversion increases during high-volatility episodes, reducing exposure before drawdowns deepen. --- ## 3. Stage 2 — Walk-Forward Validation (`run_validation`) ### 3.1 Expanding Window Cross-Validation The engine performs expanding-window (walk-forward) backtesting via `backtest.expanding_window_backtest()`: 1. An initial training window of `OOS_TRAIN_DAYS` (total days − 252) is established. 2. The model is trained on the expanding window and produces out-of-sample weights. 3. Weights are rebalanced every `trading_days / 4` periods (quarterly). 4. An out-of-sample equity curve is constructed from realised returns. This methodology prevents look-ahead bias and is the gold standard for strategy validation in quantitative finance (Bailey et al., 2014). ### 3.2 Econometric Tests The validation stage runs four statistical tests on the out-of-sample returns: #### Christoffersen Conditional Coverage Test Tests whether Value-at-Risk (VaR) exceedances are both correctly calibrated (unconditional coverage) and serially independent (no volatility clustering in violations). A joint likelihood ratio statistic is computed: ``` LR_cc = LR_uc + LR_ind ~ χ²(2) ``` **Pass Criterion:** p-value > 0.05 for both components. #### Diebold-Mariano Test Tests whether the engine's expected return model statistically outperforms a naive historical mean baseline in terms of out-of-sample prediction accuracy: ``` DM = d̄ / σ̂(d) ~ N(0, 1) ``` where d_t = |e₁_t| − |e₂_t| is the loss differential (MAE loss function). The test is robust to heteroskedasticity via Newey-West variance estimation. **Pass Criterion:** p-value < 0.05 and the engine's model wins. #### Probabilistic Sharpe Ratio (PSR) Accounts for the non-normality of returns (skewness and kurtosis) when evaluating whether the observed Sharpe ratio is statistically distinguishable from a benchmark value of zero (Bailey & López de Prado, 2012): ``` PSR = Φ[(SR − SR*) · √(n-1) / √(1 − γ₃·SR + (γ₄−1)/4 · SR²)] ``` where γ₃ and γ₄ are the sample skewness and kurtosis. **Pass Criterion:** PSR > 0.95 (95% confidence that the true Sharpe exceeds zero). #### Deflated Sharpe Ratio (DSR) Adjusts for multiple testing bias when the engine evaluates K candidate models (Bailey & López de Prado, 2014). The expected maximum Sharpe ratio under the null hypothesis (all models have zero alpha) is: ``` E[max(SR)] ≈ √(2·ln(K)) − [γ + ln(π/2)] / [2·√(2·ln(K))] ``` The DSR then tests whether the observed Sharpe significantly exceeds this multiple-testing threshold. **Pass Criterion:** DSR > 0.95. ### 3.3 Output The validation stage produces a `ValidationBundle` dataclass: ```python @dataclass class ValidationBundle: oos_eq: pd.Series # Out-of-sample equity curve oos_bench_curve: pd.Series # Benchmark equity curve oos_port_rets: pd.Series # Out-of-sample portfolio returns wf_ann_ret: float # Walk-forward annualised return var_results: dict # Christoffersen test results dm_results: dict # Diebold-Mariano test results psr_results: dict # Probabilistic Sharpe Ratio dsr_results: dict # Deflated Sharpe Ratio ``` --- ## 4. Stage 3 — Full-Sample Optimisation (`optimize`) ### 4.1 Solver Invocation The full historical dataset is passed to `solver.build_and_optimize()`, which: 1. Computes expected returns using the selected model (CAPM, BL, Fama-French, Bayesian, or ML Stacking). 2. Estimates the covariance matrix with Ledoit-Wolf shrinkage and optional GARCH scaling. 3. Formulates and solves the convex optimisation problem via the CVXPY engine. 4. Applies the 7-stage constraint relaxation cascade if the initial formulation is infeasible (see `docs/RELAXATION_CASCADE.md`). ### 4.2 Sensitivity & Stress Analysis Post-optimisation, the engine runs two diagnostic analyses: - **Sensitivity Analysis** (`analytics.portfolio_sensitivity`): Perturbs expected returns by ±10% and re-solves, measuring the weight response range per asset. Assets with >15pp swings are flagged as "fragile." - **Stress Testing** (`analytics.portfolio_stress_test`): Evaluates portfolio impact under historical crash scenarios (e.g., 2008 GFC, 2020 COVID, rate shock, tech crash). If fragile allocations are detected and the allocation engine is Mean-Variance (engine 1), a stability penalty is added to the objective function and the solver is re-invoked. ### 4.3 Output ```python @dataclass class OptimizationBundle: weights: pd.Series # Final target weights exp_rets: pd.Series # Expected returns per asset cov_mat: pd.DataFrame # Covariance matrix vol: float # Portfolio volatility corr_matrix: pd.DataFrame # Correlation matrix betas: pd.Series # Market betas model_info: dict # Model metadata sens_report: dict # Sensitivity analysis stress_report: dict # Stress test results n_fragile: int # Count of fragile allocations ``` --- ## 5. Stage 4 — Report Generation (`generate_reports`) ### 5.1 Architecture Report generation follows a three-layer architecture: ``` ┌─────────────────────────────────────────────────┐ │ report.py (Orchestrator) │ │ Coordinates data → template → file pipeline │ ├────────────────┬────────────────────────────────┤ │ report_data.py │ report_html.py │ │ (Data Layer) │ (Rendering Layer) │ │ Formats all │ Injects variables into │ │ mathematical │ report_template.html │ │ outputs into │ static template │ │ template vars │ │ └────────────────┴────────────────────────────────┘ ``` ### 5.2 Report Data Layer — `report_data.py` The `prepare_template_variables()` function is the largest single function in the codebase (~675 lines). It transforms raw mathematical outputs into presentation-ready HTML fragments and Chart.js data payloads. Key computations include: - **Advanced Risk Metrics:** CVaR (95%), Conditional Drawdown-at-Risk (CDaR), Mean Absolute Deviation (MAD), and semi-deviation. - **Transition Comparisons:** When the user provides current holdings, the report computes before/after comparisons for all metrics. - **Chart Payload:** A JSON dictionary consumed by Chart.js for interactive equity curves, allocation pie charts, efficient frontier plots, Monte Carlo fan charts, and risk contribution bar charts. - **Narrative Generation:** `narrative.py` produces a natural-language summary of the portfolio strategy, market conditions, and key risk factors. ### 5.3 HTML Rendering — `report_html.py` The rendering layer substitutes template variables into `report_template.html`, a 26KB static template with Chart.js initialisation scripts. The template uses CSS-in-HTML styling with a dark theme optimised for screen presentation. ### 5.4 Export Formats | Format | Module | Content | |----------|----------------|--------------------------------------------| | HTML | `report.py` | Interactive report with Chart.js | | PDF | `exports.py` | Static rendering via headless browser | | CSV | `exports.py` | Tabular weight/allocation summary | | Excel | `exports.py` | Multi-sheet workbook (optional) | --- ## 6. Data Flow Diagram ``` External APIs ──▶ data.py ──▶ PostgreSQL/SQLite │ ┌──────┴──────┐ │ core_engine │ │ load_data() │ └──────┬───────┘ │ ┌───────────┼───────────┐ ▼ ▼ ▼ solver.py backtest.py validation.py │ │ │ ▼ ▼ ▼ OptBundle ValBundle Test Results │ │ │ └─────┬─────┘───────────┘ ▼ report_data.py ──▶ report_html.py │ ▼ output/*.html output/*.csv output/*.pdf ``` --- ## 7. Configuration-Driven Behaviour The pipeline's behaviour is heavily parameterised via `config.py`. Key configuration axes include: | Parameter | Effect | |--------------------------|-------------------------------------------------------| | `model` (1–7) | Selects expected return model (see `docs/MODELS.md`) | | `allocation_engine` (1–3)| Mean-Variance (CVXPY), HRP, or Exact Risk Parity (see `docs/ALLOCATION_ENGINES.md`) | | `max_assets` | Cardinality constraint: max number of non-zero positions | | `garch_enabled` | Enables GARCH(1,1) covariance scaling | | `cvar_enabled` | Adds CVaR tail-risk constraint to CVXPY formulation | | `tax_enabled` | Activates tax-aware optimisation with cost-basis tracking | | `hmm_regime` | Enables HMM regime detection | | `dynamic_risk` | Enables VIX-based risk aversion adjustment | | `with_futures` | Enables futures overlay optimisation | | `return_frequency` | Daily or monthly return aggregation | --- ## 8. Error Handling & Graceful Degradation The pipeline employs multiple fallback mechanisms: 1. **Constraint Relaxation Cascade:** 7-stage progressive constraint relaxation (see `RELAXATION_CASCADE.md`). 2. **Data Fallback:** If PostgreSQL is unreachable, the engine falls back to local SQLite. 3. **Model Fallback:** If ML ensemble training fails, the engine falls back to CAPM. 4. **Report Fallback:** If PDF export fails (no headless browser), only HTML is generated. These mechanisms ensure the pipeline always produces output, even under degraded conditions. --- ## References - Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-mathematics and financial charlatanism: The effects of backtest overfitting on out-of-sample performance. *Notices of the AMS*, 61(5), 458–471. - Bailey, D. H., & López de Prado, M. (2012). The Sharpe ratio efficient frontier. *Journal of Risk*, 15(2), 3–44. - Bailey, D. H., & López de Prado, M. (2014). The deflated Sharpe ratio: Correcting for selection bias, backtest overfitting, and non-normality. *Journal of Portfolio Management*, 40(5), 94–107. - Christoffersen, P. (1998). Evaluating interval forecasts. *International Economic Review*, 39(4), 841–862. - Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. *Journal of Business & Economic Statistics*, 13(3), 253–263. - Ledoit, O., & Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. *Journal of Multivariate Analysis*, 88(2), 365–411. - Markowitz, H. (1952). Portfolio selection. *Journal of Finance*, 7(1), 77–91.