Upload README.md

README.md

@@ -1,154 +1,163 @@
-# 🏦 AlphaForge - Autonomous Quant Fund OS
+# AlphaForge - Multi-Asset Quantitative Trading System v2.0
 
-A **production-grade** quantitative trading system with 22 integrated modules — multi-source alpha generation, risk management, portfolio optimization, and derivatives pricing.
+A comprehensive quantitative trading system combining real-time data streaming, advanced feature engineering, multi-model alpha signals, sentiment analysis, volatility forecasting, portfolio optimization, and ML options pricing.
 
-## 🎯 Why This Is Different
+## What's New in v2.0
 
-| Feature | Typical Student Project | AlphaForge |
-|---------|------------------------|------------|
-| Alpha Sources | Only price data | Price + Sentiment + Factors |
-| Model Architecture | Single model | Ensemble + Meta-Model |
-| Risk | Ignored | VaR, CVaR, Tail Risk, Drawdown Control |
-| Regime | Assumed constant | HMM Regime Switching |
-| Uncertainty | Point estimates | Bayesian Probabilistic |
-| Options | None | ML Pricing + Mispricing |
-| Hedging | None | Delta-Neutral Dynamic |
-| Explainability | None | SHAP + Feature Importance |
-| Anomaly Detection | None | Isolation Forest + Autoencoder |
-| Stress Testing | None | 2008, COVID, Flash Crash, etc. |
-| Online Learning | Static | Adaptive with Drift Detection |
-| Dashboard | Console print | Gradio Live Dashboard |
+### Real-Time Data
+- **Alpaca Markets** WebSocket streaming (free tier, real-time IEX)
+- **Polygon.io** professional WebSocket (NBBO, trades, aggregates)
+- **Yahoo Finance** polling (free, 15-min delayed)
+- **FRED macro data** (yield curve, VIX, credit spreads)
+- **Live news streaming** with FinBERT sentiment processing
+- **Order flow estimation** from tick data
 
-## 🏗 Architecture
+### Advanced Feature Engineering (90+ Features)
+- **Microstructure**: Amihud illiquidity, Kyle's lambda, bid-ask spread proxy, VWAP, Roll spread
+- **Cross-sectional**: Momentum ranking, mean reversion, return dispersion
+- **Macro overlay**: Yield curve (10Y-2Y spread, inversion), VIX regime, credit spreads
+- **Stat-arb**: Cointegration spread, half-life, relative value
+- **Regime detection**: Volatility regime, trend regime, liquidity regime
+- **Advanced technicals**: Ichimoku, Supertrend, Keltner channels, Volume profile
+
+### Online Learning
+- **Drift detection**: Kolmogorov-Smirnov test, CUSUM change point
+- **Adaptive retraining**: Automatic model update when drift detected
+- **IC tracking**: Real-time information coefficient monitoring
+
+## Architecture
 
 ```
-┌─────────────────────────────────────────────────────────────┐
-│                    A L P H A F O R G E                       │
-│                   Autonomous Quant Fund OS                   │
-├─────────────────┬──────────────────┬────────────────────────┤
-│   DATA LAYER    │   MODEL LAYER    │   EXECUTION LAYER      │
-│                 │                  │                         │
-│  Market Data    │  Alpha Ensemble  │  Portfolio Optimizer   │
-│  (OHLCV)        │  (LSTM+Trans+    │  (Mean-Var, Max Sharpe,│
-│       │         │   XGBoost)       │   Robust, Black-Lit)   │
-│       ▼         │       │          │         │              │
-│  Technical      │  Sentiment Model │  Backtest Engine       │
-│  Indicators     │  (FinBERT)       │  (PnL, Sharpe, Metrics)│
-│       │         │       │          │         │              │
-│       ▼         │       ▼          │         ▼              │
-│  Cross-Asset    │  Meta-Model      │  Strategy Ensemble     │
-│  Features       │  (Signal Weights) │  (Dynamic Allocation) │
-│                 │                  │                         │
-├─────────────────┼──────────────────┼────────────────────────┤
-│  RISK LAYER     │  ADVANCED ML     │  MONITORING LAYER     │
-│                 │                  │                         │
-│  Regime Detect  │  Online Learning │  Live Dashboard        │
-│  (HMM)          │  (Adaptive AI)   │  (Gradio)              │
-│       │         │       │          │         │              │
-│       ▼         │       ▼          │         ▼              │
-│  Risk Engine    │  Explainability  │  Factor Attribution    │
-│  (VaR/CVaR)     │  (SHAP)          │  (Decomposition)       │
-│       │         │       │          │         │              │
-│       ▼         │       ▼          │         ▼              │
-│  Stress Test    │  Anomaly Detect  │  Hedging Engine        │
-│  Engine         │  (IsoForest+AE)  │  (Options-based)       │
-│                 │                  │                         │
-│  Drawdown Ctrl  │  Bayesian Layer  │  Options Pricer        │
-│  (Scaling)      │  (Uncertainty)   │  (NN + Mispricing)     │
-└─────────────────┴──────────────────┴────────────────────────┘
+Real-Time Data Feeds (Alpaca/Polygon/Yahoo)
+    |
+    ├──► Advanced Feature Engine (90+ features)
+    |     ├── Microstructure (bid-ask, Kyle lambda, VWAP)
+    |     ├── Cross-Sectional (momentum, dispersion)
+    |     ├── Macro Overlay (VIX, yield curve, credit)
+    |     ├── Regime Detection (vol/trend/liquidity)
+    |     └── Advanced Technicals (Ichimoku, Supertrend)
+    |
+    ├──► News Stream ──► FinBERT ──► Sentiment Alpha (S_t)
+    |
+    ├──► Alpha Model (LSTM + Transformer + XGBoost Ensemble)
+    |     └──► Combined Alpha = w1*Price Alpha + w2*Sentiment Alpha
+    |
+    ├──► Volatility Engine (GARCH + LSTM) ──► Covariance (Σ)
+    |
+    ├──► Portfolio Optimizer
+    |     ├── Max Sharpe
+    |     ├── Min Volatility
+    |     ├── Robust Optimization
+    |     └── Black-Litterman
+    |
+    └──► Backtest Engine ──► PnL, Sharpe, Sortino, Max DD
 ```
 
-## 📦 Modules (22 files)
-
-### Core Pipeline (Papers-backed)
-| # | Module | File | Description |
-|---|--------|------|-------------|
-| 1 | **Alpha Model** | `alpha_model.py` | LSTM + Transformer + XGBoost ensemble |
-| 2 | **Sentiment** | `sentiment_model.py` | FinBERT pipeline (`ProsusAI/finbert`) |
-| 3 | **Volatility** | `volatility_model.py` | GARCH(1,1) + LSTM with skewed-t distribution |
-| 4 | **Portfolio** | `portfolio_optimizer.py` | MV, Max Sharpe, Min Vol, Robust, Black-Litterman |
-| 5 | **Options** | `options_pricer.py` | 4-layer NN pricing + IV + mispricing signals |
-| 6 | **Backtest** | `backtest_engine.py` | Full metrics (Sharpe, Sortino, Calmar, IC) |
-| 7 | **Data** | `market_data.py` | Fetch + 20+ technical indicators |
-
-### 🔥 High-Value Additions
-| # | Module | File | Description |
-|---|--------|------|-------------|
-| 8 | **Meta-Model** | `meta_model.py` | Learns which model to trust (Renaissance-style) |
-| 9 | **Regime HMM** | `regime_detector.py` | Hidden Markov Model + strategy switching |
-| 10 | **Risk Engine** | `risk_engine.py` | VaR (historical/parametric/Cornish-Fisher), CVaR, tail |
-| 11 | **Factor Decomp** | `factor_decomposition.py` | Momentum, Value, Size, Vol, Quality factors |
-| 12 | **Online Learning** | `online_learning.py` | Incremental SGD + concept drift detection |
-| 13 | **Explainability** | `explainability.py` | SHAP-style feature importance |
-| 14 | **Anomaly Det.** | `anomaly_detector.py` | Isolation Forest + Autoencoder |
-| 15 | **Stress Test** | `stress_test.py` | 2008, COVID, Flash Crash, Monte Carlo |
-| 16 | **Bayesian** | `bayesian_layer.py` | Probabilistic forecasts + shrinkage |
-| 17 | **Hedging** | `hedging_engine.py` | Delta-neutral dynamic hedging |
-| 18 | **Strategy Ensemble** | `strategy_ensemble.py` | Multi-strategy capital allocation |
-| 19 | **Drawdown Ctrl** | `risk_engine.py` | Adaptive position scaling |
-| 20 | **Orchestrator** | `main.py` | Wires everything together |
-| 21 | **Original** | `main_original.py` | Original modular main script |
-
-## 🚀 Quick Start
+## Installation
 
 ```bash
 git clone https://huggingface.co/Premchan369/alphaforge-quant-system
 cd alphaforge-quant-system
 pip install -r requirements.txt
 
-# Run the full pipeline
-python main.py --tickers SPY QQQ AAPL MSFT GOOGL AMZN META NVDA TSLA JPM \
-               --start 2020-01-01 --end 2024-01-01 \
-               --epochs 30 --capital 1000000
+# Optional: For FRED macro data
+export FRED_API_KEY=your_key_here
+
+# Optional: For Alpaca real-time streaming
+export ALPACA_API_KEY=your_key_here
+export ALPACA_SECRET_KEY=your_secret_here
+```
+
+## Usage
+
+### Full Backtest with Advanced Features
+```bash
+# Standard backtest
+python main.py --mode backtest --start 2020-01-01 --end 2024-01-01
+
+# With advanced features + macro + sentiment + online learning
+python main.py --mode backtest --start 2020-01-01 --end 2024-01-01 \
+    --advanced-features --include-macro --include-sentiment --online-learning
+```
+
+### Real-Time Streaming
+```bash
+# Yahoo Finance (free, 15-min delayed)
+python main.py --mode realtime --source yahoo --tickers SPY QQQ AAPL MSFT
+
+# Alpaca (free tier, real-time IEX)
+python main.py --mode realtime --source alpaca \
+    --api-key YOUR_KEY --secret-key YOUR_SECRET
+
+# Polygon.io (professional)
+python main.py --mode realtime --source polygon --api-key YOUR_KEY
+```
+
+### Train Model
+```bash
+python main.py --mode train --tickers SPY QQQ AAPL MSFT GOOGL AMZN META NVDA TSLA JPM \
+    --epochs 50 --advanced-features --include-macro
+```
+
+### Options Pricing
+```bash
+python main.py --mode options
 ```
 
-## 📊 Dashboard
-
-Live monitoring dashboard at: **https://huggingface.co/spaces/Premchan369/alphaforge-dashboard**
-
-Features:
-- Portfolio equity curve with drawdown
-- Live PnL and risk metrics
-- Covariance matrix heatmap
-- Factor exposure breakdown
-- Options IV surface
-- Anomaly detection tracker
-- Model performance comparison
-- Configuration panel
-
-## 📈 Key Metrics Tracked
-
-| Metric | Description | Where |
-|--------|-------------|-------|
-| Sharpe Ratio | Risk-adjusted return | Backtest |
-| Sortino Ratio | Downside risk-adjusted | Backtest |
-| Max Drawdown | Peak-to-trough decline | Risk Engine |
-| Calmar Ratio | Return / Max DD | Backtest |
-| VaR (95%, 99%) | Value at Risk | Risk Engine |
-| CVaR | Conditional VaR | Risk Engine |
-| IC | Information Coefficient | Alpha Model |
-| Alpha / Beta | Market-adjusted metrics | Factor Decomp |
-| Factor Exposures | Style factor loadings | Factor Decomp |
-| Win Rate | % of profitable days | Backtest |
-| Hedge Ratio | Delta hedge level | Hedging |
-
-## 🧠 Research Backing
-
-- **Alpha**: xLSTM-TS (Lopez Gil et al., 2024) + QuantaAlpha (Han et al., 2026)
-- **Sentiment**: FinBERT (Araci, 2019), ChatGPT benchmarking (Fatouros et al., 2023)
-- **Volatility**: LSTM-SSTD (Michankow, 2025)
-- **Portfolio**: MTL-TSMOM (Ong & Herremans, 2023)
-- **Options**: Feed-Forward NN (Berger et al., 2023), PINN (Dhiman et al., 2023)
-
-## 🏆 Real-World Alignment
-
-This system mirrors production architectures used by:
-- **Renaissance Technologies**: Multi-signal meta-model
-- **Two Sigma**: ML-driven alpha + risk decomposition
-- **Bridgewater**: Regime-based allocation
-- **AQR**: Factor decomposition + systematic strategies
-- **Citadel**: Options pricing + delta hedging
-
-## 📄 License
-
-MIT
+## File Structure
+
+| File | Description |
+|------|-------------|
+| `main.py` | Entry point - train, backtest, or real-time mode |
+| `market_data.py` | OHLCV data fetching + basic features (RSI, MACD, BB) |
+| `alpha_model.py` | LSTM/Transformer/XGBoost ensemble with IC tracking |
+| `sentiment_model.py` | FinBERT sentiment with batch processing |
+| `volatility_model.py` | GARCH(1,1) + LSTM volatility forecasting |
+| `portfolio_optimizer.py` | Mean-variance, max-Sharpe, robust, Black-Litterman |
+| `options_pricer.py` | ML options pricing + mispricing detection |
+| `backtest_engine.py` | Full backtest with Sharpe, Sortino, max DD, regime detection |
+| `advanced_features_part1.py` | Microstructure + cross-sectional features |
+| `macro_features.py` | FRED macro + yield curve + VIX + credit spreads |
+| `regime_features.py` | Volatility/trend/liquidity regime detection |
+| `technical_indicators.py` | Ichimoku, Supertrend, Keltner, Volume Profile |
+| `stat_arb_features.py` | Cointegration, spread, relative value, half-life |
+| `online_learning.py` | Drift detection (KS, CUSUM) + adaptive retraining |
+| `realtime_data.py` | Alpaca/Polygon/Yahoo streaming + news + order flow |
+
+## Data Sources
+
+| Source | Type | Cost | Real-Time |
+|--------|------|------|-----------|
+| **Yahoo Finance** | OHLCV + News | Free | 15min delayed |
+| **Alpaca Markets** | Trades + Bars | Free tier | Real-time (IEX) |
+| **Polygon.io** | NBBO + Trades + Aggs | Paid | Real-time |
+| **FRED** | Macro (rates, VIX) | Free | Daily |
+| **FMP** | News + Financials | Free tier | Daily |
+| **FinBERT** | Sentiment | Free (local) | Batch |
+
+## Metrics
+
+| Metric | Description |
+|--------|-------------|
+| **IC** | Information Coefficient (rank correlation predicted vs actual) |
+| **IC IR** | IC Information Ratio (mean IC / std IC) |
+| **Sharpe** | Risk-adjusted return (excess return / volatility) |
+| **Sortino** | Downside risk-adjusted return |
+| **Max DD** | Maximum peak-to-trough decline |
+| **Calmar** | Annualized return / max drawdown |
+| **Alpha/Beta** | Excess return and market sensitivity |
+| **Turnover** | Portfolio rebalance intensity |
+
+## Research Backing
+
+- **Alpha Models**: xLSTM-TS with wavelet denoising (Lopez Gil et al., 2024)
+- **Sentiment**: FinBERT (Araci, 2019) with ChatGPT benchmarking
+- **Volatility**: LSTM with skewed Student's t (Michańkow, 2025)
+- **Portfolio**: Multi-task learning joint optimization (Ong & Herremans, 2023)
+- **Options**: 5-layer FNN outperforming Black-Scholes (Berger et al., 2023)
+- **Microstructure**: Amihud (2002), Kyle (1985), Corwin-Schultz (2012), Roll (1984)
+- **Online Learning**: CUSUM change detection, KS drift test
+
+## License
+
+MIT