feat: add Yahoo ingest on the FinRL tree and stop extra branch creation

Keep csv/alpaca/synthetic/FinRL paths unchanged. Add yfinance>=1.0 as an optional data_source, rewrite the README, and remove Dependabot plus the weekly dependency-updates workflow so only main and dev remain.

Co-authored-by: Cursor <cursoragent@cursor.com>

.github/dependabot.yml

@@ -1,59 +0,0 @@
-version: 2
-updates:
-  # Python dependencies
-  - package-ecosystem: "pip"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-      day: "monday"
-      time: "09:00"
-    open-pull-requests-limit: 10
-    reviewers:
-      - "dataen10"
-    assignees:
-      - "dataen10"
-    commit-message:
-      prefix: "pip"
-      prefix-development: "pip-dev"
-      include: "scope"
-    labels:
-      - "dependencies"
-      - "python"
-      
-  # Docker dependencies
-  - package-ecosystem: "docker"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-      day: "monday"
-      time: "09:00"
-    open-pull-requests-limit: 5
-    reviewers:
-      - "dataen10"
-    assignees:
-      - "dataen10"
-    commit-message:
-      prefix: "docker"
-      include: "scope"
-    labels:
-      - "dependencies"
-      - "docker"
-      
-  # GitHub Actions
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-      day: "monday"
-      time: "09:00"
-    open-pull-requests-limit: 5
-    reviewers:
-      - "dataen10"
-    assignees:
-      - "dataen10"
-    commit-message:
-      prefix: "github-actions"
-      include: "scope"
-    labels:
-      - "dependencies"
-      - "github-actions" 

.github/workflows/ci-cd.yml

@@ -2,7 +2,7 @@ name: Algorithmic Trading CI/CD Pipeline
 
 on:
   push:
-    branches: [ main, develop ]
+    branches: [ main, dev ]
   pull_request:
     branches: [ main ]
   release:

.github/workflows/dependency-update.yml

@@ -1,65 +0,0 @@
-name: Dependency Updates
-
-on:
-  schedule:
-    - cron: '0 2 * * 1'  # Every Monday at 2 AM
-  workflow_dispatch:
-
-jobs:
-  update-dependencies:
-    name: Update Dependencies
-    runs-on: ubuntu-latest
-    
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v4
-      
-    - name: Set up Python
-      uses: actions/setup-python@v5
-      with:
-        python-version: '3.11'
-        
-    - name: Install pip-tools
-      run: |
-        python -m pip install --upgrade pip
-        pip install pip-tools
-        
-    - name: Update requirements
-      run: |
-        pip-compile --upgrade requirements.in
-        pip-compile --upgrade requirements-dev.in
-        
-    - name: Check for security vulnerabilities
-      run: |
-        pip install safety
-        safety check --json --output safety-report.json
-        
-    - name: Create Pull Request
-      uses: peter-evans/create-pull-request@v7
-      with:
-        token: ${{ secrets.GITHUB_TOKEN }}
-        commit-message: 'chore: update dependencies'
-        title: '🔧 Automated dependency updates'
-        body: |
-          ## Automated Dependency Updates
-          
-          This PR updates dependencies to their latest versions.
-          
-          ### 📋 Changes
-          - Updated Python packages to latest versions
-          - Security vulnerability fixes
-          - Performance improvements
-          
-          ### 🔍 Security Report
-          - [ ] No critical vulnerabilities
-          - [ ] No high severity issues
-          - [ ] Dependencies up to date
-          
-          ### 🧪 Testing
-          - [ ] All tests pass
-          - [ ] No breaking changes
-          - [ ] Performance maintained
-          
-          **Auto-generated by GitHub Actions**
-        branch: dependency-updates
-        delete-branch: true 

CI_CD_SETUP.md

@@ -11,7 +11,7 @@ The CI/CD pipeline provides automated quality assurance, testing, deployment, an
 ### 1. **Main CI/CD Pipeline** (`.github/workflows/ci-cd.yml`)
 
 **Triggers:**
-- Push to `main` or `develop` branches
+- Push to `main` or `dev` branches
 - Pull requests to `main`
 - Release creation
 

README.md

@@ -1,600 +1,130 @@
-# Algorithmic Trading System with FinRL and Alpaca Integration
+# Algorithmic Trading
 
-A sophisticated algorithmic trading system that combines reinforcement learning (FinRL) with real-time market data and order execution through Alpaca Markets. This system supports both paper trading and live trading with advanced risk management and technical analysis.
+FinRL reinforcement-learning trading with Alpaca execution, plus optional Yahoo Finance OHLCV for unlabeled real-price research. Parallel LLC.
 
-## 🚀 Features
+This is **research and paper-trading infrastructure**. Live capital requires a separate evaluation contract, feature-parity tests, and a rewritten execution path. Do not treat `paper_trading: false` as a promotion gate.
 
-### Core Trading System
+---
 
-- **Multi-source Data Ingestion**: CSV files, Alpaca Markets API, and synthetic data generation
-- **Technical Analysis**: 20+ technical indicators including RSI, MACD, Bollinger Bands, and more
-- **Risk Management**: Position sizing, drawdown limits, and portfolio protection
-- **Real-time Execution**: Live order placement and portfolio monitoring
+## 1. Title and Summary
 
-### FinRL Reinforcement Learning
+**Algorithmic Trading**  
+Northwestern-trained data-engineering practice applied to a trading loop: ingest OHLCV, compute indicators or train a FinRL policy, size orders under position and drawdown caps, route to paper or live Alpaca.
 
-- **Multiple Algorithms**: PPO, A2C, DDPG, and TD3 support
-- **Custom Trading Environment**: Gymnasium-compatible environment for RL training
-- **Real-time Integration**: Can execute real trades during training and inference
-- **Model Persistence**: Save and load trained models for consistent performance
+GitHub `main` is the FinRL / Docker / Streamlit tree. `dev` is the integration branch. Yahoo is an additive `data_source.type`, not a replacement for Alpaca or FinRL.
 
-### Alpaca Broker Integration
+**Design themes**
 
-- **Paper Trading**: Risk-free testing with virtual money
-- **Live Trading**: Real market execution (use with caution!)
-- **Market Data**: Real-time and historical data from Alpaca
-- **Account Management**: Portfolio monitoring and position tracking
-- **Order Types**: Market orders, limit orders, and order cancellation
+* Four ingest paths: CSV replay, synthetic GBM, Alpaca REST, Yahoo (`yfinance>=1.0`)
+* FinRL policies (PPO, A2C, DDPG, TD3) on a Gymnasium-style environment
+* Alpaca for authenticated market data and order routing (paper by default)
+* Yahoo for delayed public bars when no broker key is available
+* Secrets from environment (`ALPACA_API_KEY`, `ALPACA_SECRET_KEY`), never committed
+* Tests and Docker/CI as already present on this tree
 
-### 🎨 Comprehensive UI System
+---
 
-- **Streamlit UI**: Quick prototyping and data science workflows
-- **Dash UI**: Enterprise-grade interactive dashboards
-- **Jupyter UI**: Interactive notebook-based interfaces
-- **WebSocket API**: Real-time trading data streaming
-- **Multi-interface Support**: Choose the right UI for your needs
+## 2. Concepts and Methods
 
-### Advanced Features
+### Market data
 
-- **Docker Support**: Containerized deployment for consistency
-- **Comprehensive Logging**: Detailed logs for debugging and performance analysis
-- **Backtesting Engine**: Historical performance evaluation
-- **Live Trading Simulation**: Real-time trading with configurable duration
-- **Performance Metrics**: Returns, Sharpe ratio, drawdown analysis
+| Source | When to use | Failure modes |
+| ------ | ----------- | ------------- |
+| **CSV** | Offline replay; default in `config.yaml` | Missing path or OHLCV columns → `None` |
+| **Synthetic** | Unit tests and demos | GBM is not tradable edge |
+| **Alpaca** | Authenticated bars and live/paper orders | Auth, feed, and rate-limit failures |
+| **Yahoo** | Real Close without a broker account | Unofficial API, ~15 min delay, interval lookback caps (1m ≈ 7 days). Pin `yfinance>=1.0`; 0.2.x fails against the current chart API |
 
-## 📋 Prerequisites
+`load_data` dispatches on `data_source.type`. Existing `alpaca` / `csv` / `synthetic` branches are unchanged.
 
-- Python 3.8+
-- Alpaca Markets account (free paper trading available)
-- Docker (optional, for containerized deployment)
+### Strategy and FinRL
 
-## 🛠️ Installation
+* `StrategyAgent`: SMA, RSI, Bollinger, MACD on Close; teaching rule, not an alpha claim
+* `FinRLAgent`: PPO / A2C / DDPG / TD3 via Stable-Baselines3; persist under `models/`
+* `ExecutionAgent` / `AlpacaBroker`: paper simulation or Alpaca market/limit orders
 
-### 1. Clone the Repository
+Backtests in this repo are in-sample passes unless you add a purged walk-forward yourself. Leakage is the null hypothesis.
 
-```bash
-git clone https://github.com/ParallelLLC/algorithmic_trading.git
-cd algorithmic_trading
-```
+---
 
-### 2. Install Dependencies
+## 3. Stack
 
-```bash
-pip install -r requirements.txt
-```
+| Layer | Tools |
+| ----- | ----- |
+| Language | Python 3.11 (CI); 3.8+ stated for local |
+| RL | FinRL / Stable-Baselines3, Gym/Gymnasium, PyTorch |
+| Broker | alpaca-py |
+| Market data | Alpaca REST; yfinance ≥ 1.0 (Yahoo) |
+| Tabular | pandas, NumPy, scikit-learn |
+| UI | Streamlit, Dash, Jupyter widgets |
+| Deploy | Docker Compose, GitHub Actions |
+| Tests | pytest, pytest-cov |
 
-### 3. Set Up Alpaca API Credentials
+---
 
-Create a `.env` file in the project root:
+## 4. Structure
 
-```bash
-cp env.example .env
+```
+algorithmic_trading/
+├── agentic_ai_system/     # ingest, strategy, FinRL, Alpaca, Yahoo
+├── ui/                    # Streamlit, Dash, Jupyter, WebSocket
+├── tests/
+├── models/                # trained artifacts (gitignored bodies)
+├── data/                  # generated CSV (gitignored)
+├── scripts/               # Docker / deploy helpers
+├── .github/workflows/     # CI/CD, release, backtesting
+├── config.yaml
+├── requirements.txt
+├── Dockerfile
+└── docker-compose*.yml
 ```
 
-Edit `.env` with your Alpaca credentials:
+Branch policy: **`main`** (protected) and **`dev`** only. Do not re-enable Dependabot or the Monday `dependency-updates` workflow; those created extra branches.
 
-```env
-# Get these from https://app.alpaca.markets/paper/dashboard/overview
-ALPACA_API_KEY=your_paper_api_key_here
-ALPACA_SECRET_KEY=your_paper_secret_key_here
+---
 
-# For live trading (use with caution!)
-# ALPACA_API_KEY=your_live_api_key_here
-# ALPACA_SECRET_KEY=your_live_secret_key_here
-```
+## 5. Quick start
 
-### 4. Configure Trading Parameters
+```bash
+git clone https://github.com/ParallelLLC/algorithmic_trading.git
+cd algorithmic_trading
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+cp .env.example .env   # Alpaca keys if using alpaca ingest or orders
+```
 
-Edit `config.yaml` to customize your trading strategy:
+Default ingest is CSV. For Yahoo daily bars without a broker:
 
 ```yaml
-# Data source configuration
 data_source:
-  type: 'alpaca'  # Options: 'alpaca', 'csv', 'synthetic'
-
-# Trading parameters
+  type: 'yahoo'
 trading:
   symbol: 'AAPL'
-  timeframe: '1m'
-  capital: 100000
-
-# Risk management
-risk:
-  max_position: 100
-  max_drawdown: 0.05
-
-# Execution settings
-execution:
-  broker_api: 'alpaca_paper'  # Options: 'paper', 'alpaca_paper', 'alpaca_live'
-  order_size: 10
-
-# FinRL configuration
-finrl:
-  algorithm: 'PPO'
-  learning_rate: 0.0003
-  training:
-    total_timesteps: 100000
-    save_best_model: true
+  timeframe: '1d'
 ```
 
-## 🚀 Quick Start
-
-### 1. Launch the UI (Recommended)
-
-```bash
-# Launch Streamlit UI (best for beginners)
-python ui_launcher.py streamlit
-
-# Launch Dash UI (best for production)
-python ui_launcher.py dash
-
-# Launch Jupyter Lab
-python ui_launcher.py jupyter
-
-# Launch all UIs
-python ui_launcher.py all
-```
-
-### 2. Run the Demo
-
 ```bash
 python demo.py
+python -m agentic_ai_system.main --mode backtest --start-date 2024-01-01 --end-date 2024-12-31
+pytest tests/ -q
 ```
 
-This will:
-- Test data ingestion from Alpaca
-- Demonstrate FinRL training
-- Show trading workflow execution
-- Run backtesting on historical data
-
-### 3. Start Paper Trading
-
-```bash
-python -m agentic_ai_system.main --mode live --duration 60
-```
-
-### 4. Run Backtesting
-
-```bash
-python -m agentic_ai_system.main --mode backtest --start-date 2024-01-01 --end-date 2024-01-31
-```
-
-## 📊 Usage Examples
-
-### Basic Trading Workflow
-
-```python
-from agentic_ai_system.main import load_config
-from agentic_ai_system.orchestrator import run
-
-# Load configuration
-config = load_config()
-
-# Run single trading cycle
-result = run(config)
-print(f"Trading result: {result}")
-```
-
-### FinRL Training
-
-```python
-from agentic_ai_system.finrl_agent import FinRLAgent, FinRLConfig
-from agentic_ai_system.data_ingestion import load_data
-
-# Load data and configuration
-config = load_config()
-data = load_data(config)
-
-# Initialize FinRL agent
-finrl_config = FinRLConfig(algorithm='PPO', learning_rate=0.0003)
-agent = FinRLAgent(finrl_config)
-
-# Train the agent
-result = agent.train(
-    data=data,
-    config=config,
-    total_timesteps=100000,
-    use_real_broker=False  # Use simulation for training
-)
-
-print(f"Training completed: {result}")
-```
-
-### Alpaca Integration
-
-```python
-from agentic_ai_system.alpaca_broker import AlpacaBroker
-
-# Initialize Alpaca broker
-config = load_config()
-broker = AlpacaBroker(config)
-
-# Get account information
-account_info = broker.get_account_info()
-print(f"Account balance: ${account_info['buying_power']:,.2f}")
-
-# Place a market order
-result = broker.place_market_order(
-    symbol='AAPL',
-    quantity=10,
-    side='buy'
-)
-print(f"Order result: {result}")
-```
-
-### Real-time Trading with FinRL
-
-```python
-from agentic_ai_system.finrl_agent import FinRLAgent
-
-# Load trained model
-agent = FinRLAgent(FinRLConfig())
-agent.model = agent._load_model('models/finrl_best/best_model', config)
-
-# Make predictions with real execution
-result = agent.predict(
-    data=recent_data,
-    config=config,
-    use_real_broker=True  # Execute real trades!
-)
-```
-
-## 🏗️ Architecture
-
-### System Components
-
-```
-┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
-│   Data Sources  │    │  Strategy Agent │    │ Execution Agent │
-│                 │    │                 │    │                 │
-│ • Alpaca API    │───▶│ • Technical     │───▶│ • Alpaca Broker │
-│ • CSV Files     │    │   Indicators    │    │ • Order Mgmt    │
-│ • Synthetic     │    │ • Signal Gen    │    │ • Risk Control  │
-└─────────────────┘    └─────────────────┘    └─────────────────┘
-         │                       │                       │
-         ▼                       ▼                       ▼
-┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
-│  Data Ingestion │    │   FinRL Agent   │    │   Portfolio     │
-│                 │    │                 │    │   Management    │
-│ • Validation    │    │ • PPO/A2C/DDPG  │    │ • Positions     │
-│ • Indicators    │    │ • Training      │    │ • P&L Tracking  │
-│ • Preprocessing │    │ • Prediction    │    │ • Risk Metrics  │
-└─────────────────┘    └─────────────────┘    └─────────────────┘
-```
-
-### Data Flow
-
-1. **Data Ingestion**: Market data from Alpaca, CSV, or synthetic sources
-2. **Preprocessing**: Technical indicators, data validation, and feature engineering
-3. **Strategy Generation**: Traditional technical analysis or FinRL predictions
-4. **Risk Management**: Position sizing and portfolio protection
-5. **Order Execution**: Real-time order placement through Alpaca
-6. **Performance Tracking**: Continuous monitoring and logging
-
-## 📁 Project Directory Structure
-
-```
-algorithmic_trading/
-├── 📄 README.md                    # Project documentation
-├── 📄 LICENSE                      # Apache License 2.0
-├── 📄 requirements.txt             # Python dependencies
-├── 📄 config.yaml                  # Main configuration file
-├── 📄 env.example                  # Environment variables template
-├── 📄 .gitignore                   # Git ignore rules
-├── 📄 pytest.ini                  # Test configuration
-│
-├── 🐳 Docker/
-│   ├── 📄 Dockerfile              # Container definition
-│   ├── 📄 docker-entrypoint.sh    # Container startup script
-│   ├── 📄 .dockerignore           # Docker ignore rules
-│   ├── 📄 docker-compose.yml      # Default compose file
-│   ├── 📄 docker-compose.dev.yml  # Development environment
-│   ├── 📄 docker-compose.prod.yml # Production environment
-│   ├── 📄 docker-compose.hub.yml  # Docker Hub deployment
-│   └── 📄 docker-compose.prod.yml # Production environment
-│
-├── 🤖 agentic_ai_system/          # Core AI trading system
-│   ├── 📄 main.py                 # Main entry point
-│   ├── 📄 orchestrator.py         # System coordination
-│   ├── 📄 agent_base.py           # Base agent class
-│   ├── 📄 data_ingestion.py       # Market data processing
-│   ├── 📄 strategy_agent.py       # Trading strategy logic
-│   ├── 📄 execution_agent.py      # Order execution
-│   ├── 📄 finrl_agent.py          # FinRL reinforcement learning
-│   ├── 📄 alpaca_broker.py        # Alpaca API integration
-│   ├── 📄 synthetic_data_generator.py # Test data generation
-│   └── 📄 logger_config.py        # Logging configuration
-│
-├── 🎨 ui/                         # User interface system
-│   ├── 📄 __init__.py            # UI package initialization
-│   ├── 📄 streamlit_app.py       # Streamlit web application
-│   ├── 📄 dash_app.py            # Dash enterprise dashboard
-│   ├── 📄 jupyter_widgets.py     # Jupyter interactive widgets
-│   └── 📄 websocket_server.py    # Real-time WebSocket server
-│
-├── 🧪 tests/                      # Test suite
-│   ├── 📄 __init__.py
-│   ├── 📄 test_data_ingestion.py
-│   ├── 📄 test_strategy_agent.py
-│   ├── 📄 test_execution_agent.py
-│   ├── 📄 test_finrl_agent.py
-│   ├── 📄 test_synthetic_data_generator.py
-│   └── 📄 test_integration.py
-│
-├── 📊 data/                       # Market data storage
-│   └── 📄 synthetic_market_data.csv
-│
-├── 🧠 models/                     # Trained AI models
-│   └── 📁 finrl_best/            # Best FinRL models
-│
-├── 📈 plots/                      # Generated charts/visualizations
-│
-├── 📝 logs/                       # System logs
-│   ├── 📄 trading_system.log
-│   ├── 📄 trading.log
-│   ├── 📄 performance.log
-│   ├── 📄 errors.log
-│   ├── 📁 finrl_tensorboard/     # FinRL training logs
-│   └── 📁 finrl_eval/            # Model evaluation logs
-│
-├── 🔧 scripts/                    # Utility scripts
-│   ├── 📄 docker-build.sh        # Docker build automation
-│   └── 📄 docker-hub-deploy.sh   # Docker Hub deployment
-│
-├── 📄 demo.py                     # Main demo script
-├── 📄 finrl_demo.py              # FinRL-specific demo
-├── 📄 ui_launcher.py             # UI launcher script
-├── 📄 UI_SETUP.md                # UI setup documentation
-├── 📄 DOCKER_HUB_SETUP.md        # Docker Hub documentation
-│
-└── 🐍 .venv/                     # Python virtual environment
-```
-
-## 🏗️ Architecture Overview
-
-### Core Components:
-
-- **Data Layer**: Market data ingestion and preprocessing
-- **Strategy Layer**: Technical analysis and signal generation
-- **AI Layer**: FinRL reinforcement learning agents
-- **Execution Layer**: Order management and broker integration
-- **Orchestration**: System coordination and workflow management
-
-### Key Features:
-
-- **Modular Design**: Each component is independent and testable
-- **Docker Support**: Complete containerization for deployment
-- **Testing**: Comprehensive test suite for all components
-- **Logging**: Detailed logging for monitoring and debugging
-- **Configuration**: Centralized configuration management
-- **Documentation**: Extensive documentation and examples
-
-### Development Workflow:
-
-1. **Data Ingestion** → Market data from Alpaca/CSV/synthetic sources
-2. **Strategy Generation** → Technical indicators and FinRL predictions
-3. **Risk Management** → Position sizing and portfolio protection
-4. **Order Execution** → Real-time trading through Alpaca
-5. **Performance Tracking** → Continuous monitoring and logging
-
-## 🔧 Configuration
-
-### Alpaca Settings
-
-```yaml
-alpaca:
-  api_key: ''  # Set via environment variable
-  secret_key: ''  # Set via environment variable
-  paper_trading: true
-  base_url: 'https://paper-api.alpaca.markets'
-  live_url: 'https://api.alpaca.markets'
-  data_url: 'https://data.alpaca.markets'
-  account_type: 'paper'  # 'paper' or 'live'
-```
-
-### FinRL Settings
-
-```yaml
-finrl:
-  algorithm: 'PPO'  # PPO, A2C, DDPG, TD3
-  learning_rate: 0.0003
-  batch_size: 64
-  buffer_size: 1000000
-  training:
-    total_timesteps: 100000
-    eval_freq: 10000
-    save_best_model: true
-    model_save_path: 'models/finrl_best/'
-  inference:
-    use_trained_model: false
-    model_path: 'models/finrl_best/best_model'
-```
-
-### Risk Management
-
-```yaml
-risk:
-  max_position: 100
-  max_drawdown: 0.05
-  stop_loss: 0.02
-  take_profit: 0.05
-```
-
-## 🎨 User Interface System
-
-The project includes a comprehensive UI system with multiple interface options:
-
-### Available UIs
-
-#### Streamlit UI (Recommended for beginners)
-- **URL**: http://localhost:8501
-- **Features**: Interactive widgets, real-time data visualization, easy configuration
-- **Best for**: Data scientists, quick experiments, rapid prototyping
-
-#### Dash UI (Recommended for production)
-- **URL**: http://localhost:8050
-- **Features**: Enterprise-grade dashboards, advanced charts, professional styling
-- **Best for**: Production dashboards, real-time monitoring, complex analytics
-
-#### Jupyter UI (For research)
-- **URL**: http://localhost:8888
-- **Features**: Interactive notebooks, code execution, rich documentation
-- **Best for**: Research, experimentation, educational purposes
-
-#### WebSocket API (For developers)
-- **URL**: ws://localhost:8765
-- **Features**: Real-time data streaming, trading signals, portfolio updates
-- **Best for**: Real-time trading signals, live data streaming
-
-### Quick UI Launch
-
-```bash
-# Launch individual UIs
-python ui_launcher.py streamlit    # Streamlit UI
-python ui_launcher.py dash         # Dash UI
-python ui_launcher.py jupyter      # Jupyter Lab
-python ui_launcher.py websocket    # WebSocket server
-
-# Launch all UIs at once
-python ui_launcher.py all
-```
-
-### UI Features
-
-- **Real-time Data Visualization**: Live market data charts and indicators
-- **Portfolio Monitoring**: Real-time portfolio value and P&L tracking
-- **Trading Controls**: Start/stop trading, backtesting, risk management
-- **FinRL Training**: Interactive model training and evaluation
-- **Alpaca Integration**: Account management and order execution
-- **Configuration Management**: Easy parameter tuning and strategy setup
-
-For detailed UI documentation, see [UI_SETUP.md](UI_SETUP.md).
-
-## 📈 Performance Monitoring
-
-### Logging
-
-The system provides comprehensive logging:
-
-- `logs/trading_system.log`: Main system logs
-- `logs/trading.log`: Trading-specific events
-- `logs/performance.log`: Performance metrics
-- `logs/finrl_tensorboard/`: FinRL training logs
-
-### Metrics Tracked
-
-- Portfolio value and returns
-- Trade execution statistics
-- Risk metrics (Sharpe ratio, drawdown)
-- FinRL training progress
-- Alpaca account status
-
-### Real-time Monitoring
-
-```python
-# Get account information
-account_info = broker.get_account_info()
-print(f"Portfolio Value: ${account_info['portfolio_value']:,.2f}")
-
-# Get current positions
-positions = broker.get_positions()
-for pos in positions:
-    print(f"{pos['symbol']}: {pos['quantity']} shares")
-
-# Check market status
-market_open = broker.is_market_open()
-print(f"Market: {'OPEN' if market_open else 'CLOSED'}")
-```
-
-## 🐳 Docker Deployment
-
-### Build and Run
-
-```bash
-# Build the image
-docker build -t algorithmic-trading .
-
-# Run with environment variables
-docker run -it --env-file .env algorithmic-trading
-
-# Run with Jupyter Lab for development
-docker-compose -f docker-compose.dev.yml up
-```
-
-### Production Deployment
-
-```bash
-# Use production compose file
-docker-compose -f docker-compose.prod.yml up -d
-
-# Monitor logs
-docker-compose -f docker-compose.prod.yml logs -f
-```
-
-## 🧪 Testing
-
-### Run All Tests
-
-```bash
-pytest tests/ -v
-```
-
-### Test Specific Components
-
-```bash
-# Test Alpaca integration
-pytest tests/test_alpaca_integration.py -v
-
-# Test FinRL agent
-pytest tests/test_finrl_agent.py -v
-
-# Test trading workflow
-pytest tests/test_integration.py -v
-```
-
-## ⚠️ Important Notes
-
-### Paper Trading vs Live Trading
-
-- **Paper Trading**: Uses virtual money, safe for testing
-- **Live Trading**: Uses real money, use with extreme caution
-- Always test strategies thoroughly in paper trading before going live
-
-### Risk Management
-
-- Set appropriate position limits and drawdown thresholds
-- Monitor your portfolio regularly
-- Use stop-loss orders to limit potential losses
-- Never risk more than you can afford to lose
-
-### API Rate Limits
-
-- Alpaca has rate limits on API calls
-- The system includes built-in delays to respect these limits
-- Monitor your API usage in the Alpaca dashboard
-
-## 🤝 Contributing
-
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Add tests for new functionality
-5. Submit a pull request
-
-## 📄 License
+UI launchers and Docker are documented in `UI_SETUP.md` and `DOCKER_HUB_SETUP.md`. Paper-trade before live. Yahoo is not a SIP tape.
 
-This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
+---
 
-## 🆘 Support
+## 6. Configuration (additive Yahoo keys)
 
-- **Documentation**: Check the logs and configuration files
-- **Issues**: Report bugs and feature requests on GitHub
-- **Alpaca Support**: Contact Alpaca for API-related issues
-- **Community**: Join our Discord/Telegram for discussions
+| Key | Meaning |
+| --- | ------- |
+| `data_source.type` | `csv` \| `synthetic` \| `alpaca` \| `yahoo` |
+| `yahoo.start_date` / `end_date` | Historical window; clamped per Yahoo interval limits |
+| `yahoo.auto_adjust` | Passed to `yfinance` |
+| `execution.broker_api` | `paper` \| `alpaca_paper` \| `alpaca_live` |
+| `finrl.algorithm` | PPO, A2C, DDPG, TD3 |
 
-## 🔗 Useful Links
+---
 
-- [Alpaca Markets Documentation](https://alpaca.markets/docs/)
-- [FinRL Documentation](https://finrl.readthedocs.io/)
-- [Stable Baselines3 Documentation](https://stable-baselines3.readthedocs.io/)
-- [Gymnasium Documentation](https://gymnasium.farama.org/)
+**License:** Apache License 2.0  
+**Organization:** [Parallel LLC](https://github.com/ParallelLLC)  
+**Repository:** <https://github.com/ParallelLLC/algorithmic_trading>

agentic_ai_system/data_ingestion.py

@@ -27,6 +27,8 @@ def load_data(config: Dict[str, Any]) -> Optional[pd.DataFrame]:
             return _load_csv_data(config)
         elif data_source == 'synthetic':
             return _load_synthetic_data(config)
+        elif data_source == 'yahoo':
+            return _load_yahoo_data(config)
         else:
             logger.error(f"Unsupported data source: {data_source}")
             return None
@@ -75,6 +77,30 @@ def _load_alpaca_data(config: Dict[str, Any]) -> Optional[pd.DataFrame]:
         logger.error(f"Error loading Alpaca data: {e}")
         return None
 
+def _load_yahoo_data(config: Dict[str, Any]) -> Optional[pd.DataFrame]:
+    """Load OHLCV bars from Yahoo Finance via yfinance. Does not replace Alpaca."""
+    try:
+        from .yahoo_data_stream import YahooDataStream
+
+        stream = YahooDataStream(config)
+        trading = config.get('trading', {})
+        symbol = trading.get('symbol') or (trading.get('symbols') or ['AAPL'])[0]
+        yahoo_cfg = config.get('yahoo', {})
+        start_date = yahoo_cfg.get('start_date', '2024-01-01')
+        end_date = yahoo_cfg.get('end_date', '2026-12-31')
+
+        data = stream.get_historical_data(symbol, start_date, end_date)
+        if data is None or data.empty:
+            logger.error("No Yahoo data returned for %s", symbol)
+            return None
+
+        logger.info("Loaded %s Yahoo bars for %s", len(data), symbol)
+        return data
+    except Exception as e:
+        logger.error("Error loading Yahoo data: %s", e)
+        return None
+
+
 def _load_csv_data(config: Dict[str, Any]) -> Optional[pd.DataFrame]:
     """Load market data from CSV file"""
     try:

agentic_ai_system/yahoo_data_stream.py

@@ -0,0 +1,329 @@
+import logging
+import threading
+import time
+from typing import Any, Callable, Dict, List, Optional
+
+import pandas as pd
+
+logger = logging.getLogger(__name__)
+
+_INTERVAL_MAP = {
+    '1min': '1m',
+    '1m': '1m',
+    '5min': '5m',
+    '5m': '5m',
+    '15min': '15m',
+    '15m': '15m',
+    '30min': '30m',
+    '30m': '30m',
+    '1H': '1h',
+    '1h': '1h',
+    '60min': '1h',
+    '1D': '1d',
+    '1d': '1d',
+    '1day': '1d',
+}
+
+# Yahoo lookback limits by interval. Requesting more returns empty or errors.
+_MAX_LOOKBACK = {
+    '1m': pd.Timedelta(days=7),
+    '2m': pd.Timedelta(days=60),
+    '5m': pd.Timedelta(days=60),
+    '15m': pd.Timedelta(days=60),
+    '30m': pd.Timedelta(days=60),
+    '60m': pd.Timedelta(days=730),
+    '90m': pd.Timedelta(days=60),
+    '1h': pd.Timedelta(days=730),
+    '1d': None,
+    '5d': None,
+    '1wk': None,
+    '1mo': None,
+    '3mo': None,
+}
+
+
+class YahooDataStream:
+    """
+    Market data from Yahoo Finance via yfinance.
+
+    Yahoo has no public equities WebSocket. This polls OHLCV bars.
+    Quotes are typically delayed (~15 minutes for US equities).
+    Unofficial API: rate limits and schema changes are expected failure modes.
+    """
+
+    def __init__(self, config: Dict[str, Any]):
+        self.config = config
+        trading = config.get('trading', {})
+        if trading.get('symbols'):
+            self.symbols = list(trading['symbols'])
+        elif trading.get('symbol'):
+            self.symbols = [trading['symbol']]
+        else:
+            self.symbols = ['AAPL']
+        yahoo_cfg = config.get('yahoo', {})
+        self.poll_interval = int(yahoo_cfg.get('poll_interval_seconds', 60))
+        self.auto_adjust = bool(yahoo_cfg.get('auto_adjust', False))
+        self.interval = self._map_interval(config.get('trading', {}).get('timeframe', '1d'))
+        self.data_callbacks: List[Callable] = []
+        self.is_connected = False
+        self.data_buffer: Dict[str, Dict[str, Any]] = {}
+        self._stop_event = threading.Event()
+        self._poll_thread: Optional[threading.Thread] = None
+        self._last_bar_ts: Dict[str, pd.Timestamp] = {}
+        self._buffer_size = int(config.get('realtime_data', {}).get('buffer_size', 100))
+
+        for symbol in self.symbols:
+            self.data_buffer[symbol] = {
+                'trades': [],
+                'quotes': [],
+                'bars': [],
+                'latest_bar': None,
+            }
+
+        logger.info(
+            "Initialized YahooDataStream symbols=%s interval=%s poll_interval=%ss",
+            self.symbols,
+            self.interval,
+            self.poll_interval,
+        )
+
+    @staticmethod
+    def _map_interval(timeframe: str) -> str:
+        mapped = _INTERVAL_MAP.get(str(timeframe), None)
+        if mapped is None:
+            logger.warning("Unknown timeframe %s, defaulting to 1d", timeframe)
+            return '1d'
+        return mapped
+
+    def connect(self) -> None:
+        """Start polling Yahoo for new bars."""
+        if self.is_connected:
+            logger.info("Yahoo data stream already connected")
+            return
+
+        self._stop_event.clear()
+        self._poll_once()
+        self._poll_thread = threading.Thread(target=self._poll_loop, name='yahoo-poll', daemon=True)
+        self._poll_thread.start()
+        self.is_connected = True
+        logger.info("Yahoo data stream polling started")
+
+    def disconnect(self) -> None:
+        self._stop_event.set()
+        self.is_connected = False
+        if self._poll_thread and self._poll_thread.is_alive():
+            self._poll_thread.join(timeout=min(5, self.poll_interval + 1))
+        logger.info("Disconnected from Yahoo data stream")
+
+    def is_streaming(self) -> bool:
+        return self.is_connected and self._poll_thread is not None and self._poll_thread.is_alive()
+
+    def add_data_callback(self, callback: Callable) -> None:
+        self.data_callbacks.append(callback)
+
+    def get_latest_data(self, symbol: str) -> Dict[str, Any]:
+        if symbol not in self.data_buffer:
+            return {}
+        buffer = self.data_buffer[symbol]
+        return {
+            'latest_trade': buffer['trades'][-1] if buffer['trades'] else None,
+            'latest_quote': buffer['quotes'][-1] if buffer['quotes'] else None,
+            'latest_bar': buffer['latest_bar'],
+            'recent_trades': buffer['trades'][-10:] if buffer['trades'] else [],
+            'recent_quotes': buffer['quotes'][-10:] if buffer['quotes'] else [],
+        }
+
+    def get_historical_data(self, symbol: str, start_date: str, end_date: str) -> pd.DataFrame:
+        start, end = self._clamp_window(start_date, end_date, self.interval)
+        try:
+            raw = self._download(symbol, start=start, end=end, interval=self.interval)
+            df = self._normalize_ohlcv(raw)
+            if df.empty:
+                logger.warning("No Yahoo historical data for %s between %s and %s", symbol, start, end)
+            else:
+                logger.info("Loaded %s Yahoo bars for %s (%s to %s)", len(df), symbol, start, end)
+            return df
+        except Exception as e:
+            logger.error("Error fetching Yahoo historical data for %s: %s", symbol, e, exc_info=True)
+            return pd.DataFrame()
+
+    def test_connection(self) -> bool:
+        try:
+            symbol = self.symbols[0] if self.symbols else 'AAPL'
+            df = self._download(symbol, period='5d', interval='1d')
+            if df is None or df.empty:
+                logger.warning("Yahoo connection test returned no data for %s", symbol)
+                return False
+            logger.info("Yahoo connection test succeeded for %s (%s rows)", symbol, len(df))
+            return True
+        except Exception as e:
+            logger.error("Yahoo connection test failed: %s", e)
+            return False
+
+    def get_connection_status(self) -> Dict[str, Any]:
+        return {
+            'is_connected': self.is_connected,
+            'provider': 'yahoo',
+            'interval': self.interval,
+            'poll_interval_seconds': self.poll_interval,
+            'symbols': self.symbols,
+            'data_buffers': {
+                symbol: len(buffer['bars']) for symbol, buffer in self.data_buffer.items()
+            },
+        }
+
+    def generate_simulated_data(self, symbol: str) -> Dict[str, Any]:
+        import random
+
+        latest_data = self.get_latest_data(symbol)
+        base_price = 150.0
+        if latest_data.get('latest_bar'):
+            base_price = latest_data['latest_bar']['close']
+        elif latest_data.get('latest_trade'):
+            base_price = latest_data['latest_trade']['price']
+
+        price_change = random.uniform(-0.01, 0.01) * base_price
+        new_price = base_price + price_change
+        simulated_bar = {
+            'symbol': symbol,
+            'open': base_price,
+            'high': max(base_price, new_price),
+            'low': min(base_price, new_price),
+            'close': new_price,
+            'volume': random.randint(100, 1000),
+            'timestamp': int(time.time() * 1_000_000),
+        }
+        self._store_bar(symbol, simulated_bar, emit=False)
+        return simulated_bar
+
+    def _poll_loop(self) -> None:
+        while not self._stop_event.wait(self.poll_interval):
+            try:
+                self._poll_once()
+            except Exception as e:
+                logger.error("Yahoo poll loop error: %s", e, exc_info=True)
+
+    def _poll_once(self) -> None:
+        for symbol in self.symbols:
+            try:
+                raw = self._download(symbol, period='5d', interval=self.interval)
+                df = self._normalize_ohlcv(raw)
+                if df.empty:
+                    logger.warning("Yahoo poll returned no bars for %s", symbol)
+                    continue
+                self._ingest_new_bars(symbol, df)
+            except Exception as e:
+                logger.error("Yahoo poll failed for %s: %s", symbol, e)
+
+    def _ingest_new_bars(self, symbol: str, df: pd.DataFrame) -> None:
+        last_ts = self._last_bar_ts.get(symbol)
+        rows = df
+        if last_ts is not None:
+            rows = df[df['timestamp'] > last_ts]
+        if rows.empty:
+            return
+
+        for _, row in rows.iterrows():
+            ts = pd.Timestamp(row['timestamp'])
+            bar = {
+                'symbol': symbol,
+                'open': float(row['open']),
+                'high': float(row['high']),
+                'low': float(row['low']),
+                'close': float(row['close']),
+                'volume': float(row['volume']),
+                'timestamp': int(ts.timestamp() * 1_000_000),
+            }
+            self._store_bar(symbol, bar, emit=True)
+            self._last_bar_ts[symbol] = ts
+
+    def _store_bar(self, symbol: str, bar: Dict[str, Any], emit: bool) -> None:
+        buffer = self.data_buffer[symbol]
+        buffer['bars'].append(bar)
+        buffer['latest_bar'] = bar
+        if len(buffer['bars']) > self._buffer_size:
+            buffer['bars'] = buffer['bars'][-self._buffer_size:]
+        if emit:
+            self._notify_callbacks('bar', bar)
+
+    def _notify_callbacks(self, data_type: str, data: Dict[str, Any]) -> None:
+        for callback in self.data_callbacks:
+            try:
+                callback(data_type, data)
+            except Exception as e:
+                logger.error("Error in data callback: %s", e)
+
+    def _clamp_window(self, start_date: str, end_date: str, interval: str) -> tuple:
+        start = pd.to_datetime(start_date, utc=True).tz_convert(None)
+        end = pd.to_datetime(end_date, utc=True).tz_convert(None)
+        max_lookback = _MAX_LOOKBACK.get(interval)
+        if max_lookback is not None:
+            earliest = pd.Timestamp.now(tz='UTC').tz_convert(None) - max_lookback
+            if start < earliest:
+                logger.warning(
+                    "Yahoo %s bars only cover ~%s; clamping start from %s to %s",
+                    interval,
+                    max_lookback,
+                    start.date(),
+                    earliest.date(),
+                )
+                start = earliest
+        if end < start:
+            end = start + pd.Timedelta(days=1)
+        return start.strftime('%Y-%m-%d'), end.strftime('%Y-%m-%d')
+
+    def _download(
+        self,
+        symbol: str,
+        start: Optional[str] = None,
+        end: Optional[str] = None,
+        period: Optional[str] = None,
+        interval: str = '1d',
+    ) -> pd.DataFrame:
+        import yfinance as yf
+
+        kwargs: Dict[str, Any] = {
+            'tickers': symbol,
+            'interval': interval,
+            'auto_adjust': self.auto_adjust,
+            'progress': False,
+            'threads': False,
+        }
+        if period:
+            kwargs['period'] = period
+        else:
+            kwargs['start'] = start
+            kwargs['end'] = end
+        return yf.download(**kwargs)
+
+    @staticmethod
+    def _normalize_ohlcv(df: pd.DataFrame) -> pd.DataFrame:
+        if df is None or df.empty:
+            return pd.DataFrame(columns=['timestamp', 'open', 'high', 'low', 'close', 'volume'])
+
+        out = df.copy()
+        if isinstance(out.columns, pd.MultiIndex):
+            out.columns = [str(col[0]).strip().lower() for col in out.columns]
+        else:
+            out.columns = [str(c).strip().lower() for c in out.columns]
+        rename = {}
+        if 'datetime' in out.columns:
+            rename['datetime'] = 'timestamp'
+        out = out.rename(columns=rename)
+
+        if 'timestamp' not in out.columns:
+            out = out.reset_index()
+            time_col = out.columns[0]
+            out = out.rename(columns={time_col: 'timestamp'})
+
+        out['timestamp'] = pd.to_datetime(out['timestamp'], utc=True).dt.tz_localize(None)
+
+        required = ['timestamp', 'open', 'high', 'low', 'close', 'volume']
+        missing = [c for c in required if c not in out.columns]
+        if missing:
+            logger.error("Yahoo response missing columns: %s", missing)
+            return pd.DataFrame(columns=required)
+
+        out = out[required].dropna()
+        out = out.sort_values('timestamp').drop_duplicates(subset=['timestamp'])
+        return out.reset_index(drop=True)

config.yaml

@@ -29,6 +29,14 @@ alpaca:
   websocket_url: 'wss://stream.data.alpaca.markets/v2/iex'  # WebSocket URL
   account_type: 'paper'  # 'paper' or 'live'
 
+# Yahoo Finance (optional). Set data_source.type: 'yahoo' to use.
+# Unofficial API, typically delayed; 1m lookback is ~7 days.
+yahoo:
+  poll_interval_seconds: 60
+  auto_adjust: false
+  start_date: '2024-01-01'
+  end_date: '2026-12-31'
+
 # Synthetic data generation settings
 synthetic_data:
   base_price: 150.0

requirements.txt

@@ -15,7 +15,7 @@ torch>=1.9.0
 
 # Alpaca integration
 alpaca-py>=0.42.0
-yfinance>=0.2.0
+yfinance>=1.0
 
 # Testing
 pytest>=6.0.0

tests/test_data_ingestion.py

@@ -95,7 +95,25 @@ class TestDataIngestion:
             
             assert isinstance(result, pd.DataFrame)
             mock_generate.assert_called_once_with(config)
-    
+
+    def test_load_data_yahoo_type(self, config):
+        config['data_source']['type'] = 'yahoo'
+        mock_df = pd.DataFrame({
+            'timestamp': pd.date_range('2024-01-01', periods=10, freq='D'),
+            'open': [150] * 10,
+            'high': [155] * 10,
+            'low': [145] * 10,
+            'close': [152] * 10,
+            'volume': [1000] * 10,
+        })
+        mock_stream = MagicMock()
+        mock_stream.get_historical_data.return_value = mock_df
+        with patch('agentic_ai_system.yahoo_data_stream.YahooDataStream', return_value=mock_stream):
+            result = load_data(config)
+        assert isinstance(result, pd.DataFrame)
+        assert len(result) == 10
+        mock_stream.get_historical_data.assert_called_once()
+
     def test_load_data_invalid_type(self, config):
         """Test loading data with invalid type"""
         config['data_source']['type'] = 'invalid_type'

tests/test_yahoo_data_stream.py

@@ -0,0 +1,60 @@
+import pandas as pd
+import pytest
+from unittest.mock import patch
+
+from agentic_ai_system.yahoo_data_stream import YahooDataStream
+
+
+@pytest.fixture
+def yahoo_config():
+    return {
+        'data_source': {'type': 'yahoo'},
+        'yahoo': {'poll_interval_seconds': 1, 'auto_adjust': False},
+        'trading': {
+            'symbol': 'AAPL',
+            'timeframe': '1d',
+        },
+        'realtime_data': {'buffer_size': 10},
+    }
+
+
+def _sample_yahoo_frame():
+    idx = pd.date_range('2024-06-03', periods=3, freq='D', tz='America/New_York')
+    return pd.DataFrame(
+        {
+            'Open': [190.0, 191.0, 192.0],
+            'High': [191.5, 192.5, 193.5],
+            'Low': [189.0, 190.0, 191.0],
+            'Close': [191.0, 192.0, 193.0],
+            'Volume': [1_000_000, 1_100_000, 1_200_000],
+        },
+        index=idx,
+    )
+
+
+class TestYahooDataStream:
+    def test_initialization_from_symbol(self, yahoo_config):
+        stream = YahooDataStream(yahoo_config)
+        assert stream.symbols == ['AAPL']
+        assert stream.interval == '1d'
+
+    def test_normalize_ohlcv(self, yahoo_config):
+        stream = YahooDataStream(yahoo_config)
+        df = stream._normalize_ohlcv(_sample_yahoo_frame())
+        assert list(df.columns) == ['timestamp', 'open', 'high', 'low', 'close', 'volume']
+        assert len(df) == 3
+        assert df['close'].iloc[-1] == 193.0
+
+    def test_clamp_intraday_lookback(self, yahoo_config):
+        yahoo_config['trading']['timeframe'] = '1m'
+        stream = YahooDataStream(yahoo_config)
+        start, end = stream._clamp_window('2020-01-01', '2026-01-01', '1m')
+        assert start > '2020-01-01'
+        assert end >= start
+
+    def test_get_historical_data(self, yahoo_config):
+        stream = YahooDataStream(yahoo_config)
+        with patch.object(stream, '_download', return_value=_sample_yahoo_frame()):
+            df = stream.get_historical_data('AAPL', '2024-01-01', '2024-12-31')
+        assert len(df) == 3
+        assert 'open' in df.columns