sractch.md

# ML-3m-trader: XAUUSDc 3-Minute Timeframe ML Trading System

End-to-end machine learning pipeline for trading XAUUSDc (Gold) on the 3-minute timeframe. Uses MetaTrader 5 for data acquisition, LightGBM for classification, and a vectorized backtesting engine with realistic execution modeling.

## User Review Required

> [!IMPORTANT]
> **ML Framework Choice: LightGBM** — LightGBM is the best-suited framework for this task because:
> - Tabular classification (Buy/Sell/Hold/DoNothing) is LightGBM's strongest domain
> - Extremely fast training, even on CPU (i5-7200U will handle it fine)
> - Low memory footprint (well within 12 GB RAM)
> - No GPU required (your MX110 is not needed)
> - Outperforms deep learning on structured/tabular data in virtually all benchmarks
>
> **This will run entirely on your local machine. No Google Colab needed.**

> [!WARNING]
> **MetaTrader 5 Python API** only works on Windows (which you have). MT5 must be open and logged in when running the data fetch script. The `MetaTrader5` pip package handles communication.

> [!NOTE]
> **VIX Feature**: Since the CBOE VIX index is not directly available from MT5, the system will compute a **synthetic VIX proxy** using a rolling standard deviation of returns (realized volatility), which is the standard approach in non-US-equity trading systems. If you want the actual VIX, we would need a separate data source.

---

## Proposed Changes

### Project Structure

```
ML-3m-trader/
├── config.py             # All configuration constants
├── data_fetcher.py       # MT5 data acquisition
├── features.py           # Technical indicator computation
├── labeler.py            # Trade label generation (Buy/Sell/Hold/DoNothing)
├── model.py              # LightGBM training, prediction, persistence
├── backtester.py         # Vectorized backtesting engine
├── metrics.py            # Performance evaluation
├── main.py               # CLI entry point
├── requirements.txt
├── LICENSE
├── README.md
├── GUIDE.md              # Step-by-step usage guide with tables
└── .gitignore
```

---

### Configuration

#### [NEW] [config.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/config.py)

Central configuration file containing all tunable parameters:
- `SYMBOL = "XAUUSDc"`, `TIMEFRAME = mt5.TIMEFRAME_M3`
- Feature list, lookback periods for SMA (14, 50), VROC (14), ADX (14), Momentum SI (10)
- Risk/reward ratio = 1.0, default bet percentage logic
- Slippage range (0–2 units), spread filter (`stoploss_size >= spread * 10`)
- Train/test split ratio, model hyperparameters
- Starting equity/balance

---

### Data Acquisition

#### [NEW] [data_fetcher.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/data_fetcher.py)

- Connects to MT5 terminal via `MetaTrader5` Python package
- Fetches 1-year of 3-minute OHLCV bars for XAUUSDc
- Returns a `pandas.DataFrame` with columns: `time, open, high, low, close, volume, spread`
- Saves raw data to `data/raw_xauusdc_3m.csv` for reproducibility
- Handles MT5 connection errors gracefully

---

### Feature Engineering

#### [NEW] [features.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/features.py)

Computes all required technical indicators using pure NumPy/Pandas (no TA-Lib dependency):

| Feature | Method |
|---------|--------|
| SMA | Simple Moving Average (14-period) |
| Double Moving Average | SMA(14) and SMA(50), plus crossover signal |
| VROC | Volume Rate of Change (14-period) |
| Synthetic VIX | Rolling std of log-returns (20-period) as volatility proxy |
| Momentum Strength Index | Custom momentum oscillator (10-period, 0–100 scale) |
| ADX | Average Directional Index (14-period) via Wilder's smoothing |
| Time features | Hour-of-day, minute-of-hour, day-of-week (cyclical encoded) |

All computations are vectorized with NumPy for maximum speed. NaN rows from lookback periods are dropped.

---

### Labeling Engine

#### [NEW] [labeler.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/labeler.py)

Generates ground-truth labels for supervised learning:

1. For each bar, compute a potential **Buy** and **Sell** trade:
   - **Buy**: entry at `close`, SL below recent swing low (ATR-based), TP = entry + (entry - SL) (1:1 RR)
   - **Sell**: entry at `close`, SL above recent swing high (ATR-based), TP = entry - (SL - entry) (1:1 RR)
2. Walk forward through subsequent bars to determine outcome (TP hit, SL hit, or neither within N bars)
3. Apply **spread filter**: if `SL_distance < spread * 10`, label = `DO_NOTHING`
4. Final labels: `BUY_WIN`, `BUY_LOSS`, `SELL_WIN`, `SELL_LOSS`, `HOLD`, `DO_NOTHING` → simplified to 4-class: `BUY (1)`, `SELL (2)`, `HOLD (3)`, `DO_NOTHING (0)`
5. Only winning setups are labeled as BUY/SELL; losing setups become HOLD

---

### ML Model

#### [NEW] [model.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/model.py)

- **LightGBM** multi-class classifier (4 classes)
- Hyperparameters tuned for tabular financial data:
  - `num_leaves=63`, `max_depth=8`, `learning_rate=0.05`, `n_estimators=500`
  - `subsample=0.8`, `colsample_bytree=0.8`, `min_child_samples=20`
  - `class_weight='balanced'` to handle label imbalance
- Train/validation split: 80/20 chronological (no shuffle — time series)
- Feature importance output
- Model persistence via `joblib` (save/load `.pkl`)
- Early stopping on validation set

---

### Backtesting Engine

#### [NEW] [backtester.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/backtester.py)

Vectorized backtesting with realistic execution:

- Takes model predictions and raw price data
- **Position sizing**: bet % of current balance, accounting for full SL distance
  - `lot_value = balance * bet_pct / sl_distance`
- **Random slippage**: uniform 0–2 XAUUSDc units applied to entry price
- **Spread filter**: skip trade if `sl_distance < spread * 10`
- **1:1 Risk-Reward**: TP distance = SL distance
- Walk forward bar-by-bar on test set, track equity curve
- No trade limit — takes every valid signal
- Records all trades with entry/exit prices, PnL, timestamps

---

### Metrics & Evaluation

#### [NEW] [metrics.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/metrics.py)

| Metric | Description |
|--------|-------------|
| Win Rate | % of trades closed at TP |
| Average Win % | Mean profit per winning trade as % of balance |
| Average Loss % | Mean loss per losing trade as % of balance |
| Sharpe Ratio | Annualized risk-adjusted return |
| Sortino Ratio | Downside-risk-adjusted return |
| Max Drawdown | Largest peak-to-trough equity decline |
| Profit Factor | Gross profit / Gross loss |
| Start Equity | Initial balance |
| End Equity | Final balance after all trades |
| Total Trades | Number of executed trades |
| Avg Trade Duration | Mean holding time in bars/minutes |
| Daily PnL Stats | Intraday mean, std, min, max PnL |
| Calmar Ratio | Annualized return / Max Drawdown |
| Expectancy | Average PnL per trade |

Outputs a formatted console report and saves to `results/report.txt`.

---

### CLI Entry Point

#### [NEW] [main.py](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/main.py)

Unified CLI with subcommands:

```
python main.py fetch       # Fetch 1-year data from MT5
python main.py train       # Engineer features, label, train model
python main.py backtest    # Run backtest on test set
python main.py evaluate    # Print metrics report
python main.py run         # Full pipeline: fetch → train → backtest → evaluate
```

Uses `argparse` with clear help text.

---

### Project Files

#### [NEW] [requirements.txt](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/requirements.txt)

```
MetaTrader5>=5.0.45
lightgbm>=4.0.0
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
joblib>=1.3.0
```

#### [NEW] [LICENSE](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/LICENSE)

MIT License, author: Rembrant Oyangoren Albeos, year: 2026.

#### [NEW] [README.md](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/README.md)

Professional README with badges (Python, License, LightGBM), project description, features list, quick start, architecture overview, and configuration reference. No emojis.

#### [NEW] [GUIDE.md](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/GUIDE.md)

Step-by-step usage guide with tables for all commands, parameters, and expected outputs.

#### [NEW] [.gitignore](file:///c:/Users/User/Desktop/debugrem/ML-3m-trader/.gitignore)

Standard Python gitignore plus `data/`, `results/`, `models/`, `*.pkl`.

---

## Verification Plan

### Automated Tests

1. **Syntax validation** — run `python -m py_compile <file>` on every `.py` file to confirm no syntax errors
2. **Import validation** — run `python -c "import config; import features; import labeler; import model; import backtester; import metrics"` to confirm all modules load correctly
3. **Dry-run test** — run `python main.py --help` to confirm CLI is functional

### Manual Verification

1. **User runs `python main.py fetch`** with MT5 open and logged in, confirms data CSV is created in `data/`
2. **User runs `python main.py run`** for the full pipeline, reviews the metrics report output
3. **User inspects `results/report.txt`** for the performance metrics