Spaces:

chen470
/

drl-trading-bot-dev2

Paused

Asset	Trades	Total P&L from DB	State P&L (Before Fix)	Discrepancy
BTCUSDT	8	$-40.92	$-59.60	$-18.68
ETHUSDT	11	$-84.13	MISSING	$-84.13
SOLUSDT	7	$+131.82	MISSING	$+131.82
XRPUSDT	20	$-99.09	MISSING	$-99.09
TOTAL	46	$-92.32	$-59.60	$+32.71

Root Causes Identified

Issue 1: System Started with Partial Assets

# Expected (from start.sh):
python live_trading_multi.py --assets BTCUSDT ETHUSDT SOLUSDT XRPUSDT --balance 5000

# Actual running process:
python live_trading_multi.py --assets BTCUSDT --balance 5000

Result: Only BTCUSDT bot exists in self.bots, so save_state() only saves BTCUSDT data.

Issue 2: No P&L Reconstruction on Startup

When a bot is created, realized_pnl starts at 0:

self.realized_pnl = 0.0  # ❌ Should be loaded from trade history

Even if state is loaded, if the state is missing assets or incorrect, the bot doesn't validate against trade history.

Result: Bot P&L doesn't match historical trades, causing permanent discrepancy.

The Fix

Code Changes (live_trading_multi.py)

Added 3 new methods to MultiAssetOrchestrator:

1. Enhanced `load_state()` - Auto-Reconciliation

def load_state(self):
    """Load state from storage and reconcile with trade history."""
    state = self.storage.load_state()

    if not state:
        # Reconstruct from trade history if no state exists
        self._reconstruct_state_from_trades()
        return

    assets = state.get('assets', {})
    for symbol, asset_state in assets.items():
        if symbol in self.bots:
            self.bots[symbol].restore_state(asset_state)

    # CRITICAL FIX: Reconcile missing assets from trade history
    for symbol, bot in self.bots.items():
        if symbol not in assets:
            # Asset missing from state - reconstruct from trades
            self._reconstruct_bot_state(symbol, bot)
        else:
            # Validate saved P&L matches trade history
            trades_pnl = self._calculate_pnl_from_trades(symbol)
            state_pnl = assets[symbol].get('pnl', 0)

            if abs(trades_pnl - state_pnl) > 0.01:
                logger.error(f"P&L MISMATCH for {symbol}: Correcting...")
                bot.realized_pnl = trades_pnl

2. New `_reconstruct_bot_state()` - Trade History Reconstruction

def _reconstruct_bot_state(self, symbol: str, bot):
    """Reconstruct bot state from trade history."""
    all_trades = self.storage.get_trades(limit=10000)
    symbol_trades = [t for t in all_trades if t.get('symbol') == symbol]

    if not symbol_trades:
        return

    # Calculate total P&L from all historical trades
    total_pnl = sum(t.get('pnl', 0) for t in symbol_trades)

    # Restore state
    bot.realized_pnl = total_pnl

    # ... (restore position, balance, etc.)

3. New Validation in `save_state()` - Ongoing Monitoring

def save_state(self):
    """Save state with P&L validation."""
    # ... existing code ...

    # VALIDATION: Check P&L every 5 minutes
    for symbol, bot in self.bots.items():
        trades_pnl = self._calculate_pnl_from_trades(symbol)
        if abs(bot.realized_pnl - trades_pnl) > 0.01:
            logger.error(
                f"P&L VALIDATION FAILED for {symbol}: "
                f"Bot=${bot.realized_pnl:+.2f}, Trades=${trades_pnl:+.2f}"
            )

Manual Fix Applied

Created and ran fix_portfolio_state.py:

$ ./venv/bin/python fix_portfolio_state.py

✅ Fix applied successfully!
   Before: $-102.89
   After:  $-92.32
   Correction: $+10.58

Validation & Testing

Test Script: `test_portfolio_validation.py`

Tests:

✅ All assets present in state
✅ Per-asset P&L matches trade history
✅ Total P&L matches
✅ Fix metadata preserved

$ ./venv/bin/python test_portfolio_validation.py

✅✅✅ ALL TESTS PASSED ✅✅✅

How to Prevent Recurrence

1. Always Start with All Assets

Update start.sh to always use full asset list:

python -u live_trading_multi.py --assets BTCUSDT ETHUSDT SOLUSDT XRPUSDT --balance 5000

2. Restart Required

The fix is in the code, but the running system must be restarted:

# Kill existing process
pkill -f live_trading_multi.py

# Start with all 4 assets
./start.sh

3. Auto-Validation Active

The code now:

✅ Auto-reconstructs missing asset P&L from trade history
✅ Validates P&L every 5 minutes
✅ Logs errors if discrepancies are detected
✅ Self-heals on startup if state is corrupted

Files Changed

File	Change	Purpose
`live_trading_multi.py`	Enhanced `load_state()`, added `_reconstruct_bot_state()`, added validation in `save_state()`	Auto-reconcile P&L with trade history
`fix_portfolio_state.py`	New script	One-time manual fix for current production state
`test_portfolio_validation.py`	New test	Validate fix is working correctly
`audit_portfolio_bug.py`	New audit tool	Detailed P&L audit across all trades

Next Steps

Restart System with all 4 assets:

pkill -f live_trading_multi.py
./start.sh

Verify Fix after restart:

./venv/bin/python test_portfolio_validation.py

Monitor Logs for P&L validation errors:

tail -f process.log | grep "P&L VALIDATION"

Dashboard Check: Verify portfolio P&L shows $-92.32 (not $-102.89)

Lessons Learned

Always validate on startup: Bot state should always be reconciled with database
Don't trust in-memory state: Trade history is source of truth
Log asset changes: System should log when assets are added/removed
Test state persistence: Automated tests should verify P&L matches trades

Timeline

2026-03-13 10:00 - Bug reported: Portfolio shows -$102.89 vs -$92.32 in trades
2026-03-13 11:00 - Investigation: Created audit scripts
2026-03-13 11:30 - Root cause identified: Missing assets + no P&L reconstruction
2026-03-13 12:00 - Fix implemented in code
2026-03-13 12:56 - Manual fix applied to production state
2026-03-13 13:00 - STATUS: RESOLVED (pending system restart)

Contact

QA Engineer: Claude Sonnet 4.5 Branch: dev Commit: (pending)

END OF REPORT

CRITICAL BUG: Portfolio P&L Mismatch ($10.57 Discrepancy)

Status: FIXED ✅

Summary

Investigation Results

Audit of All 46 Trades