PLAN.md

Architecture Migration Plan: Local Server + HF Client

Date: 2026-03-19

---

Current Architecture (Problem)

HuggingFace Space
├── live_trading_multi.py  (bot, all logic)
├── src/ui/api_server.py   (Flask REST API, port 5001)
└── src/ui/app.py          (Streamlit UI, port 8501)
     └── calls http://127.0.0.1:5001/api/*

Problems:

• Binance Testnet geo-blocked from HuggingFace's datacenter (HTTP 451)
• Cloudflare Workers proxy is unreliable and adds latency
• Bot cannot reach testnet.binance.vision from HF

---

Target Architecture (Solution)

Local Machine
├── live_trading_multi.py  (bot, trading logic)
├── src/ui/api_server.py   (Flask REST API, port 5001)
└── [tunnel tool]          (exposes :5001 as public HTTPS URL)
     └── e.g. https://abc123.ngrok.io → localhost:5001

HuggingFace Space
└── src/ui/app.py          (Streamlit UI, CLIENT only)
     └── calls $API_SERVER_URL/api/*
          └── API_SERVER_URL = https://abc123.ngrok.io (HF Secret)

Benefits:

• Bot runs locally → direct Binance testnet access (no geo-blocks)
• HF Space becomes a pure UI/client, no trading logic
• Testnet tab works via server-side testnet_server.py (direct connection locally)

---

Implementation Plan

Step 1: Configure Flask API Server for Remote Access

File: src/ui/api_server.py

Changes:

1. Enable CORS properly so HuggingFace domain can call the API
2. Add a /api/ping health endpoint (lightweight check for client)
3. Add startup banner showing the public URL

from flask_cors import CORS
CORS(app, origins=["*"])  # Allow HF and any origin

File: start_local_server.sh (new)

A convenience script to:

1. Start the Flask API server (port 5001)
2. Optionally start a tunnel (ngrok or Cloudflare)
3. Print the public URL to copy into HF Secrets

---

Step 2: Make Streamlit UI Configurable

File: src/ui/app.py

Changes:

1. Replace all hardcoded http://127.0.0.1:5001 with get_api_url() helper
2. get_api_url() reads from os.environ.get('API_SERVER_URL', 'http://127.0.0.1:5001')
3. Show connection status in UI (connected/disconnected + server URL)
4. Graceful degradation when server is unreachable (not just red error)

Occurrences of hardcoded URL to replace:

• Line 1183: requests.get('http://127.0.0.1:5001/api/state', ...)
• Line 1221: requests.get(f'http://127.0.0.1:5001/api/market?symbol=...', ...)
• Line 1427: requests.get('http://127.0.0.1:5001/api/state', ...)
• Line 1438: requests.get(f'http://127.0.0.1:5001/api/market?symbol=...', ...)
• Line 1459: requests.get('http://127.0.0.1:5001/api/trades', ...)
• Line 1539: requests.get('http://127.0.0.1:5001/api/state', ...)
• Line 1553: requests.get('http://127.0.0.1:5001/api/trades', ...)

---

Step 3: Fix Testnet Tab

File: src/ui/testnet_server.py

Changes:

1. Remove the HuggingFace detection/warning block (it no longer applies)
2. Remove Cloudflare proxy fallback (not needed locally)
3. Simplify to direct BinanceConnector with testnet=True
4. Server now calls https://testnet.binance.vision directly

File: src/ui/app.py (testnet tab section)

Changes:

1. Always use server-side (testnet_server.py) since server runs locally
2. Remove the client-side JS fallback (optional, can keep as backup)

---

Step 4: Add Tunnel Support (ngrok)

File: requirements.txt

Add: pyngrok>=7.0.0

File: start_local_server.py (new script)

# Start Flask server with optional ngrok tunnel
# Usage:
#   python start_local_server.py              # local only
#   python start_local_server.py --tunnel     # with ngrok tunnel

Logic:

1. Start api_server.py in a background thread
2. If --tunnel: start pyngrok tunnel, print public URL
3. Print instructions to add URL to HF Secrets

---

Step 5: Integration Tests

File: tests/test_integration/test_server_client.py (new)

Tests:

1. test_server_starts() - Flask server starts, /health returns 200
2. test_api_state() - /api/state returns valid JSON
3. test_api_trades() - /api/trades returns list
4. test_api_market() - /api/market?symbol=BTCUSDT returns market data
5. test_cors_headers() - Response includes CORS headers
6. test_binance_testnet_connectivity() - testnet.binance.vision ping returns 200
7. test_binance_testnet_with_keys() - API keys work for account balance
8. test_client_url_configurable() - API_SERVER_URL env var is read correctly
9. test_end_to_end_data_flow() - Bot saves state → API serves it → client reads it

---

Step 6: Update HuggingFace Space

File: src/ui/app.py (already updated in Step 2)

HF Space Secrets to add:

• API_SERVER_URL = <ngrok-or-tunnel-url>
• Keep existing: BINANCE_TESTNET_API_KEY, BINANCE_TESTNET_API_SECRET, etc.

File: README.md or HF Space README - update deployment instructions

Push target: Chen4700/drl-trading-bot-dev (DEV space only)

---

File Changes Summary

File	Action	Description
src/ui/api_server.py	Modify	Enable CORS, add /api/ping
src/ui/app.py	Modify	Replace hardcoded localhost:5001 with API_SERVER_URL env var
src/ui/testnet_server.py	Modify	Remove proxy/HF-specific code, direct testnet access
start_local_server.py	Create	Convenience script to start server + optional tunnel
tests/test_integration/test_server_client.py	Create	Integration tests
tests/test_integration/__init__.py	Create	Package init
requirements.txt	Modify	Add pyngrok>=7.0.0
.env.example	Modify	Add API_SERVER_URL example

---

Blockers & Solutions

Blocker 1: HuggingFace Space Needs Public URL

Problem: The local Flask server on :5001 is not reachable from HF by default. Solution: Use ngrok (pyngrok) or Cloudflare Tunnel (cloudflared) to expose it.

• ngrok: requires account but easy setup, free tier works
• Cloudflare Tunnel: free, persistent, but requires cloudflared binary
• Chosen: ngrok via pyngrok (Python package, easiest to automate)

Blocker 2: CORS on Flask

Problem: HuggingFace serves the Streamlit app from a different origin. Solution: Add flask-cors (already in requirements) and configure CORS(app, origins=["*"]).

Blocker 3: Testnet Keys in HF Secrets vs .env

Problem: The testnet tab reads keys from os.getenv() — works both locally (.env) and on HF (secrets). Solution: No change needed. Keep reading from env vars.

Blocker 4: Streamlit on HF can't do ngrok

Problem: Streamlit on HF is the CLIENT, not the server — it doesn't need ngrok. Solution: ngrok only runs on the LOCAL server side.

---

Execution Order

1. Write PLAN.md (this file)
2. Modify src/ui/api_server.py — Enable CORS + add /api/ping
3. Modify src/ui/app.py — Replace hardcoded URLs with API_SERVER_URL
4. Modify src/ui/testnet_server.py — Remove proxy/HF blocks (direct testnet)
5. Create start_local_server.py — Server startup script with optional ngrok
6. Add pyngrok to requirements.txt
7. Add API_SERVER_URL to .env.example
8. Create integration tests (tests/test_integration/test_server_client.py)
9. [~] Run tests locally — BLOCKED: this machine has no project venv (no flask/dotenv/etc.)
   • Static checks passed: all 5 modified files parse cleanly (ast.parse)
   • Logic check passed: get_api_url() default/custom/trailing-slash behavior verified
   • Storage check passed: JsonFileStorage save/load verified (when deps available)
   • Binance testnet check passed: testnet.binance.vision/api/v3/ping reachable locally
   • Full pytest suite must be run in the project's virtualenv: pip install -r requirements.txt && pytest tests/test_integration/ -v
10. Push to Chen4700/drl-trading-bot-dev