ALGORITHM_RESEARCH.md

RL Algorithm Research: Upgrade Analysis for Crypto Trading

Summary

This document analyses three reinforcement learning algorithms for the walk-forward crypto trading pipeline, justifies the selection of RecurrentPPO (primary) and QRDQN (secondary) over plain PPO, and provides implementation rationale with honest caveats.

---

1. Why Plain PPO Falls Short for Sequential Financial Data

PPO (MlpPolicy) processes each observation as a stateless, independent sample. At each step the policy sees only a fixed-length lookback window (48 bars) concatenated into a flat vector. This has two structural weaknesses:

1. Markov assumption violated: financial markets have long-range dependencies—regime changes, funding rate cycles, whale accumulation patterns—that span hundreds of bars. A 48-bar window misses these.
2. No memory across steps: the policy cannot distinguish "I've been holding this position for 3 hours" from "I just opened". It must re-infer context from raw prices every step.

Both RecurrentPPO and QRDQN address different aspects of these shortcomings.

---

2. RecurrentPPO (LSTM) — Primary Recommendation

What it is

RecurrentPPO (from sb3-contrib) replaces the MLP policy with an LSTM-based policy (MlpLstmPolicy). The LSTM hidden state propagates across timesteps within an episode, giving the agent a memory of recent history beyond the fixed observation window.

Why it fits financial time-series

• Regime awareness: LSTM can learn to detect bull/bear regimes and adjust risk exposure accordingly without explicit regime labels.
• Position tracking: the hidden state naturally encodes "how long have I been in this trade", enabling better exit timing.
• Sequential inductive bias: financial data is a sequence. Imposing a recurrent structure matches the domain.

Academic references

• Moody et al. (1998) — "Performance Functions and Reinforcement Learning for Trading Systems and Portfolios" — early demonstration that recurrent networks outperform feedforward for trading.
• Lim & Zohren (2021) — "Time-series forecasting with deep learning: a survey" (Philosophical Transactions A) — survey of temporal architectures for financial prediction.
• Théate & Ernst (2021) — "An application of deep reinforcement learning to algorithmic trading" (Expert Systems with Applications) — shows LSTM-based DRL agents outperform MLP baselines on equity data.
• Carapuço et al. (2018) — "Reinforcement learning applied to Forex trading" — demonstrates recurrent policies learning multi-step strategies.

Expected improvements (with honest caveats)

Metric	Expectation	Caveat
Sharpe ratio	+5–20% vs PPO baseline	Highly data-dependent; noisy markets can hurt
Overfit resistance	Similar or slightly worse	More parameters → risk of memorisation; mitigated by smaller net_arch
Training stability	Slightly worse	LSTM gradients can explode; mitigated by lower lr=3e-5, clip_range=0.1
Training speed	~30% slower	LSTM forward pass is heavier than MLP

Key caveat: RecurrentPPO can overfit more severely if the LSTM memorises specific historical sequences. The walk-forward validation Sharpe early stopping is critical here. Do not deploy if val/test Sharpe ratio exceeds 3×.

Hyperparameter rationale

Parameter	Value	Reason
lstm_hidden_size	64	Enough memory capacity without exploding parameter count
n_lstm_layers	1	Single layer avoids vanishing gradients; multiple layers rarely help for RL
learning_rate	3e-5	3× lower than PPO — LSTM loss landscape is steeper
clip_range	0.1	More conservative than PPO's 0.15 — LSTM policies are more sensitive to large updates
ent_coef	0.02	Lower than PPO's 0.05 — LSTM explores implicitly through memory
n_steps	512	Shorter rollouts — LSTM state carries information, so full 2048-step rollouts are not needed
net_arch	[64]	Small feedforward head after LSTM — total params remain ~50K

---

3. QRDQN — Secondary Recommendation

What it is

Quantile Regression DQN (Dabney et al., 2017) is a distributional RL algorithm. Instead of learning the expected Q-value E[R], it learns the full return distribution via N quantile heads. The agent implicitly models risk alongside expected return.

Why it fits risk-sensitive trading

• Risk-aware decisions: by knowing the full distribution of outcomes, the policy can prefer lower-variance actions — critical when managing drawdown.
• No Gaussian assumption: markets have fat tails. Distributional RL captures this without explicit modelling.
• Off-policy efficiency: QRDQN uses an experience replay buffer (100K steps), enabling data-efficient learning and stable updates. Useful when training data is limited.

Academic references

• Dabney et al. (2017) — "Distributional Reinforcement Learning with Quantile Regression" (AAAI 2018) — original QRDQN paper.
• Lim et al. (2022) — "Distributional Reinforcement Learning for Risk-Aware Portfolio Optimization" — shows quantile-based agents achieve better Sharpe/CVaR tradeoffs than standard DRL.
• Yang et al. (2020) — "Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy" (ACM ICAIF) — ensemble of distributional agents outperforms single-agent baselines.

Expected improvements (with honest caveats)

Metric	Expectation	Caveat
Max drawdown	-10–25% reduction vs PPO	Depends on market regime; only holds if risk-aversion is rewarded
Sharpe ratio	Comparable or +5–15%	Distributional benefit requires sufficient exploration; learning_starts=5000
Training stability	Better than PPO in volatile regimes	Replay buffer smooths out noisy reward signals
Sample efficiency	Better (off-policy)	Buffer size=100K means early training uses replayed experience

Key caveat: QRDQN treats the problem as discrete action selection (HOLD / LONG / SHORT). It does not model continuous position sizing. The risk-awareness manifests in which discrete action to take, not in how much capital to allocate — which limits the practical risk management benefit.

Hyperparameter rationale

Parameter	Value	Reason
n_quantiles	50	50 quantile heads give a detailed distribution estimate; diminishing returns beyond 100
buffer_size	100_000	~4 months of hourly data; large enough to decorrelate samples
learning_starts	5_000	Collect baseline experience before first gradient update
batch_size	256	Larger batches for stable quantile regression
train_freq	4	Update every 4 steps — balance between data efficiency and compute
target_update_interval	1_000	Slow target network updates for stability
net_arch	[128, 64]	Slightly larger than RecurrentPPO — no LSTM overhead
learning_rate	1e-4	Standard for DQN-family

---

4. Algorithm Comparison Table

Property	PPO (baseline)	RecurrentPPO	QRDQN
Type	On-policy	On-policy	Off-policy
Memory	None (flat obs)	LSTM hidden state	Replay buffer
Distribution modelling	No	No	Yes (quantile)
Temporal memory	Observation window only	Learned LSTM state	None
Training parallelism	4 envs (used here)	4 envs	1 env (off-policy)
Sample efficiency	Low	Low	High
Risk-awareness	None	Implicit (via memory)	Explicit (quantile)
Complexity	Low (~30K params)	Medium (~50K params)	Medium (~40K params)
Stability	High	Medium	High
Recommended for	Baseline / fast iteration	Primary production	Risk-sensitive markets
sb3-contrib required	No	Yes	Yes
Early stop mechanism	Rollout-based (val Sharpe)	Rollout-based (val Sharpe)	Step-based (val Sharpe)
Ensemble compatible	Yes	Yes (LSTM state tracked)	Yes

---

5. Algorithms Considered but Not Recommended

Algorithm	Reason not selected
SAC	Continuous action space only; discrete trading actions require discretisation hacks
TD3	Same as SAC — continuous-only
A2C	Synchronous updates, inferior to PPO in most benchmarks, no benefit here
DQN (standard)	QRDQN strictly dominates — same architecture, better by distributional learning
Rainbow	Overkill for this problem; adds n-step, prioritised replay, noisy nets — complex to tune
Dreamer / MBRL	Model-based RL requires accurate world model; financial markets too non-stationary
TRPO	Superceded by PPO; slower, harder to implement with SB3

---

6. Implementation Plan

Phase 1 (complete): Algorithm layer integration

• Add --algorithm flag: ppo, recurrent_ppo, qrdqn
• Add --compare flag running all 3 with tabular output
• LSTM state management in ValidationSharpCallback and evaluate_model_on_env
• Off-policy compatibility (QRDQN uses same step-based callback, handles empty ep_info_buffer gracefully)
• Ensemble majority voting (works across algorithm types; per-model LSTM state tracked)
• Algorithm saved per fold (algorithm.txt) for correct loading

Phase 2 (next steps)

• Hyperparameter search per algorithm using Optuna on a single fold
• Cross-algorithm ensemble: combine best PPO + best RecurrentPPO + best QRDQN models
• Attention-based policy (Transformer) as a future RecurrentPPO successor
• Per-asset algorithm selection based on walk-forward OOS Sharpe

Phase 3 (research)

• Continuous action space (position sizing): switch to SAC/TD3 with a bounded continuous action
• Multi-asset recurrent policy with shared LSTM and asset-specific heads
• Distributional ensemble: combine QRDQN quantile outputs across folds before argmax

---

7. Usage Reference

# Train with RecurrentPPO (primary recommendation)
python train_walkforward_v2.py --asset BTCUSDT --algorithm recurrent_ppo

# Train with QRDQN (secondary — risk-aware)
python train_walkforward_v2.py --asset BTCUSDT --algorithm qrdqn

# PPO baseline (for comparison)
python train_walkforward_v2.py --asset BTCUSDT --algorithm ppo

# Run all 3 and get comparison table
python train_walkforward_v2.py --asset BTCUSDT --compare

# Quick test (1 fold, 50K steps)
python train_walkforward_v2.py --asset BTCUSDT --algorithm recurrent_ppo \
    --max-folds 1 --total-timesteps 50000

# Build ensemble after training
python train_walkforward_v2.py --asset BTCUSDT --algorithm recurrent_ppo \
    --eval-only --ensemble

---

8. Important Caveats

1. No algorithm is guaranteed to outperform PPO on any specific asset or time period. The walk-forward OOS Sharpe is the only honest measure. Run --compare and pick the winner empirically.

2. RecurrentPPO's LSTM can memorise specific price sequences if trained too long. The validation Sharpe early stopping (patience=6) is the primary defence. Monitor val/test Sharpe ratio in overfit_report.json.

3. QRDQN's risk-awareness is limited to discrete action selection. True risk management (position sizing, Kelly criterion) requires a continuous action space or a separate risk module.

4. Training time increases significantly: RecurrentPPO is ~30–40% slower than PPO per step. QRDQN is comparable to PPO per step but needs more total steps due to learning_starts.

5. These algorithms do not change the reward function or environment. All gains or losses are purely from the policy architecture. The environment's stop-loss and take-profit rules still apply.