File size: 4,410 Bytes
3339913 2d2e42a 3339913 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | ---
title: Backtest Reality Check
emoji: 🎲
colorFrom: blue
colorTo: gray
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: true
license: apache-2.0
short_description: Your backtest is probably lying to you. This proves it.
tags:
- finance
- quantitative-finance
- algorithmic-trading
- backtesting
- statistics
- time-series
---
# Backtest Reality Check
**Your backtest is probably lying to you.**
Pick a market and a trading rule. This Space runs the backtest — and then spends
the rest of its effort trying to prove the result was luck.
Most backtesting tools answer *"how much would this have made?"*. That is the easy
question, and the answer is almost always flattering. This one answers the question
you need before risking money: **how much of that was luck?**
## The four ways a backtest lies, and the test for each
| The lie | The test |
|---|---|
| The market had no structure to find | **Permutation test** — re-run your rule on hundreds of shuffled markets |
| You tried 200 things and reported the best | **Deflated Sharpe Ratio** — charge for every variant you tried |
| The parameters were fitted to the past | **PBO + walk-forward** — does the in-sample winner keep winning? |
| The edge is smaller than the costs | **Cost stress test** — triple the friction and see what survives |
Each contributes to a single **Reality Score** out of 100, with a grade from A to F.
The scale is deliberately harsh. Most strategies people post online score below 40.
## Two labs
**The Lab** validates a timing rule on one asset. **The Portfolio Lab** validates a
book that ranks many names — and it gets a harder null: we keep every date's gross
exposure, net exposure and position count exactly as they were and randomise only
**which name got which weight**. A book that beats that is picking names. One that
doesn't was being paid for style exposure you can buy in an ETF, which the factor
regression measures directly.
It also measures survivorship rather than assuming it away. A universe where every
name is still trading after ten years was chosen after the fact, and every number
computed on it is an upper bound.
## Try this first
Run the **Arena** tab on `SPY`. On most markets and most date ranges, plain
**buy & hold** tops the leaderboard, and the **coin flip** control out-ranks
several respectable-looking strategies. That is not a bug in the app — it is the
finding.
## How the permutation test works
We take the real price series and shuffle it. Each bar's gap, high, low, body and
volume are kept intact, but their **order** is destroyed. The result is a market
with the same volatility and the same fat tails, and no exploitable structure at
all. Then we re-run *your exact rule* on hundreds of these shuffled markets.
If your Sharpe ratio sits comfortably inside that cloud, your rule found nothing
a coin-flip market would not also have handed it.
## No look-ahead, by construction
A strategy emits a target exposure at each bar's close using only data up to that
bar. The engine holds `position[t] = target[t - lag]` with `lag >= 1`, so a signal
computed on Tuesday's close cannot earn Tuesday's move. That is the single line
where look-ahead could enter, and the test suite asserts it directly.
## Use it from Python
```python
from algotrader import LabConfig, run_lab
report = run_lab(LabConfig(symbol="SPY", strategy="sma_cross"))
print(report.verdict["grade"], report.verdict["score"])
print(report.permutation.p_value, report.dsr["dsr"], report.pbo["pbo"])
```
Or from the command line:
```bash
python -m algotrader.cli lab --symbol SPY --strategy donchian_breakout --permutations 500
python -m algotrader.cli arena --symbol BTC-USD
```
## Data
Live prices come from Yahoo Finance. When the network is unavailable or rate-limited,
the app falls back to a deterministic market simulator with regime switching, fat
tails and volatility clustering — and says so, clearly, on every result. The
statistics remain valid; they are just measured on a simulated market.
## References
- Bailey & López de Prado (2014), *The Deflated Sharpe Ratio*
- Bailey, Borwein, López de Prado & Zhu (2016), *The Probability of Backtest Overfitting*
- Masters (2018), *Permutation and Randomization Tests for Trading System Development*
---
Apache-2.0. Research tooling, not investment advice. Nothing here is a
recommendation to trade.
|