tags:
- ml-intern
- crypto
- finance
- backtesting
- evaluation
crypto-quant-eval
Institutional-grade evaluation harness for crypto quant trading strategies, plus systematic strategy sweeps (hourly single-asset and daily multi-asset). The companion dataset lives at Avi-L/crypto-quant-eval-data.
What this eval measures
A strategy "passes" only if it survives all of:
| Check | Where | Rule |
|---|---|---|
| No lookahead (purge + embargo) | crypto_eval/cv.py |
label window [t, t+h] must never overlap a test block; embargo >= horizon (8h = one funding interval) enforced |
| Leakage audit | crypto_eval/cv.py |
independent per-segment audit of every CPCV fold; run fails loudly on violation |
| Fee-aware PnL | crypto_eval/costs.py |
5 bps taker / 2 bps maker (Binance VIP0), funding accrued at 8h ticks |
| Slippage | crypto_eval/costs.py |
square-root impact k·σ_daily·√(Q/ADV), σ_daily = realized daily vol from the data (317 bps on this dataset), 2 bps floor |
| Deflated Sharpe (DSR) | crypto_eval/stats.py |
accept only if DSR > 0.95 |
| Probability of Backtest Overfitting (PBO) | crypto_eval/stats.py |
accept only if PBO <= 0.5 |
| Min Track Record Length | crypto_eval/stats.py |
how long the track record must be before SR is significant |
Splitting uses CPCV (N=6 blocks, K=2 held out → 15 combinations) via
purgedcv 0.1.11, following
López de Prado, Advances in Financial Machine Learning (Wiley 2018),
ch. 7 (purged K-fold + embargo) and ch. 11–12 (DSR, PBO, MinBTL).
Quickstart
pip install -r requirements.txt
python run_eval.py --data /path/to/curated --symbol BTCUSDT --freq 1h \
--strategy funding_carry
--data is a directory containing the dataset repo's parquet files
(bars_BTCUSDT_1h.parquet, funding_BTCUSDT.parquet, manifest.json).
Strategies are functions (bars, funding) -> positions; add yours to
crypto_eval/strategies.py. Decision-time discipline is enforced
structurally: a bar starting at t closes at t+freq, so signals use
shift_bars (last closed bar at t-1); the engine and audit catch peeking.
Strategy sweep v2 — daily multi-asset panel (2026-10-25): PASS
66 configurations (panel TSMOM/Donchian × 24+8, XS momentum ± trend filter × 18, single-asset BTC/ETH long-only trend × 16) through the full gates on daily data 2014-01 → 2026-10, 4,526 days × 8 majors (coinmetrics VWAP index, 5bps/2bps fees, per-asset ADV impact, 0.1095 APR funding drag).
Winner: Donchian 20d long-only on ETH — the first config to clear every gate:
| Gate | Value | Required | |
|---|---|---|---|
| Deflated Sharpe (66 trials) | 0.9981 | > 0.95 | ✅ |
| PBO (CSCV, 8 splits) | 0.257 | ≤ 0.5 | ✅ |
| Leakage audit (15 CPCV combos) | OK | no leakage | ✅ |
| OOS mean SR (CPCV, purge+embargo 5d) | 1.48 | — | 15/15 combos positive |
| Full-sample SR / MDD | 1.56 / −61% | — | skew +1.28, kurt 18 |
Era analysis (mandatory honesty check): the ETH winner's edge concentrates pre-2022 (2023-26 era SR 0.50). The era-stable picks are BTC long-only trend: Donchian 200d (recent-era SR 0.92, full 0.93), TSMOM 50d (recent 0.89), Donchian 20d (recent 0.87) — lower full-sample SR (~0.9) but no era decay. Engine fix found during this sweep: the portfolio divisor previously granted implicit early-era leverage; corrected (unlisted capital = cash), which lowered all panel numbers — the PASS survived the fix.
Read: long-only trend following on daily crypto is a statistically significant, era-stable (BTC) edge after full institutional costs — with −60/−80% drawdowns inherent to the unlevered profile and an edge magnitude that decays as the market matures.
Strategy sweep v1 — hourly single-asset (2026-10-25): FAIL
Run on real curated data: BTCUSDT 1h, 59,654 bars (2020-01 → 2026-10), 5,295 funding ticks, $1M initial capital, CPCV N=6/K=2, horizon=embargo=8h.
A 136-configuration grid (TSMOM × lookback/long-only/vol-target/band, Donchian,
funding carry, funding-extreme events, mean-reversion controls) was run through
the full harness on real data — every config a counted trial. Full 136-config
ranking in results/sweep_report_v1.json; runner: sweep.py.
VERDICT: FAIL — no configuration passes the institutional gate.
| Rank | Config | OOS mean SR (CPCV) | Combos positive | Net PnL | Gates |
|---|---|---|---|---|---|
| 1 | Donchian 55d long-only | 0.41 | 12/15 | +$61,557 | DSR≈0 ✗, PBO 0.63 ✗ |
| 2 | TSMOM 65d long-only | 0.35 | 11/15 | +$54,597 | DSR≈0 ✗ |
| 3 | TSMOM 200d long-only + vol-target 30% + 20% band | 0.39 | 11/15 | +$35,178 | DSR≈0 ✗ |
The best strategies are modestly positive OOS (SR ~0.3-0.4, essentially long-bias trend capture in a 2020-2026 bull market) but cannot clear the multiple-testing bar: after 136 counted trials, DSR ≈ 0, and PBO 0.63 says config selection inside this grid is worse than a coin flip. Mean-reversion and short-horizon momentum controls are strongly negative, as the literature predicts. Funding carry is positive (+$22k) but individually insignificant (PSR 0.53).
That is the honest institutional answer: a deployable edge is not found by grid search on one asset's 1h bars. The harness is doing its job.
Demo-strategy validation results (costs corrected 2026-10-25)
NOTE: the original 2026-10-23 run used a mis-scaled slippage model (missing the σ_daily term, impact ~50× overstated). All results below use the corrected model. The correction is reported openly because it changed the carry verdict.
| Strategy | Net PnL | Fees | Slippage | Funding | Mean OOS SR | PBO | Verdict |
|---|---|---|---|---|---|---|---|
funding_carry |
+$22,242 | $2,856 | $3,016 | +$5,940 | +0.17 | 0.100 | REJECT (PSR 0.53) |
ma_crossover |
−$62,096 | $29,208 | $47,872 | −$3,469 | −0.21 | 0.057 | REJECT |
The MA crossover remains a cost-loser. The carry trade flips positive once slippage is realistic, but PSR 0.53 is nowhere near significance.
Note on the DSR column: per-combination values are PSR (n_trials=1 within
a combination); DSR applies when multiple strategy configurations are
compared — eval_trial_set computes it with the across-trial SR variance.
Validation performed
- Synthetic-data smoke: 20k GBM bars, both strategies, numeric DSR/PBO (
synth_gen.py) - Leakage test:
--embargo 0with horizon 8 raisesValueError(guard inmake_cpcv) - Sanity: zero-position strategy produces exactly 0.0 net PnL; any turnover produces strictly positive fees + slippage
- No-lookahead audit passes on all 15 CPCV combinations, both real and synthetic data
Known gaps
- ETH enters via funding only; ETH spot 1m (HaianCao/Binance-Public-OHLCV, 34.6 GB) was not ingested
- On-chain metrics are daily (coinmetrics-community), no usable HF on-chain repo exists
- PBO uses n_splits=8 (70 combos) rather than 16 (12,870) — the latter is prohibitively slow on CPU with purge/embargo enabled
- Cross-venue funding divergence (Chainticks/funding-divergence) is verified as a source but not yet wired into the harness
References
- López de Prado, Advances in Financial Machine Learning, Wiley 2018 (ch. 7, 11, 12)
purgedcv— MIT-licensed implementation of the splitting and statistics used here