| --- |
| tags: [ml-intern, crypto, finance, backtesting, evaluation] |
| --- |
| |
| # crypto-quant-eval |
|
|
| [](https://hf.co/chat?mode=ml-intern) |
|
|
| Institutional-grade evaluation harness for crypto quant trading strategies, |
| plus systematic strategy sweeps (hourly single-asset and daily multi-asset). |
| The companion dataset lives at |
| [Avi-L/crypto-quant-eval-data](https://huggingface.co/datasets/Avi-L/crypto-quant-eval-data). |
|
|
| ## What this eval measures |
|
|
| A strategy "passes" only if it survives all of: |
|
|
| | Check | Where | Rule | |
| |---|---|---| |
| | No lookahead (purge + embargo) | `crypto_eval/cv.py` | label window `[t, t+h]` must never overlap a test block; embargo >= horizon (8h = one funding interval) enforced | |
| | Leakage audit | `crypto_eval/cv.py` | independent per-segment audit of every CPCV fold; run fails loudly on violation | |
| | Fee-aware PnL | `crypto_eval/costs.py` | 5 bps taker / 2 bps maker (Binance VIP0), funding accrued at 8h ticks | |
| | Slippage | `crypto_eval/costs.py` | square-root impact `k·σ_daily·√(Q/ADV)`, σ_daily = realized daily vol from the data (317 bps on this dataset), 2 bps floor | |
| | Deflated Sharpe (DSR) | `crypto_eval/stats.py` | accept only if DSR > 0.95 | |
| | Probability of Backtest Overfitting (PBO) | `crypto_eval/stats.py` | accept only if PBO <= 0.5 | |
| | Min Track Record Length | `crypto_eval/stats.py` | how long the track record must be before SR is significant | |
|
|
| Splitting uses CPCV (N=6 blocks, K=2 held out → 15 combinations) via |
| [`purgedcv`](https://pypi.org/project/purgedcv/) 0.1.11, following |
| López de Prado, *Advances in Financial Machine Learning* (Wiley 2018), |
| ch. 7 (purged K-fold + embargo) and ch. 11–12 (DSR, PBO, MinBTL). |
|
|
| ## Quickstart |
|
|
| ```bash |
| pip install -r requirements.txt |
| python run_eval.py --data /path/to/curated --symbol BTCUSDT --freq 1h \ |
| --strategy funding_carry |
| ``` |
|
|
| `--data` is a directory containing the dataset repo's parquet files |
| (`bars_BTCUSDT_1h.parquet`, `funding_BTCUSDT.parquet`, `manifest.json`). |
| Strategies are functions `(bars, funding) -> positions`; add yours to |
| `crypto_eval/strategies.py`. Decision-time discipline is enforced |
| structurally: a bar starting at `t` closes at `t+freq`, so signals use |
| `shift_bars` (last closed bar at `t-1`); the engine and audit catch peeking. |
|
|
| ## Strategy sweep v2 — daily multi-asset panel (2026-10-25): **PASS** |
|
|
| 66 configurations (panel TSMOM/Donchian × 24+8, XS momentum ± trend filter |
| × 18, single-asset BTC/ETH long-only trend × 16) through the full gates on |
| **daily data 2014-01 → 2026-10, 4,526 days × 8 majors** (coinmetrics VWAP |
| index, 5bps/2bps fees, per-asset ADV impact, 0.1095 APR funding drag). |
|
|
| **Winner: Donchian 20d long-only on ETH — the first config to clear every gate:** |
|
|
| | Gate | Value | Required | | |
| |---|---|---|---| |
| | Deflated Sharpe (66 trials) | **0.9981** | > 0.95 | ✅ | |
| | PBO (CSCV, 8 splits) | **0.257** | ≤ 0.5 | ✅ | |
| | Leakage audit (15 CPCV combos) | OK | no leakage | ✅ | |
| | OOS mean SR (CPCV, purge+embargo 5d) | **1.48** | — | 15/15 combos positive | |
| | Full-sample SR / MDD | 1.56 / −61% | — | skew +1.28, kurt 18 | |
|
|
| **Era analysis (mandatory honesty check):** the ETH winner's edge |
| concentrates pre-2022 (2023-26 era SR **0.50**). The **era-stable** picks are |
| BTC long-only trend: Donchian 200d (recent-era SR **0.92**, full 0.93), |
| TSMOM 50d (recent 0.89), Donchian 20d (recent 0.87) — lower full-sample SR |
| (~0.9) but no era decay. Engine fix found during this sweep: the portfolio |
| divisor previously granted implicit early-era leverage; corrected (unlisted |
| capital = cash), which *lowered* all panel numbers — the PASS survived the fix. |
|
|
| Read: long-only trend following on daily crypto is a statistically |
| significant, era-stable (BTC) edge after full institutional costs — with |
| −60/−80% drawdowns inherent to the unlevered profile and an edge magnitude |
| that decays as the market matures. |
|
|
| ## Strategy sweep v1 — hourly single-asset (2026-10-25): **FAIL** |
|
|
| Run on real curated data: BTCUSDT 1h, 59,654 bars (2020-01 → 2026-10), |
| 5,295 funding ticks, $1M initial capital, CPCV N=6/K=2, horizon=embargo=8h. |
|
|
| A 136-configuration grid (TSMOM × lookback/long-only/vol-target/band, Donchian, |
| funding carry, funding-extreme events, mean-reversion controls) was run through |
| the full harness on real data — every config a counted trial. Full 136-config |
| ranking in `results/sweep_report_v1.json`; runner: `sweep.py`. |
|
|
| **VERDICT: FAIL — no configuration passes the institutional gate.** |
|
|
| | Rank | Config | OOS mean SR (CPCV) | Combos positive | Net PnL | Gates | |
| |---|---|---|---|---|---| |
| | 1 | Donchian 55d long-only | 0.41 | 12/15 | +$61,557 | DSR≈0 ✗, PBO 0.63 ✗ | |
| | 2 | TSMOM 65d long-only | 0.35 | 11/15 | +$54,597 | DSR≈0 ✗ | |
| | 3 | TSMOM 200d long-only + vol-target 30% + 20% band | 0.39 | 11/15 | +$35,178 | DSR≈0 ✗ | |
|
|
| The best strategies are modestly positive OOS (SR ~0.3-0.4, essentially |
| long-bias trend capture in a 2020-2026 bull market) but **cannot clear the |
| multiple-testing bar**: after 136 counted trials, DSR ≈ 0, and PBO 0.63 says |
| config selection inside this grid is worse than a coin flip. Mean-reversion |
| and short-horizon momentum controls are strongly negative, as the literature |
| predicts. Funding carry is positive (+$22k) but individually insignificant |
| (PSR 0.53). |
|
|
| That is the honest institutional answer: a deployable edge is not found by |
| grid search on one asset's 1h bars. The harness is doing its job. |
|
|
| ## Demo-strategy validation results (costs corrected 2026-10-25) |
|
|
| NOTE: the original 2026-10-23 run used a mis-scaled slippage model (missing |
| the σ_daily term, impact ~50× overstated). All results below use the corrected |
| model. The correction is reported openly because it changed the carry verdict. |
| |
| | Strategy | Net PnL | Fees | Slippage | Funding | Mean OOS SR | PBO | Verdict | |
| |---|---|---|---|---|---|---|---| |
| | `funding_carry` | **+$22,242** | $2,856 | $3,016 | +$5,940 | +0.17 | 0.100 | **REJECT** (PSR 0.53) | |
| | `ma_crossover` | **−$62,096** | $29,208 | $47,872 | −$3,469 | −0.21 | 0.057 | **REJECT** | |
|
|
| The MA crossover remains a cost-loser. The carry trade flips positive once |
| slippage is realistic, but PSR 0.53 is nowhere near significance. |
|
|
| Note on the DSR column: per-combination values are PSR (n_trials=1 within |
| a combination); DSR applies when multiple strategy configurations are |
| compared — `eval_trial_set` computes it with the across-trial SR variance. |
| |
| ## Validation performed |
| |
| - Synthetic-data smoke: 20k GBM bars, both strategies, numeric DSR/PBO (`synth_gen.py`) |
| - Leakage test: `--embargo 0` with horizon 8 raises `ValueError` (guard in `make_cpcv`) |
| - Sanity: zero-position strategy produces exactly 0.0 net PnL; any turnover produces strictly positive fees + slippage |
| - No-lookahead audit passes on all 15 CPCV combinations, both real and synthetic data |
|
|
| ## Known gaps |
|
|
| - ETH enters via funding only; ETH spot 1m (HaianCao/Binance-Public-OHLCV, |
| 34.6 GB) was not ingested |
| - On-chain metrics are daily (coinmetrics-community), no usable HF on-chain repo exists |
| - PBO uses n_splits=8 (70 combos) rather than 16 (12,870) — the latter is |
| prohibitively slow on CPU with purge/embargo enabled |
| - Cross-venue funding divergence ([Chainticks/funding-divergence](https://huggingface.co/datasets/Chainticks/funding-divergence)) is verified |
| as a source but not yet wired into the harness |
| |
| ## References |
| |
| - López de Prado, *Advances in Financial Machine Learning*, Wiley 2018 (ch. 7, 11, 12) |
| - [`purgedcv`](https://pypi.org/project/purgedcv/) — MIT-licensed implementation of the splitting and statistics used here |
| |
