README.md

---
tags: [ml-intern, crypto, finance, backtesting, evaluation]
---

# crypto-quant-eval

[![Built with ML Intern](https://huggingface.co/datasets/huggingface/badges/resolve/main/built-with-ml-intern-sm.svg)](https://hf.co/chat?mode=ml-intern)

Institutional-grade evaluation harness for crypto quant trading strategies,
plus systematic strategy sweeps (hourly single-asset and daily multi-asset).
The companion dataset lives at
[Avi-L/crypto-quant-eval-data](https://huggingface.co/datasets/Avi-L/crypto-quant-eval-data).

## What this eval measures

A strategy "passes" only if it survives all of:

| Check | Where | Rule |
|---|---|---|
| No lookahead (purge + embargo) | `crypto_eval/cv.py` | label window `[t, t+h]` must never overlap a test block; embargo >= horizon (8h = one funding interval) enforced |
| Leakage audit | `crypto_eval/cv.py` | independent per-segment audit of every CPCV fold; run fails loudly on violation |
| Fee-aware PnL | `crypto_eval/costs.py` | 5 bps taker / 2 bps maker (Binance VIP0), funding accrued at 8h ticks |
| Slippage | `crypto_eval/costs.py` | square-root impact `k·σ_daily·√(Q/ADV)`, σ_daily = realized daily vol from the data (317 bps on this dataset), 2 bps floor |
| Deflated Sharpe (DSR) | `crypto_eval/stats.py` | accept only if DSR > 0.95 |
| Probability of Backtest Overfitting (PBO) | `crypto_eval/stats.py` | accept only if PBO <= 0.5 |
| Min Track Record Length | `crypto_eval/stats.py` | how long the track record must be before SR is significant |

Splitting uses CPCV (N=6 blocks, K=2 held out → 15 combinations) via
[`purgedcv`](https://pypi.org/project/purgedcv/) 0.1.11, following
López de Prado, *Advances in Financial Machine Learning* (Wiley 2018),
ch. 7 (purged K-fold + embargo) and ch. 11–12 (DSR, PBO, MinBTL).

## Quickstart

```bash
pip install -r requirements.txt
python run_eval.py --data /path/to/curated --symbol BTCUSDT --freq 1h \
    --strategy funding_carry
```

`--data` is a directory containing the dataset repo's parquet files
(`bars_BTCUSDT_1h.parquet`, `funding_BTCUSDT.parquet`, `manifest.json`).
Strategies are functions `(bars, funding) -> positions`; add yours to
`crypto_eval/strategies.py`. Decision-time discipline is enforced
structurally: a bar starting at `t` closes at `t+freq`, so signals use
`shift_bars` (last closed bar at `t-1`); the engine and audit catch peeking.

## Strategy sweep v2 — daily multi-asset panel (2026-10-25): **PASS**

66 configurations (panel TSMOM/Donchian × 24+8, XS momentum ± trend filter
× 18, single-asset BTC/ETH long-only trend × 16) through the full gates on
**daily data 2014-01 → 2026-10, 4,526 days × 8 majors** (coinmetrics VWAP
index, 5bps/2bps fees, per-asset ADV impact, 0.1095 APR funding drag).

**Winner: Donchian 20d long-only on ETH — the first config to clear every gate:**

| Gate | Value | Required | |
|---|---|---|---|
| Deflated Sharpe (66 trials) | **0.9981** | > 0.95 | ✅ |
| PBO (CSCV, 8 splits) | **0.257** | ≤ 0.5 | ✅ |
| Leakage audit (15 CPCV combos) | OK | no leakage | ✅ |
| OOS mean SR (CPCV, purge+embargo 5d) | **1.48** | — | 15/15 combos positive |
| Full-sample SR / MDD | 1.56 / −61% | — | skew +1.28, kurt 18 |

**Era analysis (mandatory honesty check):** the ETH winner's edge
concentrates pre-2022 (2023-26 era SR **0.50**). The **era-stable** picks are
BTC long-only trend: Donchian 200d (recent-era SR **0.92**, full 0.93),
TSMOM 50d (recent 0.89), Donchian 20d (recent 0.87) — lower full-sample SR
(~0.9) but no era decay. Engine fix found during this sweep: the portfolio
divisor previously granted implicit early-era leverage; corrected (unlisted
capital = cash), which *lowered* all panel numbers — the PASS survived the fix.

Read: long-only trend following on daily crypto is a statistically
significant, era-stable (BTC) edge after full institutional costs — with
−60/−80% drawdowns inherent to the unlevered profile and an edge magnitude
that decays as the market matures.

## Strategy sweep v1 — hourly single-asset (2026-10-25): **FAIL**

Run on real curated data: BTCUSDT 1h, 59,654 bars (2020-01 → 2026-10),
5,295 funding ticks, $1M initial capital, CPCV N=6/K=2, horizon=embargo=8h.

A 136-configuration grid (TSMOM × lookback/long-only/vol-target/band, Donchian,
funding carry, funding-extreme events, mean-reversion controls) was run through
the full harness on real data — every config a counted trial. Full 136-config
ranking in `results/sweep_report_v1.json`; runner: `sweep.py`.

**VERDICT: FAIL — no configuration passes the institutional gate.**

| Rank | Config | OOS mean SR (CPCV) | Combos positive | Net PnL | Gates |
|---|---|---|---|---|---|
| 1 | Donchian 55d long-only | 0.41 | 12/15 | +$61,557 | DSR≈0 ✗, PBO 0.63 ✗ |
| 2 | TSMOM 65d long-only | 0.35 | 11/15 | +$54,597 | DSR≈0 ✗ |
| 3 | TSMOM 200d long-only + vol-target 30% + 20% band | 0.39 | 11/15 | +$35,178 | DSR≈0 ✗ |

The best strategies are modestly positive OOS (SR ~0.3-0.4, essentially
long-bias trend capture in a 2020-2026 bull market) but **cannot clear the
multiple-testing bar**: after 136 counted trials, DSR ≈ 0, and PBO 0.63 says
config selection inside this grid is worse than a coin flip. Mean-reversion
and short-horizon momentum controls are strongly negative, as the literature
predicts. Funding carry is positive (+$22k) but individually insignificant
(PSR 0.53).

That is the honest institutional answer: a deployable edge is not found by
grid search on one asset's 1h bars. The harness is doing its job.

## Demo-strategy validation results (costs corrected 2026-10-25)

NOTE: the original 2026-10-23 run used a mis-scaled slippage model (missing
the σ_daily term, impact ~50× overstated). All results below use the corrected
model. The correction is reported openly because it changed the carry verdict.

| Strategy | Net PnL | Fees | Slippage | Funding | Mean OOS SR | PBO | Verdict |
|---|---|---|---|---|---|---|---|
| `funding_carry` | **+$22,242** | $2,856 | $3,016 | +$5,940 | +0.17 | 0.100 | **REJECT** (PSR 0.53) |
| `ma_crossover` | **−$62,096** | $29,208 | $47,872 | −$3,469 | −0.21 | 0.057 | **REJECT** |

The MA crossover remains a cost-loser. The carry trade flips positive once
slippage is realistic, but PSR 0.53 is nowhere near significance.

Note on the DSR column: per-combination values are PSR (n_trials=1 within
a combination); DSR applies when multiple strategy configurations are
compared — `eval_trial_set` computes it with the across-trial SR variance.

## Validation performed

- Synthetic-data smoke: 20k GBM bars, both strategies, numeric DSR/PBO (`synth_gen.py`)
- Leakage test: `--embargo 0` with horizon 8 raises `ValueError` (guard in `make_cpcv`)
- Sanity: zero-position strategy produces exactly 0.0 net PnL; any turnover produces strictly positive fees + slippage
- No-lookahead audit passes on all 15 CPCV combinations, both real and synthetic data

## Known gaps

- ETH enters via funding only; ETH spot 1m (HaianCao/Binance-Public-OHLCV,
  34.6 GB) was not ingested
- On-chain metrics are daily (coinmetrics-community), no usable HF on-chain repo exists
- PBO uses n_splits=8 (70 combos) rather than 16 (12,870) — the latter is
  prohibitively slow on CPU with purge/embargo enabled
- Cross-venue funding divergence ([Chainticks/funding-divergence](https://huggingface.co/datasets/Chainticks/funding-divergence)) is verified
  as a source but not yet wired into the harness

## References

- López de Prado, *Advances in Financial Machine Learning*, Wiley 2018 (ch. 7, 11, 12)
- [`purgedcv`](https://pypi.org/project/purgedcv/) — MIT-licensed implementation of the splitting and statistics used here