README.md

metadata

title: Bit Backtest Lab
emoji: 📉
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: true
license: apache-2.0
hf_oauth: true
hf_oauth_scopes:
  - inference-api
short_description: Compare forecasting models under honest costs
tags:
  - finance
  - backtesting
  - time-series
  - forecasting

Backtest Lab

The Bit Trading Company · backtest and compare forecasting models and rule-based trading strategies against a shared, precomputed signal store.

• App: https://huggingface.co/spaces/Bit-Trading-Company/bit-backtest-lab
• Signal store: https://huggingface.co/datasets/The-Bit-Trading-Company/bit-signal-store

What it is

Most backtesting tools make it easy to produce a beautiful, false result. This one is built around the handful of things that actually decide whether a backtest means anything: when you are allowed to trade, what it costs, and whether you ever tested on data you had not already fitted.

The store holds raw model outputs and prices only — never trade decisions. Trading rules, costs and sizing are applied live, per request, so many strategies can be compared over the same forecasts without re-running inference.

The Compare tab

The app opens on Compare, a view over everything this Space has ever computed — currently 168 combinations of strategy × model × asset × timeframe, precomputed under one canonical config so the rows are actually comparable with each other.

• Leaderboard — ranked board with filters, plus returns over time overlaying the top algorithms on one axis, and a risk/return scatter where marker area is trade count.
• Models — directional accuracy against naive baselines, band calibration, and best result per model.
• Signals — a signal aggregator: every model's latest stored forecast for one asset, weighted by how accurate that model has actually been on that slice, plus a consensus verdict.
• Run history — session runs merged with runs saved to the store, so a result found on one machine is visible from another.

Two editorial rules run through it. Rows with fewer than 20 trades are flagged and kept out of the default ranking, because a Sharpe of 4 on 13 trades will otherwise own the board forever. And naive baselines are ranked alongside the learned models everywhere — which is how you find out that on daily bars the drift baseline currently calls direction more often correctly than any Chronos variant. The Models tab says so in plain language.

Architecture

┌──────────────────────────────────────────────┐
│  bit-backtest-lab  (Gradio Space)            │
│                                              │
│  Strategy Builder ─┐                         │
│                    ├─► engine.py (vectorbt)  │
│  Results Canvas ◄──┘   next-bar-open fills   │
│                        costs, stops, sizing  │
│                        walk-forward + holdout│
│         │                                    │
│         │ read-only          ┌─────────────┐ │
│         └───────────────────►│  LRU cache  │ │
│                              └──────┬──────┘ │
│  "Extend coverage"                  │        │
│   └─► @spaces.GPU (user's quota)    │        │
│        └─► CommitScheduler ──┐      │        │
└──────────────────────────────┼──────┼────────┘
                               ▼      │
        ┌──────────────────────────────────────┐
        │  bit-signal-store  (Dataset)         │
        │  manifest.json    coverage map       │
        │  signals/  raw q10/q50/q90           │
        │  prices/   OHLCV cache + source      │
        │  comparisons/  precomputed tables    │
        │  runs/     saved run summaries       │
        └──────────────────────────────────────┘
                     ▲
                     │ batch refresh only
        ┌────────────┴─────────────────────────┐
        │ ccxt (Binance→Coinbase)              │
        │ yfinance → Stooq → Tiingo            │
        └──────────────────────────────────────┘

How results are computed

Fills. Every fill executes at the next bar's open. A decision made at bar t is shifted forward one bar by the engine and filled at open[t+1]. There is no configuration that enters on the signal bar; run_backtest applies the shift itself and a strategy cannot bypass it.

No lookahead. Strategies are checked by perturbation, not by convention: the tail of the price series is scaled up and down, the indicator is re-run, and every output before the perturbation point must be bit-identical. A strategy that reads bar t+1 — or normalises by a full-sample statistic, or uses a centred rolling window — changes its earlier outputs and is caught.

Costs. On by default: commission per side plus a slippage model (fixed bps or volume-scaled). Slippage is embedded in the fill price, so it is reconstructed from the unslipped reference price and booked onto the trade row. gross − costs = net holds exactly on every trade. Turning costs off is possible and is labelled, loudly, as not real.

Validation. Simple split, walk-forward (configurable train/test/roll), and a locked last-N-months holdout. Train and test never share a bar. The holdout is excluded from selectable_index(), which is the only index any parameter-selection path is given — it cannot be fitted on by accident. When a configuration produces no out-of-sample period, the app says so instead of printing 0.00.

Metrics. Total return, CAGR, Sharpe, Sortino, max drawdown, win rate, profit factor, exposure and trade count, each computed in-sample, out-of-sample, and on the holdout. Every displayed number comes from metrics.py operating on the equity curve or the trade list.

Honest limitations

• Coverage is sparse. The seed covers 6 assets × 3 timeframes with eight models (Chronos-Bolt tiny/mini/small/base, Chronos-T5 small, and three naive baselines) — 78 signal slices, ~265k rows. Anything else needs extending.
• The learned models are not clearly beating the naive baselines on directional accuracy at daily resolution. That is a real result, and it is displayed rather than buried.
• Binance is geo-blocked from the seeding machine, so crypto prices came from Coinbase. Prices will not tick-match another venue.
• Equity intraday history is provider-capped — roughly 730 days of hourly and 60 days of 15-minute bars. This is recorded in the manifest as a coverage boundary, not hidden and not reported as an error.
• Stooq began serving an HTML block page instead of CSV during the build, so the equity fallback chain effectively runs on yfinance alone right now.
• Sentiment is a stub. Sentiment-Gated Momentum runs against a labelled, deterministic price-derived proxy, not news sentiment. It sits behind a SentimentSource interface for a real feed later.
• Pairs Trading and Custom (code) appear in the design's preset list but are not runnable. Custom-code strategies are disabled deliberately: this Space never executes user-supplied code.
• A backtest is a hypothesis. Survivorship, regime change, liquidity and your own future behaviour are not modelled.

Extending coverage

Anonymous visitors get full read and backtest access. Extending coverage requires signing in with Hugging Face, because inference runs inside a @spaces.GPU function on your ZeroGPU quota:

1. Pick a model, asset, timeframe and range in the Coverage tab.
2. Estimate shows the step count; the request is deduplicated against manifest.json, so an already-covered range recomputes nothing.
3. Per-request caps apply (2 years daily / 6 months hourly / 2 months 15m).
4. On success the new slice, the manifest, and the regenerated comparison tables are written back in a single atomic commit, attributed to you.

Add model takes an adapter family plus a Hub model id, resolves and pins the revision, and runs a 100-step smoke test on your quota before the model appears in the coverage map. Only allow-listed adapter families (chronos, timesfm) can be constructed, and model ids are validated before they reach the Hub.

If your quota is exhausted the app says so and offers a duplicate-this-Space link — the store is public, so a duplicate reads the same data.

Running locally

uv venv --python 3.10 && uv pip install -r requirements.txt
python app.py

Seeding your own store:

python scripts/seed_store.py --plan v1 --dry-run
python scripts/seed_store.py --plan smoke --push

Tests:

bash tests/run_all.sh

HF_WRITE_TOKEN is required only for write-back; reading and backtesting work without any token. See .env.example.

Disclaimer

Backtested results are hypothetical, derived from historical data, and are not indicative of future results. Nothing here is investment advice. The Bit Trading Company is not a licensed investment adviser.