README.md

---
title: Bit Backtest Lab
emoji: 📉
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: true
license: apache-2.0
hf_oauth: true
hf_oauth_scopes:
  - inference-api
short_description: Compare forecasting models under honest costs
tags:
  - finance
  - backtesting
  - time-series
  - forecasting
---

# Backtest Lab

**The Bit Trading Company** · backtest and compare forecasting models and
rule-based trading strategies against a shared, precomputed signal store.

- **App**: https://huggingface.co/spaces/Bit-Trading-Company/bit-backtest-lab
- **Signal store**: https://huggingface.co/datasets/The-Bit-Trading-Company/bit-signal-store

## What it is

Most backtesting tools make it easy to produce a beautiful, false result. This
one is built around the handful of things that actually decide whether a
backtest means anything: when you are allowed to trade, what it costs, and
whether you ever tested on data you had not already fitted.

The store holds **raw model outputs and prices only** — never trade decisions.
Trading rules, costs and sizing are applied live, per request, so many
strategies can be compared over the same forecasts without re-running inference.

## The Compare tab

The app opens on **Compare**, a view over everything this Space has ever
computed — currently **168 combinations** of strategy × model × asset ×
timeframe, precomputed under one canonical config so the rows are actually
comparable with each other.

- **Leaderboard** — ranked board with filters, plus *returns over time*
  overlaying the top algorithms on one axis, and a risk/return scatter where
  marker area is trade count.
- **Models** — directional accuracy against naive baselines, band calibration,
  and best result per model.
- **Signals** — a signal aggregator: every model's latest stored forecast for
  one asset, weighted by how accurate that model has actually been on that
  slice, plus a consensus verdict.
- **Run history** — session runs merged with runs saved to the store, so a
  result found on one machine is visible from another.

Two editorial rules run through it. Rows with fewer than 20 trades are flagged
and kept out of the default ranking, because a Sharpe of 4 on 13 trades will
otherwise own the board forever. And naive baselines are ranked alongside the
learned models everywhere — which is how you find out that on daily bars **the
drift baseline currently calls direction more often correctly than any Chronos
variant**. The Models tab says so in plain language.

## Architecture

```
┌──────────────────────────────────────────────┐
│  bit-backtest-lab  (Gradio Space)            │
│                                              │
│  Strategy Builder ─┐                         │
│                    ├─► engine.py (vectorbt)  │
│  Results Canvas ◄──┘   next-bar-open fills   │
│                        costs, stops, sizing  │
│                        walk-forward + holdout│
│         │                                    │
│         │ read-only          ┌─────────────┐ │
│         └───────────────────►│  LRU cache  │ │
│                              └──────┬──────┘ │
│  "Extend coverage"                  │        │
│   └─► @spaces.GPU (user's quota)    │        │
│        └─► CommitScheduler ──┐      │        │
└──────────────────────────────┼──────┼────────┘
                               ▼      │
        ┌──────────────────────────────────────┐
        │  bit-signal-store  (Dataset)         │
        │  manifest.json    coverage map       │
        │  signals/  raw q10/q50/q90           │
        │  prices/   OHLCV cache + source      │
        │  comparisons/  precomputed tables    │
        │  runs/     saved run summaries       │
        └──────────────────────────────────────┘
                     ▲
                     │ batch refresh only
        ┌────────────┴─────────────────────────┐
        │ ccxt (Binance→Coinbase)              │
        │ yfinance → Stooq → Tiingo            │
        └──────────────────────────────────────┘
```

## How results are computed

**Fills.** Every fill executes at the **next bar's open**. A decision made at
bar `t` is shifted forward one bar by the engine and filled at `open[t+1]`.
There is no configuration that enters on the signal bar; `run_backtest` applies
the shift itself and a strategy cannot bypass it.

**No lookahead.** Strategies are checked by *perturbation*, not by convention:
the tail of the price series is scaled up and down, the indicator is re-run, and
every output before the perturbation point must be bit-identical. A strategy
that reads bar `t+1` — or normalises by a full-sample statistic, or uses a
centred rolling window — changes its earlier outputs and is caught.

**Costs.** On by default: commission per side plus a slippage model
(fixed bps or volume-scaled). Slippage is embedded in the fill price, so it is
reconstructed from the unslipped reference price and booked onto the trade row.
`gross − costs = net` holds exactly on every trade. Turning costs off is
possible and is labelled, loudly, as not real.

**Validation.** Simple split, walk-forward (configurable train/test/roll), and a
locked last-N-months holdout. Train and test never share a bar. The holdout is
excluded from `selectable_index()`, which is the only index any
parameter-selection path is given — it cannot be fitted on by accident. When a
configuration produces no out-of-sample period, the app says so instead of
printing `0.00`.

**Metrics.** Total return, CAGR, Sharpe, Sortino, max drawdown, win rate,
profit factor, exposure and trade count, each computed in-sample,
out-of-sample, and on the holdout. Every displayed number comes from
`metrics.py` operating on the equity curve or the trade list.

## Honest limitations

- **Coverage is sparse.** The seed covers 6 assets × 3 timeframes with eight
  models (Chronos-Bolt tiny/mini/small/base, Chronos-T5 small, and three naive
  baselines) — 78 signal slices, ~265k rows. Anything else needs extending.
- **The learned models are not clearly beating the naive baselines** on
  directional accuracy at daily resolution. That is a real result, and it is
  displayed rather than buried.
- **Binance is geo-blocked from the seeding machine**, so crypto prices came
  from Coinbase. Prices will not tick-match another venue.
- **Equity intraday history is provider-capped** — roughly 730 days of hourly
  and 60 days of 15-minute bars. This is recorded in the manifest as a coverage
  boundary, not hidden and not reported as an error.
- **Stooq began serving an HTML block page** instead of CSV during the build, so
  the equity fallback chain effectively runs on yfinance alone right now.
- **Sentiment is a stub.** `Sentiment-Gated Momentum` runs against a labelled,
  deterministic price-derived proxy, *not* news sentiment. It sits behind a
  `SentimentSource` interface for a real feed later.
- **Pairs Trading and Custom (code)** appear in the design's preset list but are
  not runnable. Custom-code strategies are disabled deliberately: this Space
  never executes user-supplied code.
- **A backtest is a hypothesis.** Survivorship, regime change, liquidity and
  your own future behaviour are not modelled.

## Extending coverage

Anonymous visitors get full read and backtest access. Extending coverage
requires signing in with Hugging Face, because inference runs inside a
`@spaces.GPU` function on **your** ZeroGPU quota:

1. Pick a model, asset, timeframe and range in the **Coverage** tab.
2. **Estimate** shows the step count; the request is deduplicated against
   `manifest.json`, so an already-covered range recomputes nothing.
3. Per-request caps apply (2 years daily / 6 months hourly / 2 months 15m).
4. On success the new slice, the manifest, and the regenerated comparison
   tables are written back in a single atomic commit, attributed to you.

**Add model** takes an adapter family plus a Hub model id, resolves and pins the
revision, and runs a 100-step smoke test on your quota before the model appears
in the coverage map. Only allow-listed adapter families (`chronos`, `timesfm`)
can be constructed, and model ids are validated before they reach the Hub.

If your quota is exhausted the app says so and offers a duplicate-this-Space
link — the store is public, so a duplicate reads the same data.

## Running locally

```bash
uv venv --python 3.10 && uv pip install -r requirements.txt
python app.py
```

Seeding your own store:

```bash
python scripts/seed_store.py --plan v1 --dry-run
python scripts/seed_store.py --plan smoke --push
```

Tests:

```bash
bash tests/run_all.sh
```

`HF_WRITE_TOKEN` is required only for write-back; reading and backtesting work
without any token. See `.env.example`.

## Disclaimer

Backtested results are hypothetical, derived from historical data, and are not
indicative of future results. Nothing here is investment advice. The Bit Trading
Company is not a licensed investment adviser.