README.md

---
title: TopoBench Space
emoji: 🧩
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false
---

# TopoBench

TopoBench now includes a Hugging Face Space app for playing, timing, and verifying
the benchmark puzzles in addition to the original benchmark runner.

## Space App

The Space uses:

- `FastAPI` for the backend API and static asset serving
- `React + Vite` for the frontend
- `SQLite` under `/data/topobench.sqlite` for session storage
- The existing native puzzle verifiers from this repo for correctness checking

### Local development

Backend dependencies:

```bash
python3 -m pip install -e .
```

Frontend dependencies:

```bash
cd frontend
npm install
npm run build
```

Run the app:

```bash
uvicorn space_app.main:app --host 0.0.0.0 --port 7860
```

Optional environment variables:

- `ADMIN_API_TOKEN`
- `TOPOBENCH_DATA_DIR`
- `TOPOBENCH_DATABASE_PATH`
- `HF_HOME`

Admin exports:

- `GET /api/admin/solves`
- `GET /api/admin/aggregates`

Both require `Authorization: Bearer <ADMIN_API_TOKEN>`.

## Benchmark Runner

![TopoBench](figure1.png)

This repo provides the code to run the main TopoBench benchmark with different input formats.

Dataset Links:

- [Plain](https://huggingface.co/datasets/topobench/topobench)
- [Intformat](https://huggingface.co/datasets/topobench/topobench_intformat)
- [Intformat_json](https://huggingface.co/datasets/topobench/topobench_intformat_json)

## Setup (docker required)

Build the container:

```bash
docker build -t topobench -f docker/Dockerfile . \
  --build-arg DEBIAN_MIRROR=https://ftp.us.debian.org/debian \
  --build-arg DEBIAN_SECURITY_MIRROR=https://security.debian.org/debian-security
```

Troubleshooting Docker builds:

- If the build fails while fetching Debian packages, you might want to change the mirrors

Run it:

```bash
docker run --rm -it \
  -e OPENROUTER_KEY=your_key_here \
  topobench \
  python evals/src/main.py run-and-verify \
  --provider openrouter \
  --model inception/mercury-2 \
  --variant intformat_json \
  --difficulty easy \
  --puzzle bridges \
  --limit 1
```

Options for keys (only set the keys you need for the provider you plan to use):

- `OPENAI_API_KEY`
- `OPENROUTER_API_KEY`
- `DEEPSEEK_API_KEY`
- `ANTHROPIC_API_KEY`
- `GOOGLE_API_KEY`

Run all six puzzles on the plain release:

```bash
python evals/src/main.py run \
  --provider openai \
  --model gpt-5-mini \
  --variant plain \
  --difficulty all
  --limit 50
```

Run only bridges on intformat_json and immediately verify in one command:

```bash
python evals/src/main.py run-and-verify \
  --provider openrouter \
  --model inception/mercury-2 \
  --variant intformat_json \
  --difficulty easy \
  --puzzle bridges \
  --limit 50
```

Format options:

- plain
- intformat
- intformat_json

Puzzle options:

- bridges
- flow_free
- galaxies
- loopy
- pattern
- undead

## Verify Existing Runs

Each run is saved under `results/runs/<run-name>/` with:

- `manifest.json`
- `responses.jsonl`

Verify a run and export CSV summaries:

```bash
python evals/src/main.py verify \
  --run-dir results/runs/<run-name>
```

Verification writes:

- `results/reports/<run-name>_details.csv`
- `results/reports/<run-name>_summary.csv`

The verifier also prints a summary table to the terminal.