README.md

metadata

title: TopoBench Space
emoji: 🧩
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false

TopoBench

TopoBench now includes a Hugging Face Space app for playing, timing, and verifying the benchmark puzzles in addition to the original benchmark runner.

Space App

The Space uses:

• FastAPI for the backend API and static asset serving
• React + Vite for the frontend
• SQLite under /data/topobench.sqlite for session storage
• The existing native puzzle verifiers from this repo for correctness checking

Local development

Backend dependencies:

python3 -m pip install -e .

Frontend dependencies:

cd frontend
npm install
npm run build

Run the app:

uvicorn space_app.main:app --host 0.0.0.0 --port 7860

Optional environment variables:

• ADMIN_API_TOKEN
• TOPOBENCH_DATA_DIR
• TOPOBENCH_DATABASE_PATH
• HF_HOME

Admin exports:

• GET /api/admin/solves
• GET /api/admin/aggregates

Both require Authorization: Bearer <ADMIN_API_TOKEN>.

Benchmark Runner

This repo provides the code to run the main TopoBench benchmark with different input formats.

Dataset Links:

• Plain
• Intformat
• Intformat_json

Setup (docker required)

Build the container:

docker build -t topobench -f docker/Dockerfile . \
  --build-arg DEBIAN_MIRROR=https://ftp.us.debian.org/debian \
  --build-arg DEBIAN_SECURITY_MIRROR=https://security.debian.org/debian-security

Troubleshooting Docker builds:

• If the build fails while fetching Debian packages, you might want to change the mirrors

Run it:

docker run --rm -it \
  -e OPENROUTER_KEY=your_key_here \
  topobench \
  python evals/src/main.py run-and-verify \
  --provider openrouter \
  --model inception/mercury-2 \
  --variant intformat_json \
  --difficulty easy \
  --puzzle bridges \
  --limit 1

Options for keys (only set the keys you need for the provider you plan to use):

• OPENAI_API_KEY
• OPENROUTER_API_KEY
• DEEPSEEK_API_KEY
• ANTHROPIC_API_KEY
• GOOGLE_API_KEY

Run all six puzzles on the plain release:

python evals/src/main.py run \
  --provider openai \
  --model gpt-5-mini \
  --variant plain \
  --difficulty all
  --limit 50

Run only bridges on intformat_json and immediately verify in one command:

python evals/src/main.py run-and-verify \
  --provider openrouter \
  --model inception/mercury-2 \
  --variant intformat_json \
  --difficulty easy \
  --puzzle bridges \
  --limit 50

Format options:

• plain
• intformat
• intformat_json

Puzzle options:

• bridges
• flow_free
• galaxies
• loopy
• pattern
• undead

Verify Existing Runs

Each run is saved under results/runs/<run-name>/ with:

• manifest.json
• responses.jsonl

Verify a run and export CSV summaries:

python evals/src/main.py verify \
  --run-dir results/runs/<run-name>

Verification writes:

• results/reports/<run-name>_details.csv
• results/reports/<run-name>_summary.csv

The verifier also prints a summary table to the terminal.