Spaces:
Running
Running
File size: 3,273 Bytes
1f50695 67acd34 1f50695 67acd34 1f50695 67acd34 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | ---
title: TopoBench Space
emoji: 🧩
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false
---
# TopoBench
TopoBench now includes a Hugging Face Space app for playing, timing, and verifying
the benchmark puzzles in addition to the original benchmark runner.
## Space App
The Space uses:
- `FastAPI` for the backend API and static asset serving
- `React + Vite` for the frontend
- `SQLite` under `/data/topobench.sqlite` for session storage
- The existing native puzzle verifiers from this repo for correctness checking
### Local development
Backend dependencies:
```bash
python3 -m pip install -e .
```
Frontend dependencies:
```bash
cd frontend
npm install
npm run build
```
Run the app:
```bash
uvicorn space_app.main:app --host 0.0.0.0 --port 7860
```
Optional environment variables:
- `ADMIN_API_TOKEN`
- `TOPOBENCH_DATA_DIR`
- `TOPOBENCH_DATABASE_PATH`
- `HF_HOME`
Admin exports:
- `GET /api/admin/solves`
- `GET /api/admin/aggregates`
Both require `Authorization: Bearer <ADMIN_API_TOKEN>`.
## Benchmark Runner
This repo provides the code to run the main TopoBench benchmark with different input formats.
Dataset Links:
- [Plain](https://huggingface.co/datasets/topobench/topobench)
- [Intformat](https://huggingface.co/datasets/topobench/topobench_intformat)
- [Intformat_json](https://huggingface.co/datasets/topobench/topobench_intformat_json)
## Setup (docker required)
Build the container:
```bash
docker build -t topobench -f docker/Dockerfile . \
--build-arg DEBIAN_MIRROR=https://ftp.us.debian.org/debian \
--build-arg DEBIAN_SECURITY_MIRROR=https://security.debian.org/debian-security
```
Troubleshooting Docker builds:
- If the build fails while fetching Debian packages, you might want to change the mirrors
Run it:
```bash
docker run --rm -it \
-e OPENROUTER_KEY=your_key_here \
topobench \
python evals/src/main.py run-and-verify \
--provider openrouter \
--model inception/mercury-2 \
--variant intformat_json \
--difficulty easy \
--puzzle bridges \
--limit 1
```
Options for keys (only set the keys you need for the provider you plan to use):
- `OPENAI_API_KEY`
- `OPENROUTER_API_KEY`
- `DEEPSEEK_API_KEY`
- `ANTHROPIC_API_KEY`
- `GOOGLE_API_KEY`
Run all six puzzles on the plain release:
```bash
python evals/src/main.py run \
--provider openai \
--model gpt-5-mini \
--variant plain \
--difficulty all
--limit 50
```
Run only bridges on intformat_json and immediately verify in one command:
```bash
python evals/src/main.py run-and-verify \
--provider openrouter \
--model inception/mercury-2 \
--variant intformat_json \
--difficulty easy \
--puzzle bridges \
--limit 50
```
Format options:
- plain
- intformat
- intformat_json
Puzzle options:
- bridges
- flow_free
- galaxies
- loopy
- pattern
- undead
## Verify Existing Runs
Each run is saved under `results/runs/<run-name>/` with:
- `manifest.json`
- `responses.jsonl`
Verify a run and export CSV summaries:
```bash
python evals/src/main.py verify \
--run-dir results/runs/<run-name>
```
Verification writes:
- `results/reports/<run-name>_details.csv`
- `results/reports/<run-name>_summary.csv`
The verifier also prints a summary table to the terminal.
|