| # Reproduce Pivot's public results | |
| Run the standalone `jevbench_public.py` script from this repository. It checks out the official JevBench v1.4.1 source at commit `24b9b5c1609a7a9e8fa14f49e5985a836c9dc842`, verifies a clean checkout, and uses its public tasks and official per-task scorer. | |
| Install Python, Git, and a PyTorch build for your hardware. The requirements file records the versions used for the reported run. On Windows PowerShell: | |
| ```powershell | |
| py -m pip install -r benchmarks/requirements.txt | |
| py benchmarks/jevbench_public.py --device cpu --threads 4 --batch-size 4 --output pivot_cpu_results.json | |
| ``` | |
| On a CUDA machine use `--device cuda --batch-size 32`. If you only need timings, add `--mode speed`. For public task accuracy only, add `--mode accuracy`. The model is pinned to commit `14bf8c26bf344ebdf88e22a4b6152dc5f75f3578` unless you deliberately provide another full SHA via `--revision`. Allow time and disk space for the approximately 1.43 GB checkpoint. | |
| The script uses the same candidate wording, input rendering, token limits, FP32 scoring, warmup count, and timing protocol as the published run. Timings still vary with hardware, software and background load. Only public JevBench tasks are available here; `official_jevbench_score` is always null. The full official score requires organizer-held sealed and judge tasks and cost measurement. See [the benchmark protocol](../docs/BENCHMARK.md). | |