benchmarks/README.md

Reproduce Pivot's public results

Run the standalone jevbench_public.py script from this repository. It checks out the official JevBench v1.4.1 source at commit 24b9b5c1609a7a9e8fa14f49e5985a836c9dc842, verifies a clean checkout, and uses its public tasks and official per-task scorer.

Install Python, Git, and a PyTorch build for your hardware. The requirements file records the versions used for the reported run. On Windows PowerShell:

py -m pip install -r benchmarks/requirements.txt
py benchmarks/jevbench_public.py --device cpu --threads 4 --batch-size 4 --output pivot_cpu_results.json

On a CUDA machine use --device cuda --batch-size 32. If you only need timings, add --mode speed. For public task accuracy only, add --mode accuracy. The model is pinned to commit 14bf8c26bf344ebdf88e22a4b6152dc5f75f3578 unless you deliberately provide another full SHA via --revision. Allow time and disk space for the approximately 1.43 GB checkpoint.

The script uses the same candidate wording, input rendering, token limits, FP32 scoring, warmup count, and timing protocol as the published run. Timings still vary with hardware, software and background load. Only public JevBench tasks are available here; official_jevbench_score is always null. The full official score requires organizer-held sealed and judge tasks and cost measurement. See the benchmark protocol.