docs/BENCH.md

Ink Bench

A fast, model-agnostic and reproducible way to compare ink detectors on real segments of PHerc. 139, and to turn what you find into candidates an expert can check.

Open it from the bar chart button in the Unroll Space, or go to ?bench. Everything runs in your browser. Nothing is uploaded.

What it does

1. Load your ink prediction for one winding (w023 to w059) as PNG or TIFF: 8 or 16 bit gray, 8 bit RGB or palette, or 8, 16 or 32 bit (float) TIFF with no compression, LZW, Deflate or PackBits. Any resolution and aspect. JPEG and WebP are refused on purpose: decoders differ, so scores would not reproduce.
2. Align it in UV space to the published ink layer (or to the X-ray layer), automatically: a coarse search over 25 scales from 0.6x to 1.6x and shifts up to a quarter of the image at about 80 px, then a pattern search on a 4-level pyramid down to 0.02 px. The model is a scale about the grid center plus a shift, with independent x and y scale. Nudge buttons and number fields give a manual fallback.
3. Look: side by side, a swipe, and a difference heatmap (white both, amber published only, cyan yours only), with a 5 mm scale bar and 2x and 4x zoom.
4. Score against the published ink: Pearson r; Pearson after blurring the reference to your prediction's sharpness, with that blur in µm; IoU, Dice, precision and recall at adjustable thresholds; a precision and recall curve over 51 thresholds, its average precision (AP) and the best F1.
5. Annotate letter-shaped strokes: draw boxes, tag a candidate Greek letter and a confidence. Every box is marked "candidate for expert review".
6. Export one zip: bundle.json (format below), figure.png (reference, aligned prediction, difference, scale bar, numbers, credits), CITATION.cff, inkbench.bib and a share link that restores the comparison. Public layers restore fully from the link; a local file restores once you drop the same file (checked by SHA-256).

Every score is agreement with another model's output, the Vesuvius Challenge published ink detection on the 2.4 µm scan. It is not ground truth. A model can score lower because it is different, or even better. Use the numbers to find where detectors agree and disagree, then send the interesting regions to people who read Greek papyri.

Researcher takeaway

On the public ink_9um run over all 37 windings (docs/examples/bench/), against the published 2.4 µm ink: median AP 0.319 for ink_9um, 0.117 for raw X-ray intensity and 0.089 for uniform noise, and ink_9um beats X-ray intensity on AP on every winding. On Pearson r the medians are 0.422, 0.266 and 0.000, but X-ray intensity outranks ink_9um on w039, w053, w054 and w057. So: rank ink models by AP (threshold-free and robust to brightness), always report the X-ray floor next to your model, and look at windings where the floor wins before trusting a correlation. These are agreement scores with another model's output, not ground truth.

The ink_9um model was trained by the Vesuvius Challenge and run by Chase Hendrick.

Reference grid

Each winding has a reference grid of 640 px on its long side (data/bench/<id>_ink.png, <id>_ct.png, and <id>_gpu.png, the public ink_9um run), lossless 8 bit PNG, with checksums and pixels per mm in data/bench/index.json (about 8.3 px per mm, so 1 px is about 120 µm on the papyrus). They are built by tools/make_bench_refs.py from the page's own layers. All metrics are computed on this grid, after your prediction is resampled onto it (bilinear, 2x2 mean pooled first while it is more than twice the grid).

Python and CLI

tools/inkbench.py is a single file (numpy and Pillow) that implements the same steps in the same order and writes the same bundle. The test suite checks that the browser code and the Python code produce identical JSON, apart from tool and created.

pip install "git+https://huggingface.co/spaces/SharpMeow/unroll"     # or: curl -LO .../resolve/main/tools/inkbench.py
inkbench segments
inkbench compare pred.png --segment w046 --model "my-model v3" -o bundle.json
inkbench compare pred.png --segment w046 --alignment old_bundle.json   # reuse an alignment
inkbench rank bundles/*.json --metric ap
inkbench validate bundle.json
inkbench cite

Reference layers are read from a local checkout when you run the file inside the repo, otherwise fetched from the Space (--root and --base override).

In a training loop:

import inkbench
refs = inkbench.Refs()                      # fetch the reference layers once
b = inkbench.compare(prob_map, segment="w046", refs=refs, model={"name": "run 17, step 40k"})
log(b["metrics"]["pearson"], b["metrics"]["ap"])

prob_map can be a path, PNG or TIFF bytes, or a 2-D numpy array in [0, 1].

In CI (fails below a floor you choose):

- run: pip install "git+https://huggingface.co/spaces/SharpMeow/unroll"
- run: inkbench compare outputs/w046.png --segment w046 --min-pearson 0.35 -o w046.json

Adapters for common pipelines

inkbench adapt <kind> <input> -o pred.png writes a 16 bit gray PNG that both the page and the CLI read exactly.

kind	input	what it does
villa-folder	a vesuvius or villa style segment folder	finds the prediction image (name contains pred or ink, not inklabels) and applies mask.png if present
tif-stack	a multi-page .tif layer or probability stack	mean (default) or --reduce max, or one --page N, min-max scaled
nnunet	nnU-Net .npz with probabilities, or a label PNG	takes --channel (default 1), max over depth; label maps (0 and 1) are kept as is
timesformer	a TimeSformer style gray probability PNG	passes it through

Your prediction has to cover the same segment surface as the reference (the same UV layout up to scale and shift). Predictions on the public PHerc. 139 segments listed in data/scroll.json (ink_source, mesh_source) fit.

In the page, for agents

window.unrollBench (also window.unroll.bench when the page's agent API is present):

const B = window.unrollBench;
await B.segments();
const bundle = await B.compare({ segment: 'w049', layer: 'gpu' });            // a public layer
await B.compare({ segment: 'w046', url: 'https://example.org/pred.png', model: 'my model' });
await B.compare({ segment: 'w046', bytes: base64OrUint8Array, name: 'pred.png' });
await B.setThresholds({ ref: 0.5, pred: 0.4 });
await B.nudge({ dx: 0.5 }); await B.setAlignment({ tx: 3, ty: -2, sx: 1, sy: 1 });
B.annotate({ x: 120, y: 80, w: 14, h: 18, letter: 'Α', confidence: 0.6 });
B.bundle(); B.figure(); B.shareURL(); B.bibtex();
await B.exportBundle({ download: false });                                   // { name, size, bundle, zip }
await B.rank(['https://.../a.json', bundleObject], 'ap'); B.validate(bundleObject);
B.core                                                                       // decode, align, metrics without the UI

Leaderboard

The Leaderboard tab loads any number of bundles from URLs (one per line, CORS) or files and ranks them per winding by AP (the default), Pearson r, best F1, IoU or Dice, entirely in the browser. ?board=url1,url2 opens it preloaded. "Load examples" shows the public ink_9um run on all 37 windings next to two floors any ink model should beat: the X-ray intensity itself and uniform noise.

How it is tested

python3 tests/test_bench.py (numpy, Pillow, Node 20) and the Playwright run in tests/e2e_bench.py:

• alignment recovers 10 synthetic cases made from the full 2,400 px published preview with known shift (up to 40 px), scale (0.8x to 1.3x, including unequal x and y) and Gaussian blur (0 to 6 px);
• metrics match a plain numpy reference (np.corrcoef, set counts, AP) to 1e-6;
• the browser core and Python write identical bundles for PNG 8 and 16 bit, RGB, palette, label maps, TIFF 8 bit Deflate, 16 bit LZW and 32 bit float, inverted input, oversized input and noise;
• pure noise scores near zero.

Format, license, citation

Bundle format: BENCH_FORMAT.md and bench-bundle.schema.json.

The Bench code and the format spec are Apache-2.0 (LICENSE, NOTICE), like all of Unroll's code, by Chase Hendrick. The data (scan, segments, ink layers, reference grids) is Vesuvius Challenge open data under CC BY-NC 4.0: non-commercial use, with attribution. Data credits are in the About panel and in every bundle and figure.

@software{hendrick_unroll_inkbench_2026,
  author = {Hendrick, Chase},
  title = {Unroll Ink Bench: model-agnostic agreement metrics for Herculaneum ink detection},
  year = {2026},
  version = {1.0.0},
  url = {https://huggingface.co/spaces/SharpMeow/unroll},
  license = {Apache-2.0},
  note = {Data: Vesuvius Challenge open data, CC BY-NC 4.0}
}