| <!-- SPDX-License-Identifier: Apache-2.0 --> |
| # Ink Bench |
|
|
| A fast, model-agnostic and reproducible way to compare ink detectors on real segments of PHerc. 139, and to turn what you find into candidates an expert can check. |
|
|
| Open it from the bar chart button in the [Unroll Space](https://huggingface.co/spaces/SharpMeow/unroll), or go to `?bench`. Everything runs in your browser. **Nothing is uploaded.** |
|
|
| ## What it does |
|
|
| 1. **Load** your ink prediction for one winding (`w023` to `w059`) as PNG or TIFF: 8 or 16 bit gray, 8 bit RGB or palette, or 8, 16 or 32 bit (float) TIFF with no compression, LZW, Deflate or PackBits. Any resolution and aspect. JPEG and WebP are refused on purpose: decoders differ, so scores would not reproduce. |
| 2. **Align** it in UV space to the published ink layer (or to the X-ray layer), automatically: a coarse search over 25 scales from 0.6x to 1.6x and shifts up to a quarter of the image at about 80 px, then a pattern search on a 4-level pyramid down to 0.02 px. The model is a scale about the grid center plus a shift, with independent x and y scale. Nudge buttons and number fields give a manual fallback. |
| 3. **Look**: side by side, a swipe, and a difference heatmap (white both, amber published only, cyan yours only), with a 5 mm scale bar and 2x and 4x zoom. |
| 4. **Score** against the published ink: Pearson r; Pearson after blurring the reference to your prediction's sharpness, with that blur in µm; IoU, Dice, precision and recall at adjustable thresholds; a precision and recall curve over 51 thresholds, its average precision (AP) and the best F1. |
| 5. **Annotate** letter-shaped strokes: draw boxes, tag a candidate Greek letter and a confidence. Every box is marked "candidate for expert review". |
| 6. **Export** one zip: `bundle.json` (format below), `figure.png` (reference, aligned prediction, difference, scale bar, numbers, credits), `CITATION.cff`, `inkbench.bib` and a share link that restores the comparison. Public layers restore fully from the link; a local file restores once you drop the same file (checked by SHA-256). |
|
|
| > Every score is **agreement with another model's output**, the Vesuvius Challenge published ink detection on the 2.4 µm scan. It is not ground truth. A model can score lower because it is different, or even better. Use the numbers to find where detectors agree and disagree, then send the interesting regions to people who read Greek papyri. |
|
|
| ## Researcher takeaway |
| On the public ink_9um run over all 37 windings (`docs/examples/bench/`), against the published 2.4 µm ink: median AP 0.319 for ink_9um, 0.117 for raw X-ray intensity and 0.089 for uniform noise, and ink_9um beats X-ray intensity on AP on every winding. On Pearson r the medians are 0.422, 0.266 and 0.000, but X-ray intensity outranks ink_9um on `w039`, `w053`, `w054` and `w057`. So: rank ink models by AP (threshold-free and robust to brightness), always report the X-ray floor next to your model, and look at windings where the floor wins before trusting a correlation. These are agreement scores with another model's output, not ground truth. |
|
|
| The ink_9um model was trained by the Vesuvius Challenge and run by Chase Hendrick. |
| |
| ## Reference grid |
| |
| Each winding has a reference grid of 640 px on its long side (`data/bench/<id>_ink.png`, `<id>_ct.png`, and `<id>_gpu.png`, the public ink_9um run), lossless 8 bit PNG, with checksums and pixels per mm in `data/bench/index.json` (about 8.3 px per mm, so 1 px is about 120 µm on the papyrus). They are built by `tools/make_bench_refs.py` from the page's own layers. All metrics are computed on this grid, after your prediction is resampled onto it (bilinear, 2x2 mean pooled first while it is more than twice the grid). |
|
|
| ## Python and CLI |
|
|
| `tools/inkbench.py` is a single file (numpy and Pillow) that implements the same steps in the same order and writes the same bundle. The test suite checks that the browser code and the Python code produce identical JSON, apart from `tool` and `created`. |
|
|
| ```sh |
| pip install "git+https://huggingface.co/spaces/SharpMeow/unroll" # or: curl -LO .../resolve/main/tools/inkbench.py |
| inkbench segments |
| inkbench compare pred.png --segment w046 --model "my-model v3" -o bundle.json |
| inkbench compare pred.png --segment w046 --alignment old_bundle.json # reuse an alignment |
| inkbench rank bundles/*.json --metric ap |
| inkbench validate bundle.json |
| inkbench cite |
| ``` |
|
|
| Reference layers are read from a local checkout when you run the file inside the repo, otherwise fetched from the Space (`--root` and `--base` override). |
|
|
| In a training loop: |
|
|
| ```python |
| import inkbench |
| refs = inkbench.Refs() # fetch the reference layers once |
| b = inkbench.compare(prob_map, segment="w046", refs=refs, model={"name": "run 17, step 40k"}) |
| log(b["metrics"]["pearson"], b["metrics"]["ap"]) |
| ``` |
|
|
| `prob_map` can be a path, PNG or TIFF bytes, or a 2-D numpy array in [0, 1]. |
|
|
| In CI (fails below a floor you choose): |
|
|
| ```yaml |
| - run: pip install "git+https://huggingface.co/spaces/SharpMeow/unroll" |
| - run: inkbench compare outputs/w046.png --segment w046 --min-pearson 0.35 -o w046.json |
| ``` |
|
|
| ## Adapters for common pipelines |
|
|
| `inkbench adapt <kind> <input> -o pred.png` writes a 16 bit gray PNG that both the page and the CLI read exactly. |
|
|
| | kind | input | what it does | |
| | --- | --- | --- | |
| | `villa-folder` | a vesuvius or villa style segment folder | finds the prediction image (name contains `pred` or `ink`, not `inklabels`) and applies `mask.png` if present | |
| | `tif-stack` | a multi-page `.tif` layer or probability stack | mean (default) or `--reduce max`, or one `--page N`, min-max scaled | |
| | `nnunet` | nnU-Net `.npz` with `probabilities`, or a label PNG | takes `--channel` (default 1), max over depth; label maps (0 and 1) are kept as is | |
| | `timesformer` | a TimeSformer style gray probability PNG | passes it through | |
|
|
| Your prediction has to cover the same segment surface as the reference (the same UV layout up to scale and shift). Predictions on the public PHerc. 139 segments listed in `data/scroll.json` (`ink_source`, `mesh_source`) fit. |
|
|
| ## In the page, for agents |
|
|
| `window.unrollBench` (also `window.unroll.bench` when the page's agent API is present): |
|
|
| ```js |
| const B = window.unrollBench; |
| await B.segments(); |
| const bundle = await B.compare({ segment: 'w049', layer: 'gpu' }); // a public layer |
| await B.compare({ segment: 'w046', url: 'https://example.org/pred.png', model: 'my model' }); |
| await B.compare({ segment: 'w046', bytes: base64OrUint8Array, name: 'pred.png' }); |
| await B.setThresholds({ ref: 0.5, pred: 0.4 }); |
| await B.nudge({ dx: 0.5 }); await B.setAlignment({ tx: 3, ty: -2, sx: 1, sy: 1 }); |
| B.annotate({ x: 120, y: 80, w: 14, h: 18, letter: 'Α', confidence: 0.6 }); |
| B.bundle(); B.figure(); B.shareURL(); B.bibtex(); |
| await B.exportBundle({ download: false }); // { name, size, bundle, zip } |
| await B.rank(['https://.../a.json', bundleObject], 'ap'); B.validate(bundleObject); |
| B.core // decode, align, metrics without the UI |
| ``` |
|
|
| ## Leaderboard |
|
|
| The Leaderboard tab loads any number of bundles from URLs (one per line, CORS) or files and ranks them per winding by AP (the default), Pearson r, best F1, IoU or Dice, entirely in the browser. `?board=url1,url2` opens it preloaded. "Load examples" shows the public ink_9um run on all 37 windings next to two floors any ink model should beat: the X-ray intensity itself and uniform noise. |
| |
| ## How it is tested |
| |
| `python3 tests/test_bench.py` (numpy, Pillow, Node 20) and the Playwright run in `tests/e2e_bench.py`: |
| |
| - alignment recovers 10 synthetic cases made from the full 2,400 px published preview with known shift (up to 40 px), scale (0.8x to 1.3x, including unequal x and y) and Gaussian blur (0 to 6 px); |
| - metrics match a plain numpy reference (`np.corrcoef`, set counts, AP) to 1e-6; |
| - the browser core and Python write identical bundles for PNG 8 and 16 bit, RGB, palette, label maps, TIFF 8 bit Deflate, 16 bit LZW and 32 bit float, inverted input, oversized input and noise; |
| - pure noise scores near zero. |
| |
| ## Format, license, citation |
| |
| Bundle format: [BENCH_FORMAT.md](BENCH_FORMAT.md) and [bench-bundle.schema.json](bench-bundle.schema.json). |
| |
| The Bench code and the format spec are Apache-2.0 (`LICENSE`, `NOTICE`), like all of Unroll's code, by Chase Hendrick. The data (scan, segments, ink layers, reference grids) is Vesuvius Challenge open data under CC BY-NC 4.0: non-commercial use, with attribution. Data credits are in the About panel and in every bundle and figure. |
| |
| ```bibtex |
| @software{hendrick_unroll_inkbench_2026, |
| author = {Hendrick, Chase}, |
| title = {Unroll Ink Bench: model-agnostic agreement metrics for Herculaneum ink detection}, |
| year = {2026}, |
| version = {1.0.0}, |
| url = {https://huggingface.co/spaces/SharpMeow/unroll}, |
| license = {Apache-2.0}, |
| note = {Data: Vesuvius Challenge open data, CC BY-NC 4.0} |
| } |
| ``` |
| |
