AFV V2 fiber probability maps for six scrolls

The voxel maps behind the six public AFV fiber datasets: for every voxel of the scan, the probability that Qual's AFV V2 model gives to a vertical fiber and to a horizontal fiber. The traced fibers (.afv curves) are in the datasets; this bucket holds the maps they were traced from, so you can trace again with other settings, use the maps as features, or check a fiber against the probability around it. Model output about papyrus fiber texture, not a reading.

Layout

SCAN/volume.json              scan, volume, level, CT URL, volume shape, block and codec, the 1024-voxel tiles
SCAN/native/Z_Y_X.blosc       one 512^3 block of the map at the model's level, origin Z_Y_X
SCAN/pooled/Z_Y_X.blosc       the same block 2x average pooled (256^3), the input the spline generator prefers
SCAN/regions/TILE.afv         the curves traced in one 1024-voxel tile before stitching (+ TILE.afv.json)
SCAN/tiles/TILE.json          the model run for that tile: windows run, timings
SCAN/level2/                  PHerc0332 and PHerc1667 only: the stitched fibers in level-2 voxels, before the
                              level-0 copies in the datasets were made from them

Each .blosc file is raw uint8 bytes compressed with numcodecs Blosc(cname="zstd", clevel=3, shuffle=BITSHUFFLE), shaped (2, dz, dy, dx): channel 0 is the vertical fiber probability, channel 1 the horizontal one, each times 255. Blocks start at multiples of 512 and are 512 voxels on a side except at the volume's far edges (dz = min(512, shape_z - Z), likewise y and x, with shape_zyx from volume.json). A pooled block has half each side (dz // 2). A block that is absent is all zero. Coordinates are voxels of the level in volume.json (level 0 for four scans, level 2 for PHerc0332 and PHerc1667; multiply by 4 for level 0).

Read one block

import json
import numcodecs
import numpy as np
from huggingface_hub import HfApi

bucket, scan, z, y, x = "SharpMeow/afv-fiber-probabilities", "PHerc0139", 10752, 5120, 3584
HfApi().download_bucket_files(bucket, [(f"{scan}/volume.json", "volume.json"),
                                       (f"{scan}/native/{z}_{y}_{x}.blosc", "block.blosc")])
shape = json.load(open("volume.json"))["shape_zyx"]
d = [min(512, s - o) for s, o in zip(shape, (z, y, x))]
v, h = np.frombuffer(numcodecs.Blosc().decode(open("block.blosc", "rb").read()), np.uint8).reshape([2] + d)
print(v.shape, round(v.mean() / 255, 4), round(h.mean() / 255, 4))

List a scan's blocks with HfApi().list_bucket_tree(bucket, prefix="PHerc0139/native").

How it was made

• Model: Qualzz20/afv_fiber_9um_v2 (Apache-2.0), loaded with its own model.py; Qual's recipe: 128-voxel windows at stride 64 on one lattice fixed to the whole volume, Gaussian blending, bf16, per-window normalization, no mirroring. The probabilities are the softmax of the model's vertical and horizontal channels. Run on Hugging Face Jobs (RTX PRO 6000), 1024-voxel tiles.
• Region curves: AVF_SPLINE_GENERATOR at commit 3c8b7362, one run per 1024-voxel tile with a 128-voxel halo and seeds only inside the tile. The datasets' files are these curves cut at the tile faces, joined across faces and cleaned of curves touching the scan's outside.
• Same files as the run: everything here was copied server-side from the run's working storage, byte for byte.

License and credit

The scans are Vesuvius Challenge open data, licensed CC BY-NC 4.0; these maps are derived from them and carry the same terms. The model and the spline generator are Qual's, under Apache-2.0. Please credit the Vesuvius Challenge, Qual's AFV model, and this bucket.