Buckets:
| # AFV V2 fiber probability maps for six scrolls | |
| The voxel maps behind the six public [AFV fiber datasets](https://huggingface.co/collections/SharpMeow/afv-fibers-papyrus-fibers-for-six-scrolls-6ae0d3c8540ed245dd349c91): | |
| for every voxel of the scan, the probability that [Qual](https://huggingface.co/Qualzz20)'s **AFV V2** model gives to a | |
| vertical fiber and to a horizontal fiber. The traced fibers (`.afv` curves) are in the datasets; this bucket holds the | |
| maps they were traced from, so you can trace again with other settings, use the maps as features, or check a fiber | |
| against the probability around it. Model output about papyrus fiber texture, not a reading. | |
| | Scan | Volume (Vesuvius Challenge open data) | Level | Voxel | Native maps | Pooled maps | Region curves | | |
| | --- | --- | --- | --- | --- | --- | --- | | |
| | PHerc0009B | `20250521125136-8.640um-1.2m-116keV-masked.zarr` | 0 | 8.640 um | 32.4 GB, 1,128 blocks | 5.2 GB | 4.6 GB, 223 regions | | |
| | PHerc0139 | `20250728140407-9.362um-1.2m-113keV-masked.zarr` | 0 | 9.362 um | 163.1 GB, 2,878 blocks | 26.5 GB | 24.0 GB, 467 regions | | |
| | PHerc0332 | `20251211183505-2.399um-0.2m-78keV-masked.zarr` | 2 | 9.596 um | 22.2 GB, 456 blocks | 3.6 GB | 3.0 GB, 89 regions | | |
| | PHerc0814 | `20250804134230-9.362um-1.2m-113keV-masked.zarr` | 0 | 9.362 um | 122.5 GB, 2,122 blocks | 19.8 GB | 18.6 GB, 351 regions | | |
| | PHerc0841 | `20250821151531-9.366um-1.2m-113keV-masked.zarr` | 0 | 9.366 um | 176.9 GB, 3,089 blocks | 28.2 GB | 22.8 GB, 501 regions | | |
| | PHerc1667 | `20251217075048-2.399um-0.2m-78keV-masked.zarr` | 2 | 9.596 um | 20.8 GB, 509 blocks | 3.4 GB | 2.4 GB, 94 regions | | |
| ## Layout | |
| ``` | |
| SCAN/volume.json scan, volume, level, CT URL, volume shape, block and codec, the 1024-voxel tiles | |
| SCAN/native/Z_Y_X.blosc one 512^3 block of the map at the model's level, origin Z_Y_X | |
| SCAN/pooled/Z_Y_X.blosc the same block 2x average pooled (256^3), the input the spline generator prefers | |
| SCAN/regions/TILE.afv the curves traced in one 1024-voxel tile before stitching (+ TILE.afv.json) | |
| SCAN/tiles/TILE.json the model run for that tile: windows run, timings | |
| SCAN/level2/ PHerc0332 and PHerc1667 only: the stitched fibers in level-2 voxels, before the | |
| level-0 copies in the datasets were made from them | |
| ``` | |
| Each `.blosc` file is raw uint8 bytes compressed with numcodecs `Blosc(cname="zstd", clevel=3, shuffle=BITSHUFFLE)`, | |
| shaped `(2, dz, dy, dx)`: channel 0 is the vertical fiber probability, channel 1 the horizontal one, each times 255. | |
| Blocks start at multiples of 512 and are 512 voxels on a side except at the volume's far edges | |
| (`dz = min(512, shape_z - Z)`, likewise y and x, with `shape_zyx` from `volume.json`). A pooled block has half each | |
| side (`dz // 2`). A block that is absent is all zero. Coordinates are voxels of the level in | |
| `volume.json` (level 0 for four scans, level 2 for PHerc0332 and PHerc1667; multiply by 4 for level 0). | |
| ## Read one block | |
| ```python | |
| import json | |
| import numcodecs | |
| import numpy as np | |
| from huggingface_hub import HfApi | |
| bucket, scan, z, y, x = "SharpMeow/afv-fiber-probabilities", "PHerc0139", 10752, 5120, 3584 | |
| HfApi().download_bucket_files(bucket, [(f"{scan}/volume.json", "volume.json"), | |
| (f"{scan}/native/{z}_{y}_{x}.blosc", "block.blosc")]) | |
| shape = json.load(open("volume.json"))["shape_zyx"] | |
| d = [min(512, s - o) for s, o in zip(shape, (z, y, x))] | |
| v, h = np.frombuffer(numcodecs.Blosc().decode(open("block.blosc", "rb").read()), np.uint8).reshape([2] + d) | |
| print(v.shape, round(v.mean() / 255, 4), round(h.mean() / 255, 4)) | |
| ``` | |
| List a scan's blocks with `HfApi().list_bucket_tree(bucket, prefix="PHerc0139/native")`. | |
| ## How it was made | |
| - **Model**: [Qualzz20/afv_fiber_9um_v2](https://huggingface.co/Qualzz20/afv_fiber_9um_v2) (Apache-2.0), loaded with | |
| its own `model.py`; Qual's recipe: 128-voxel windows at stride 64 on one lattice fixed to the whole volume, Gaussian | |
| blending, bf16, per-window normalization, no mirroring. The probabilities are the softmax of the model's vertical | |
| and horizontal channels. Run on Hugging Face Jobs (RTX PRO 6000), 1024-voxel tiles. | |
| - **Region curves**: [AVF_SPLINE_GENERATOR](https://github.com/Qualzz/AVF_SPLINE_GENERATOR) at commit `3c8b7362`, one | |
| run per 1024-voxel tile with a 128-voxel halo and seeds only inside the tile. The datasets' files are these curves | |
| cut at the tile faces, joined across faces and cleaned of curves touching the scan's outside. | |
| - **Same files as the run**: everything here was copied server-side from the run's working storage, byte for byte. | |
| ## License and credit | |
| The scans are Vesuvius Challenge open data, licensed [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/); | |
| these maps are derived from them and carry the same terms. The model and the spline generator are Qual's, under | |
| Apache-2.0. Please credit the Vesuvius Challenge, Qual's AFV model, and this bucket. | |
Xet Storage Details
- Size:
- 4.99 kB
- Xet hash:
- 86c589f00c29bbb15c161f0facb16c96a51695e44e75394a0e9d62421e332c66
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.