README.md

---
license: cc-by-nc-4.0
tags:
- vesuvius-challenge
- herculaneum
- papyrology
- image-classification
- pytorch
library_name: pytorch
pipeline_tag: image-classification
---

# Herculaneum legibility proxy (proxy_v4 / proxy_s2s3_v1)

A small ResNet-18 binary classifier that scores ~1 cm windows of Vesuvius
Challenge **ink-probability maps** for *legible-text likelihood*: does this
window contain connected Greek letterforms, or just fiber texture / noise /
damage?

It is a **triage tool, not an ink detector**. It runs *downstream* of the
official ink-detection models, on their output maps, and answers a different
question: "of all this detected ink, where should a human look first?"

Built and used to produce a full legibility index of Scroll 1 (PHerc. Paris
4): **125,298 windows scored across all 78 of Scroll 1's currently-published
official ink maps** (complete coverage, verified live against the S3 bucket
2026-07-08 — see "Coverage" below for how this grew from an initial 16-panel
pass). 5,678 high-confidence text windows on the deduplicated wrap-series union
(w010–w129); classic-lineage segments add 2,460 more on 18 physical
surfaces (kept separate — their overlap with the series is unresolved), and
7,248 further gold windows come from redundant re-renders, counted only as
consistency checks (see Coverage below). The same checkpoint, unmodified, was
then run on three more scrolls with no fine-tuning in between:

| Scroll | Windows scored | Segments/maps used | Coverage of official maps | Gold (≥0.9) | Independent check |
|---|---|---|---|---|---|
| Scroll 1 (PHerc. Paris 4) | 125,298 | 78 official segments (16 original panels + 62 added 2026-07-08) | 78 of 78 — complete | 5,678 on the deduplicated wrap-series union w010–w129 (+2,460 on classic-lineage segments, kept separate; +7,248 redundant re-renders — see Coverage) | 3 held-out panels, AUROC 0.985; wrap w028-037 independently gold-confirmed in 2 unrelated renders (50.9% and 40.0% gold density) |
| PHerc 0139 | 4,905 | 38 | 38 of 38 — complete | 63 | title segment ranked 2nd of 38 by gold density; 63/63 human-reviewed (single reviewer): 54 clear-text / 9 possible / 0 rejected |
| Scroll 5 (PHerc 172) | 24,528 | 53 × 2 ink-detector models | 53 of 53 — complete | 3 (by `july_retreat`-model score; 0 by the more conservative `november19` model — see cross-render caveat below) | max score 0.938 on a window with plainly visible, large uppercase Greek letters (segment w066) |
| Scroll 4 (PHerc 1667) | 4,761 | 19 official maps | 19 of 19 segments that carry an official ink map (1 of the 20 public segments has none) — complete | 430 | per-wrap gold fraction correlates at Spearman r = 0.843 (p < 0.0001, n = 18) with the count of Greek letters *already transcribed* in that wrap — ground truth external to this model |

### Coverage — how the Scroll 1 index reached 78/78

The index originally covered only 16 of Scroll 1's 81 public segments (one
continuous panel series, `w010`–`w100`, uploaded 2026-06-23) — 29,853
windows, 4,491 gold. A live S3 check on 2026-07-08 found the scroll had
grown to 78 populated official maps (3 of the 81 public segments still have
none). The remaining 62 were scored with the identical unmodified pipeline
(same `proxy_v4.pt`, same WIN=512/STRIDE=256 grid, same ds8-resolution
preprocessing — verified byte-for-byte SHA256-identical against the
official pre-made `ink-detection/downsampled/*-ds8.jpg` product before
trusting it on new segments), split into four groups:

| Group | Segments | Windows | Gold (≥0.9) | Territory |
|---|---|---|---|---|
| 23 "classic"-lineage panels (name-level dedup: 18 physical surfaces) | 23 | 8,978 | 2,684 (2,460 on representatives + 224 on re-renders) | **separate bucket** — overlap with the panel series unresolved, so never summed into the wrap-series denominator |
| 11 more panels from the same June-2026 series, `w101`–`w129` | 11 | 14,998 | 1,056 | **distinct** — extends past the original `w100` boundary |
| 2 alternate-processing panels, `w046-052_jordi` / `w053-058_jordi` | 2 | 3,929 | 739 | **overlaps** — re-renders of `w046-052`/`w053-058`, already in the original 16 panels (confirmed by near-identical gold-density-by-row fingerprint against the originals) |
| 26 panels from a second complete series uploaded 2026-07-01, covering `w010`–`w129` | 26 | 67,540 | 6,416 | **one panel is unique territory**: the June `w104-106` segment has no published ink map; the July `w104-106` panel (2,648 windows, 131 gold) covers that range and joins the deduplicated union. The other 25 panels (6,285 gold) re-cover indexed wraps and stay redundant |

**Correction (2026-07-08, caught in adversarial review):** the `_jordi`
panels were originally folded into the "distinct" count because their
segment IDs looked like part of the `w101`–`w129` extension. They aren't —
they cover wrap ranges already present in the original 16-panel index. Moved
to the overlap bucket; the headline distinct-territory number dropped from
an incorrect 8,970 to a correct 8,231 as a result. The July-2026 series is
confirmed *not* a byte-identical duplicate of the June one (e.g. the
`w010-027` file is 682 MB in June vs 1.07 GB in July, same recipe tag), but
it covers the same physical wraps, so — like the `_jordi` panels — its gold
windows are reported separately rather than folded into the "distinct
territory" total, to avoid double-counting the same underlying text.
[Update, 2026-07-11: with one exception found later — the July `w104-106`
panel is unique territory; see the correction below.] Both
overlap groups are useful anyway as independent consistency checks: wrap
range `w028-037` (June vs July) scores gold at 50.9%/40.0%, and
`w046-052`/`w053-058` (original vs `_jordi`) show matching gold-density
fingerprints band-by-band — unrelated renders of the same regions agreeing
on where the text is.

**Correction (2026-07-11, peer review):** an earlier version of this card
reported "8,231 gold on distinct territory" by summing the wrap-series
panels with the classic segments. Two problems, both fixed here: the June
`w104-106` segment has no published ink map, so the July panel covering
those wraps is unique territory (131 gold) wrongly excluded as redundant; and the classic
bucket contained re-renders of the same physical surfaces (a `_copy`, a
`_v14`, a `_v2_flatboi`, a `_v8`, an offset-0 re-render — 23 panels are 18
surfaces) and its physical overlap with the panel series is unresolved.
**Canonical accounting: deduplicated wrap-series union (June w010–w129 +
July w104-106) = 5,678 gold windows; classic-lineage representatives =
2,460 gold on 18 surfaces, reported separately; redundant re-renders =
7,248 gold (224 classic + 739 `_jordi` + 6,285 July), used only as
consistency checks.** Cross-check invariant: 5,678 + 2,460 + 224 + 739 +
6,285 = 15,386 gold rows in the full 125,298-window index — **shipped as
`full_index_complete.json` in the companion dataset repo** (SHA256
`4d393d70ce886ed62b7e73e365f1d01cbe7f6efa37168fb3f27ade2b89d6e7a8`; every
bucket and denominator above is recomputed with asserts by its
`summarize_index.py`). Note the score key in that file is named `v3` for
legacy reasons; every score in it was produced by `proxy_v4.pt`.

## Files

| File | What it is | MD5 (first 12) |
|---|---|---|
| `proxy_v4.pt` | main checkpoint, trained on official Scroll 1 panel maps | `c869ce189f2a` |
| `proxy_s2s3_v1.pt` | domain-calibrated variant: same S1 knowledge + 192 windows of self-rendered Scroll 2/3 maps as explicit negatives | `95b6839c58c8` |

Both are plain `state_dict`s for `torchvision.models.resnet18()` with
`fc = nn.Linear(512, 1)` (122 tensors).

### Which one to use — the split is by *rendering pipeline*, not by scroll

`proxy_v4` generalizes across **scrolls**, as long as the map came out of
the official/community rendering pipeline (the texture statistics of an
"official-style" ink map are consistent regardless of which scroll it's
of). See the table above for the four-scroll validation: one native, one
blind positive control (0139), one cross-render-calibration caveat that's
about the ink detector, not this classifier (Scroll 5), and one check
against fully independent, non-visual ground truth — transcribed-letter
counts, not just human eyeballing (Scroll 4; project-internal validation
note — the correlation, n, and p reported above are the complete result).

`proxy_s2s3_v1` exists because that assumption breaks on maps rendered by
a **custom / non-official pipeline** — which for us happened to be Scrolls
2 and 3, only because no official ink maps existed yet for the regions we
needed and we rendered our own. On those self-rendered mosaics, `proxy_v4`
alone gives an inflated background-noise floor (~0.04); the 192
self-rendered-domain negatives used to fine-tune `proxy_s2s3_v1` drop that
to ~0.003 without hurting Scroll 1 performance (AUROC 0.984 vs 0.985). It
was never evaluated on official maps of other scrolls (0139, Scroll 5) —
there's no reason to prefer it over `proxy_v4` there, since `proxy_v4`
already has evidence of working on that domain and `proxy_s2s3_v1` doesn't.

| Map came from | Use |
|---|---|
| Official / community rendering pipeline (any scroll) | `proxy_v4.pt` |
| Your own custom rendering pipeline | `proxy_s2s3_v1.pt`, or better: fine-tune your own domain-negatives the same way (see Training summary) |

## How to use

The preprocessing is part of the model. Deviate from it and the scores
shift silently — this exact recipe is what the training used:

```python
import numpy as np
import torch, torch.nn as nn, torchvision
from PIL import Image

m = torchvision.models.resnet18()
m.fc = nn.Linear(512, 1)
m.load_state_dict(torch.load("proxy_v4.pt", map_location="cpu"))
m.eval()

def score_window(win_u8):          # win_u8: 2D uint8 crop of an ink map
    c = win_u8.astype(np.float32)
    active = c[c > 10]
    if len(active) > 50:           # percentile stretch over active pixels
        lo, hi = np.percentile(active, [2, 99.5])
        c = np.clip((c - lo) / max(hi - lo, 1e-6), 0, 1)
    else:
        c = c / 255.0
    img = Image.fromarray((c * 255).astype(np.uint8)).resize((160, 160), Image.BILINEAR)
    t = torch.from_numpy(np.array(img, np.float32) / 255.0)[None].repeat(3, 1, 1)
    with torch.no_grad():
        return torch.sigmoid(m(t[None])).item()
```

### The window must be ~1 cm physical — this is the #1 usage error

The model was trained on 512 px windows of ds8 panels at ≈19.2 µm/px
≈ **9,830 µm ≈ 1 cm** per window. On any other map, compute the window
size from the segment's real physical scale, e.g.:

```python
px_um = (area_cm2 * 1e8 / (H * W)) ** 0.5   # from the mesh meta.json area
WIN   = round(9830 / px_um)
```

Two failure modes we hit ourselves, so you don't have to:

1. **Do not assume 512 px.** On a 7.91 µm-scan ds8 map the right window is
   ~155 px; on a 2.4 µm-scan ds8 map it's ~512–560 px.
2. **Do not stretch undersized crops.** If a fragment is smaller than WIN,
   resizing what you have up to 160×160 changes the fiber texture's
   apparent frequency and *inflates scores* (we measured score
   anti-correlating with true window size, r ≈ −0.45, before fixing this).
   Pad onto a black 160×160 canvas at the training scale
   (61.4 µm per classifier pixel) instead, and treat such windows with
   extra suspicion.

### Thresholds we used

- **≥ 0.9** — "gold": on Scroll 1, round-2 human review confirmed 120/120
  of the model's top-ranked proposals. On PHerc 0139, all 63 gold windows
  were human-reviewed (single reviewer): 54 clear text (85.7%), 9 possible,
  0 rejected as noise/artifact. These are one reviewer's visual verdicts on
  model-selected windows, not population precision estimates.
- **0.35–0.70** — genuinely ambiguous band; useful for uncertainty
  sampling if you're fine-tuning.
- Scores are *not* calibrated probabilities, and they do **not transfer
  across ink-model renders**: on Scroll 5, the same three candidate windows
  (segment w066 and neighbors) scored 0.938 / 0.914 / 0.903 on
  `july_retreat`-model maps but 0.266 / 0.029 / 0.334 on `november19`-model
  maps of the identical windows — pixel correlation between the two maps
  in those windows is 0.86–0.89 (i.e. both detectors render essentially the
  same visible letterforms; only the score disagrees). Rank within one map
  family; never compare raw scores across families, and never take an
  AND-style agreement filter (`min(scoreA, scoreB) ≥ 0.9`) at face value
  without checking whether one model is just systematically deflated —
  we lost 3 true positives to exactly that before catching it.

## Training summary

- **Architecture:** torchvision ResNet-18 (ImageNet init), `fc → 1`,
  BCE-with-logits, AdamW.
- **Data (proxy_v4):** 541 positive windows from three rounds of
  human-in-the-loop review of official Scroll 1 panel maps (393 browse-all
  + 120 model-proposed/human-confirmed + 28 uncertainty-sampled), negatives
  from 50 GPU-verified fiber windows (shipped in the companion dataset
  repo as `fiber_negatives_50.jsonl`; 12 of the 50 fall in the validation
  panels), background sampling, and 87
  human-implied negatives at weight 0.7; 22 human-"unsure" windows
  excluded. Spatial validation: 3 full held-out panels → AUROC 0.985.
  (Corrected 2026-07-08: the browse-all round produced 398 raw rows, but 5
  of those were tagged "unsure," not positive — 393 + 120 + 28 = 541. The
  full, independently re-verified breakdown ships as training *data*, not
  just this prose summary — see the companion dataset repo:
  https://huggingface.co/datasets/LimeGS/herculaneum-legibility-proxy-labels.)
- **proxy_s2s3_v1:** initialized from proxy_v4, fine-tuned at LR 1e-4 with
  self-rendered Scroll 2/3 negatives: 23 human-reviewed noise windows at
  weight 1.0, plus ~169 additional implicit negatives at weight 0.7 from
  the same rendering pass. The 7 human-flagged candidate windows were
  excluded from training entirely. **Reproducibility caveat:** only the 23
  explicitly-reviewed windows are traceable in this release — the ~169
  implicit negatives were read at training time from a session-scratchpad
  file that no longer exists, so they can't currently be regenerated
  bit-for-bit. This doesn't change what was trained (the weights are
  final), it means the companion dataset repo can reproduce 23 of the
  ~192 S2/S3 negatives, not all of them — noted there in full, not glossed
  over here.
- Trains in minutes on a laptop (Apple MPS, CUDA, or CPU); this is
  deliberately a small, reproducible model.

## Training data release

The human-labeled training coordinates (not images — just `{panel, y, x,
label, weight}`, since the underlying maps are already public) are
released alongside this model at
https://huggingface.co/datasets/LimeGS/herculaneum-legibility-proxy-labels, with a
crop-regeneration script, a consolidated training script, pinned dependency
versions, and a self-test. Scope of reproducibility, stated precisely: the
full `proxy_v4` recipe (labels + 50 fiber negatives + procedural background
negatives → crops → training with the original spatial holdout) can be
re-run end to end from public data; expect a checkpoint that matches in
architecture and closely in behavior, not bit-for-bit (GPU/RNG/environment
nondeterminism; a retrained checkpoint's AUROC will vary around 0.985,
while the PUBLISHED checkpoint's validation AUROC is exactly reproducible:
the dataset repo's `eval_checkpoint.py` deterministically rebuilds the
held-out split, 81 positives / 123 negatives, and yields 0.9853457794). `proxy_s2s3_v1` additionally has the documented 169-negative
gap. The scoring INDEX built with the model ships in the same dataset repo
(`full_index_complete.json` + `summarize_index.py`); the sweep script that
produced it is project-internal, but its recipe is fully specified above
(the `score_window()` preprocessing on a WIN=512 / stride-256 grid).

## Limitations / what this does not claim

- **Not an ink detector.** Input is an ink-probability map produced by the
  official/community detectors, never raw CT.
- **Finds letterform-like structure, not meaning.** A high score means
  "connected strokes at letter scale," which is what a human reviewer
  should look at — it is not a transcription and not proof of novel text.
- **Domain is defined by rendering pipeline, not by scroll identity** — see
  "Which one to use" above. `proxy_v4` is validated on official-style maps
  from four scrolls (S1, PHerc 0139, Scroll 5, Scroll 4); a fifth scroll's
  official maps haven't been tried, though the pattern across four
  independent scrolls (including one quantitative correlation against
  transcribed-text ground truth, not just spot checks) makes transfer the
  expected outcome, not a hopeful one.
- Cross-render score shifts are real even within `proxy_v4`'s validated
  domain: two different official ink-detection models of the *same* scroll
  region can disagree by 0.1+ (see Thresholds) — that's the ink detector's
  output changing, not this classifier's domain breaking.
- Windows flagged "gold" on public maps are *locations worth expert
  attention*, on data anyone can download — whether the text there is
  actually untranscribed must be checked against the papyrological
  literature, which is a moving target.

## Provenance & license

Trained exclusively on publicly released Vesuvius Challenge data
(CC BY-NC 4.0); the weights are released under the same CC BY-NC 4.0 to
match. Human labels by the project's own reviewer. No official-team code
or checkpoints are redistributed here.