Umbilici 1.13.0: hide line ends outside the scroll; lines from center lines v2.9 (from Scrolls main 48e9346)

README.md

@@ -114,6 +114,21 @@ villa's source (`main`, read 23 Oct 2026):
 Because VC3D holds the ends and spiral-fitting extrapolates them, check a line's usable ends before relying on
 it near the top or bottom of a scan.
 
+## Ends outside the scroll
+
+Since 1.13.0 each line is drawn from its first to its last point inside the scroll. `tools/build_lines.py` tests
+every line's end points against the scan's masked CT at level 5 (32 times down; voxels outside the scroll are zero):
+a point lies outside when, in its own axial slice, it sits more than 4 mm outside the scroll's outline (the convex
+hull of the nonzero voxels) or the slice holds no scroll at all. At each end the hidden run reaches from the end to
+the innermost outside point among the outer quarter of the line's points; the middle half is never hidden. Two
+checks make the test able to fail: the line's middle point must be inside and the slice corner farthest from it
+outside, or the line is drawn whole and marked untested. Each card says per end how many points are hidden and how
+far past the scroll they lie, the Data tables have `low out` and `high out` columns, and the drift statistics
+(mean, stray, tilt, wiggle, path, pair gaps) use the inside part. **show ends outside the scroll** at bottom left
+(`outside=1` in the view link) draws the hidden runs faded and dotted. Some Center Finder v2 lines run tens of mm
+sideways at an end, where their start line was never trimmed to the papyrus; their files are unchanged here, and
+the downloads, `lines()` and the copied URLs keep every point.
+
 ## X-ray
 
 CT is not stored on this Space. Each visitor's browser reads coarse levels of the OME-Zarr volumes straight from
@@ -140,19 +155,20 @@ The page is driven from the console or a headless browser through `window.umbili
 | Call | Returns or does |
 | --- | --- |
 | `scans()` | every scan: scroll, scan id, volume, voxel size, shape, prize lists, sources, end grade |
-| `lines(scroll)` | that scan's lines with control points `[x, y, z]` in level-0 voxels |
+| `lines(scroll)` | that scan's lines with control points `[x, y, z]` in level-0 voxels, every point of the file, and `outside` (points hidden at each end, or null with `outside_why`) |
 | `at(scroll, z)` | each line's `[x, y]` at height `z` (linear between control points) |
 | `select(scroll, mode)`, `mode('array' or 'scan' or 'lasagna')`, `setZ(z)` | change the view |
-| `state()` | mode, scan, height, CT streaming progress |
+| `state()` | mode, scan, height, whether the ends outside the scroll are shown, CT streaming progress |
+| `outside(on)` | shows (`true`) or hides (`false`) the line ends that lie outside the scroll in the CT, omitted toggles; returns `{outside}`, also kept in the URL as `outside=1` |
 | `image()`, `slice()` | PNG data URLs of the 3D view and the slice |
-| `stats(scroll)` | every statistic of each line (points, z range, coverage, steps, stray, tilt, wiggle, path, stamp) and the pair gaps |
+| `stats(scroll)` | every statistic of each line (points, z range, coverage, steps, points hidden at each end, stray, tilt, wiggle, path, stamp; the drift statistics on the inside part) and the pair gaps |
 | `table('lines' or 'scans')`, `csv('lines' or 'scans' or 'points')` | the Data tables as objects or CSV text |
 | `guide(tab)`, `data(tab)` | open the guide (`basics`, `vc3d`, `research`, `data`, `credits`) or the Data tables |
 | `bench(n)` | renders `n` frames and returns milliseconds per frame and the pixel ratio |
-| `show(kind, name, on)` | shows or hides lines: `kind` `'source'` with a maker (`bucket`, `drobkov`, `tauil`, `dopico`, `finder`, `finder-polished`) or `'line'` with `SCROLL:SOURCE`; `on` true or false, omitted toggles; returns `{off}`, also kept in the URL as `off=` |
+| `show(kind, name, on)` | shows or hides lines: `kind` `'source'` with a maker (`bucket`, `drobkov`, `tauil`, `dopico`, `finder`, `finder-polished`, `finder-v2`) or `'line'` with `SCROLL:SOURCE`; `on` true or false, omitted toggles; returns `{off}`, also kept in the URL as `off=` |
 | `settings(obj)` | read or change the viewer settings: `glow` (0 to 3), `radius` (0 to 1), `quality` (`auto`, `high`, `low`), `film`, `spin`, `inset` (the slice panel shown), `turn` (the slice turns with the 3D view) |
 
-URL parameters: `?mode=scan&scan=PHerc0191&z=10400`, `?mode=lasagna&scan=PHerc0175A&z=6400`, `?ex=5`,
+URL parameters: `?mode=scan&scan=PHerc0191&z=10400`, `?mode=lasagna&scan=PHerc0175A&z=6400`, `?ex=5`, `?outside=1`,
 `?guide=vc3d`, `?data=lines`, `?glow=1.5`. The gear button holds the settings (glow strength and spread, quality,
 film grain, spin), saved in the browser.
 `data/lines.json` is the line table built at deploy by `tools/build_lines.py` from a copy of the dataset. On load the page reads the dataset's newest revision from huggingface.co and shows it instead, so new releases appear without a redeploy; the built table is the fallback.

app.js

@@ -7,7 +7,7 @@ import { RenderPass } from 'three/addons/postprocessing/RenderPass.js';
 import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js';
 import { OutputPass } from 'three/addons/postprocessing/OutputPass.js';
 
-const VERSION = '1.12.0';
+const VERSION = '1.13.0';
 const DATASET = 'https://huggingface.co/datasets/SharpMeow/vesuvius-center-lines';
 const LASAGNA_SCROLL = 'PHerc0175A';
 const ICE = new THREE.Color('#89f0ff');
@@ -27,6 +27,7 @@ const $ = (id) => document.getElementById(id);
 const S = {
   data: null, las: null, mode: 'array', sel: null, hot: null, ex: 3, z: null,
   ct: true, render: 'xray', gain: 44, sheets: true, needles: true, ct2: false, ori: false, blend: 70,
+  outside: false, // draw the end runs that lie outside the scroll in the CT (hidden by default; URL outside=1)
   dirty: true, status: '',
 };
 
@@ -90,16 +91,25 @@ const mm = (s, v) => v * s.voxel_um / 1000;
 const fmt = (v, d = 1) => Number(v).toFixed(d);
 async function getJSON(url) { const r = await fetch(url); if (!r.ok) throw new Error(`${r.status} ${url}`); return r.json(); }
 function linePts(l) { const p = l.points, out = []; for (let i = 0; i < p.length; i += 3) out.push([p[i], p[i + 1], p[i + 2]]); return out; }
-function lineAt(l, z) {
-  const p = linePts(l);
-  if (z < p[0][2] || z > p[p.length - 1][2]) return null;
+// The CT test of tools/build_lines.py (outside_ends): l.outside counts, per end, the control points from that end to
+// the innermost one that lies outside the scroll in the CT; null when the line could not be tested (l.outside_why).
+// Those end runs are hidden unless S.outside, and every drift statistic uses the inside part only. The file, its
+// points and its sha256 are the dataset's, unchanged.
+function cut(l) { const n = l.points.length / 3, o = l.outside; return o ? [o.low.hidden, n - o.high.hidden] : [0, n]; }
+function insidePts(l) { const [a, b] = cut(l); return linePts(l).slice(a, b); }
+// whether height z falls in a hidden end run of l
+function outsideAt(l, z) { const o = l.outside; if (!o) return false; const p = l.points, n = p.length / 3, [a, b] = cut(l); return (a > 0 && z < p[3 * a + 2]) || (b < n && z > p[3 * (b - 1) + 2]); }
+function lineAt(l, z, pts) {
+  const p = pts || linePts(l);
+  if (!p.length || z < p[0][2] || z > p[p.length - 1][2]) return null;
   for (let i = 1; i < p.length; i++) {
     if (p[i][2] >= z) { const a = p[i - 1], b = p[i], t = b[2] === a[2] ? 0 : (z - a[2]) / (b[2] - a[2]); return [a[0] + t * (b[0] - a[0]), a[1] + t * (b[1] - a[1])]; }
   }
   return null;
 }
-function lineMean(l) { const p = linePts(l); let x = 0, y = 0; for (const q of p) { x += q[0]; y += q[1]; } return [x / p.length, y / p.length]; }
-function drift(s, l) { const [mx, my] = lineMean(l); let d = 0; for (const q of linePts(l)) d = Math.max(d, Math.hypot(q[0] - mx, q[1] - my)); return mm(s, d); }
+// the mean and the stray use the inside part (insidePts), so an end that runs off past the scroll moves neither
+function lineMean(l) { const p = insidePts(l); let x = 0, y = 0; for (const q of p) { x += q[0]; y += q[1]; } return [x / p.length, y / p.length]; }
+function drift(s, l) { const [mx, my] = lineMean(l); let d = 0; for (const q of insidePts(l)) d = Math.max(d, Math.hypot(q[0] - mx, q[1] - my)); return mm(s, d); }
 const isFinder = (l) => l.source === 'finder';
 // an older version (superseded_by names the newer line's file) or a withdrawn line is drawn faded
 const isOld = (l) => !!l.superseded_by || l.note.includes('withdrawn');
@@ -155,8 +165,10 @@ const RAW = (l) => `${DATASET}/resolve/main/${l.file}`;
 const lic = (l) => l.licence;
 const median = (a) => { if (!a.length) return null; const b = [...a].sort((u, v) => u - v); const k = b.length >> 1; return b.length % 2 ? b[k] : (b[k - 1] + b[k]) / 2; };
 function lineStats(s, l) {
-  const p = linePts(l), n = p.length;
-  const dz = []; for (let i = 1; i < n; i++) dz.push(p[i][2] - p[i - 1][2]);
+  // points, z range, coverage and steps describe the whole file; mean, stray, tilt, wiggle and path the inside part
+  const all = linePts(l), N = all.length;
+  const dz = []; for (let i = 1; i < N; i++) dz.push(all[i][2] - all[i - 1][2]);
+  const p = insidePts(l), n = p.length;
   const [mx, my] = lineMean(l);
   // least-squares x(z), y(z)
   let sz = 0, szz = 0, sx = 0, sy = 0, sxz = 0, syz = 0;
@@ -167,18 +179,21 @@ function lineStats(s, l) {
   for (let i = 0; i < n; i++) { const [x, y, z] = p[i]; r2 += (x - ax - bx * z) ** 2 + (y - ay - by * z) ** 2; if (i) path += Math.hypot(x - p[i - 1][0], y - p[i - 1][1], z - p[i - 1][2]); }
   const st = l.stamp || {};
   const stamped = st.volume_width === s.shape[2] && st.volume_height === s.shape[1] && st.volume_slices === s.shape[0];
-  const z0 = p[0][2], z1 = p[n - 1][2];
+  const z0 = all[0][2], z1 = all[N - 1][2], o = l.outside;
   return {
-    points: n, z_start: z0, z_end: z1, height_mm: mm(s, z1 - z0), coverage: (z1 - z0) / s.shape[0],
+    points: N, z_start: z0, z_end: z1, height_mm: mm(s, z1 - z0), coverage: (z1 - z0) / s.shape[0],
+    outside_low: o ? o.low.hidden : null, outside_high: o ? o.high.hidden : null, inside_z_start: p[0][2], inside_z_end: p[n - 1][2],
     median_step_vox: median(dz), max_step_vox: dz.length ? Math.max(...dz) : null,
     mean_x: mx, mean_y: my, off_center_mm: mm(s, Math.hypot(mx - s.shape[2] / 2, my - s.shape[1] / 2)),
     stray_mm: drift(s, l), tilt_deg: Math.atan(Math.hypot(bx, by)) * 180 / Math.PI, wiggle_mm: mm(s, Math.sqrt(r2 / n)),
-    path_mm: mm(s, path), path_over_height: path / Math.max(1, z1 - z0), stamped,
+    path_mm: mm(s, path), path_over_height: path / Math.max(1, p[n - 1][2] - p[0][2]), stamped,
   };
 }
+// pair gaps over the heights where both lines' inside parts exist
 function pairStats(s, a, b) {
-  const lo = Math.max(a.z[0], b.z[0]), hi = Math.min(a.z[1], b.z[1]); const g = [];
-  for (let z = lo; z <= hi; z += 64) { const p = lineAt(a, z), q = lineAt(b, z); if (p && q) g.push(Math.hypot(p[0] - q[0], p[1] - q[1])); }
+  const pa = insidePts(a), pb = insidePts(b);
+  const lo = Math.max(pa[0][2], pb[0][2]), hi = Math.min(pa[pa.length - 1][2], pb[pb.length - 1][2]); const g = [];
+  for (let z = lo; z <= hi; z += 64) { const p = lineAt(a, z, pa), q = lineAt(b, z, pb); if (p && q) g.push(Math.hypot(p[0] - q[0], p[1] - q[1])); }
   if (!g.length) return null;
   return { a: a.source, b: b.source, overlap_mm: mm(s, hi - lo), samples: g.length, median_mm: mm(s, median(g)), min_mm: mm(s, Math.min(...g)), max_mm: mm(s, Math.max(...g)) };
 }
@@ -190,7 +205,7 @@ function lineRows() {
     const t = statsOf(s, l);
     return {
       scroll: s.scroll, scan: s.scan, volume: s.volume, voxel_um: s.voxel_um, source: l.source, maker: S.data.sources[l.source].label,
-      kind: l.kind, grade: l.grade || '', ct_untested: !!l.untested, ct_end_low: endText(l, 'low'), ct_end_high: endText(l, 'high'), usable_ends: isFinder(l) ? (usable(l) || '') : '', withdrawn: l.note.includes('withdrawn'), superseded: !!l.superseded_by,
+      kind: l.kind, grade: l.grade || '', ct_untested: !!l.untested, ct_end_low: endText(l, 'low'), ct_end_high: endText(l, 'high'), outside_tested: !!l.outside, outside_why: l.outside_why || '', usable_ends: isFinder(l) ? (usable(l) || '') : '', withdrawn: l.note.includes('withdrawn'), superseded: !!l.superseded_by,
       ...t, license: lic(l), sha256: l.sha256, file: RAW(l), upstream: l.upstream, note: l.note,
     };
   }));
@@ -242,6 +257,25 @@ function tube(points3, color, radius, opacity = 1) {
   const m = new THREE.MeshBasicMaterial({ color: color.clone().multiplyScalar(1.35), transparent: true, opacity, depthWrite: true, toneMapped: false });
   const mesh = new THREE.Mesh(g, m); mesh.renderOrder = 3; mesh.userData.base = opacity; return mesh;
 }
+// a hidden end run when the viewer shows it: faded dots along the same curve, one draw call
+function dots(points3, color, radius, opacity) {
+  const curve = new THREE.CatmullRomCurve3(points3.map((p) => new THREE.Vector3(...p)), false, 'centripetal');
+  const n = Math.max(2, Math.min(600, Math.floor(curve.getLength() / (radius * 3.2))));
+  const m = new THREE.MeshBasicMaterial({ color: color.clone().multiplyScalar(1.35), transparent: true, opacity, depthWrite: false, toneMapped: false });
+  const mesh = new THREE.InstancedMesh(new THREE.SphereGeometry(radius * 0.8, 8, 6), m, n);
+  const at = new THREE.Matrix4();
+  for (let i = 0; i < n; i++) mesh.setMatrixAt(i, at.makeTranslation(curve.getPointAt(i / (n - 1))));
+  mesh.renderOrder = 3; mesh.userData.base = opacity; return mesh;
+}
+// a line as drawn: its inside part as a tube, and, when S.outside, each hidden end run as dots from the last inside point
+function lineMeshes(l, pts3, radius, opacity) {
+  const [a, b] = cut(l), out = [tube(pts3.slice(a, b), colOf(l), radius, opacity)];
+  if (S.outside) {
+    if (a > 0) out.push(dots(pts3.slice(0, a + 1), colOf(l), radius, opacity * 0.4));
+    if (b < pts3.length) out.push(dots(pts3.slice(b - 1), colOf(l), radius, opacity * 0.4));
+  }
+  return out;
+}
 function ring(r, y, color, opacity, seg = 96) {
   const pts = []; for (let i = 0; i <= seg; i++) { const a = i / seg * Math.PI * 2; pts.push(new THREE.Vector3(Math.cos(a) * r, y, Math.sin(a) * r)); }
   const l = new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), new THREE.LineBasicMaterial({ color, transparent: true, opacity, toneMapped: false }));
@@ -268,7 +302,7 @@ function buildArray() {
       for (const l of s.lines) {
         if (!shown(s, l)) continue;
         const pts = linePts(l).map(([x, y, z]) => [mm(s, x - cx) * S.ex, mm(s, z), mm(s, y - cy) * S.ex]);
-        g.add(tube(pts, colOf(l), 0.9, isOld(l) ? 0.45 : 1));
+        g.add(...lineMeshes(l, pts, 0.9, isOld(l) ? 0.45 : 1));
       }
       const tint = hasFinder(s) ? AMBER : ICE;
       g.add(ring(26, 0, tint, 0.28)); g.add(ring(26, H, tint, 0.12));
@@ -455,7 +489,7 @@ function toWorld(s, x, y, z) { const [ox, oy] = scanOrigin(s); return [mm(s, x -
 const scanLines = new THREE.Group();
 function buildScanLines(s) {
   scanLines.clear();
-  for (const l of s.lines) if (shown(s, l)) scanLines.add(tube(linePts(l).map(([x, y, z]) => toWorld(s, x, y, z)), colOf(l), 0.55, isOld(l) ? 0.5 : 1));
+  for (const l of s.lines) if (shown(s, l)) scanLines.add(...lineMeshes(l, linePts(l).map(([x, y, z]) => toWorld(s, x, y, z)), 0.55, isOld(l) ? 0.5 : 1));
   S.dirty = true;
 }
 function buildScan(s) {
@@ -563,12 +597,15 @@ function drawMarks(sx, s, z, a = 0) {
   for (const l of s.lines) {
     if (!shown(s, l)) continue;
     const p = lineAt(l, z); if (!p) continue;
+    const off = outsideAt(l, z); if (off && !S.outside) continue; // an end run outside the scroll: faded and dashed when shown
     const u = p[0] * kx - c, v = p[1] * ky - c; const x = c + u * ca - v * sa, y = c + u * sa + v * ca; const col = hexOf(l);
+    sx.globalAlpha = off ? 0.45 : 1; sx.setLineDash(off ? [3, 3] : []);
     sx.strokeStyle = col; sx.lineWidth = 1.6; sx.shadowColor = col; sx.shadowBlur = 10;
     sx.beginPath(); sx.arc(x, y, 9, 0, Math.PI * 2); sx.stroke();
     sx.beginPath(); sx.moveTo(x - 22, y); sx.lineTo(x - 12, y); sx.moveTo(x + 12, y); sx.lineTo(x + 22, y); sx.moveTo(x, y - 22); sx.lineTo(x, y - 12); sx.moveTo(x, y + 12); sx.lineTo(x, y + 22); sx.stroke();
     sx.fillStyle = col; sx.beginPath(); sx.arc(x, y, 1.6, 0, Math.PI * 2); sx.fill();
     sx.shadowBlur = 0; sx.font = '500 12px JetBrains Mono, monospace'; sx.fillText(S.data.sources[l.source].short, x + 14, y - 13);
+    sx.globalAlpha = 1; sx.setLineDash([]);
   }
 }
 async function drawSlice() {
@@ -694,6 +731,19 @@ function buildPick() {
   $('c-pick').innerHTML = `<optgroup label="Prize page">${on.map(opt).join('')}</optgroup>` + (off.length ? `<optgroup label="Not on the prize page">${off.map(opt).join('')}</optgroup>` : '');
 }
 $('c-pick').addEventListener('change', (e) => { const name = e.target.value; select(name, S.mode === 'lasagna' && name !== LASAGNA_SCROLL ? 'scan' : S.mode === 'array' ? 'scan' : S.mode); e.target.blur(); });
+// what the CT test says about a line's ends, for its scan card
+function outsideHTML(l) {
+  const o = l.outside;
+  if (!o) return `<div class="note">Ends not tested against the CT: ${l.outside_why || 'this copy of the line has no CT test'}. Drawn whole.</div>`;
+  const P = linePts(l), n = P.length;
+  const end = (k) => {
+    const e = o[k]; if (!e.hidden) return `${k} end: none`;
+    const z = k === 'low' ? [P[0][2], P[e.hidden - 1][2]] : [P[n - e.hidden][2], P[n - 1][2]];
+    return `${k} end: <b>${e.hidden} point${e.hidden === 1 ? '' : 's'}</b> hidden (z ${z[0]} to ${z[1]}), ${e.far} of them outside the scroll${e.max_mm != null ? `, up to ${fmt(e.max_mm, 1)} mm past it` : ''}${e.empty ? `, ${e.empty} in slices with no scroll at all` : ''}`;
+  };
+  if (!o.low.hidden && !o.high.hidden) return '<div class="note">CT: no end point lies outside the scroll.</div>';
+  return `<div class="note out">Outside the scroll in the CT: ${end('low')}; ${end('high')}. ${S.outside ? 'Shown faded and dotted.' : 'Hidden; tick <i>show ends outside the scroll</i> to see them.'} The file is unchanged.</div>`;
+}
 function buildCard(s) {
   $('c-pick').value = s.scroll;
   $('c-kick').textContent = `SCAN ${s.scan} · ${s.voxel_um} µm · ${prizeWords(s)}`;
@@ -710,12 +760,13 @@ function buildCard(s) {
     const row = (k, v) => `<tr><th>${k}</th><td>${v}</td></tr>`;
     const k = lineKey(s, l);
     return `<div class="ln${shown(s, l) ? '' : ' off'}"><div class="who"><span class="name"><label class="show" title="show or hide this line"><input type="checkbox" data-line="${k}"${OFF.line.has(k) ? '' : ' checked'}${OFF.src.has(l.source) ? ' disabled' : ''}></label><b style="color:${hexOf(l)}">${src.label}</b></span><span class="tags">${ver}${grade}</span></div>
-      <div class="meta">${t.points} points · z ${t.z_start} to ${t.z_end} (${fmt(t.height_mm, 0)} mm, ${fmt(t.coverage * 100, 0)}% of the scan) · strays up to ${fmt(t.stray_mm, 1)} mm from its mean · ${lic(l)}</div>${was}${note}
+      <div class="meta">${t.points} points · z ${t.z_start} to ${t.z_end} (${fmt(t.height_mm, 0)} mm, ${fmt(t.coverage * 100, 0)}% of the scan) · strays up to ${fmt(t.stray_mm, 1)} mm from its mean · ${lic(l)}</div>${was}${note}${outsideHTML(l)}
       <div class="links"><a href="${RAW(l)}" download>download</a><button class="lk" data-copy="${RAW(l)}" data-what="file URL copied">copy URL</button><button class="lk" data-copy="python -m kit centerline get ${s.scroll} --source ${l.source} --into YOUR_PROJECT_FOLDER" data-what="kit command copied: it writes YOUR_PROJECT_FOLDER/umbilicus.json" title="A command for the Scrolls kit that downloads this line, checks its sha256 and writes it as umbilicus.json where VC3D looks">copy kit command</button><a href="${DATASET}/blob/main/${l.file}" target="_blank" rel="noopener">view</a>${up}</div>
       <details class="full"><summary>every number</summary><table class="kv">
         ${row('credit', l.credit)}${row('kind', l.kind)}${row('points', t.points)}
         ${row('z range', `${t.z_start} to ${t.z_end} voxels, ${fmt(mm(s, t.z_start), 2)} to ${fmt(mm(s, t.z_end), 2)} mm`)}
         ${row('coverage', `${fmt(t.coverage * 100, 1)}% of ${s.shape[0]} slices`)}
+        ${row('inside the scroll', l.outside ? `z ${t.inside_z_start} to ${t.inside_z_end}; ${t.outside_low} low and ${t.outside_high} high end points hidden (CT level ${l.outside.level}, more than ${l.outside.far_mm} mm past the scroll's outline)` : `not tested: ${l.outside_why || 'no CT test'}`)}
         ${row('step', `median ${fmt(t.median_step_vox, 0)}, max ${fmt(t.max_step_vox, 0)} voxels (${fmt(mm(s, t.max_step_vox), 1)} mm)`)}
         ${row('mean x, y', `${fmt(t.mean_x, 0)}, ${fmt(t.mean_y, 0)} voxels, ${fmt(t.off_center_mm, 2)} mm off grid center`)}
         ${row('stray', `${fmt(t.stray_mm, 2)} mm`)}${row('tilt', `${fmt(t.tilt_deg, 2)}°`)}${row('wiggle', `${fmt(t.wiggle_mm, 3)} mm RMS`)}
@@ -733,12 +784,12 @@ function buildCard(s) {
     <div class="dim">every level ${s.pyramid[0].dtype === '|u1' ? 'uint8' : s.pyramid[0].dtype}, chunks ${s.pyramid[0].chunks.join(' x ')}, ${s.pyramid[0].compressor || 'no compressor'}</div>${streamable(s) ? '' : `<div class="note">This page reads one plane of an uncompressed chunk at a time, so it shows no CT for this scan; its line is drawn in the scan's frame. zarr, VC3D and the neuroglancer link read the compressed chunks.</div>`}`;
   $('c-vol').innerHTML = `CT <a href="${s.zarr}/.zattrs" target="_blank" rel="noopener">${s.volume}</a> · Vesuvius Challenge open data, CC BY-NC 4.0`;
 }
-function hereRows(s, z) { return s.lines.map((l) => { const p = lineAt(l, z); return p ? { source: l.source, x: p[0], y: p[1], z } : null; }).filter(Boolean); }
+function hereRows(s, z) { return s.lines.map((l) => { const p = lineAt(l, z); return p ? { source: l.source, x: p[0], y: p[1], z, outside: outsideAt(l, z) } : null; }).filter(Boolean); }
 function drawHere() {
   const s = S.sel; if (!s || S.z == null) return;
   const z = Math.round(S.z); const rows = hereRows(s, z);
   $('c-here').innerHTML = `<div class="mono small dim">AT z ${z} (${fmt(mm(s, z), 2)} mm), LEVEL-0 VOXELS</div>` + (rows.length
-    ? rows.map((r) => `<div class="hr"><span class="${r.source.startsWith('finder') ? 'amber' : 'ice'}">${S.data.sources[r.source].short}</span> <code>${fmt(r.x, 1)}, ${fmt(r.y, 1)}, ${z}</code><button class="lk" data-copy="${fmt(r.x, 1)}, ${fmt(r.y, 1)}, ${z}" data-what="x, y, z copied">copy</button></div>`).join('')
+    ? rows.map((r) => `<div class="hr"><span class="${r.source.startsWith('finder') ? 'amber' : 'ice'}">${S.data.sources[r.source].short}</span> <code>${fmt(r.x, 1)}, ${fmt(r.y, 1)}, ${z}</code><button class="lk" data-copy="${fmt(r.x, 1)}, ${fmt(r.y, 1)}, ${z}" data-what="x, y, z copied">copy</button>${r.outside ? ' <span class="dim">outside the scroll</span>' : ''}</div>`).join('')
     : '<div class="dim small">no line reaches this height</div>');
 }
 function setMode(m) {
@@ -777,6 +828,8 @@ $('b-connect').addEventListener('click', () => { markSeen(); openGuide('connect'
 let exQueued = false;
 $('ex').addEventListener('input', (e) => { S.ex = +e.target.value; $('ex-v').textContent = S.ex; if (!exQueued) { exQueued = true; requestAnimationFrame(() => { exQueued = false; buildArray(); }); } syncURLSoon(); });
 $('t-ct').addEventListener('change', (e) => { S.ct = e.target.checked; applyVolUniforms(); });
+function setOutside(on) { S.outside = on === undefined ? !S.outside : !!on; $('t-outside').checked = S.outside; linesChanged(); return { outside: S.outside }; }
+$('t-outside').addEventListener('change', (e) => setOutside(e.target.checked));
 $('t-ct2').addEventListener('change', (e) => { S.ct2 = e.target.checked; applyVolUniforms(); });
 $('render').addEventListener('change', (e) => { S.render = e.target.value; applyVolUniforms(); });
 $('gain').addEventListener('input', (e) => { S.gain = +e.target.value; applyVolUniforms(); });
@@ -851,7 +904,7 @@ $('about').querySelector('.tabs').addEventListener('click', (e) => { const b = e
 
 // data sheet
 const COLS = {
-  lines: [['scroll', 'scroll'], ['maker', 'maker'], ['grade', 'grade'], ['ct_end_low', 'low end'], ['ct_end_high', 'high end'], ['points', 'pts'], ['z_start', 'z start'], ['z_end', 'z end'], ['height_mm', 'height mm'],
+  lines: [['scroll', 'scroll'], ['maker', 'maker'], ['grade', 'grade'], ['ct_end_low', 'low end'], ['ct_end_high', 'high end'], ['outside_low', 'low out'], ['outside_high', 'high out'], ['points', 'pts'], ['z_start', 'z start'], ['z_end', 'z end'], ['height_mm', 'height mm'],
     ['coverage', 'cover'], ['median_step_vox', 'step'], ['max_step_vox', 'max step'], ['off_center_mm', 'off ctr mm'], ['stray_mm', 'stray mm'], ['tilt_deg', 'tilt °'],
     ['wiggle_mm', 'wiggle mm'], ['path_over_height', 'path/ht'], ['stamped', 'stamp'], ['usable_ends', 'usable ends'], ['superseded', 'superseded'], ['license', 'license'], ['sha256', 'sha256']],
   scans: [['scroll', 'scroll'], ['scan', 'scan'], ['voxel_um', 'µm'], ['slices', 'z'], ['height_px', 'y'], ['width_px', 'x'], ['height_mm', 'height mm'], ['width_mm', 'width mm'],
@@ -972,18 +1025,20 @@ function syncURL() {
   if (S.mode !== 'array') { q.set('mode', S.mode); q.set('scan', S.sel.scroll); q.set('z', Math.round(S.z)); }
   if (S.ex !== 3) q.set('ex', S.ex);
   const off = [...OFF.src, ...OFF.line]; if (off.length) q.set('off', off.join(','));
+  if (S.outside) q.set('outside', '1');
   history.replaceState(null, '', q.toString() ? `?${q}` : location.pathname);
 }
 window.umbilici = {
   version: () => VERSION,
   show: (kind, name, on) => showLines(kind, name, on),
+  outside: (on) => setOutside(on),
   scans: () => S.data.scrolls.map((s) => ({ scroll: s.scroll, scan: s.scan, volume: s.volume, voxel_um: s.voxel_um, shape_zyx: s.shape, grand_prize: s.grand_prize, first_letters: s.first_letters, sources: s.lines.map((l) => l.source), grade: gradeOf(s) })),
   lines: (name) => { const s = S.data.scrolls.find((q) => q.scroll === name); return s ? s.lines.map((l) => ({ ...l, points: linePts(l) })) : null; },
   at: (name, z) => { const s = S.data.scrolls.find((q) => q.scroll === name); return s ? s.lines.map((l) => ({ source: l.source, xy: lineAt(l, z) })) : null; },
   select: (name, mode = 'scan') => select(name, mode),
   mode: (m) => setMode(m),
   setZ: (z) => setZ(z, true),
-  state: () => ({ mode: S.mode, scan: S.sel?.scroll ?? null, z: S.z, ex: S.ex, ct: { loaded: vol?.loaded ?? 0, total: vol?.total ?? 0 }, render: S.render, pixel_ratio: renderer.getPixelRatio(), auto_level: autoLevel }),
+  state: () => ({ mode: S.mode, scan: S.sel?.scroll ?? null, z: S.z, ex: S.ex, outside: S.outside, ct: { loaded: vol?.loaded ?? 0, total: vol?.total ?? 0 }, render: S.render, pixel_ratio: renderer.getPixelRatio(), auto_level: autoLevel }),
   image: () => { composer.render(); return renderer.domElement.toDataURL('image/png'); },
   slice: () => sc.toDataURL('image/png'),
   stats: (name) => { const s = S.data.scrolls.find((q) => q.scroll === name); return s ? { lines: s.lines.map((l) => ({ source: l.source, file: RAW(l), ...statsOf(s, l) })), pairs: pairsOf(s) } : null; },
@@ -1081,7 +1136,10 @@ async function liveLine(f, had, at) {
     usable = use && typeof use === 'object' ? ['low', 'high'].filter((k) => use[k]) : null;
   }
   const l = Object.fromEntries(LINE_KEYS.map((k) => [k, f[k] ?? (k === 'note' ? '' : f[k])]));
-  return { ...l, grade, ends, untested: !!f.ct_untested_scan, usable, stamp, points, superseded_by: f.superseded_by ?? null };
+  // the CT test of a line's ends runs when the page is built; a file newer than the build is drawn whole
+  const outside = had ? had.outside ?? null : null;
+  const outside_why = had ? had.outside_why ?? (outside ? null : 'this copy of the page was built before the CT test') : 'the file is newer than this page\'s build, which ran the CT test';
+  return { ...l, grade, ends, untested: !!f.ct_untested_scan, usable, stamp, points, superseded_by: f.superseded_by ?? null, outside, outside_why };
 }
 
 // ---------- boot ----------
@@ -1114,6 +1172,7 @@ async function liveLine(f, had, at) {
   const q = new URLSearchParams(location.search);
   if (q.get('ex')) { S.ex = Math.max(1, Math.min(30, +q.get('ex'))); $('ex').value = S.ex; $('ex-v').textContent = S.ex; }
   for (const k of (q.get('off') || '').split(',').filter(Boolean)) (k.includes(':') ? OFF.line : OFF.src).add(k);
+  S.outside = q.get('outside') === '1'; $('t-outside').checked = S.outside;
   buildLegend();
   applySettings(false); buildRail(); buildPick(); buildArray();
   $('boot').remove();

index.html

@@ -191,6 +191,10 @@
     <label><input type="checkbox" id="t-ct2"> live CT</label>
     <span class="dim">villa lasagna predict3d, model output</span>
   </div>
+  <div class="ctl" id="ctl-out">
+    <label title="Line ends whose control points lie outside the scroll in the CT are hidden by default; tick to draw them faded and dotted. The files are unchanged."><input type="checkbox" id="t-outside"> show ends outside the scroll</label>
+    <span class="dim">hidden by default: points the CT puts outside the scroll</span>
+  </div>
   <div class="data" id="data-links">
     <span class="dim">DATA</span>
     <a href="https://huggingface.co/datasets/SharpMeow/vesuvius-center-lines" target="_blank" rel="noopener">vesuvius-center-lines</a>
@@ -230,6 +234,7 @@
     </ol>
     <h3>Colors and grades</h3>
     <p>Each maker has its own color: <span style="color:#89f0ff">Vesuvius Challenge</span>, <span style="color:#ff7eb6">Alexey Drobkov</span>, <span style="color:#b69cff">Abd Elilah</span>, <span style="color:#f2f5f7">Iyan Dopico</span>, all made by people. <span class="amber">Amber</span> lines are model output from Chase Hendrick's Center Finder, made for the nine 1.2 m prize scans nobody had traced and for nine scans not on the prize page (seven since 1.8.0, PHerc1667 and PHerc0172 since 1.10.0); no person has checked their ends on CT. <span style="color:#c6f36b">Lime</span> lines are the Center Finder lines after the coverage polish (Scrolls <code>kit axispolish</code>), marked NEWEST: the twelve on 1.2 m scans since 1.9.0, and since 1.10.0 three on 2.4 µm scans (PHerc1299, PHercMAN5, PHercMANB), where the polish ran on level 2 and is untested. The originals stay on the page, faded and marked SUPERSEDED, so a later polish can replace a line without losing the one before: on all ten scans of a frozen test the 1.2 m polished lines sat deeper in villa's traced sheet than the line before the polish, and kit takes them first, but they are jumpier and their ends were not re-checked, so the grades below are the original line's. An automatic CT end check graded each end of each Center Finder line (release v2.2, 23 Oct 2026). It compares the end with the scroll center the CT itself shows there, estimated from the scan alone (sheet orientations in three slices, never the line), 0, 1 and 2 mm inside the end, with a 4 mm bar. A planted fake tilt is caught at all 18 ends. 14 of the 18 ends pass and 4 are flagged. The same check on the seven newer lines (24 Oct 2026) passes all 14 ends, 2 after a CT recap, but four of those scans are 2.4 µm scans the check was never tuned on, so their grades are marked untested and no end there is usable. Of the two lines added in release v2.6 (25 Oct 2026), PHerc1667's (a 2.4 µm scan) passes both ends, untested, and PHerc0172's (a 7.91 µm scan) is flagged at both ends, so treat its ends with care.</p>
+    <p>Since 1.13.0 a line is drawn from its first to its last point inside the scroll. Where its end runs outside the scroll in the CT (more than 4 mm outside the scroll's outline in that slice, or in a slice with no scroll at all), those end points are hidden, the scan card says how many at each end, and <b>show ends outside the scroll</b> at bottom left draws them faded and dotted. The files are unchanged; For researchers has the rule.</p>
     <p>Scan names and grades use a traffic light: <span class="grade lt-good">green</span> when both ends passed, <span class="grade lt-caution">yellow</span> for caution (a pass that rests on a CT recap, an end the check could not judge, or an untested scan), <span class="grade lt-bad">red</span> when an end was flagged.</p>
     <ul>
       <li><span class="grade PASS">PASS</span> both ends passed.</li>
@@ -267,25 +272,28 @@
       <tr><th>z start, end</th><td>first and last control point height, level-0 voxels</td></tr>
       <tr><th>height</th><td>z end minus z start, in mm</td></tr>
       <tr><th>coverage</th><td>height as a share of the scan's z extent</td></tr>
+      <tr><th>low out, high out</th><td>control points hidden at the low and the high end because the line runs outside the scroll there, from the line's end to its innermost point that lies outside (blank: not tested). A point lies outside when, in its own axial slice of the scan's masked CT at level 5 (32 times coarser than level 0; voxels outside the scroll are zero), it sits more than 4 mm outside the outline (convex hull) of the scroll, or the slice holds no scroll at all. Only the outer quarter of a line's points at each end can be hidden; the middle half never is. The test runs when the page is built (<code>tools/build_lines.py</code>) and passes two checks per line, or the line is drawn whole and marked untested: its middle point must be inside, and the slice corner farthest from it outside. The files, their points and sha256 stay the dataset's.</td></tr>
+      <tr><th>inside part</th><td>the line without its hidden end runs; every statistic below from mean x, y on uses it, so an end that runs off past the scroll does not count as drift</td></tr>
       <tr><th>median, max step</th><td>median and largest z gap between neighboring control points, voxels; a big max step is a stretch the line interpolates across</td></tr>
-      <tr><th>mean x, y</th><td>average position of the control points, voxels</td></tr>
+      <tr><th>mean x, y</th><td>average position of the inside part's control points, voxels</td></tr>
       <tr><th>off center</th><td>distance from that mean to the middle of the scan grid, mm</td></tr>
-      <tr><th>stray</th><td>largest distance of any control point from the mean, mm</td></tr>
-      <tr><th>tilt</th><td>angle from vertical of a least-squares straight line through the points, degrees</td></tr>
-      <tr><th>wiggle</th><td>RMS distance of the points from that straight line, mm</td></tr>
-      <tr><th>path</th><td>length along the line, mm; path over height above 1 means it wanders</td></tr>
+      <tr><th>stray</th><td>largest distance of any control point of the inside part from the mean, mm</td></tr>
+      <tr><th>tilt</th><td>angle from vertical of a least-squares straight line through the inside part's points, degrees</td></tr>
+      <tr><th>wiggle</th><td>RMS distance of the inside part's points from that straight line, mm</td></tr>
+      <tr><th>path</th><td>length along the inside part, mm; path over its height above 1 means it wanders</td></tr>
       <tr><th>stamped</th><td>whether the file's metadata names the voxel size and grid VC3D reads</td></tr>
-      <tr><th>pair gap</th><td>for two lines on one scan: median, min and max horizontal distance between them every 64 voxels where both exist, mm</td></tr>
+      <tr><th>pair gap</th><td>for two lines on one scan: median, min and max horizontal distance between them every 64 voxels where both inside parts exist, mm</td></tr>
     </table>
     <h3>How to read them</h3>
     <ul>
       <li>Where a scan has two lines, the gap shows how far two independent tracings disagree. It does not say which one is closer to the true center.</li>
+      <li>A hidden end (low out, high out) says only that those control points lie outside the scroll in the published mask at level 5; it does not say where the center is there. Some lines start their search over the mount or cap slices, where no scroll is, and run off sideways; people's lines can also overshoot a tapering end by a few millimeters. The raw files keep every point: the download links, <code>lines()</code> and the copied URLs give the dataset's file unchanged.</li>
       <li>A line's own spread (stray, wiggle) is not an error. Scrolls are crushed and bent, so the center can wander; a straight line through a bent scroll would be wrong.</li>
       <li>The end check grade applies only to Center Finder lines. Person-made lines have no grade here; that is not a pass.</li>
       <li>The radiograph is a view, not a measurement: it uses scan level 5 (32 times coarser than level 0), with a z stride, and a display curve.</li>
     </ul>
     <h3>For agents</h3>
-    <p>Everything is scriptable through <code>window.umbilici</code> in a browser or headless Chromium: <code>scans()</code>, <code>lines(scroll)</code>, <code>at(scroll, z)</code>, <code>stats(scroll)</code>, <code>table()</code>, <code>csv('lines' | 'scans' | 'points')</code>, <code>select(scroll, mode)</code>, <code>mode(m)</code>, <code>setZ(z)</code>, <code>state()</code>, <code>image()</code>, <code>slice()</code>. URL parameters: <code>?mode=scan&amp;scan=PHerc0191&amp;z=10400</code>, <code>?guide=vc3d</code>, <code>?data=lines</code>.</p>
+    <p>Everything is scriptable through <code>window.umbilici</code> in a browser or headless Chromium: <code>scans()</code>, <code>lines(scroll)</code>, <code>at(scroll, z)</code>, <code>stats(scroll)</code>, <code>table()</code>, <code>csv('lines' | 'scans' | 'points')</code>, <code>select(scroll, mode)</code>, <code>mode(m)</code>, <code>outside(on)</code>, <code>setZ(z)</code>, <code>state()</code>, <code>image()</code>, <code>slice()</code>. URL parameters: <code>?mode=scan&amp;scan=PHerc0191&amp;z=10400</code>, <code>&amp;outside=1</code> (show the ends outside the scroll), <code>?guide=vc3d</code>, <code>?data=lines</code>.</p>
   </section>
 
   <section data-pane="data" hidden>

tools/build_lines.py

@@ -1,12 +1,19 @@
 """Build data/lines.json for the Space from the public center-lines dataset folder.
 
-Usage: python3 tools/build_lines.py DATASET_DIR OUT.json
+Usage: python3 tools/build_lines.py DATASET_DIR OUT.json [--no-ct]
 DATASET_DIR is a copy of SharpMeow/vesuvius-center-lines (index.json plus lines/). Only the control points, the
 public fields of index.json (with each Center Finder line's per-end CT end check, ct_ends, ct_untested_scan
 as untested, and superseded_by, the newer version of a line), each file's frame
 stamp (voxel size, volume name and grid) and its usable_ends are copied; the rest of each file's metadata is not. Scan shapes are read from the
 Vesuvius Challenge open-data bucket.
+
+Since 1.13.0 each line also gets "outside": how many control points at each end lie outside the scroll in the CT,
+by the test in outside_ends() of the Center Finder's scripts/umbilicus/lineends.py (the one copy of the
+test), read from level 5 of the scan's masked OME-Zarr in the same bucket (about 16 KB per range read). The page hides
+those end runs by default. The test never changes a line's points, file or sha256. A line it cannot test gets
+"outside": null and the reason in "outside_why"; --no-ct skips the test.
 """
+import importlib.util
 import json
 import re
 import sys
@@ -33,8 +40,34 @@ def get_json(url):
         return json.load(r)
 
 
-def main(src, out):
+# ---------- the CT test for line ends ----------
+# The test lives in the Center Finder (scripts/umbilicus/lineends.py, whose trim command cuts the same end runs from a
+# line file), so the page hides exactly what a trim would cut. It is loaded from its file, not through sys.path, so
+# the Center Finder's other generic module names never shadow anything: tools/lineends.py beside this file when there
+# is one (the Space ships a copy there), else the repository's.
+def _load_lineends():
+    name = "umbilicus_lineends"
+    if name not in sys.modules:
+        here = Path(__file__).resolve().parent
+        path = here / "lineends.py"
+        if not path.exists():
+            path = here.parents[2] / "scripts" / "umbilicus" / "lineends.py"
+        spec = importlib.util.spec_from_file_location(name, path)
+        mod = importlib.util.module_from_spec(spec)
+        sys.modules[name] = mod
+        spec.loader.exec_module(mod)
+    return sys.modules[name]
+
+
+_le = _load_lineends()
+CORNER, END_SHARE, FAR_MM, LEVEL = _le.CORNER, _le.END_SHARE, _le.FAR_MM, _le.LEVEL
+Slices, Volume, hull, hull_distance = _le.Slices, _le.Volume, _le.hull, _le.hull_distance
+check_line, open_slices, outside_ends = _le.check_line, _le.open_slices, _le.outside_ends
+
+
+def main(src, out, *flags):
     src = Path(src)
+    ct = "--no-ct" not in flags
     ix = json.loads((src / "index.json").read_text())
     vol_of = {}
     for f in ix["files"]:
@@ -55,6 +88,15 @@ def main(src, out):
                             "compressor": comp.get("id") if comp else None})
         zarr0 = pyramid[0]
         um = float(re.search(r"-([0-9.]+)um-", vol).group(1))
+        if not ct:
+            scan = (None, "the page was built without the CT test (--no-ct)")
+        else:
+            try:
+                scan = open_slices(base, pyramid, um)
+            except ImportError:
+                scan = (None, "this scan's chunks are compressed and the build had no numcodecs to read them")
+            except Exception as e:
+                scan = (None, f"the scan's CT could not be opened ({type(e).__name__})")
         lines = []
         for f in ix["files"]:
             if f["scroll"] != name:
@@ -66,6 +108,7 @@ def main(src, out):
             flat = []
             for p in sorted(pts, key=lambda p: p["z"]):
                 flat += [round(p["x"]), round(p["y"]), round(p["z"])]
+            outside, why = check_line([tuple(flat[i:i + 3]) for i in range(0, len(flat), 3)], scan)
             # v2.2 and later grade each end on CT (index ct_ends); a line is FLAG if either end is. Older releases
             # carried one grade in the note.
             ends = {k: {q: e[q] for q in ("grade", "recapped", "D_mm") if q in e} for k, e in (f.get("ct_ends") or {}).items()}
@@ -87,6 +130,9 @@ def main(src, out):
                 "stamp": stamp, "points": flat,
                 # v2.5 and later: the file of the newer version of this line, if there is one (the page fades it)
                 "superseded_by": f.get("superseded_by"),
+                # 1.13.0: the end runs that lie outside the scroll in the CT (outside_ends); null with the reason
+                # when the line could not be tested
+                "outside": outside, "outside_why": why,
             })
         lines.sort(key=lambda l: list(SOURCES).index(l["source"]))
         scrolls.append({
@@ -94,7 +140,9 @@ def main(src, out):
             "shape": zarr0["shape"], "levels": len(levels), "pyramid": pyramid,
             "first_letters": s["first_letters"], "grand_prize": s["grand_prize"], "lines": lines,
         })
-        print(name, vol, zarr0["shape"], len(levels), [l["source"] for l in lines])
+        if scan[0] is not None:
+            scan[0].vol.cache.clear()
+        print(name, vol, zarr0["shape"], len(levels), [(l["source"], l["outside"] and (l["outside"]["low"]["hidden"], l["outside"]["high"]["hidden"])) for l in lines], flush=True)
     Path(out).write_text(json.dumps({"made": ix["made"], "frame": ix["frame"], "sources": SOURCES,
                                      "scrolls": scrolls}, separators=(",", ":")))
 

tools/lineends.py

@@ -0,0 +1,480 @@
+"""Center line ends against the scroll's outline in the masked CT: find the end runs that leave the papyrus and trim them.
+
+    python scripts/umbilicus/lineends.py check FILE [FILE ...] [--volume SCROLL/volumes/VOL] [--report R.json] [--json]
+    python scripts/umbilicus/lineends.py trim FILE [FILE ...] --out DIR [--volume SCROLL/volumes/VOL] [--report R.json]
+
+FILE is an umbilicus file in the published format (control_points x, y, z in level-0 voxels of its scan). Its scan
+is metadata.source_volume ("SCROLL/volumes/VOL.zarr", as the Center Finder writes it) unless --volume names it. The
+scan must be a masked OME-Zarr in the Vesuvius Challenge open-data bucket: voxels outside the scroll are zero.
+
+The test (outside_ends) reads level 5 of the scan (32 times down) by HTTP range requests. A control point lies
+outside the scroll when, in its own axial slice, no nonzero voxel lies within FAR_MM (4 mm) of it and it sits more
+than FAR_MM outside the convex hull of the slice's nonzero voxels, or the slice has no nonzero voxel at all. Only the
+outer END_SHARE (a quarter) of a line's points at each end can be cut: at each end the cut reaches from the end to
+the innermost outside point in that zone, so an excursion that comes back inside is cut with it, and the middle half
+of a line is never cut. Two checks make the test able to fail; a line that fails either is left whole and reported
+untested: its middle point must be inside (else the frame or the mask is wrong) and the slice corner farthest from
+it must be outside (else the scan's air is not zero).
+
+check   prints, per file, the points and the height each end would lose; writes nothing but the --report JSON.
+trim    also writes DIR/<file name> for every file: the same JSON with the cut end points removed and
+        metadata.papyrus_trim saying what was cut and by which rule. A file with nothing to cut, or untested, is
+        written unchanged except for that block. Points keep their order and every key they had.
+
+The same test hides the end runs in the Umbilici Space (spaces/umbilici/tools/build_lines.py imports it from here),
+so a trimmed line shows whole there. It measures the line against the published mask, not where the center is: a
+trimmed line is still model output. Standard library only (numcodecs to read a scan stored with compressed chunks).
+"""
+import argparse
+import json
+import math
+import re
+import sys
+import threading
+import urllib.error
+import urllib.request
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+S3 = "https://vesuvius-challenge-open-data.s3.us-east-1.amazonaws.com"
+VERSION = "1.0.0"
+
+# A control point lies outside the scroll when, in its own axial slice of the scan's masked CT (voxels outside the
+# scroll are zero), it sits more than FAR_MM outside the convex hull of the nonzero voxels, or the slice has no
+# nonzero voxel at all. The hull, not the nearest voxel, is the bar because a scroll's ends taper and break into
+# fragments: a point between two fragments is near no voxel but still inside the scroll's outline (a survey of all 57
+# shown lines on 4 Nov 2026 found such gaps of up to 10.9 mm on people's lines, all inside the hull).
+# Only the outer END_SHARE of a line's points at each end can be hidden: at each end the hidden run reaches from the
+# end to the innermost outside point in that zone, so an excursion a few points in from an end that comes back inside
+# is hidden with it. The middle half of a line is never hidden. Two built-in checks make the test able to fail, and
+# a line whose checks fail is drawn whole and marked untested: the line's middle point must be inside (else the frame
+# or the mask is wrong), and the slice corner farthest from it must be outside (else the scan's air is not zero).
+LEVEL = 5          # 32 times down: 0.28 to 0.30 mm voxels on the 1.2 m scans, 77 um on the 2.4 um scans
+FAR_MM = 4.0
+END_SHARE = 0.25
+CORNER = 0.02      # the corner control sits this share of the width in from each edge
+RULE = ("end points cut where they leave the scroll in the masked CT (level 5): no nonzero voxel within 4 mm in "
+        "the point's slice and more than 4 mm outside the convex hull of the slice's nonzero voxels, or a slice "
+        "with no scroll; only the outer quarter of points at each end can be cut, from the end to the innermost "
+        "outside point; the middle point must be inside and the slice corner farthest from it outside, else nothing "
+        "is cut")
+
+
+def get_json(url):
+    with urllib.request.urlopen(url, timeout=30) as r:
+        return json.load(r)
+
+
+class Volume:
+    """One pyramid level of a uint8 OME-Zarr: shape (z, y, x), read one axial plane of a chunk at a time.
+
+    tile(zi, cy, cx) returns that plane of chunk (zi // cz, cy, cx) as bytes (cy_size * cx_size), or None for a
+    missing chunk, which reads as zeros (the masked scans' fill value)."""
+
+    def __init__(self, shape, chunks, tile, threads=1):
+        self.shape, self.chunks, self._tile, self.threads = list(shape), list(chunks), tile, threads
+        self.cache, self.lock = {}, threading.Lock()
+
+    def tiles(self, zi, y0, y1, x0, x1):
+        _, ch_y, ch_x = self.chunks
+        if not 0 <= zi < self.shape[0]:
+            return []
+        y0, y1 = max(0, y0), min(self.shape[1], y1)
+        x0, x1 = max(0, x0), min(self.shape[2], x1)
+        if y0 >= y1 or x0 >= x1:
+            return []
+        return [(zi, a, b) for a in range(y0 // ch_y, (y1 - 1) // ch_y + 1) for b in range(x0 // ch_x, (x1 - 1) // ch_x + 1)]
+
+    def prefetch(self, keys):
+        todo = sorted(set(keys) - set(self.cache))
+        if not todo:
+            return
+        def one(k):
+            v = self._tile(*k)
+            with self.lock:
+                self.cache[k] = v
+        if self.threads > 1 and len(todo) > 1:
+            with ThreadPoolExecutor(self.threads) as ex:
+                list(ex.map(one, todo))
+        else:
+            for k in todo:
+                one(k)
+
+    def rows(self, zi, y0, y1, x0, x1):
+        """Rows y0..y1-1 of slice zi between x0 and x1, as bytes, zero outside the volume."""
+        _, ch_y, ch_x = self.chunks
+        self.prefetch(self.tiles(zi, y0, y1, x0, x1))
+        out = []
+        for y in range(y0, y1):
+            if not (0 <= zi < self.shape[0] and 0 <= y < self.shape[1]):
+                out.append(bytes(x1 - x0))
+                continue
+            row = bytearray(x1 - x0)
+            xa, xb = max(0, x0), min(self.shape[2], x1)
+            for b in (range(xa // ch_x, (xb - 1) // ch_x + 1) if xa < xb else ()):
+                t = self.cache.get((zi, y // ch_y, b))
+                if t is None:
+                    continue
+                tx0 = b * ch_x
+                a0, a1 = max(x0, tx0), min(x1, tx0 + ch_x, self.shape[2])
+                r = (y % ch_y) * ch_x
+                row[a0 - x0:a1 - x0] = t[r + a0 - tx0:r + a1 - tx0]
+            out.append(bytes(row))
+        return out
+
+
+def _cross(o, a, b):
+    return (a[0] - o[0]) * (b[1] - o[1]) - (a[1] - o[1]) * (b[0] - o[0])
+
+
+def hull(pts):
+    """Convex hull, counterclockwise, of (x, y) points (Andrew's monotone chain)."""
+    pts = sorted(set(pts))
+    if len(pts) <= 2:
+        return pts
+    lo, hi = [], []
+    for p in pts:
+        while len(lo) >= 2 and _cross(lo[-2], lo[-1], p) <= 0:
+            lo.pop()
+        lo.append(p)
+    for p in reversed(pts):
+        while len(hi) >= 2 and _cross(hi[-2], hi[-1], p) <= 0:
+            hi.pop()
+        hi.append(p)
+    return lo[:-1] + hi[:-1]
+
+
+def _seg_dist(p, a, b):
+    dx, dy = b[0] - a[0], b[1] - a[1]
+    L = dx * dx + dy * dy
+    t = 0.0 if L == 0 else max(0.0, min(1.0, ((p[0] - a[0]) * dx + (p[1] - a[1]) * dy) / L))
+    return math.hypot(p[0] - a[0] - t * dx, p[1] - a[1] - t * dy)
+
+
+def hull_distance(h, p):
+    """Distance from p to the convex polygon h (0 inside or on it)."""
+    if len(h) >= 3 and all(_cross(h[i], h[(i + 1) % len(h)], p) >= 0 for i in range(len(h))):
+        return 0.0
+    if len(h) == 1:
+        return math.hypot(p[0] - h[0][0], p[1] - h[0][1])
+    return min(_seg_dist(p, h[i], h[(i + 1) % len(h)]) for i in range(len(h)))
+
+
+class Slices:
+    """The CT test on one scan level: vol (a Volume), f (level-0 voxels per voxel there), vox_mm (its voxel size)."""
+
+    def __init__(self, vol, f, vox_mm):
+        self.vol, self.f, self.vox_mm, self.hulls = vol, f, vox_mm, {}
+
+    def _window(self, x, y, z, r):
+        zi, yc, xc = int(z) // self.f, y / self.f, x / self.f
+        return zi, yc, xc, (int(math.floor(yc - r)) - 1, int(math.ceil(yc + r)) + 1, int(math.floor(xc - r)) - 1, int(math.ceil(xc + r)) + 1)
+
+    def near_tiles(self, x, y, z):
+        zi, _, _, (y0, y1, x0, x1) = self._window(x, y, z, FAR_MM / self.vox_mm)
+        return self.vol.tiles(zi, y0, y1, x0, x1)
+
+    def slice_tiles(self, z):
+        return self.vol.tiles(int(z) // self.f, 0, self.vol.shape[1], 0, self.vol.shape[2])
+
+    def material_near(self, x, y, z):
+        """True when a nonzero voxel center lies within FAR_MM of the point in its slice."""
+        r = FAR_MM / self.vox_mm
+        zi, yc, xc, (y0, y1, x0, x1) = self._window(x, y, z, r)
+        for j, row in zip(range(y0, y1), self.vol.rows(zi, y0, y1, x0, x1)):
+            dy = j + 0.5 - yc
+            if abs(dy) > r:
+                continue
+            half = math.sqrt(r * r - dy * dy)
+            a, b = math.ceil(xc - half - 0.5), math.floor(xc + half - 0.5)
+            seg = row[max(0, a - x0):max(0, b - x0 + 1)]
+            if seg.count(0) < len(seg):
+                return True
+        return False
+
+    def hull_of(self, zi):
+        """Convex hull of slice zi's nonzero voxel centers, in voxels of this level (None for an empty slice)."""
+        if zi not in self.hulls:
+            pts = []
+            Y, X = self.vol.shape[1], self.vol.shape[2]
+            for y, row in enumerate(self.vol.rows(zi, 0, Y, 0, X)):
+                s = row.lstrip(b"\0")
+                if s:
+                    pts += [(X - len(s) + 0.5, y + 0.5), (len(row.rstrip(b"\0")) - 0.5, y + 0.5)]
+            self.hulls[zi] = hull(pts) if pts else None
+        return self.hulls[zi]
+
+    def outside(self, x, y, z):
+        """None when the point is inside the scroll; else the distance past the slice's hull, mm, or 'empty'."""
+        if self.material_near(x, y, z):
+            return None
+        h = self.hull_of(int(z) // self.f)
+        if h is None:
+            return "empty"
+        d = hull_distance(h, (x / self.f, y / self.f)) * self.vox_mm
+        return d if d > FAR_MM else None
+
+
+def outside_ends(points, sl):
+    """The CT test on one line: points as (x, y, z) level-0 voxels sorted by z, sl a Slices.
+
+    Returns (result, None) or (None, why). result holds, per end, the hidden run's length (hidden), how many of its
+    points lie outside (far), how many of those are in a slice with no scroll at all (empty) and the largest distance
+    past the hull of the others (max_mm)."""
+    n = len(points)
+    if n < 3:
+        return None, "the line has fewer than three points"
+    zone = max(1, int(n * END_SHARE))
+    idx = sorted(set(range(zone)) | set(range(n - zone, n)) | {n // 2})
+    vol = sl.vol
+    vol.prefetch([k for i in idx for k in sl.near_tiles(*points[i])])
+    mid = points[n // 2]
+    if sl.outside(*mid) is not None:
+        return None, "its middle point lies outside the scroll in the CT, so the frame or the mask is in doubt"
+    X, Y = vol.shape[2] * sl.f, vol.shape[1] * sl.f
+    cx = [CORNER * X, (1 - CORNER) * X]
+    cy = [CORNER * Y, (1 - CORNER) * Y]
+    corner = max(((a, b) for a in cx for b in cy), key=lambda c: math.hypot(c[0] - mid[0], c[1] - mid[1]))
+    vol.prefetch(sl.slice_tiles(mid[2]))
+    if sl.outside(corner[0], corner[1], mid[2]) is None:
+        return None, "a corner of the scan's slice reads as inside the scroll, so the scan's air is not zero"
+    far = {}
+    for i in idx:
+        if i == n // 2:
+            continue
+        if not sl.material_near(*points[i]):
+            vol.prefetch(sl.slice_tiles(points[i][2]))
+        far[i] = sl.outside(*points[i])
+    res = {}
+    for end, order in (("low", range(zone)), ("high", range(n - 1, n - 1 - zone, -1))):
+        hit = [k for k, i in enumerate(order) if far[i] is not None]
+        hidden = hit[-1] + 1 if hit else 0
+        vals = [far[i] for k, i in enumerate(order) if k < hidden and far[i] is not None]
+        mms = [v for v in vals if v != "empty"]
+        res[end] = {"hidden": hidden, "far": len(vals), "empty": sum(1 for v in vals if v == "empty"),
+                    "max_mm": round(max(mms), 1) if mms else None}
+    return dict(res, level=None, far_mm=FAR_MM, end_share=END_SHARE), None
+
+
+def _range_tile(url, chunks, compressor):
+    """tile() for a level in the bucket: one range read per plane of an uncompressed chunk; a compressed chunk is read
+    whole once and decoded with numcodecs."""
+    ch_z, ch_y, ch_x = chunks
+    whole, lock = {}, threading.Lock()
+    codec = None
+    if compressor:
+        import numcodecs  # ImportError: the caller marks the line untested
+        codec = numcodecs.get_codec(compressor)
+
+    def get(u, rng=None):
+        req = urllib.request.Request(u, headers={"Range": "bytes=%d-%d" % rng} if rng else {})
+        for attempt in range(4):
+            try:
+                with urllib.request.urlopen(req, timeout=60) as r:
+                    return r.read()
+            except urllib.error.HTTPError as e:
+                if e.code in (403, 404):
+                    return None
+                err = e
+            except OSError as e:
+                err = e
+        raise err
+
+    def tile(zi, cy, cx):
+        key = f"{url}/{zi // ch_z}/{cy}/{cx}"
+        plane = ch_y * ch_x
+        if codec is None:
+            off = (zi % ch_z) * plane
+            return get(key, (off, off + plane - 1))
+        with lock:
+            have = key in whole
+        if not have:
+            raw = get(key)
+            data = bytes(codec.decode(raw)) if raw is not None else None
+            with lock:
+                whole[key] = data
+        data = whole[key]
+        return None if data is None else data[(zi % ch_z) * plane:(zi % ch_z + 1) * plane]
+    return tile
+
+
+def open_slices(base, pyramid, um):
+    """The CT test's view of one scan: the Slices for level LEVEL (or the coarsest level there is) of its zarr."""
+    k = min(LEVEL, len(pyramid) - 1)
+    p = pyramid[k]
+    comp = None
+    if p["compressor"]:
+        comp = get_json(f"{base}/{p['path']}/.zarray")["compressor"]
+    tile = _range_tile(f"{base}/{p['path']}", p["chunks"], comp)
+    f = 2 ** k
+    return Slices(Volume(p["shape"], p["chunks"], tile, threads=32), f, um * f / 1000), k
+
+
+def read_pyramid(base):
+    """Every level of the OME-Zarr at base: path, shape, chunks, dtype and compressor id."""
+    attrs = get_json(f"{base}/.zattrs")
+    out = []
+    for lv in [d["path"] for d in attrs["multiscales"][0]["datasets"]]:
+        za = get_json(f"{base}/{lv}/.zarray")
+        comp = za.get("compressor")
+        out.append({"path": lv, "shape": za["shape"], "chunks": za["chunks"], "dtype": za["dtype"],
+                    "compressor": comp.get("id") if comp else None})
+    return out
+
+
+def open_scan(source_volume):
+    """(Slices, level) for a bucket scan named "SCROLL/volumes/VOL.zarr", or (None, why) when it cannot be read."""
+    m = re.search(r"-([0-9.]+)um-", source_volume or "")
+    if not m:
+        return None, f"no voxel size in the volume name {source_volume!r}"
+    base = f"{S3}/{source_volume.strip('/')}"
+    try:
+        return open_slices(base, read_pyramid(base), float(m.group(1)))
+    except ImportError:
+        return None, "this scan's chunks are compressed and numcodecs is not installed to read them"
+    except Exception as e:  # a scan that cannot be opened: its lines are left whole and say so
+        return None, f"the scan's CT could not be opened ({type(e).__name__})"
+
+
+def check_line(points, scan):
+    """(result, why) for one line; scan is (Slices, level) or (None, why). result["level"] is the level read."""
+    sl, lev = scan
+    if sl is None:
+        return None, lev
+    try:
+        res, why = outside_ends(points, sl)
+    except Exception as e:  # a read that failed: the line is left whole and says so
+        return None, f"the CT could not be read ({type(e).__name__})"
+    if res:
+        res["level"] = lev
+    return res, why
+
+
+# ---------- trimming a file ----------
+
+def _path_mm(pts, um):
+    return sum(math.dist(a, b) for a, b in zip(pts, pts[1:])) * um / 1000
+
+
+def cut_of(n, res):
+    """The kept slice [a, b) of a line of n points sorted by z, from a result of outside_ends (None keeps all)."""
+    if not res:
+        return 0, n
+    return res["low"]["hidden"], n - res["high"]["hidden"]
+
+
+def trim_doc(doc, res, why, um=None):
+    """A copy of an umbilicus document with the end runs res cuts removed and metadata.papyrus_trim recording it.
+
+    Points keep their file order and keys; the cut is made on their order by z (the order outside_ends tested).
+    res None (untested) cuts nothing and records why. um, the voxel size in micrometers, gives the cut in mm."""
+    cps = doc["control_points"]
+    order = sorted(range(len(cps)), key=lambda i: cps[i]["z"])
+    a, b = cut_of(len(cps), res)
+    keep = set(order[a:b])
+    pts = [(cps[i]["x"], cps[i]["y"], cps[i]["z"]) for i in order]
+    out = dict(doc)
+    out["control_points"] = [p for i, p in enumerate(cps) if i in keep]
+    md = dict(doc.get("metadata") or {})
+    um = um if um is not None else md.get("voxelsize_um")
+    rec = {"version": VERSION, "rule": RULE, "points_before": len(cps), "points_after": len(keep)}
+    if res is None:
+        rec.update({"tested": False, "why": why})
+    else:
+        rec.update({"tested": True, "level": res.get("level"), "far_mm": res["far_mm"], "end_share": res["end_share"]})
+        for end, run, edge in (("low", pts[:a + 1] if a else [], a), ("high", pts[b - 1:] if b < len(pts) else [], b - 1)):
+            e = res[end]
+            rec[end] = {"points": e["hidden"], "outside": e["far"], "empty_slices": e["empty"],
+                        "max_mm_past_outline": e["max_mm"]}
+            if e["hidden"] and edge >= 0 and edge < len(pts):
+                dz = abs(pts[edge][2] - (pts[0][2] if end == "low" else pts[-1][2]))
+                rec[end]["z_voxels"] = round(dz, 2)
+                if um:
+                    rec[end]["z_mm"] = round(dz * um / 1000, 2)
+                    rec[end]["path_mm"] = round(_path_mm(run, um), 2)
+    md["papyrus_trim"] = rec
+    out["metadata"] = md
+    return out
+
+
+def _points(doc):
+    return [(p["x"], p["y"], p["z"]) for p in sorted(doc["control_points"], key=lambda p: p["z"])]
+
+
+def run(files, volume=None, out=None, scans=None, opener=None):
+    """check (out None) or trim (out a folder) each file; returns one report row per file.
+
+    scans caches (Slices, level) per volume across calls; opener(source_volume) opens one (open_scan by default)."""
+    scans = {} if scans is None else scans
+    opener = opener or open_scan
+    rows = []
+    for f in files:
+        f = Path(f)
+        doc = json.loads(f.read_text())
+        md = doc.get("metadata") or {}
+        vol = volume or md.get("source_volume")
+        if not vol:
+            res, why = None, "the file names no scan (metadata.source_volume); give --volume"
+        else:
+            if vol not in scans:
+                scans[vol] = opener(vol)
+            res, why = check_line(_points(doc), scans[vol])
+        m = re.search(r"-([0-9.]+)um-", vol or "")
+        um = md.get("voxelsize_um") or (float(m.group(1)) if m else None)
+        new = trim_doc(doc, res, why, um)
+        rec = new["metadata"]["papyrus_trim"]
+        rows.append(dict({"file": str(f), "volume": vol}, **rec))
+        if out is not None:
+            Path(out).mkdir(parents=True, exist_ok=True)
+            (Path(out) / f.name).write_text(json.dumps(new, indent=1) + "\n")
+    return rows
+
+
+def _say(r):
+    if not r["tested"]:
+        return f"{Path(r['file']).name}: untested, {r['why']}; left whole"
+    parts = []
+    for end in ("low", "high"):
+        e = r[end]
+        if e["points"]:
+            parts.append(f"{end} {e['points']} points" + (f", {e['z_mm']} mm of height" if "z_mm" in e else "")
+                         + (f", up to {e['max_mm_past_outline']} mm past the outline" if e["max_mm_past_outline"] else "")
+                         + (f", {e['empty_slices']} in slices with no scroll" if e["empty_slices"] else ""))
+    return f"{Path(r['file']).name}: {r['points_before']} to {r['points_after']} points" + (": " + "; ".join(parts) if parts else ", nothing outside")
+
+
+def main(argv=None):
+    if sys.stdout.isatty():                  # the Scrolls look (AGENTS.md, terminal look rule)
+        try:
+            sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+            from kit import termstyle
+            termstyle.script()
+        except ImportError:
+            pass
+    ap = argparse.ArgumentParser(prog="lineends.py", description=__doc__.split("\n\n")[0])
+    ap.add_argument("cmd", choices=["check", "trim"])
+    ap.add_argument("files", nargs="+", help="umbilicus files in the published format")
+    ap.add_argument("--out", help="trim: the folder for the trimmed copies")
+    ap.add_argument("--volume", help="the scan, SCROLL/volumes/VOL.zarr in the open-data bucket, for every file "
+                                     "(default: each file's metadata.source_volume)")
+    ap.add_argument("--report", help="write the per-file report here as JSON")
+    ap.add_argument("--json", action="store_true", help="print the report as JSON")
+    a = ap.parse_args(argv)
+    if a.cmd == "trim" and not a.out:
+        ap.error("trim needs --out DIR")
+    out = Path(a.out) if a.cmd == "trim" else None
+    if out is not None and any(Path(f).resolve().parent == out.resolve() for f in a.files):
+        ap.error("--out is the folder of an input file; write the trimmed copies elsewhere")
+    rows = run(a.files, a.volume, out)
+    if a.report:
+        Path(a.report).write_text(json.dumps(rows, indent=1) + "\n")
+    if a.json:
+        print(json.dumps(rows, indent=1))
+    else:
+        for r in rows:
+            print(_say(r))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())