AGENT-HANDOFF.md

# Agent handoff — continuing this work on the AMD workstation

Companion to [`README.md`](README.md) (the public model card, which holds the results, method,
literature, and traps). **Read that first.** This file is the operational half: environment, layout,
and what to actually do next.

Written 2026-08-04, at the migration off the Gilbreth cluster.

---

## 1. The machine changed, and it is AMD

| | |
|---|---|
| CPU | AMD Ryzen AI Max+ 395 (Strix Halo), 16C/32T |
| GPU | **Radeon 8060S iGPU — gfx1151, RDNA 3.5** |
| memory | 128 GB unified (61 GB system + 64 GB VRAM carve-out), 8 GB swap |
| disk | 1.9 TB NVMe, ~1.4 TB free |
| OS | Ubuntu 26.04 |

Everything before this ran on an **A100 80GB with CUDA 12.6**. The port was assessed by reading
imports, not by assumption:

**✅ What is fine**
- `scripts/train_vecset.py` imports only numpy / torch / h5py + project modules. **Training does not
  need pytorch3d.**
- **No custom CUDA kernels anywhere in project code** — no `CUDAExtension`, no `cpp_extension.load`.
- No flash-attn, xformers, deepspeed, apex, or bitsandbytes. The usual ROCm blockers are all absent.
- 🔑 **The ~40 hard-coded `.cuda()` / `device='cuda'` call sites are NOT a problem.** ROCm PyTorch keeps
  the `torch.cuda` namespace and maps it onto HIP. **Do not "fix" them** — rewriting them to
  `device_type` strings is churn that buys nothing and risks breaking the CUDA path.

**⚠️ The one real blocker — pytorch3d**
- No ROCm wheels exist. Needed by `utils/util_3d.py` and `scripts/foundations/eval_massing_arms.py`
  (**the harness**) for `MeshRasterizer` / `MeshRenderer`.
- Build it **CPU-only** (`FORCE_CUDA=0`). At n=48 buildings the rasterisation cost is tolerable.
- If the CPU build also fights you, the rendering is separable from the metrics — `fp_iou`, `missing`,
  `extra`, `vol_iou` are voxel operations and do not need pytorch3d. Only the montages do.

**⚠️ Do NOT `pip install -r requirements-frozen.txt` verbatim.** It pins 14 `nvidia-*-cu12` packages,
`torch==2.8.0+cu126`, `torchvision==0.23.0+cu126`, and `triton==3.4.0`. Install ROCm torch wheels first,
then the non-torch remainder. Note ROCm's official Ubuntu support may lag 26.04; gfx1151 support is
recent (ROCm 6.4+).

**⚠️ Expect materially slower training.** The denoiser is only 49M params and fits the 64 GB carve-out
with enormous headroom, but an integrated GPU is not an A100. Budget accordingly before planning a
240k-step run — the runs behind this work were ~10–11 GPU-hours *each* on A100.

---

## 2. What to bring over

**Do not copy the cluster tree.** It is 1.4 TB: `data/` 493 GB, `logs_building/` 406 GB, `legacy/`
369 GB. Roughly 700 GB of that is superseded dense-grid snapshots — **delete, don't copy.**

| item | how |
|---|---|
| code, docs, results, montages | `git clone`, branch `massing-solid-gate-retrain` |
| the corpus | **regenerate** — 25 MB in git rebuilds 35 GB SDF + 17.4 GB latents (`REPRODUCING.md` §4) |
| model weights | this folder — **945 MB** for the five vecset checkpoints |
| `stage3a_lod2_deployed.pth` | 7.2 GB, **optional** — superseded baseline, comparison arm only |
| Dora-VAE (`dora_vae_1_1.ckpt`) | 2.1 GB, re-download from Hugging Face |

Regenerated `data/` lands near ~55 GB, not 493 — `REPRODUCING.md` §4d notes `nrw.h5` and `plateau.h5`
are intermediate staging nothing current uses.

Verify after transfer: `sha256sum -c SHA256SUMS`.

---

## 3. Where things are

| what | path |
|---|---|
| **THE harness** — 48 pinned ids, all arms, one pass | `scripts/foundations/eval_massing_arms.py` |
| cheap tracker (has a no-op detector; **not authoritative**) | `scripts/foundations/probe_vecset_checkpoint.py` |
| training | `scripts/train_vecset.py` (`--resume`, `--surf_weight`, `--surf_t_center`, `--archive_every`) |
| the model | `models/networks/vecset_denoiser.py` |
| the gradient path | `DoraCodec(differentiable=True).freeze()` — off by default |
| corpus loading | `load_surfaces` — ⚠️ **never read the h5 directly**, it is inward-wound |
| frame conversion | `scene.surface_sampling.to_array_frame`, guarded by `verify_frame` |
| results artifacts | `execution/artifacts/massing_arms_eval_*.json` |
| reasoning trail | `docs/wayfinding/vecset-convergence/` |

`--ids_from` replays a pinned id set so runs stay comparable. **Use it.** Every number in the model
card is on the same 48 ids; a number from a different sample is not comparable to any of them.

---

## 4. What to do next

**The open question: what separates the 29 solid buildings from the 19 hollow ones?**

This is the first *specific* investigation this project has had — every prior step was a blind lever
pull. If the split correlates with something legible, it is a targeted fix rather than a sweep.

- ❌ **Building size is ruled out** *(tested 2026-08-04)*. Median GT volume 50,515 (solid) vs 43,554
  (hollow), Mann-Whitney **p=0.246**, point-biserial r=0.186 (p=0.204). Not significant at n=48.
- ⬜ **Untested:** footprint complexity (vertex count, concavity, aspect ratio), source corpus
  (3DBAG / NRW / PLATEAU), height, roof-form class.

Secondary, cheaper: **lower `--surf_weight` at the same band** — the collapses may simply be the term
overshooting on harder cases.

**Then, in priority order:**

1. **Criterion 2 — footprint fidelity 0.962 → 1.000.** This is the *hard, non-negotiable* gate and it
   was under-weighted for a whole cycle in favour of 3D IoU, which the specification marks
   diagnostic-only. Weight it first.
2. **Add a collapse rate to the harness.** It reports medians, which is correct for unimodal noise and
   actively misleading here. The bimodality was caught by eye, from a render — not by the metrics.
3. **#79 — SNE (sharp-normal-error).** The only proposed instrument that might separate crisp from
   melted, and still absent from the harness. ⚠️ `guard_roughness` is **not cross-arm comparable** —
   do not use it to compare models.
4. **#82 — footprint-only height inference.** Until this lands, the claim is *"footprint + height →
   mass"*, not *"footprint alone"*. **Keep that exact wherever the work is written up.**

---

## 5. Read this before trusting any number

The full list is in the model card's **Measurement traps** section. The three that will actually cost
you time:

1. **Never extrapolate the training curve.** It went 0.719 → 0.657 → 0.532 → **0.840**. Two separate
   runs were nearly killed during multi-checkpoint collapses that recovered. A 30,000-step window of
   garbage output is not evidence of a dead run *in this model*.
2. **Always report `vs input` beside any quality number.** The generator scores well by *declining to
   act* — at s=0.45 it returned its input at 99.9% and inherited its score. A model that does nothing
   looks excellent on every aggregate metric here.
3. **Render before you conclude.** The aggregate went 0.195 → 0.200 ("better") while a building went
   from a box to a shredded cage. Both bimodality and the cage failure were caught by eye, never by a
   scalar. This project is judged visually and the metrics have repeatedly failed to see what mattered.

⚠️ **One historical correction to be aware of:** a "beat 0.840 3D IoU" bar appears throughout older
documents. It is **retired**. The criteria are: (1) human visual judgement — primary, (2) footprint
match — hard gate, (3) 3D IoU — diagnostic only. A retired criterion was re-imposed and dominated the
framing of two tickets before this was caught.