exp011a shipped: real data pays adapters 2-3x more; structure replicates; blob targets built (schema lesson disclosed)
Browse files
README.md
CHANGED
|
@@ -76,6 +76,7 @@ resolution) so gaps live in a narrow band — the paired design is load-bearing.
|
|
| 76 |
| exp003_sigma_registers | register probe: **sigma is the ONLY live axis** — caption/type/cond all compress to ≤0; ordering prereg MISS with design consequences | **shipped (candidate, s0)** |
|
| 77 |
| exp004_anima_relay | relays on the 2B DiT via diffusion-pipe (bf16 per dtype law) | staged |
|
| 78 |
| exp005_sdxl_tree4a | the SDXL capacity battery (guidepost/scaffold/skeleton) | designed |
|
|
|
|
| 79 |
| exp010_controller | **StepGatedSampler ships**: controller lifts grounding +0.089 over frozen; monotonic lesion ladder; HIGH lesion 14x LP-dominant (coarse-to-fine confirmed in image space); eps-trained HIGH band concentrates (diversity = open training goal) | **shipped (candidate)** |
|
| 80 |
| exp009_bandroles | role objectives: **directional hit 4/4 but noise-adjacent — frequency reweighting too collinear; needs qualitatively different supervision + generation-side gauges (exp010)** | **shipped (2 seeds + rejudge)** |
|
| 81 |
| exp008_multiband | band-ASSIGNED experts: **mechanism certifies — 3/3 surgical band lesions (50-200x), HIGH-noise band expert wins its band (label-corrected); uniform objective doesn't pay (roles enter exp009)** | **shipped (candidate, s0; s1 running)** |
|
|
|
|
| 76 |
| exp003_sigma_registers | register probe: **sigma is the ONLY live axis** — caption/type/cond all compress to ≤0; ordering prereg MISS with design consequences | **shipped (candidate, s0)** |
|
| 77 |
| exp004_anima_relay | relays on the 2B DiT via diffusion-pipe (bf16 per dtype law) | staged |
|
| 78 |
| exp005_sdxl_tree4a | the SDXL capacity battery (guidepost/scaffold/skeleton) | designed |
|
| 79 |
+
| exp011a_fused_multiband | multiband on REAL fused data: **adapters pay 2-3x more; structural story replicates (3/3 surgical, monolith edge persists); blob targets built 100% after a schema lesson** | **shipped (candidate, s0; s1 running)** |
|
| 80 |
| exp010_controller | **StepGatedSampler ships**: controller lifts grounding +0.089 over frozen; monotonic lesion ladder; HIGH lesion 14x LP-dominant (coarse-to-fine confirmed in image space); eps-trained HIGH band concentrates (diversity = open training goal) | **shipped (candidate)** |
|
| 81 |
| exp009_bandroles | role objectives: **directional hit 4/4 but noise-adjacent — frequency reweighting too collinear; needs qualitatively different supervision + generation-side gauges (exp010)** | **shipped (2 seeds + rejudge)** |
|
| 82 |
| exp008_multiband | band-ASSIGNED experts: **mechanism certifies — 3/3 surgical band lesions (50-200x), HIGH-noise band expert wins its band (label-corrected); uniform objective doesn't pay (roles enter exp009)** | **shipped (candidate, s0; s1 running)** |
|
exp011_fused_multiband/README.md
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# exp011a_fused_multiband — the multiband mechanism on REAL DATA
|
| 2 |
+
# (+ the segmentation blob-target cache) — CANDIDATE s0; s1 running
|
| 3 |
+
|
| 4 |
+
**Questions.** (1) Does real fused data (multi-entity photos, natural
|
| 5 |
+
captions) change the band economics that uniform synthetic data couldn't pay
|
| 6 |
+
for? (2) Build the segmentation blob targets for the HIGH-band blobbing role
|
| 7 |
+
(loss design held for review — new mechanism, pacing law).
|
| 8 |
+
|
| 9 |
+
**Design.** qwen-deepfashion-fused, 4352 rows (dataset audit gates enforced;
|
| 10 |
+
all rows passed), caption_joycaption at nl77, same certified beds as exp008
|
| 11 |
+
(multiband3 vs mono48, uniform eps objective, 3000 steps, pure Adam wd=0).
|
| 12 |
+
|
| 13 |
+
**Results** (`results.json`):
|
| 14 |
+
|
| 15 |
+
| arm | overall val | band0 (LOW) | band1 (MID) | band2 (HIGH) |
|
| 16 |
+
|---|---|---|---|---|
|
| 17 |
+
| frozen | 0.142435 | 0.293931 | 0.071764 | 0.010422 |
|
| 18 |
+
| multiband3 | 0.135215 (−5.1%) | 0.280633 | 0.066728 | **0.009562** |
|
| 19 |
+
| monolith r48 | **0.134184 (−5.8%)** | **0.278893** | **0.065772** | 0.009564 |
|
| 20 |
+
|
| 21 |
+
Lesions: 3/3 surgical, with LARGER asymmetries than synthetic (band0 own
|
| 22 |
+
0.0130 vs 0.000062 cross — 200×).
|
| 23 |
+
|
| 24 |
+
**Findings (candidate, s0).**
|
| 25 |
+
1. **Real data pays adapters 2–3× more** than synthetic (−5.1/−5.8% vs
|
| 26 |
+
−2.2/−2.4% in exp008) — the richer distribution has more for small
|
| 27 |
+
adapters to learn.
|
| 28 |
+
2. **The structural story replicates on real data**: surgical band
|
| 29 |
+
specialists, monolith edge persists under a uniform objective, HIGH-band
|
| 30 |
+
own-band win reappears at a hair (0.009562 vs 0.009564).
|
| 31 |
+
3. **Blob targets: built, verified, cached** — after a schema lesson: the
|
| 32 |
+
first rasterizer pass returned 100% EMPTY (fused_json polygons are FLAT
|
| 33 |
+
[x1,y1,...] lists in a declared `coord_space`, not the assumed pair-list
|
| 34 |
+
NORM_0_1000; the emptiness counter caught it — silent-zero discipline).
|
| 35 |
+
Corrected rasterizer: 100% non-empty, mean occupancy 0.293, coord_space
|
| 36 |
+
norm_0_1000 confirmed on every row. `fix_blob_targets.py` in-package.
|
| 37 |
+
|
| 38 |
+
**Next.** The blob LOSS (how coarse occupancy couples to the HIGH band's
|
| 39 |
+
x0-estimate) goes to the 8am design review with targets ready. s1
|
| 40 |
+
replication of this bed lands ~07:30Z.
|
| 41 |
+
|
| 42 |
+
**Caveats.** s0; 4352 rows single dataset; blob maps unused in this
|
| 43 |
+
training (uniform objective by design). Cost ≈ 2.9 GPU-h incl. cache.
|
exp011_fused_multiband/dexp011_fused_multiband.py
ADDED
|
@@ -0,0 +1,331 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""dexp011_fused_multiband.py — exp011a: multiband on REAL FUSED DATA
|
| 2 |
+
(+ the blob-target cache for the 8am blob-loss design review).
|
| 3 |
+
|
| 4 |
+
Two jobs (stage plan phase 4 prep, D2 from the reconstruction brief):
|
| 5 |
+
1. Does REAL data (qwen-deepfashion-fused: real photos, joycaption NL,
|
| 6 |
+
multi-entity scenes) change the band economics that synthetic
|
| 7 |
+
single-object data couldn't pay for? Same certified beds (multiband3 vs
|
| 8 |
+
mono48, uniform objective), new substrate data, exp010 battery after.
|
| 9 |
+
2. Build the SEGMENTATION BLOB TARGETS: fused_json entities[].mask.polygon
|
| 10 |
+
(NORM_0_1000) rasterized to the 64x64 latent grid, cached per row.
|
| 11 |
+
The blob LOSS mechanism is DELIBERATELY NOT implemented tonight —
|
| 12 |
+
it is a new mechanism and goes to Phil's 8am review with targets ready
|
| 13 |
+
(pacing law: don't step into something unprepared).
|
| 14 |
+
|
| 15 |
+
Data gates: rows filtered on the dataset's own audit columns when present
|
| 16 |
+
(age_audit/audit approved-only; counts printed; empty result fails LOUDLY —
|
| 17 |
+
never silently train on people data past an available gate).
|
| 18 |
+
|
| 19 |
+
Pod: bash pod2/run_exp011.sh
|
| 20 |
+
"""
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
import io
|
| 24 |
+
import json
|
| 25 |
+
import os
|
| 26 |
+
import sys
|
| 27 |
+
import time
|
| 28 |
+
|
| 29 |
+
sys.path[:0] = ["pod2", "."]
|
| 30 |
+
|
| 31 |
+
import numpy as np
|
| 32 |
+
import torch
|
| 33 |
+
import torch.nn.functional as F
|
| 34 |
+
|
| 35 |
+
from pod_ledger import ledger_run, note, burn_down
|
| 36 |
+
from d1_substrate import MEM_FRACTION
|
| 37 |
+
from dexp006_sd15core_relay import make_schedule, add_noise, encode_nl77
|
| 38 |
+
from dexp008_multiband import (MultibandDelta, MonoDelta, band_weights,
|
| 39 |
+
band_of, attach, load_unet, N_BANDS)
|
| 40 |
+
|
| 41 |
+
DATASET = "AbstractPhil/qwen-deepfashion-fused"
|
| 42 |
+
SD_BASE = "stable-diffusion-v1-5/stable-diffusion-v1-5"
|
| 43 |
+
VAE_SCALE = 0.18215
|
| 44 |
+
N_TRAIN, N_VAL = 4096, 256
|
| 45 |
+
BATCH = int(os.environ.get("DEXP11_BATCH", "16"))
|
| 46 |
+
STEPS = int(os.environ.get("DEXP11_STEPS", "3000"))
|
| 47 |
+
SEED = int(os.environ.get("DEXP11_SEED", "0"))
|
| 48 |
+
LR, CFG_DROPOUT = 1e-3, 0.1
|
| 49 |
+
DATA_DIR = ("/workspace/data/dexp011" if os.path.isdir("/workspace")
|
| 50 |
+
else os.path.join(os.environ.get("GEOLIP_DATA", "./data"),
|
| 51 |
+
"dexp011"))
|
| 52 |
+
CKPT_DIR = ("/workspace/ckpts2/dexp011" if os.path.isdir("/workspace")
|
| 53 |
+
else DATA_DIR)
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def rasterize_blob(fused_json_str: str, size: int = 64) -> np.ndarray:
|
| 57 |
+
"""Entity mask polygons (NORM_0_1000) -> union occupancy on the latent
|
| 58 |
+
grid. Returns (size, size) float32 in [0,1]. Rows without polygons
|
| 59 |
+
return zeros (counted by the caller)."""
|
| 60 |
+
from PIL import Image, ImageDraw
|
| 61 |
+
try:
|
| 62 |
+
fj = json.loads(fused_json_str)
|
| 63 |
+
except Exception:
|
| 64 |
+
return np.zeros((size, size), dtype=np.float32)
|
| 65 |
+
img = Image.new("L", (size, size), 0)
|
| 66 |
+
draw = ImageDraw.Draw(img)
|
| 67 |
+
n = 0
|
| 68 |
+
for ent in (fj.get("entities") or []):
|
| 69 |
+
mask = ent.get("mask") or {}
|
| 70 |
+
poly = mask.get("polygon") or []
|
| 71 |
+
if len(poly) >= 3:
|
| 72 |
+
pts = [(p[0] / 1000.0 * (size - 1), p[1] / 1000.0 * (size - 1))
|
| 73 |
+
for p in poly if isinstance(p, (list, tuple))
|
| 74 |
+
and len(p) >= 2]
|
| 75 |
+
if len(pts) >= 3:
|
| 76 |
+
draw.polygon(pts, fill=255)
|
| 77 |
+
n += 1
|
| 78 |
+
arr = np.asarray(img, dtype=np.float32) / 255.0
|
| 79 |
+
return arr
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def _iter_fused(need: int):
|
| 83 |
+
from huggingface_hub import HfApi, hf_hub_download
|
| 84 |
+
import pyarrow.parquet as pq
|
| 85 |
+
api = HfApi()
|
| 86 |
+
files = sorted(f for f in api.list_repo_files(DATASET, repo_type="dataset")
|
| 87 |
+
if f.endswith(".parquet"))
|
| 88 |
+
got, kept, gated = 0, 0, 0
|
| 89 |
+
cols_printed = False
|
| 90 |
+
for fname in files:
|
| 91 |
+
path = hf_hub_download(DATASET, fname, repo_type="dataset")
|
| 92 |
+
tbl = pq.read_table(path)
|
| 93 |
+
if not cols_printed:
|
| 94 |
+
print(f"[fused] columns: {tbl.column_names}", flush=True)
|
| 95 |
+
cols_printed = True
|
| 96 |
+
for r in tbl.to_pylist():
|
| 97 |
+
got += 1
|
| 98 |
+
gate_vals = [str(r.get(c)) for c in ("age_audit", "audit")
|
| 99 |
+
if c in r and r.get(c) is not None]
|
| 100 |
+
if gate_vals and not any(("approved" in v.lower()
|
| 101 |
+
or "pass" in v.lower())
|
| 102 |
+
for v in gate_vals):
|
| 103 |
+
gated += 1
|
| 104 |
+
continue
|
| 105 |
+
kept += 1
|
| 106 |
+
yield r
|
| 107 |
+
if kept >= need:
|
| 108 |
+
print(f"[fused] kept {kept} / seen {got} (gated out {gated})",
|
| 109 |
+
flush=True)
|
| 110 |
+
return
|
| 111 |
+
raise AssertionError(
|
| 112 |
+
f"exhausted dataset with only {kept} gated rows of {got} seen — "
|
| 113 |
+
f"inspect gate columns before proceeding")
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def build_cache(device="cuda"):
|
| 117 |
+
cache_f = os.path.join(DATA_DIR, "cache.pt")
|
| 118 |
+
if os.path.exists(cache_f):
|
| 119 |
+
print(f"[cache] exists: {cache_f}")
|
| 120 |
+
return cache_f
|
| 121 |
+
os.makedirs(DATA_DIR, exist_ok=True)
|
| 122 |
+
from PIL import Image
|
| 123 |
+
from diffusers import AutoencoderKL
|
| 124 |
+
from transformers import CLIPTextModel, CLIPTokenizer
|
| 125 |
+
vae = AutoencoderKL.from_pretrained(
|
| 126 |
+
SD_BASE, subfolder="vae", torch_dtype=torch.float32).to(device).eval()
|
| 127 |
+
tok = CLIPTokenizer.from_pretrained(SD_BASE, subfolder="tokenizer")
|
| 128 |
+
te = CLIPTextModel.from_pretrained(
|
| 129 |
+
SD_BASE, subfolder="text_encoder",
|
| 130 |
+
torch_dtype=torch.float32).to(device).eval()
|
| 131 |
+
g = torch.Generator(device=device).manual_seed(SEED)
|
| 132 |
+
lat, ehs, blobs, jpegs = [], [], [], []
|
| 133 |
+
buf_img, buf_txt = [], []
|
| 134 |
+
n_blob_empty = 0
|
| 135 |
+
|
| 136 |
+
@torch.no_grad()
|
| 137 |
+
def flush():
|
| 138 |
+
if not buf_img:
|
| 139 |
+
return
|
| 140 |
+
px = torch.stack(buf_img).to(device)
|
| 141 |
+
lat.append((vae.encode(px).latent_dist.sample(generator=g)
|
| 142 |
+
* VAE_SCALE).cpu())
|
| 143 |
+
ehs.append(encode_nl77(buf_txt, tok, te, device).cpu())
|
| 144 |
+
buf_img.clear()
|
| 145 |
+
buf_txt.clear()
|
| 146 |
+
|
| 147 |
+
n = 0
|
| 148 |
+
for r in _iter_fused(N_TRAIN + N_VAL):
|
| 149 |
+
img = r["image"]
|
| 150 |
+
img = img.get("bytes") if isinstance(img, dict) else img
|
| 151 |
+
im = Image.open(io.BytesIO(img)).convert("RGB").resize((512, 512))
|
| 152 |
+
buf_img.append(torch.from_numpy(np.asarray(im)).float()
|
| 153 |
+
.permute(2, 0, 1) / 127.5 - 1.0)
|
| 154 |
+
cap = r.get("caption_joycaption") or r.get("prompt_fused") or ""
|
| 155 |
+
buf_txt.append(str(cap))
|
| 156 |
+
b = rasterize_blob(str(r.get("fused_json", "")))
|
| 157 |
+
if b.max() == 0:
|
| 158 |
+
n_blob_empty += 1
|
| 159 |
+
blobs.append(torch.from_numpy(b))
|
| 160 |
+
if n < 24:
|
| 161 |
+
jpegs.append(img)
|
| 162 |
+
n += 1
|
| 163 |
+
if len(buf_img) == 16:
|
| 164 |
+
flush()
|
| 165 |
+
if n % 512 == 0:
|
| 166 |
+
print(f"[cache] {n}/{N_TRAIN + N_VAL} "
|
| 167 |
+
f"(blob-empty {n_blob_empty})", flush=True)
|
| 168 |
+
flush()
|
| 169 |
+
lat, ehs = torch.cat(lat), torch.cat(ehs)
|
| 170 |
+
blobs = torch.stack(blobs)
|
| 171 |
+
gv = torch.Generator().manual_seed(SEED + 1)
|
| 172 |
+
torch.save({"lat": lat[:N_TRAIN], "ehs": ehs[:N_TRAIN],
|
| 173 |
+
"blob": blobs[:N_TRAIN],
|
| 174 |
+
"val_lat": lat[N_TRAIN:], "val_ehs": ehs[N_TRAIN:],
|
| 175 |
+
"val_blob": blobs[N_TRAIN:],
|
| 176 |
+
"val_noise": torch.randn(N_VAL, 4, 64, 64, generator=gv),
|
| 177 |
+
"val_t": torch.randint(0, 1000, (N_VAL,), generator=gv),
|
| 178 |
+
"judge_jpegs": jpegs,
|
| 179 |
+
"blob_empty_count": n_blob_empty}, cache_f)
|
| 180 |
+
print(f"[cache] built: {N_TRAIN}+{N_VAL} rows, blob-empty {n_blob_empty}",
|
| 181 |
+
flush=True)
|
| 182 |
+
del vae, te
|
| 183 |
+
torch.cuda.empty_cache()
|
| 184 |
+
return cache_f
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
def run(device="cuda"):
|
| 188 |
+
torch.cuda.set_per_process_memory_fraction(MEM_FRACTION, 0)
|
| 189 |
+
os.makedirs(CKPT_DIR, exist_ok=True)
|
| 190 |
+
acp = make_schedule(device)
|
| 191 |
+
with ledger_run("dexp011 fused cache + blob targets", budget_h=1.5):
|
| 192 |
+
build_cache(device)
|
| 193 |
+
cache = torch.load(os.path.join(DATA_DIR, "cache.pt"),
|
| 194 |
+
map_location="cpu", weights_only=True)
|
| 195 |
+
|
| 196 |
+
def set_w(wraps, s01):
|
| 197 |
+
w = band_weights(s01)
|
| 198 |
+
for wr in wraps:
|
| 199 |
+
wr.w_bands = w
|
| 200 |
+
|
| 201 |
+
def loss_of(unet, wraps, lat, ehs, gen):
|
| 202 |
+
bsz = lat.shape[0]
|
| 203 |
+
drop = torch.rand(bsz, generator=gen, device=device) < CFG_DROPOUT
|
| 204 |
+
ehs = ehs.clone()
|
| 205 |
+
ehs[drop] = 0
|
| 206 |
+
t = torch.randint(0, 1000, (bsz,), generator=gen, device=device)
|
| 207 |
+
set_w(wraps, t.float() / 1000.0)
|
| 208 |
+
noise = torch.randn(lat.shape, generator=gen, device=device)
|
| 209 |
+
pred = unet(add_noise(lat, noise, t, acp), t, ehs,
|
| 210 |
+
return_dict=False)[0]
|
| 211 |
+
return F.mse_loss(pred, noise)
|
| 212 |
+
|
| 213 |
+
@torch.no_grad()
|
| 214 |
+
def val(unet, wraps):
|
| 215 |
+
tot, per_band = [], {0: [], 1: [], 2: []}
|
| 216 |
+
for i in range(0, N_VAL, 32):
|
| 217 |
+
lat = cache["val_lat"][i:i + 32].to(device)
|
| 218 |
+
ehs = cache["val_ehs"][i:i + 32].to(device)
|
| 219 |
+
noise = cache["val_noise"][i:i + 32].to(device)
|
| 220 |
+
t = cache["val_t"][i:i + 32].to(device)
|
| 221 |
+
set_w(wraps, t.float() / 1000.0)
|
| 222 |
+
pred = unet(add_noise(lat, noise, t, acp), t, ehs,
|
| 223 |
+
return_dict=False)[0]
|
| 224 |
+
mse = ((pred - noise) ** 2).mean(dim=(1, 2, 3))
|
| 225 |
+
tot += mse.tolist()
|
| 226 |
+
for j, tv in enumerate((t.float() / 1000.0).tolist()):
|
| 227 |
+
per_band[band_of(tv)].append(mse[j].item())
|
| 228 |
+
return (sum(tot) / len(tot),
|
| 229 |
+
{f"band{b}": round(sum(v) / max(len(v), 1), 6)
|
| 230 |
+
for b, v in per_band.items()})
|
| 231 |
+
|
| 232 |
+
def train(unet, wraps, params, label):
|
| 233 |
+
opt = torch.optim.Adam(params, lr=LR, weight_decay=0.0)
|
| 234 |
+
gen = torch.Generator(device=device).manual_seed(SEED + 42)
|
| 235 |
+
idx = torch.Generator().manual_seed(SEED + 7)
|
| 236 |
+
t0 = time.time()
|
| 237 |
+
for step in range(1, STEPS + 1):
|
| 238 |
+
sel = torch.randint(0, N_TRAIN, (BATCH,), generator=idx)
|
| 239 |
+
loss = loss_of(unet, wraps, cache["lat"][sel].to(device),
|
| 240 |
+
cache["ehs"][sel].to(device), gen)
|
| 241 |
+
loss.backward()
|
| 242 |
+
opt.step()
|
| 243 |
+
opt.zero_grad(set_to_none=True)
|
| 244 |
+
if step == 50 or step % 500 == 0:
|
| 245 |
+
print(f"[{label}] step {step}: loss {loss.item():.4f} | "
|
| 246 |
+
f"{(time.time() - t0) / step:.2f}s/step", flush=True)
|
| 247 |
+
return round((time.time() - t0) / STEPS, 3)
|
| 248 |
+
|
| 249 |
+
results = {"config": {"dataset": DATASET, "steps": STEPS, "batch": BATCH,
|
| 250 |
+
"seed": SEED, "cond": "caption_joycaption nl77",
|
| 251 |
+
"blob_empty": int(cache["blob_empty_count"])}}
|
| 252 |
+
with ledger_run("dexp011 frozen", budget_h=0.2) as h:
|
| 253 |
+
unet = load_unet(device)
|
| 254 |
+
v, pb = val(unet, [])
|
| 255 |
+
results["frozen"] = {"val": v, "per_band": pb}
|
| 256 |
+
del unet
|
| 257 |
+
torch.cuda.empty_cache()
|
| 258 |
+
h["verdict"] = f"val {v:.5f}"
|
| 259 |
+
|
| 260 |
+
for label, mk, r_note in (("multiband3", lambda d: MultibandDelta(d),
|
| 261 |
+
"3 band experts"),
|
| 262 |
+
("monolith", lambda d: MonoDelta(d), "r48")):
|
| 263 |
+
with ledger_run(f"dexp011 {label} s{SEED}", budget_h=2.5) as h:
|
| 264 |
+
unet = load_unet(device)
|
| 265 |
+
mods, wraps = attach(unet, mk)
|
| 266 |
+
n_params = sum(p.numel() for p in mods.parameters())
|
| 267 |
+
spd = train(unet, wraps, mods.parameters(), label)
|
| 268 |
+
v, pb = val(unet, wraps)
|
| 269 |
+
entry = {"n_params": n_params, "val": v, "per_band": pb,
|
| 270 |
+
"s_per_step": spd}
|
| 271 |
+
if label == "multiband3":
|
| 272 |
+
lesions = {}
|
| 273 |
+
for b in range(N_BANDS):
|
| 274 |
+
for m in mods:
|
| 275 |
+
m.band_enabled[b] = False
|
| 276 |
+
_, pb_les = val(unet, wraps)
|
| 277 |
+
lesions[f"lesion_band{b}"] = pb_les
|
| 278 |
+
for m in mods:
|
| 279 |
+
m.band_enabled[b] = True
|
| 280 |
+
entry["lesions"] = lesions
|
| 281 |
+
torch.save({"mods": [m.state_dict() for m in mods]},
|
| 282 |
+
os.path.join(CKPT_DIR, f"{label}_s{SEED}.pt"))
|
| 283 |
+
results[label] = entry
|
| 284 |
+
del unet, mods
|
| 285 |
+
torch.cuda.empty_cache()
|
| 286 |
+
h["verdict"] = f"val {v:.5f}"
|
| 287 |
+
|
| 288 |
+
mb, mo = results["multiband3"], results["monolith"]
|
| 289 |
+
lesion_sur = {}
|
| 290 |
+
for b in range(N_BANDS):
|
| 291 |
+
les = mb["lesions"][f"lesion_band{b}"]
|
| 292 |
+
own = les[f"band{b}"] - mb["per_band"][f"band{b}"]
|
| 293 |
+
oth = sum(les[f"band{o}"] - mb["per_band"][f"band{o}"]
|
| 294 |
+
for o in range(N_BANDS) if o != b) / (N_BANDS - 1)
|
| 295 |
+
lesion_sur[f"band{b}"] = {"own": round(own, 6),
|
| 296 |
+
"other": round(oth, 6),
|
| 297 |
+
"surgical": own > 3 * abs(oth) and own > 0}
|
| 298 |
+
results["verdict"] = {
|
| 299 |
+
"mb_vs_mono": "multiband" if mb["val"] <= mo["val"] else "monolith",
|
| 300 |
+
"per_band_wins": {b: mb["per_band"][b] < mo["per_band"][b]
|
| 301 |
+
for b in ("band0", "band1", "band2")},
|
| 302 |
+
"lesions": lesion_sur,
|
| 303 |
+
"note": "REAL fused data (D2); blob targets cached for the 8am "
|
| 304 |
+
"blob-loss design review; 1-seed CANDIDATE",
|
| 305 |
+
}
|
| 306 |
+
with open(os.path.join(DATA_DIR, "results.json" if SEED == 0
|
| 307 |
+
else f"results_s{SEED}.json"), "w") as f:
|
| 308 |
+
json.dump(results, f, indent=2)
|
| 309 |
+
note(f"dexp011: {json.dumps(results['verdict']['mb_vs_mono'])}")
|
| 310 |
+
print(json.dumps(results["verdict"], indent=2))
|
| 311 |
+
burn_down()
|
| 312 |
+
return results
|
| 313 |
+
|
| 314 |
+
|
| 315 |
+
def smoke():
|
| 316 |
+
fj = json.dumps({"entities": [
|
| 317 |
+
{"mask": {"polygon": [[100, 100], [900, 100], [900, 900],
|
| 318 |
+
[100, 900]]}},
|
| 319 |
+
{"mask": {"polygon": [[0, 0], [200, 0], [100, 200]]}}]})
|
| 320 |
+
b = rasterize_blob(fj)
|
| 321 |
+
assert b.shape == (64, 64) and 0.5 < b.mean() < 0.9, b.mean()
|
| 322 |
+
assert rasterize_blob("not json").max() == 0
|
| 323 |
+
assert rasterize_blob(json.dumps({"entities": []})).max() == 0
|
| 324 |
+
print("dexp011 smoke PASSED (blob rasterizer; GPU run is pod work)")
|
| 325 |
+
|
| 326 |
+
|
| 327 |
+
if __name__ == "__main__":
|
| 328 |
+
if "--run" in sys.argv:
|
| 329 |
+
run()
|
| 330 |
+
else:
|
| 331 |
+
smoke()
|
exp011_fused_multiband/fix_blob_targets.py
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""fix_blob_targets.py — rebuild exp011's blob targets with the CORRECT
|
| 2 |
+
fused_json schema (diagnosed 2026-07-18: polygon is a FLAT [x1,y1,x2,y2,...]
|
| 3 |
+
list, and coordinates live in the declared coord_space with image_size — not
|
| 4 |
+
the assumed NORM_0_1000 pair-list). Row order replicates _iter_fused exactly
|
| 5 |
+
(same files, same gates); patches blob/val_blob into the existing cache.
|
| 6 |
+
LOUD assert: >50% of rows must produce non-empty blobs. CPU-only.
|
| 7 |
+
"""
|
| 8 |
+
from __future__ import annotations
|
| 9 |
+
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
import sys
|
| 13 |
+
|
| 14 |
+
sys.path[:0] = ["pod2", "."]
|
| 15 |
+
|
| 16 |
+
import numpy as np
|
| 17 |
+
import torch
|
| 18 |
+
|
| 19 |
+
from dexp011_fused_multiband import _iter_fused, N_TRAIN, N_VAL, DATA_DIR
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def rasterize_v2(fused_json_str: str, size: int = 64) -> np.ndarray:
|
| 23 |
+
from PIL import Image, ImageDraw
|
| 24 |
+
try:
|
| 25 |
+
fj = json.loads(fused_json_str)
|
| 26 |
+
except Exception:
|
| 27 |
+
return np.zeros((size, size), dtype=np.float32)
|
| 28 |
+
cs = str(fj.get("coord_space", "")).lower()
|
| 29 |
+
im_size = fj.get("image_size")
|
| 30 |
+
if isinstance(im_size, (list, tuple)) and len(im_size) >= 2:
|
| 31 |
+
W, H = float(im_size[0]) or None, float(im_size[1]) or None
|
| 32 |
+
elif isinstance(im_size, dict):
|
| 33 |
+
W = float(im_size.get("width") or im_size.get("w") or 0) or None
|
| 34 |
+
H = float(im_size.get("height") or im_size.get("h") or 0) or None
|
| 35 |
+
else:
|
| 36 |
+
W = H = None
|
| 37 |
+
img = Image.new("L", (size, size), 0)
|
| 38 |
+
draw = ImageDraw.Draw(img)
|
| 39 |
+
for ent in (fj.get("entities") or []):
|
| 40 |
+
poly = (ent.get("mask") or {}).get("polygon") or []
|
| 41 |
+
if len(poly) >= 6 and all(isinstance(v, (int, float))
|
| 42 |
+
for v in poly[:6]):
|
| 43 |
+
xs, ys = poly[0::2], poly[1::2]
|
| 44 |
+
if "1000" in cs or (max(xs + ys) <= 1000 and W is None):
|
| 45 |
+
sx = sy = 1.0 / 1000.0
|
| 46 |
+
elif W and H:
|
| 47 |
+
sx, sy = 1.0 / W, 1.0 / H
|
| 48 |
+
else:
|
| 49 |
+
m = max(max(xs), max(ys), 1.0)
|
| 50 |
+
sx = sy = 1.0 / m
|
| 51 |
+
pts = [(min(max(x * sx, 0), 1) * (size - 1),
|
| 52 |
+
min(max(y * sy, 0), 1) * (size - 1))
|
| 53 |
+
for x, y in zip(xs, ys)]
|
| 54 |
+
if len(pts) >= 3:
|
| 55 |
+
draw.polygon(pts, fill=255)
|
| 56 |
+
return np.asarray(img, dtype=np.float32) / 255.0
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def main():
|
| 60 |
+
blobs, n_empty, cs_seen = [], 0, {}
|
| 61 |
+
for r in _iter_fused(N_TRAIN + N_VAL):
|
| 62 |
+
s = str(r.get("fused_json", ""))
|
| 63 |
+
b = rasterize_v2(s)
|
| 64 |
+
if b.max() == 0:
|
| 65 |
+
n_empty += 1
|
| 66 |
+
try:
|
| 67 |
+
cs = json.loads(s).get("coord_space", "?")
|
| 68 |
+
cs_seen[str(cs)] = cs_seen.get(str(cs), 0) + 1
|
| 69 |
+
except Exception:
|
| 70 |
+
pass
|
| 71 |
+
blobs.append(torch.from_numpy(b))
|
| 72 |
+
blobs = torch.stack(blobs)
|
| 73 |
+
frac = 1 - n_empty / len(blobs)
|
| 74 |
+
print(f"[fix_blob] non-empty {frac:.1%} (empty {n_empty}/{len(blobs)}); "
|
| 75 |
+
f"coord_spaces {cs_seen}; mean occupancy "
|
| 76 |
+
f"{blobs.mean().item():.3f}", flush=True)
|
| 77 |
+
assert frac > 0.5, f"still mostly empty ({frac:.1%}) — schema wrong again"
|
| 78 |
+
f = os.path.join(DATA_DIR, "cache.pt")
|
| 79 |
+
cache = torch.load(f, map_location="cpu", weights_only=True)
|
| 80 |
+
cache["blob"] = blobs[:N_TRAIN]
|
| 81 |
+
cache["val_blob"] = blobs[N_TRAIN:]
|
| 82 |
+
cache["blob_empty_count"] = n_empty
|
| 83 |
+
torch.save(cache, f)
|
| 84 |
+
print(f"[fix_blob] cache patched: {f}", flush=True)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
if __name__ == "__main__":
|
| 88 |
+
main()
|
exp011_fused_multiband/results.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"dataset": "AbstractPhil/qwen-deepfashion-fused",
|
| 4 |
+
"steps": 3000,
|
| 5 |
+
"batch": 16,
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"cond": "caption_joycaption nl77",
|
| 8 |
+
"blob_empty": 4352
|
| 9 |
+
},
|
| 10 |
+
"frozen": {
|
| 11 |
+
"val": 0.14243539798007987,
|
| 12 |
+
"per_band": {
|
| 13 |
+
"band0": 0.293931,
|
| 14 |
+
"band1": 0.071764,
|
| 15 |
+
"band2": 0.010422
|
| 16 |
+
}
|
| 17 |
+
},
|
| 18 |
+
"multiband3": {
|
| 19 |
+
"n_params": 1235568,
|
| 20 |
+
"val": 0.1352152533709159,
|
| 21 |
+
"per_band": {
|
| 22 |
+
"band0": 0.280633,
|
| 23 |
+
"band1": 0.066728,
|
| 24 |
+
"band2": 0.009562
|
| 25 |
+
},
|
| 26 |
+
"s_per_step": 1.688,
|
| 27 |
+
"lesions": {
|
| 28 |
+
"lesion_band0": {
|
| 29 |
+
"band0": 0.293597,
|
| 30 |
+
"band1": 0.066852,
|
| 31 |
+
"band2": 0.009562
|
| 32 |
+
},
|
| 33 |
+
"lesion_band1": {
|
| 34 |
+
"band0": 0.280774,
|
| 35 |
+
"band1": 0.071368,
|
| 36 |
+
"band2": 0.00964
|
| 37 |
+
},
|
| 38 |
+
"lesion_band2": {
|
| 39 |
+
"band0": 0.280633,
|
| 40 |
+
"band1": 0.066757,
|
| 41 |
+
"band2": 0.010283
|
| 42 |
+
}
|
| 43 |
+
}
|
| 44 |
+
},
|
| 45 |
+
"monolith": {
|
| 46 |
+
"n_params": 1210576,
|
| 47 |
+
"val": 0.1341835820844608,
|
| 48 |
+
"per_band": {
|
| 49 |
+
"band0": 0.278893,
|
| 50 |
+
"band1": 0.065772,
|
| 51 |
+
"band2": 0.009564
|
| 52 |
+
},
|
| 53 |
+
"s_per_step": 1.654
|
| 54 |
+
},
|
| 55 |
+
"verdict": {
|
| 56 |
+
"mb_vs_mono": "monolith",
|
| 57 |
+
"per_band_wins": {
|
| 58 |
+
"band0": false,
|
| 59 |
+
"band1": false,
|
| 60 |
+
"band2": true
|
| 61 |
+
},
|
| 62 |
+
"lesions": {
|
| 63 |
+
"band0": {
|
| 64 |
+
"own": 0.012964,
|
| 65 |
+
"other": 6.2e-05,
|
| 66 |
+
"surgical": true
|
| 67 |
+
},
|
| 68 |
+
"band1": {
|
| 69 |
+
"own": 0.00464,
|
| 70 |
+
"other": 0.00011,
|
| 71 |
+
"surgical": true
|
| 72 |
+
},
|
| 73 |
+
"band2": {
|
| 74 |
+
"own": 0.000721,
|
| 75 |
+
"other": 1.5e-05,
|
| 76 |
+
"surgical": true
|
| 77 |
+
}
|
| 78 |
+
},
|
| 79 |
+
"note": "REAL fused data (D2); blob targets cached for the 8am blob-loss design review; 1-seed CANDIDATE"
|
| 80 |
+
}
|
| 81 |
+
}
|