AbstractPhil commited on
Commit
5386334
·
verified ·
1 Parent(s): e277496

exp011a shipped: real data pays adapters 2-3x more; structure replicates; blob targets built (schema lesson disclosed)

Browse files
README.md CHANGED
@@ -76,6 +76,7 @@ resolution) so gaps live in a narrow band — the paired design is load-bearing.
76
  | exp003_sigma_registers | register probe: **sigma is the ONLY live axis** — caption/type/cond all compress to ≤0; ordering prereg MISS with design consequences | **shipped (candidate, s0)** |
77
  | exp004_anima_relay | relays on the 2B DiT via diffusion-pipe (bf16 per dtype law) | staged |
78
  | exp005_sdxl_tree4a | the SDXL capacity battery (guidepost/scaffold/skeleton) | designed |
 
79
  | exp010_controller | **StepGatedSampler ships**: controller lifts grounding +0.089 over frozen; monotonic lesion ladder; HIGH lesion 14x LP-dominant (coarse-to-fine confirmed in image space); eps-trained HIGH band concentrates (diversity = open training goal) | **shipped (candidate)** |
80
  | exp009_bandroles | role objectives: **directional hit 4/4 but noise-adjacent — frequency reweighting too collinear; needs qualitatively different supervision + generation-side gauges (exp010)** | **shipped (2 seeds + rejudge)** |
81
  | exp008_multiband | band-ASSIGNED experts: **mechanism certifies — 3/3 surgical band lesions (50-200x), HIGH-noise band expert wins its band (label-corrected); uniform objective doesn't pay (roles enter exp009)** | **shipped (candidate, s0; s1 running)** |
 
76
  | exp003_sigma_registers | register probe: **sigma is the ONLY live axis** — caption/type/cond all compress to ≤0; ordering prereg MISS with design consequences | **shipped (candidate, s0)** |
77
  | exp004_anima_relay | relays on the 2B DiT via diffusion-pipe (bf16 per dtype law) | staged |
78
  | exp005_sdxl_tree4a | the SDXL capacity battery (guidepost/scaffold/skeleton) | designed |
79
+ | exp011a_fused_multiband | multiband on REAL fused data: **adapters pay 2-3x more; structural story replicates (3/3 surgical, monolith edge persists); blob targets built 100% after a schema lesson** | **shipped (candidate, s0; s1 running)** |
80
  | exp010_controller | **StepGatedSampler ships**: controller lifts grounding +0.089 over frozen; monotonic lesion ladder; HIGH lesion 14x LP-dominant (coarse-to-fine confirmed in image space); eps-trained HIGH band concentrates (diversity = open training goal) | **shipped (candidate)** |
81
  | exp009_bandroles | role objectives: **directional hit 4/4 but noise-adjacent — frequency reweighting too collinear; needs qualitatively different supervision + generation-side gauges (exp010)** | **shipped (2 seeds + rejudge)** |
82
  | exp008_multiband | band-ASSIGNED experts: **mechanism certifies — 3/3 surgical band lesions (50-200x), HIGH-noise band expert wins its band (label-corrected); uniform objective doesn't pay (roles enter exp009)** | **shipped (candidate, s0; s1 running)** |
exp011_fused_multiband/README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # exp011a_fused_multiband — the multiband mechanism on REAL DATA
2
+ # (+ the segmentation blob-target cache) — CANDIDATE s0; s1 running
3
+
4
+ **Questions.** (1) Does real fused data (multi-entity photos, natural
5
+ captions) change the band economics that uniform synthetic data couldn't pay
6
+ for? (2) Build the segmentation blob targets for the HIGH-band blobbing role
7
+ (loss design held for review — new mechanism, pacing law).
8
+
9
+ **Design.** qwen-deepfashion-fused, 4352 rows (dataset audit gates enforced;
10
+ all rows passed), caption_joycaption at nl77, same certified beds as exp008
11
+ (multiband3 vs mono48, uniform eps objective, 3000 steps, pure Adam wd=0).
12
+
13
+ **Results** (`results.json`):
14
+
15
+ | arm | overall val | band0 (LOW) | band1 (MID) | band2 (HIGH) |
16
+ |---|---|---|---|---|
17
+ | frozen | 0.142435 | 0.293931 | 0.071764 | 0.010422 |
18
+ | multiband3 | 0.135215 (−5.1%) | 0.280633 | 0.066728 | **0.009562** |
19
+ | monolith r48 | **0.134184 (−5.8%)** | **0.278893** | **0.065772** | 0.009564 |
20
+
21
+ Lesions: 3/3 surgical, with LARGER asymmetries than synthetic (band0 own
22
+ 0.0130 vs 0.000062 cross — 200×).
23
+
24
+ **Findings (candidate, s0).**
25
+ 1. **Real data pays adapters 2–3× more** than synthetic (−5.1/−5.8% vs
26
+ −2.2/−2.4% in exp008) — the richer distribution has more for small
27
+ adapters to learn.
28
+ 2. **The structural story replicates on real data**: surgical band
29
+ specialists, monolith edge persists under a uniform objective, HIGH-band
30
+ own-band win reappears at a hair (0.009562 vs 0.009564).
31
+ 3. **Blob targets: built, verified, cached** — after a schema lesson: the
32
+ first rasterizer pass returned 100% EMPTY (fused_json polygons are FLAT
33
+ [x1,y1,...] lists in a declared `coord_space`, not the assumed pair-list
34
+ NORM_0_1000; the emptiness counter caught it — silent-zero discipline).
35
+ Corrected rasterizer: 100% non-empty, mean occupancy 0.293, coord_space
36
+ norm_0_1000 confirmed on every row. `fix_blob_targets.py` in-package.
37
+
38
+ **Next.** The blob LOSS (how coarse occupancy couples to the HIGH band's
39
+ x0-estimate) goes to the 8am design review with targets ready. s1
40
+ replication of this bed lands ~07:30Z.
41
+
42
+ **Caveats.** s0; 4352 rows single dataset; blob maps unused in this
43
+ training (uniform objective by design). Cost ≈ 2.9 GPU-h incl. cache.
exp011_fused_multiband/dexp011_fused_multiband.py ADDED
@@ -0,0 +1,331 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """dexp011_fused_multiband.py — exp011a: multiband on REAL FUSED DATA
2
+ (+ the blob-target cache for the 8am blob-loss design review).
3
+
4
+ Two jobs (stage plan phase 4 prep, D2 from the reconstruction brief):
5
+ 1. Does REAL data (qwen-deepfashion-fused: real photos, joycaption NL,
6
+ multi-entity scenes) change the band economics that synthetic
7
+ single-object data couldn't pay for? Same certified beds (multiband3 vs
8
+ mono48, uniform objective), new substrate data, exp010 battery after.
9
+ 2. Build the SEGMENTATION BLOB TARGETS: fused_json entities[].mask.polygon
10
+ (NORM_0_1000) rasterized to the 64x64 latent grid, cached per row.
11
+ The blob LOSS mechanism is DELIBERATELY NOT implemented tonight —
12
+ it is a new mechanism and goes to Phil's 8am review with targets ready
13
+ (pacing law: don't step into something unprepared).
14
+
15
+ Data gates: rows filtered on the dataset's own audit columns when present
16
+ (age_audit/audit approved-only; counts printed; empty result fails LOUDLY —
17
+ never silently train on people data past an available gate).
18
+
19
+ Pod: bash pod2/run_exp011.sh
20
+ """
21
+ from __future__ import annotations
22
+
23
+ import io
24
+ import json
25
+ import os
26
+ import sys
27
+ import time
28
+
29
+ sys.path[:0] = ["pod2", "."]
30
+
31
+ import numpy as np
32
+ import torch
33
+ import torch.nn.functional as F
34
+
35
+ from pod_ledger import ledger_run, note, burn_down
36
+ from d1_substrate import MEM_FRACTION
37
+ from dexp006_sd15core_relay import make_schedule, add_noise, encode_nl77
38
+ from dexp008_multiband import (MultibandDelta, MonoDelta, band_weights,
39
+ band_of, attach, load_unet, N_BANDS)
40
+
41
+ DATASET = "AbstractPhil/qwen-deepfashion-fused"
42
+ SD_BASE = "stable-diffusion-v1-5/stable-diffusion-v1-5"
43
+ VAE_SCALE = 0.18215
44
+ N_TRAIN, N_VAL = 4096, 256
45
+ BATCH = int(os.environ.get("DEXP11_BATCH", "16"))
46
+ STEPS = int(os.environ.get("DEXP11_STEPS", "3000"))
47
+ SEED = int(os.environ.get("DEXP11_SEED", "0"))
48
+ LR, CFG_DROPOUT = 1e-3, 0.1
49
+ DATA_DIR = ("/workspace/data/dexp011" if os.path.isdir("/workspace")
50
+ else os.path.join(os.environ.get("GEOLIP_DATA", "./data"),
51
+ "dexp011"))
52
+ CKPT_DIR = ("/workspace/ckpts2/dexp011" if os.path.isdir("/workspace")
53
+ else DATA_DIR)
54
+
55
+
56
+ def rasterize_blob(fused_json_str: str, size: int = 64) -> np.ndarray:
57
+ """Entity mask polygons (NORM_0_1000) -> union occupancy on the latent
58
+ grid. Returns (size, size) float32 in [0,1]. Rows without polygons
59
+ return zeros (counted by the caller)."""
60
+ from PIL import Image, ImageDraw
61
+ try:
62
+ fj = json.loads(fused_json_str)
63
+ except Exception:
64
+ return np.zeros((size, size), dtype=np.float32)
65
+ img = Image.new("L", (size, size), 0)
66
+ draw = ImageDraw.Draw(img)
67
+ n = 0
68
+ for ent in (fj.get("entities") or []):
69
+ mask = ent.get("mask") or {}
70
+ poly = mask.get("polygon") or []
71
+ if len(poly) >= 3:
72
+ pts = [(p[0] / 1000.0 * (size - 1), p[1] / 1000.0 * (size - 1))
73
+ for p in poly if isinstance(p, (list, tuple))
74
+ and len(p) >= 2]
75
+ if len(pts) >= 3:
76
+ draw.polygon(pts, fill=255)
77
+ n += 1
78
+ arr = np.asarray(img, dtype=np.float32) / 255.0
79
+ return arr
80
+
81
+
82
+ def _iter_fused(need: int):
83
+ from huggingface_hub import HfApi, hf_hub_download
84
+ import pyarrow.parquet as pq
85
+ api = HfApi()
86
+ files = sorted(f for f in api.list_repo_files(DATASET, repo_type="dataset")
87
+ if f.endswith(".parquet"))
88
+ got, kept, gated = 0, 0, 0
89
+ cols_printed = False
90
+ for fname in files:
91
+ path = hf_hub_download(DATASET, fname, repo_type="dataset")
92
+ tbl = pq.read_table(path)
93
+ if not cols_printed:
94
+ print(f"[fused] columns: {tbl.column_names}", flush=True)
95
+ cols_printed = True
96
+ for r in tbl.to_pylist():
97
+ got += 1
98
+ gate_vals = [str(r.get(c)) for c in ("age_audit", "audit")
99
+ if c in r and r.get(c) is not None]
100
+ if gate_vals and not any(("approved" in v.lower()
101
+ or "pass" in v.lower())
102
+ for v in gate_vals):
103
+ gated += 1
104
+ continue
105
+ kept += 1
106
+ yield r
107
+ if kept >= need:
108
+ print(f"[fused] kept {kept} / seen {got} (gated out {gated})",
109
+ flush=True)
110
+ return
111
+ raise AssertionError(
112
+ f"exhausted dataset with only {kept} gated rows of {got} seen — "
113
+ f"inspect gate columns before proceeding")
114
+
115
+
116
+ def build_cache(device="cuda"):
117
+ cache_f = os.path.join(DATA_DIR, "cache.pt")
118
+ if os.path.exists(cache_f):
119
+ print(f"[cache] exists: {cache_f}")
120
+ return cache_f
121
+ os.makedirs(DATA_DIR, exist_ok=True)
122
+ from PIL import Image
123
+ from diffusers import AutoencoderKL
124
+ from transformers import CLIPTextModel, CLIPTokenizer
125
+ vae = AutoencoderKL.from_pretrained(
126
+ SD_BASE, subfolder="vae", torch_dtype=torch.float32).to(device).eval()
127
+ tok = CLIPTokenizer.from_pretrained(SD_BASE, subfolder="tokenizer")
128
+ te = CLIPTextModel.from_pretrained(
129
+ SD_BASE, subfolder="text_encoder",
130
+ torch_dtype=torch.float32).to(device).eval()
131
+ g = torch.Generator(device=device).manual_seed(SEED)
132
+ lat, ehs, blobs, jpegs = [], [], [], []
133
+ buf_img, buf_txt = [], []
134
+ n_blob_empty = 0
135
+
136
+ @torch.no_grad()
137
+ def flush():
138
+ if not buf_img:
139
+ return
140
+ px = torch.stack(buf_img).to(device)
141
+ lat.append((vae.encode(px).latent_dist.sample(generator=g)
142
+ * VAE_SCALE).cpu())
143
+ ehs.append(encode_nl77(buf_txt, tok, te, device).cpu())
144
+ buf_img.clear()
145
+ buf_txt.clear()
146
+
147
+ n = 0
148
+ for r in _iter_fused(N_TRAIN + N_VAL):
149
+ img = r["image"]
150
+ img = img.get("bytes") if isinstance(img, dict) else img
151
+ im = Image.open(io.BytesIO(img)).convert("RGB").resize((512, 512))
152
+ buf_img.append(torch.from_numpy(np.asarray(im)).float()
153
+ .permute(2, 0, 1) / 127.5 - 1.0)
154
+ cap = r.get("caption_joycaption") or r.get("prompt_fused") or ""
155
+ buf_txt.append(str(cap))
156
+ b = rasterize_blob(str(r.get("fused_json", "")))
157
+ if b.max() == 0:
158
+ n_blob_empty += 1
159
+ blobs.append(torch.from_numpy(b))
160
+ if n < 24:
161
+ jpegs.append(img)
162
+ n += 1
163
+ if len(buf_img) == 16:
164
+ flush()
165
+ if n % 512 == 0:
166
+ print(f"[cache] {n}/{N_TRAIN + N_VAL} "
167
+ f"(blob-empty {n_blob_empty})", flush=True)
168
+ flush()
169
+ lat, ehs = torch.cat(lat), torch.cat(ehs)
170
+ blobs = torch.stack(blobs)
171
+ gv = torch.Generator().manual_seed(SEED + 1)
172
+ torch.save({"lat": lat[:N_TRAIN], "ehs": ehs[:N_TRAIN],
173
+ "blob": blobs[:N_TRAIN],
174
+ "val_lat": lat[N_TRAIN:], "val_ehs": ehs[N_TRAIN:],
175
+ "val_blob": blobs[N_TRAIN:],
176
+ "val_noise": torch.randn(N_VAL, 4, 64, 64, generator=gv),
177
+ "val_t": torch.randint(0, 1000, (N_VAL,), generator=gv),
178
+ "judge_jpegs": jpegs,
179
+ "blob_empty_count": n_blob_empty}, cache_f)
180
+ print(f"[cache] built: {N_TRAIN}+{N_VAL} rows, blob-empty {n_blob_empty}",
181
+ flush=True)
182
+ del vae, te
183
+ torch.cuda.empty_cache()
184
+ return cache_f
185
+
186
+
187
+ def run(device="cuda"):
188
+ torch.cuda.set_per_process_memory_fraction(MEM_FRACTION, 0)
189
+ os.makedirs(CKPT_DIR, exist_ok=True)
190
+ acp = make_schedule(device)
191
+ with ledger_run("dexp011 fused cache + blob targets", budget_h=1.5):
192
+ build_cache(device)
193
+ cache = torch.load(os.path.join(DATA_DIR, "cache.pt"),
194
+ map_location="cpu", weights_only=True)
195
+
196
+ def set_w(wraps, s01):
197
+ w = band_weights(s01)
198
+ for wr in wraps:
199
+ wr.w_bands = w
200
+
201
+ def loss_of(unet, wraps, lat, ehs, gen):
202
+ bsz = lat.shape[0]
203
+ drop = torch.rand(bsz, generator=gen, device=device) < CFG_DROPOUT
204
+ ehs = ehs.clone()
205
+ ehs[drop] = 0
206
+ t = torch.randint(0, 1000, (bsz,), generator=gen, device=device)
207
+ set_w(wraps, t.float() / 1000.0)
208
+ noise = torch.randn(lat.shape, generator=gen, device=device)
209
+ pred = unet(add_noise(lat, noise, t, acp), t, ehs,
210
+ return_dict=False)[0]
211
+ return F.mse_loss(pred, noise)
212
+
213
+ @torch.no_grad()
214
+ def val(unet, wraps):
215
+ tot, per_band = [], {0: [], 1: [], 2: []}
216
+ for i in range(0, N_VAL, 32):
217
+ lat = cache["val_lat"][i:i + 32].to(device)
218
+ ehs = cache["val_ehs"][i:i + 32].to(device)
219
+ noise = cache["val_noise"][i:i + 32].to(device)
220
+ t = cache["val_t"][i:i + 32].to(device)
221
+ set_w(wraps, t.float() / 1000.0)
222
+ pred = unet(add_noise(lat, noise, t, acp), t, ehs,
223
+ return_dict=False)[0]
224
+ mse = ((pred - noise) ** 2).mean(dim=(1, 2, 3))
225
+ tot += mse.tolist()
226
+ for j, tv in enumerate((t.float() / 1000.0).tolist()):
227
+ per_band[band_of(tv)].append(mse[j].item())
228
+ return (sum(tot) / len(tot),
229
+ {f"band{b}": round(sum(v) / max(len(v), 1), 6)
230
+ for b, v in per_band.items()})
231
+
232
+ def train(unet, wraps, params, label):
233
+ opt = torch.optim.Adam(params, lr=LR, weight_decay=0.0)
234
+ gen = torch.Generator(device=device).manual_seed(SEED + 42)
235
+ idx = torch.Generator().manual_seed(SEED + 7)
236
+ t0 = time.time()
237
+ for step in range(1, STEPS + 1):
238
+ sel = torch.randint(0, N_TRAIN, (BATCH,), generator=idx)
239
+ loss = loss_of(unet, wraps, cache["lat"][sel].to(device),
240
+ cache["ehs"][sel].to(device), gen)
241
+ loss.backward()
242
+ opt.step()
243
+ opt.zero_grad(set_to_none=True)
244
+ if step == 50 or step % 500 == 0:
245
+ print(f"[{label}] step {step}: loss {loss.item():.4f} | "
246
+ f"{(time.time() - t0) / step:.2f}s/step", flush=True)
247
+ return round((time.time() - t0) / STEPS, 3)
248
+
249
+ results = {"config": {"dataset": DATASET, "steps": STEPS, "batch": BATCH,
250
+ "seed": SEED, "cond": "caption_joycaption nl77",
251
+ "blob_empty": int(cache["blob_empty_count"])}}
252
+ with ledger_run("dexp011 frozen", budget_h=0.2) as h:
253
+ unet = load_unet(device)
254
+ v, pb = val(unet, [])
255
+ results["frozen"] = {"val": v, "per_band": pb}
256
+ del unet
257
+ torch.cuda.empty_cache()
258
+ h["verdict"] = f"val {v:.5f}"
259
+
260
+ for label, mk, r_note in (("multiband3", lambda d: MultibandDelta(d),
261
+ "3 band experts"),
262
+ ("monolith", lambda d: MonoDelta(d), "r48")):
263
+ with ledger_run(f"dexp011 {label} s{SEED}", budget_h=2.5) as h:
264
+ unet = load_unet(device)
265
+ mods, wraps = attach(unet, mk)
266
+ n_params = sum(p.numel() for p in mods.parameters())
267
+ spd = train(unet, wraps, mods.parameters(), label)
268
+ v, pb = val(unet, wraps)
269
+ entry = {"n_params": n_params, "val": v, "per_band": pb,
270
+ "s_per_step": spd}
271
+ if label == "multiband3":
272
+ lesions = {}
273
+ for b in range(N_BANDS):
274
+ for m in mods:
275
+ m.band_enabled[b] = False
276
+ _, pb_les = val(unet, wraps)
277
+ lesions[f"lesion_band{b}"] = pb_les
278
+ for m in mods:
279
+ m.band_enabled[b] = True
280
+ entry["lesions"] = lesions
281
+ torch.save({"mods": [m.state_dict() for m in mods]},
282
+ os.path.join(CKPT_DIR, f"{label}_s{SEED}.pt"))
283
+ results[label] = entry
284
+ del unet, mods
285
+ torch.cuda.empty_cache()
286
+ h["verdict"] = f"val {v:.5f}"
287
+
288
+ mb, mo = results["multiband3"], results["monolith"]
289
+ lesion_sur = {}
290
+ for b in range(N_BANDS):
291
+ les = mb["lesions"][f"lesion_band{b}"]
292
+ own = les[f"band{b}"] - mb["per_band"][f"band{b}"]
293
+ oth = sum(les[f"band{o}"] - mb["per_band"][f"band{o}"]
294
+ for o in range(N_BANDS) if o != b) / (N_BANDS - 1)
295
+ lesion_sur[f"band{b}"] = {"own": round(own, 6),
296
+ "other": round(oth, 6),
297
+ "surgical": own > 3 * abs(oth) and own > 0}
298
+ results["verdict"] = {
299
+ "mb_vs_mono": "multiband" if mb["val"] <= mo["val"] else "monolith",
300
+ "per_band_wins": {b: mb["per_band"][b] < mo["per_band"][b]
301
+ for b in ("band0", "band1", "band2")},
302
+ "lesions": lesion_sur,
303
+ "note": "REAL fused data (D2); blob targets cached for the 8am "
304
+ "blob-loss design review; 1-seed CANDIDATE",
305
+ }
306
+ with open(os.path.join(DATA_DIR, "results.json" if SEED == 0
307
+ else f"results_s{SEED}.json"), "w") as f:
308
+ json.dump(results, f, indent=2)
309
+ note(f"dexp011: {json.dumps(results['verdict']['mb_vs_mono'])}")
310
+ print(json.dumps(results["verdict"], indent=2))
311
+ burn_down()
312
+ return results
313
+
314
+
315
+ def smoke():
316
+ fj = json.dumps({"entities": [
317
+ {"mask": {"polygon": [[100, 100], [900, 100], [900, 900],
318
+ [100, 900]]}},
319
+ {"mask": {"polygon": [[0, 0], [200, 0], [100, 200]]}}]})
320
+ b = rasterize_blob(fj)
321
+ assert b.shape == (64, 64) and 0.5 < b.mean() < 0.9, b.mean()
322
+ assert rasterize_blob("not json").max() == 0
323
+ assert rasterize_blob(json.dumps({"entities": []})).max() == 0
324
+ print("dexp011 smoke PASSED (blob rasterizer; GPU run is pod work)")
325
+
326
+
327
+ if __name__ == "__main__":
328
+ if "--run" in sys.argv:
329
+ run()
330
+ else:
331
+ smoke()
exp011_fused_multiband/fix_blob_targets.py ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """fix_blob_targets.py — rebuild exp011's blob targets with the CORRECT
2
+ fused_json schema (diagnosed 2026-07-18: polygon is a FLAT [x1,y1,x2,y2,...]
3
+ list, and coordinates live in the declared coord_space with image_size — not
4
+ the assumed NORM_0_1000 pair-list). Row order replicates _iter_fused exactly
5
+ (same files, same gates); patches blob/val_blob into the existing cache.
6
+ LOUD assert: >50% of rows must produce non-empty blobs. CPU-only.
7
+ """
8
+ from __future__ import annotations
9
+
10
+ import json
11
+ import os
12
+ import sys
13
+
14
+ sys.path[:0] = ["pod2", "."]
15
+
16
+ import numpy as np
17
+ import torch
18
+
19
+ from dexp011_fused_multiband import _iter_fused, N_TRAIN, N_VAL, DATA_DIR
20
+
21
+
22
+ def rasterize_v2(fused_json_str: str, size: int = 64) -> np.ndarray:
23
+ from PIL import Image, ImageDraw
24
+ try:
25
+ fj = json.loads(fused_json_str)
26
+ except Exception:
27
+ return np.zeros((size, size), dtype=np.float32)
28
+ cs = str(fj.get("coord_space", "")).lower()
29
+ im_size = fj.get("image_size")
30
+ if isinstance(im_size, (list, tuple)) and len(im_size) >= 2:
31
+ W, H = float(im_size[0]) or None, float(im_size[1]) or None
32
+ elif isinstance(im_size, dict):
33
+ W = float(im_size.get("width") or im_size.get("w") or 0) or None
34
+ H = float(im_size.get("height") or im_size.get("h") or 0) or None
35
+ else:
36
+ W = H = None
37
+ img = Image.new("L", (size, size), 0)
38
+ draw = ImageDraw.Draw(img)
39
+ for ent in (fj.get("entities") or []):
40
+ poly = (ent.get("mask") or {}).get("polygon") or []
41
+ if len(poly) >= 6 and all(isinstance(v, (int, float))
42
+ for v in poly[:6]):
43
+ xs, ys = poly[0::2], poly[1::2]
44
+ if "1000" in cs or (max(xs + ys) <= 1000 and W is None):
45
+ sx = sy = 1.0 / 1000.0
46
+ elif W and H:
47
+ sx, sy = 1.0 / W, 1.0 / H
48
+ else:
49
+ m = max(max(xs), max(ys), 1.0)
50
+ sx = sy = 1.0 / m
51
+ pts = [(min(max(x * sx, 0), 1) * (size - 1),
52
+ min(max(y * sy, 0), 1) * (size - 1))
53
+ for x, y in zip(xs, ys)]
54
+ if len(pts) >= 3:
55
+ draw.polygon(pts, fill=255)
56
+ return np.asarray(img, dtype=np.float32) / 255.0
57
+
58
+
59
+ def main():
60
+ blobs, n_empty, cs_seen = [], 0, {}
61
+ for r in _iter_fused(N_TRAIN + N_VAL):
62
+ s = str(r.get("fused_json", ""))
63
+ b = rasterize_v2(s)
64
+ if b.max() == 0:
65
+ n_empty += 1
66
+ try:
67
+ cs = json.loads(s).get("coord_space", "?")
68
+ cs_seen[str(cs)] = cs_seen.get(str(cs), 0) + 1
69
+ except Exception:
70
+ pass
71
+ blobs.append(torch.from_numpy(b))
72
+ blobs = torch.stack(blobs)
73
+ frac = 1 - n_empty / len(blobs)
74
+ print(f"[fix_blob] non-empty {frac:.1%} (empty {n_empty}/{len(blobs)}); "
75
+ f"coord_spaces {cs_seen}; mean occupancy "
76
+ f"{blobs.mean().item():.3f}", flush=True)
77
+ assert frac > 0.5, f"still mostly empty ({frac:.1%}) — schema wrong again"
78
+ f = os.path.join(DATA_DIR, "cache.pt")
79
+ cache = torch.load(f, map_location="cpu", weights_only=True)
80
+ cache["blob"] = blobs[:N_TRAIN]
81
+ cache["val_blob"] = blobs[N_TRAIN:]
82
+ cache["blob_empty_count"] = n_empty
83
+ torch.save(cache, f)
84
+ print(f"[fix_blob] cache patched: {f}", flush=True)
85
+
86
+
87
+ if __name__ == "__main__":
88
+ main()
exp011_fused_multiband/results.json ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "dataset": "AbstractPhil/qwen-deepfashion-fused",
4
+ "steps": 3000,
5
+ "batch": 16,
6
+ "seed": 0,
7
+ "cond": "caption_joycaption nl77",
8
+ "blob_empty": 4352
9
+ },
10
+ "frozen": {
11
+ "val": 0.14243539798007987,
12
+ "per_band": {
13
+ "band0": 0.293931,
14
+ "band1": 0.071764,
15
+ "band2": 0.010422
16
+ }
17
+ },
18
+ "multiband3": {
19
+ "n_params": 1235568,
20
+ "val": 0.1352152533709159,
21
+ "per_band": {
22
+ "band0": 0.280633,
23
+ "band1": 0.066728,
24
+ "band2": 0.009562
25
+ },
26
+ "s_per_step": 1.688,
27
+ "lesions": {
28
+ "lesion_band0": {
29
+ "band0": 0.293597,
30
+ "band1": 0.066852,
31
+ "band2": 0.009562
32
+ },
33
+ "lesion_band1": {
34
+ "band0": 0.280774,
35
+ "band1": 0.071368,
36
+ "band2": 0.00964
37
+ },
38
+ "lesion_band2": {
39
+ "band0": 0.280633,
40
+ "band1": 0.066757,
41
+ "band2": 0.010283
42
+ }
43
+ }
44
+ },
45
+ "monolith": {
46
+ "n_params": 1210576,
47
+ "val": 0.1341835820844608,
48
+ "per_band": {
49
+ "band0": 0.278893,
50
+ "band1": 0.065772,
51
+ "band2": 0.009564
52
+ },
53
+ "s_per_step": 1.654
54
+ },
55
+ "verdict": {
56
+ "mb_vs_mono": "monolith",
57
+ "per_band_wins": {
58
+ "band0": false,
59
+ "band1": false,
60
+ "band2": true
61
+ },
62
+ "lesions": {
63
+ "band0": {
64
+ "own": 0.012964,
65
+ "other": 6.2e-05,
66
+ "surgical": true
67
+ },
68
+ "band1": {
69
+ "own": 0.00464,
70
+ "other": 0.00011,
71
+ "surgical": true
72
+ },
73
+ "band2": {
74
+ "own": 0.000721,
75
+ "other": 1.5e-05,
76
+ "surgical": true
77
+ }
78
+ },
79
+ "note": "REAL fused data (D2); blob targets cached for the 8am blob-loss design review; 1-seed CANDIDATE"
80
+ }
81
+ }