Spaces:

WitneyWW
/

wan22-vae-flow-vs-control

Running

App Files Files Community

WitneyWW commited on Jul 16

Commit

533b8a8

verified ·

1 Parent(s): bdae779

Flow-weighted LoRA vs 6fps control: recon, diff, motion split (clip + 32 val clips)

Browse files

Files changed (13) hide show

.gitattributes +8 -0
README.md +34 -4
diff_base.mp4 +3 -0
diff_flow.mp4 +3 -0
diff_fps6.mp4 +3 -0
index.html +134 -18
metrics.json +50 -0
motion_weight.mp4 +3 -0
original.mp4 +3 -0
recon_base.mp4 +3 -0
recon_flow.mp4 +3 -0
recon_fps6.mp4 +3 -0
val_split.json +14 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+diff_base.mp4 filter=lfs diff=lfs merge=lfs -text
+diff_flow.mp4 filter=lfs diff=lfs merge=lfs -text
+diff_fps6.mp4 filter=lfs diff=lfs merge=lfs -text
+motion_weight.mp4 filter=lfs diff=lfs merge=lfs -text
+original.mp4 filter=lfs diff=lfs merge=lfs -text
+recon_base.mp4 filter=lfs diff=lfs merge=lfs -text
+recon_flow.mp4 filter=lfs diff=lfs merge=lfs -text
+recon_fps6.mp4 filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -1,10 +1,40 @@
 ---
-title: Wan22 Vae Flow Vs Control
-emoji: 📈
 colorFrom: indigo
-colorTo: green
 sdk: static
 pinned: false
 ---
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

 ---
+title: Wan2.2 VAE — Flow-Weighted LoRA vs Control
+emoji: 🌊
 colorFrom: indigo
+colorTo: pink
 sdk: static
 pinned: false
 ---
+# Wan2.2 VAE — flow-weighted LoRA vs control
+A GelSight tactile clip (`CutTofu_1_tactile_left.mp4`) reconstructed through the
+Wan2.2 causal 3D VAE three ways: the **frozen base**, a **control** LoRA, and a
+**flow-weighted** LoRA.
+The two adapters' training configs differ in exactly one field — `flow_weight` (0 vs 1).
+Everything else is identical: 6fps (stride 5 from 30fps sources),
+r=32 α=32, all conv targets, 4000 steps, edge-gradient L1 + LPIPS 0.1. Holding fps fixed
+matters: fps alone is worth ~4 dB and confounded the earlier comparison.
+Flow weighting multiplies the L1 term by optical-flow magnitude, spending capacity on
+moving regions at the expense of static ones. **Overall PSNR cannot show this** — it
+averages every pixel, so a pure redistribution nets out. The page therefore splits PSNR by
+the same motion magnitude the loss was weighted with (`m̂ ≥ 0.5` moving,
+`≤ 0.1` static), which is the measurement with the power to discriminate.
+## Result: it didn't work
+Over **32 held-out val clips**, the flow-weighted adapter is **-0.28 dB** on moving regions — the regions it was built to improve — while static regions gain +0.20 dB. The intended trade ran backwards.
+The single clip on this page happens to disagree (+0.12 dB on moving), which is exactly why both are shown: a tenth of a dB on one clip is not separable from clip-to-clip variation.
+Caveat in the other direction: one training run per arm (seed 0), so a 0.28 dB gap does not rule out run-to-run variance. The honest claim is *no evidence flow weighting helps moving regions*, not proof that it hurts.
+Files: `original.mp4`, `recon_{base,fps6,flow}.mp4`, `diff_{base,fps6,flow}.mp4`
+(|original − recon| ×20, hot colormap), `motion_weight.mp4` (the m̂ map),
+`metrics.json`.
+Reconstruction is at **6fps** — the rate the adapters trained at. Evaluating them at the
+source's native 30fps would measure a domain mismatch instead of the adapter.

diff_base.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:38dcd8d8f95d6efcdc0d7d77bd282402af4494614c78c64e2b8e541100dc6ac6
+size 3187276

diff_flow.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:03c7756778317bd1825d3ddff1fcb9210f1431bff36574f1a987826567424d2f
+size 3050378

diff_fps6.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6b2c8fc6490294eb740f3c961d871a32d9190d9158ddbe192e4ec80993257c98
+size 3056291

index.html CHANGED Viewed

@@ -1,19 +1,135 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
 </html>

+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8" />
+<meta name="viewport" content="width=device-width, initial-scale=1" />
+<title>Wan2.2 VAE — Flow-Weighted LoRA vs Control</title>
+<style>
+  :root {
+    --bg:#0f1115; --card:#191c23; --border:#2a2f3a; --text:#e8eaed; --muted:#9aa1ac;
+    --accent:#a78bfa; --head:#12141a; --good:#34d399; --bad:#f87171; --warn:#fbbf24;
+  }
+  @media (prefers-color-scheme: light) {
+    :root { --bg:#f6f7f9; --card:#fff; --border:#e3e6eb; --text:#1a1d23; --muted:#5c6472;
+            --accent:#7c3aed; --head:#eef0f4; --good:#059669; --bad:#dc2626; --warn:#b45309; }
+  }
+  * { box-sizing: border-box; }
+  body { margin:0; background:var(--bg); color:var(--text); line-height:1.5;
+         font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; }
+  header { padding:32px 24px 4px; max-width:1200px; margin:0 auto; }
+  h1 { margin:0 0 4px; font-size:1.5rem; }
+  .sub { color:var(--muted); font-size:0.95rem; }
+  .bar { max-width:1200px; margin:12px auto 0; padding:0 24px; display:flex; gap:8px; align-items:center; }
+  .spacer { flex:1; }
+  button.action { background:transparent; color:var(--text); border:1px solid var(--border);
+                  border-radius:8px; padding:6px 14px; font-size:0.88rem; cursor:pointer; }
+  button.action:hover { border-color:var(--accent); color:var(--accent); }
+  .meta { max-width:1200px; margin:8px auto 0; padding:0 24px; color:var(--muted); font-size:0.85rem; }
+  .meta code, .finding code { font-family:ui-monospace, SFMono-Regular, Menlo, monospace; color:var(--accent); }
+  section { max-width:1200px; margin:0 auto; padding:4px 24px; }
+  h2 { font-size:0.8rem; text-transform:uppercase; letter-spacing:0.06em; color:var(--muted);
+       margin:22px 0 10px; padding-bottom:6px; border-bottom:1px solid var(--border); }
+  .grid { display:grid; grid-template-columns:repeat(auto-fit, minmax(300px, 1fr)); gap:18px; }
+  .card { background:var(--card); border:1px solid var(--border); border-radius:12px;
+          overflow:hidden; display:flex; flex-direction:column; }
+  .card .head { background:var(--head); padding:9px 14px; border-bottom:1px solid var(--border);
+                display:flex; align-items:baseline; gap:8px; flex-wrap:wrap; }
+  .card .head .name { font-weight:700; font-size:0.95rem;
+                      font-family:ui-monospace, SFMono-Regular, Menlo, monospace; }
+  .card .head .desc { color:var(--muted); font-size:0.8rem; }
+  .card video { width:100%; display:block; background:#000; aspect-ratio:1/1; }
+  .card .metric { padding:8px 14px; font-size:0.82rem; color:var(--muted);
+                  font-family:ui-monospace, SFMono-Regular, Menlo, monospace;
+                  border-top:1px solid var(--border); }
+  .card .metric b { color:var(--text); }
+  .delta.up { color:var(--good); } .delta.down { color:var(--bad); } .delta.flat { color:var(--muted); }
+  .tag { display:inline-block; font-size:0.68rem; padding:1px 8px; border-radius:999px;
+         border:1px solid var(--border); color:var(--accent); }
+  .finding { background:var(--card); border:1px solid var(--border); border-left:3px solid var(--warn);
+             border-radius:8px; padding:14px 16px; margin:16px 0; font-size:0.9rem; }
+  .finding h3 { margin:0 0 6px; font-size:0.9rem; }
+  .tbl-wrap { overflow-x:auto; }
+  table { border-collapse:collapse; width:100%; font-size:0.86rem;
+          font-family:ui-monospace, SFMono-Regular, Menlo, monospace; }
+  th, td { text-align:right; padding:8px 12px; border-bottom:1px solid var(--border); white-space:nowrap; }
+  th:first-child, td:first-child { text-align:left; }
+  thead th { color:var(--muted); font-weight:600; font-size:0.76rem;
+             text-transform:uppercase; letter-spacing:0.05em; }
+  tbody tr:last-child td { border-bottom:none; }
+  tr.delta-row td { border-top:1px solid var(--border); font-weight:700; }
+  footer { max-width:1200px; margin:0 auto; padding:24px; color:var(--muted); font-size:0.8rem; }
+</style>
+</head>
+<body>
+  <header>
+    <h1>Wan2.2 VAE — Flow-Weighted LoRA <span class="tag">A/B vs control</span></h1>
+    <div class="sub">Does weighting the reconstruction loss by optical-flow magnitude buy
+    sharper <em>moving</em> regions? Two adapters whose configs differ in exactly one field.</div>
+  </header>
+  <div class="bar">
+    <div class="spacer"></div>
+    <button class="action" id="toggle">⏯ Play / pause all</button>
+  </div>
+  <div class="meta">
+    Source: <code>CutTofu_1_tactile_left.mp4</code> ·
+    Clip: <code>77 frames · 256×256 · 6fps
+    (stride 5 from 390 native @ 30fps)</code> ·
+    Adapters: <code>lora_last.pt · r32 α32 targets=all · step 4000</code>
+  </div>
+  <section><div class="finding"><h3>Flow weighting did not do what it was built to do</h3>Across <b>32 held-out val clips</b>, the flow-weighted adapter is <b>-0.28 dB worse</b> on exactly the moving regions it was supposed to improve, while static regions gain +0.20 dB. That is the intended trade <em>running backwards</em>: capacity moved away from motion, not toward it.<br><br>This single clip disagrees — it shows moving +0.12 dB. That is the point of quoting both: one clip is not enough to separate a tenth of a dB from clip-to-clip variation, and picking the clip that agrees with the hypothesis would have manufactured a positive result.<br><br><b>Caveat that cuts the other way:</b> this is one training run per arm (seed 0). A 0.28 dB gap is small enough that run-to-run variance is not ruled out. The honest reading is <em>no evidence flow weighting helps moving regions</em> — not proof that it hurts.<br><br>Overall PSNR was never going to settle this: it averages every pixel, so a pure redistribution nets out. Only the <b>moving</b> column can discriminate.</div></section>
+  <section>
+    <h2>Reconstruction</h2>
+    <div class="grid"><div class="card"><div class="head"><span class="name">original</span><span class="desc">resized + subsampled input</span></div><video src="original.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">reference · 77 frames @ 256×256</div></div><div class="card"><div class="head"><span class="name">base Wan2.2</span><span class="desc">frozen VAE, no adapter</span></div><video src="recon_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>35.84 dB</b> · MSE <b>2.61e-04</b></div></div><div class="card"><div class="head"><span class="name">control · flow_weight 0</span><span class="desc">6fps, edge+LPIPS only</span></div><video src="recon_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>38.61 dB</b> · MSE <b>1.38e-04</b> · <span class="delta up">+2.77 dB vs base</span></div></div><div class="card"><div class="head"><span class="name">treatment · flow_weight 1</span><span class="desc">6fps, L1 weighted by motion</span></div><video src="recon_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>38.64 dB</b> · MSE <b>1.37e-04</b> · <span class="delta up">+2.80 dB vs base</span></div></div></div>
+  </section>
+  <section>
+    <h2>Error maps</h2>
+    <div class="grid"><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − base Wan2.2|</span></div><video src="diff_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − control · flow_weight 0|</span></div><video src="diff_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − treatment · flow_weight 1|</span></div><video src="diff_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">motion weight</span><span class="desc">m̂ from optical flow</span></div><video src="motion_weight.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">what <b>flow</b> up-weighted · 5.1% of pixels are “moving”</div></div></div>
+  </section>
+  <section>
+    <h2>PSNR split by motion · this clip</h2>
+    <div class="tbl-wrap"><table>
+      <thead><tr><th>adapter</th><th>moving (m̂≥0.5)</th>
+      <th>static (m̂≤0.1)</th><th>all pixels</th></tr></thead>
+      <tbody><tr><td>base (frozen)</td><td>31.43</td><td>37.14</td><td>35.84</td></tr><tr><td>control (flow_weight 0)</td><td>34.30</td><td>39.87</td><td>38.61</td></tr><tr><td>flow (flow_weight 1)</td><td>34.42</td><td>39.87</td><td>38.64</td></tr><tr class="delta-row"><td>flow − control</td><td class="delta up">+0.12 dB</td><td class="delta flat">-0.00 dB</td><td class="delta up">+0.04 dB</td></tr></tbody>
+    </table></div>
+  </section>
+  <section>
+    <h2>Val set · 32 held-out clips</h2>
+    <div class="meta" style="padding:0 0 10px">
+      The clip above is one sample; this is the population. Same motion split, same
+      <code>lora_last.pt</code> adapters, 32 val clips at 6fps
+      (4.8% of pixels moving) ·
+      <code>compare_flow_adapters.py</code>
+    </div>
+    <div class="tbl-wrap"><table>
+      <thead><tr><th>adapter</th><th>moving (m̂≥0.5)</th>
+      <th>static (m̂≤0.1)</th><th>all pixels</th></tr></thead>
+      <tbody><tr><td>base (frozen)</td><td>34.04</td><td>36.77</td><td>36.48</td></tr><tr><td>control (flow_weight 0)</td><td>36.83</td><td>39.96</td><td>39.62</td></tr><tr><td>flow (flow_weight 1)</td><td>36.55</td><td>40.16</td><td>39.84</td></tr><tr class="delta-row"><td>flow − control</td><td class="delta down">-0.28 dB</td><td class="delta up">+0.20 dB</td><td class="delta up">+0.22 dB</td></tr></tbody>
+    </table></div>
+  </section>
+  <footer>
+    Both adapters: LoRA on the Wan2.2 causal 3D VAE (<code>Wan-AI/Wan2.2-TI2V-5B</code>),
+    r=32 α=32, all conv targets, 4000 steps at 6fps, edge-gradient L1 + LPIPS 0.1.
+    The only difference is <code>flow_weight</code> (0 vs 1); fps is held fixed, so the
+    ~4 dB fps effect is not a confound here. Judged from <code>lora_last.pt</code> because
+    <code>lora_best.pt</code> is selected on <em>unweighted</em> val MSE, which works against
+    the flow objective. Diff videos are |original − recon| amplified ×20
+    through a hot colormap.
+  </footer>
+<script>
+  document.getElementById('toggle').addEventListener('click', () => {
+    const vids = [...document.querySelectorAll('video')];
+    const anyPaused = vids.some(v => v.paused);
+    vids.forEach(v => anyPaused ? v.play().catch(() => {}) : v.pause());
+  });
+</script>
+</body>
 </html>

metrics.json ADDED Viewed

	@@ -0,0 +1,50 @@

+{
+  "input": "/group2/ct/weihanx/world_model/data/videos/freetacman/CutTofu/CutTofu_1_tactile_left.mp4",
+  "n_native_frames": 390,
+  "stride": 5,
+  "num_frames": 77,
+  "fps": 6.0,
+  "src_fps": 30.0,
+  "height": 256,
+  "width": 256,
+  "diff_amp": 20.0,
+  "flow_floor": 1.5,
+  "hi": 0.5,
+  "lo": 0.1,
+  "moving_pixel_frac": 0.051228512078523636,
+  "arms": [
+    {
+      "name": "base",
+      "adapter": null,
+      "step": null,
+      "mse": 0.0002605195331852883,
+      "psnr": 35.8415970869247,
+      "split": {
+        "moving": 31.427666249287004,
+        "static": 37.143628060008346
+      }
+    },
+    {
+      "name": "fps6",
+      "adapter": "./lora_out_fps6/lora_last.pt",
+      "step": 4000,
+      "mse": 0.00013778584252577275,
+      "psnr": 38.60795403827545,
+      "split": {
+        "moving": 34.299859705201655,
+        "static": 39.87059917677676
+      }
+    },
+    {
+      "name": "flow",
+      "adapter": "./lora_out_flow/lora_last.pt",
+      "step": 4000,
+      "mse": 0.00013665476581081748,
+      "psnr": 38.6437521776387,
+      "split": {
+        "moving": 34.4247495927023,
+        "static": 39.86891655585387
+      }
+    }
+  ]
+}

motion_weight.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2199c4d7b31979a1bfa89b184b601172b5ad56ec5c88b100bb125d803200a082
+size 131181

original.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2f49987779bac0426c6656e4e2a5c3b5ae4aa76787800a13f40381356132322a
+size 409379

recon_base.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fd004e9be5fb2b1a6605ed953370293ab9a0613400e7961c26ad60fef8401b71
+size 315528

recon_flow.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:689f3e2b5b10103bc90bcb23249036bc0425c19f919f8b48612988c26d8566a7
+size 268690

recon_fps6.mp4 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:adfad78bad4982a5d6b32f0e140221215405087f0a1803d068568f04a8b138c9
+size 268259

val_split.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "source": "compare_flow_adapters.py --adapters ./lora_out_fps6/lora_last.pt ./lora_out_flow/lora_last.pt --fps 6 --num-clips 32",
+  "log": "slurm-logs/cmp_flow_val.log",
+  "num_clips": 32,
+  "fps": 6,
+  "hi": 0.5,
+  "lo": 0.1,
+  "moving_pixel_frac": 0.048,
+  "arms": [
+    {"name": "base",  "label": "base (frozen)",           "moving": 34.04, "static": 36.77, "all": 36.48},
+    {"name": "fps6",  "label": "control (flow_weight 0)", "moving": 36.83, "static": 39.96, "all": 39.62},
+    {"name": "flow",  "label": "flow (flow_weight 1)",    "moving": 36.55, "static": 40.16, "all": 39.84}
+  ]
+}