WitneyWW's picture
Add 0510 ep003 tactile_right, 0511 ep003 tactile_left/right (contact-picked windows)
f523bef verified
Raw
History Blame Contribute Delete
24.2 kB
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Wan2.2 VAE — Flow-Weighted LoRA vs Control</title>
<style>
:root {
--bg:#0f1115; --card:#191c23; --border:#2a2f3a; --text:#e8eaed; --muted:#9aa1ac;
--accent:#a78bfa; --head:#12141a; --good:#34d399; --bad:#f87171; --warn:#fbbf24;
}
@media (prefers-color-scheme: light) {
:root { --bg:#f6f7f9; --card:#fff; --border:#e3e6eb; --text:#1a1d23; --muted:#5c6472;
--accent:#7c3aed; --head:#eef0f4; --good:#059669; --bad:#dc2626; --warn:#b45309; }
}
* { box-sizing: border-box; }
body { margin:0; background:var(--bg); color:var(--text); line-height:1.5;
font-family:-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; }
header { padding:32px 24px 4px; max-width:1200px; margin:0 auto; }
h1 { margin:0 0 4px; font-size:1.5rem; }
.sub { color:var(--muted); font-size:0.95rem; }
.bar { max-width:1200px; margin:12px auto 0; padding:0 24px;
display:flex; flex-wrap:wrap; gap:8px; align-items:center; }
.tabs { display:flex; gap:6px; flex-wrap:wrap; }
.tab, button.action { background:transparent; color:var(--text); border:1px solid var(--border);
border-radius:8px; padding:6px 14px; font-size:0.88rem; cursor:pointer; }
.tab:hover, button.action:hover { border-color:var(--accent); color:var(--accent); }
.tab.active { background:var(--accent); border-color:var(--accent); color:#fff; font-weight:600; }
.spacer { flex:1; }
.meta { max-width:1200px; margin:8px auto 0; padding:0 24px; color:var(--muted); font-size:0.85rem; }
.meta code, .finding code { font-family:ui-monospace, SFMono-Regular, Menlo, monospace; color:var(--accent); }
.clip { display:none; } .clip.active { display:block; }
section { max-width:1200px; margin:0 auto; padding:4px 24px; }
h2 { font-size:0.8rem; text-transform:uppercase; letter-spacing:0.06em; color:var(--muted);
margin:22px 0 10px; padding-bottom:6px; border-bottom:1px solid var(--border); }
.grid { display:grid; grid-template-columns:repeat(auto-fit, minmax(300px, 1fr)); gap:18px; }
.card { background:var(--card); border:1px solid var(--border); border-radius:12px;
overflow:hidden; display:flex; flex-direction:column; }
.card .head { background:var(--head); padding:9px 14px; border-bottom:1px solid var(--border);
display:flex; align-items:baseline; gap:8px; flex-wrap:wrap; }
.card .head .name { font-weight:700; font-size:0.95rem;
font-family:ui-monospace, SFMono-Regular, Menlo, monospace; }
.card .head .desc { color:var(--muted); font-size:0.8rem; }
.card video { width:100%; display:block; background:#000; aspect-ratio:1/1; }
.card .metric { padding:8px 14px; font-size:0.82rem; color:var(--muted);
font-family:ui-monospace, SFMono-Regular, Menlo, monospace;
border-top:1px solid var(--border); }
.card .metric b { color:var(--text); }
.delta.up { color:var(--good); } .delta.down { color:var(--bad); } .delta.flat { color:var(--muted); }
.tag { display:inline-block; font-size:0.68rem; padding:1px 8px; border-radius:999px;
border:1px solid var(--border); color:var(--accent); }
.finding { background:var(--card); border:1px solid var(--border); border-left:3px solid var(--warn);
border-radius:8px; padding:14px 16px; margin:16px 0; font-size:0.9rem; }
.finding h3 { margin:0 0 6px; font-size:0.95rem; }
.note { color:var(--muted); font-size:0.85rem; margin:10px 0 0; }
.tbl-wrap { overflow-x:auto; }
table { border-collapse:collapse; width:100%; font-size:0.86rem;
font-family:ui-monospace, SFMono-Regular, Menlo, monospace; }
th, td { text-align:right; padding:8px 12px; border-bottom:1px solid var(--border); white-space:nowrap; }
th:first-child, td:first-child { text-align:left; }
thead th { color:var(--muted); font-weight:600; font-size:0.76rem;
text-transform:uppercase; letter-spacing:0.05em; }
tbody tr:last-child td { border-bottom:none; }
tr.delta-row td { border-top:1px solid var(--border); font-weight:700; }
footer { max-width:1200px; margin:0 auto; padding:24px; color:var(--muted); font-size:0.8rem; }
</style>
</head>
<body>
<header>
<h1>Wan2.2 VAE — Flow-Weighted LoRA <span class="tag">A/B vs control</span></h1>
<div class="sub">Does weighting the reconstruction loss by optical-flow magnitude buy
sharper <em>moving</em> regions? Two adapters whose configs differ in exactly one field.</div>
</header>
<div class="bar">
<div class="tabs"><button class="tab active" data-clip="clip0">CutTofu · tactile_left</button><button class="tab" data-clip="clip1">0510 ep000 · tactile_right</button><button class="tab" data-clip="clip2">0510 ep003 · tactile_right</button><button class="tab" data-clip="clip3">0511 ep003 · tactile_left</button><button class="tab" data-clip="clip4">0511 ep003 · tactile_right</button></div>
<div class="spacer"></div>
<button class="action" id="toggle">⏯ Play / pause visible</button>
</div>
<section><div class="finding"><h3>Flow weighting did not do what it was built to do</h3>Across <b>32 held-out val clips</b>, the flow-weighted adapter is <b>-0.28 dB</b> on exactly the moving regions it was supposed to improve, while static regions gain +0.20 dB. That is the intended trade <em>running backwards</em>: capacity moved away from motion, not toward it.<br><br>The <b>CutTofu</b> clip disagrees — it shows moving +0.12 dB. That is why both are shown: one clip cannot separate a tenth of a dB from clip-to-clip variation, and picking the clip that agrees with the hypothesis would have manufactured a positive result.<br><br><b>Caveat that cuts the other way:</b> one training run per arm (seed 0). A 0.28 dB gap is small enough that run-to-run variance is not ruled out. The honest reading is <em>no evidence flow weighting helps moving regions</em> — not proof that it hurts.<br><br>Overall PSNR was never going to settle this: it averages every pixel, so a pure redistribution nets out. Only the <b>moving</b> column can discriminate. Both adapters do beat the frozen base by ~2.8 dB — the LoRA finetune works; only the flow weighting is a wash.</div></section>
<div class="clip active" id="clip0">
<div class="meta">
Source: <code>CutTofu_1_tactile_left.mp4</code> ·
Clip: <code>77 frames · 256×256 · 6fps (stride 5 from 390 native
@ 30fps)</code> ·
Adapters: <code>lora_last.pt · r32 α32 targets=all · step 4000</code>
</div>
<section>
<h2>Reconstruction</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">original</span><span class="desc">resized + subsampled input</span></div><video src="original.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">reference · 77 frames @ 256×256</div></div><div class="card"><div class="head"><span class="name">base Wan2.2</span><span class="desc">frozen VAE, no adapter</span></div><video src="recon_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>35.84 dB</b> · MSE <b>2.61e-04</b></div></div><div class="card"><div class="head"><span class="name">control · flow_weight 0</span><span class="desc">6fps, edge+LPIPS only</span></div><video src="recon_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>38.61 dB</b> · MSE <b>1.38e-04</b> · <span class="delta up">+2.77 dB vs base</span></div></div><div class="card"><div class="head"><span class="name">treatment · flow_weight 1</span><span class="desc">6fps, L1 weighted by motion</span></div><video src="recon_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>38.64 dB</b> · MSE <b>1.37e-04</b> · <span class="delta up">+2.80 dB vs base</span></div></div></div>
</section>
<section>
<h2>Error maps</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − base Wan2.2|</span></div><video src="diff_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − control · flow_weight 0|</span></div><video src="diff_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − treatment · flow_weight 1|</span></div><video src="diff_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">motion weight</span><span class="desc">m̂ from optical flow</span></div><video src="motion_weight.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">what <b>flow</b> up-weighted · 5.1% of pixels moving</div></div></div>
</section>
<section>
<h2>PSNR split by motion · this clip</h2>
<div class="tbl-wrap"><table>
<thead><tr><th>adapter</th><th>moving (m̂≥0.5)</th>
<th>static (m̂≤0.1)</th><th>all pixels</th></tr></thead>
<tbody><tr><td>base (frozen)</td><td>31.43</td><td>37.14</td><td>35.84</td></tr><tr><td>control (flow_weight 0)</td><td>34.30</td><td>39.87</td><td>38.61</td></tr><tr><td>flow (flow_weight 1)</td><td>34.42</td><td>39.87</td><td>38.64</td></tr><tr class="delta-row"><td>flow − control</td><td class="delta up">+0.12 dB</td><td class="delta flat">-0.00 dB</td><td class="delta up">+0.04 dB</td></tr></tbody>
</table></div>
</section>
</div>
<div class="clip" id="clip1">
<div class="meta">
Source: <code>tactile_right.mp4</code> ·
Clip: <code>117 frames · 256×256 · 6fps · first 20s (stride 5 from 600 native
@ 30fps)</code> ·
Adapters: <code>lora_last.pt · r32 α32 targets=all · step 4000</code>
</div>
<section>
<h2>Reconstruction</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">original</span><span class="desc">resized + subsampled input</span></div><video src="mb_original.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">reference · 117 frames @ 256×256</div></div><div class="card"><div class="head"><span class="name">base Wan2.2</span><span class="desc">frozen VAE, no adapter</span></div><video src="mb_recon_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>45.76 dB</b> · MSE <b>2.65e-05</b></div></div><div class="card"><div class="head"><span class="name">control · flow_weight 0</span><span class="desc">6fps, edge+LPIPS only</span></div><video src="mb_recon_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>46.16 dB</b> · MSE <b>2.42e-05</b> · <span class="delta up">+0.40 dB vs base</span></div></div><div class="card"><div class="head"><span class="name">treatment · flow_weight 1</span><span class="desc">6fps, L1 weighted by motion</span></div><video src="mb_recon_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>46.18 dB</b> · MSE <b>2.41e-05</b> · <span class="delta up">+0.42 dB vs base</span></div></div></div>
</section>
<section>
<h2>Error maps</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − base Wan2.2|</span></div><video src="mb_diff_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − control · flow_weight 0|</span></div><video src="mb_diff_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − treatment · flow_weight 1|</span></div><video src="mb_diff_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div></div>
</section>
<section>
<h2>PSNR split by motion · this clip</h2>
<p class="note">No <code>*_flow.npz</code> exists beside this clip, so there is no
motion map to split on — only whole-frame PSNR is available here. The motion split for
these adapters is on the CutTofu tab and, at population scale, in the val table below.</p>
</section>
</div>
<div class="clip" id="clip2">
<div class="meta">
Source: <code>tactile_right.mp4</code> ·
Clip: <code>117 frames · 256×256 · 6fps · first 20s (stride 5 from 600 native
@ 30fps)</code> ·
Adapters: <code>lora_last.pt · r32 α32 targets=all · step 4000</code>
</div>
<section>
<h2>Reconstruction</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">original</span><span class="desc">resized + subsampled input</span></div><video src="e3r_original.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">reference · 117 frames @ 256×256</div></div><div class="card"><div class="head"><span class="name">base Wan2.2</span><span class="desc">frozen VAE, no adapter</span></div><video src="e3r_recon_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>45.40 dB</b> · MSE <b>2.88e-05</b></div></div><div class="card"><div class="head"><span class="name">control · flow_weight 0</span><span class="desc">6fps, edge+LPIPS only</span></div><video src="e3r_recon_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>45.85 dB</b> · MSE <b>2.60e-05</b> · <span class="delta up">+0.45 dB vs base</span></div></div><div class="card"><div class="head"><span class="name">treatment · flow_weight 1</span><span class="desc">6fps, L1 weighted by motion</span></div><video src="e3r_recon_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>45.86 dB</b> · MSE <b>2.59e-05</b> · <span class="delta up">+0.47 dB vs base</span></div></div></div>
</section>
<section>
<h2>Error maps</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − base Wan2.2|</span></div><video src="e3r_diff_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − control · flow_weight 0|</span></div><video src="e3r_diff_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − treatment · flow_weight 1|</span></div><video src="e3r_diff_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div></div>
</section>
<section>
<h2>PSNR split by motion · this clip</h2>
<p class="note">No <code>*_flow.npz</code> exists beside this clip, so there is no
motion map to split on — only whole-frame PSNR is available here. The motion split for
these adapters is on the CutTofu tab and, at population scale, in the val table below.</p>
</section>
</div>
<div class="clip" id="clip3">
<div class="meta">
Source: <code>tactile_left.mp4</code> ·
Clip: <code>117 frames · 256×256 · 6fps · first 20s (stride 5 from 600 native
@ 30fps)</code> ·
Adapters: <code>lora_last.pt · r32 α32 targets=all · step 4000</code>
</div>
<section>
<h2>Reconstruction</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">original</span><span class="desc">resized + subsampled input</span></div><video src="l11_original.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">reference · 117 frames @ 256×256</div></div><div class="card"><div class="head"><span class="name">base Wan2.2</span><span class="desc">frozen VAE, no adapter</span></div><video src="l11_recon_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>42.83 dB</b> · MSE <b>5.21e-05</b></div></div><div class="card"><div class="head"><span class="name">control · flow_weight 0</span><span class="desc">6fps, edge+LPIPS only</span></div><video src="l11_recon_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>43.55 dB</b> · MSE <b>4.41e-05</b> · <span class="delta up">+0.73 dB vs base</span></div></div><div class="card"><div class="head"><span class="name">treatment · flow_weight 1</span><span class="desc">6fps, L1 weighted by motion</span></div><video src="l11_recon_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>43.59 dB</b> · MSE <b>4.37e-05</b> · <span class="delta up">+0.77 dB vs base</span></div></div></div>
</section>
<section>
<h2>Error maps</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − base Wan2.2|</span></div><video src="l11_diff_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − control · flow_weight 0|</span></div><video src="l11_diff_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − treatment · flow_weight 1|</span></div><video src="l11_diff_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div></div>
</section>
<section>
<h2>PSNR split by motion · this clip</h2>
<p class="note">No <code>*_flow.npz</code> exists beside this clip, so there is no
motion map to split on — only whole-frame PSNR is available here. The motion split for
these adapters is on the CutTofu tab and, at population scale, in the val table below.</p>
</section>
</div>
<div class="clip" id="clip4">
<div class="meta">
Source: <code>tactile_right.mp4</code> ·
Clip: <code>117 frames · 256×256 · 6fps · 210–230s (stride 5 from 600 native
@ 30fps)</code> ·
Adapters: <code>lora_last.pt · r32 α32 targets=all · step 4000</code>
</div>
<section>
<h2>Reconstruction</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">original</span><span class="desc">resized + subsampled input</span></div><video src="r11_original.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">reference · 117 frames @ 256×256</div></div><div class="card"><div class="head"><span class="name">base Wan2.2</span><span class="desc">frozen VAE, no adapter</span></div><video src="r11_recon_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>43.27 dB</b> · MSE <b>4.71e-05</b></div></div><div class="card"><div class="head"><span class="name">control · flow_weight 0</span><span class="desc">6fps, edge+LPIPS only</span></div><video src="r11_recon_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>43.38 dB</b> · MSE <b>4.59e-05</b> · <span class="delta up">+0.12 dB vs base</span></div></div><div class="card"><div class="head"><span class="name">treatment · flow_weight 1</span><span class="desc">6fps, L1 weighted by motion</span></div><video src="r11_recon_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">PSNR <b>43.39 dB</b> · MSE <b>4.58e-05</b> · <span class="delta up">+0.13 dB vs base</span></div></div></div>
</section>
<section>
<h2>Error maps</h2>
<div class="grid"><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − base Wan2.2|</span></div><video src="r11_diff_base.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − control · flow_weight 0|</span></div><video src="r11_diff_fps6.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div><div class="card"><div class="head"><span class="name">diff ×20</span><span class="desc">|original − treatment · flow_weight 1|</span></div><video src="r11_diff_flow.mp4" muted loop playsinline controls preload="metadata"></video><div class="metric">brighter = larger error</div></div></div>
</section>
<section>
<h2>PSNR split by motion · this clip</h2>
<p class="note">No <code>*_flow.npz</code> exists beside this clip, so there is no
motion map to split on — only whole-frame PSNR is available here. The motion split for
these adapters is on the CutTofu tab and, at population scale, in the val table below.</p>
</section>
</div>
<section>
<h2>Val set · 32 held-out clips · the population read</h2>
<p class="note">The tabs above are individual samples; this is the population, and it is
what the conclusion rests on. Same motion split, same <code>lora_last.pt</code> adapters,
32 val clips at 6fps (4.8% of
pixels moving) · <code>compare_flow_adapters.py</code></p>
<div class="tbl-wrap"><table>
<thead><tr><th>adapter</th><th>moving (m̂≥0.5)</th>
<th>static (m̂≤0.1)</th><th>all pixels</th></tr></thead>
<tbody><tr><td>base (frozen)</td><td>34.04</td><td>36.77</td><td>36.48</td></tr><tr><td>control (flow_weight 0)</td><td>36.83</td><td>39.96</td><td>39.62</td></tr><tr><td>flow (flow_weight 1)</td><td>36.55</td><td>40.16</td><td>39.84</td></tr><tr class="delta-row"><td>flow − control</td><td class="delta down">-0.28 dB</td><td class="delta up">+0.20 dB</td><td class="delta up">+0.22 dB</td></tr></tbody>
</table></div>
</section>
<footer>
Both adapters: LoRA on the Wan2.2 causal 3D VAE (<code>Wan-AI/Wan2.2-TI2V-5B</code>),
r=32 α=32, all conv targets, 4000 steps at 6fps, edge-gradient L1 + LPIPS 0.1.
The only difference is <code>flow_weight</code> (0 vs 1); fps is held fixed, so the
~4 dB fps effect is not a confound. Judged from <code>lora_last.pt</code> because
<code>lora_best.pt</code> is selected on <em>unweighted</em> val MSE, which works against
the flow objective. Reconstruction runs at <b>6fps</b> — the rate the adapters trained at;
evaluating at the sources' native 30fps would measure a domain mismatch instead of the
adapter. Diff videos are |original − recon| amplified through a hot colormap.
</footer>
<script>
document.querySelectorAll('.tab').forEach(t => t.addEventListener('click', () => {
document.querySelectorAll('.tab').forEach(x => x.classList.remove('active'));
document.querySelectorAll('.clip').forEach(x => x.classList.remove('active'));
t.classList.add('active');
document.getElementById(t.dataset.clip).classList.add('active');
}));
document.getElementById('toggle').addEventListener('click', () => {
const vis = [...document.querySelectorAll('.clip.active video')];
const anyPaused = vis.some(v => v.paused);
vis.forEach(v => anyPaused ? v.play().catch(() => {}) : v.pause());
});
</script>
</body>
</html>