| --- |
| title: Wan2.2 VAE — Flow-Weighted LoRA vs Control |
| emoji: 🌊 |
| colorFrom: indigo |
| colorTo: pink |
| sdk: static |
| pinned: false |
| --- |
| |
| # Wan2.2 VAE — flow-weighted LoRA vs control |
|
|
| GelSight tactile clips reconstructed through the Wan2.2 causal 3D VAE three ways: the |
| **frozen base**, a **control** LoRA, and a **flow-weighted** LoRA. |
|
|
| The two adapters' training configs differ in exactly one field — `flow_weight` (0 vs 1). |
| Everything else is identical: 6fps (stride 5 from 30fps sources), r=32 α=32, all conv |
| targets, 4000 steps, edge-gradient L1 + LPIPS 0.1. Holding fps fixed matters: fps alone is |
| worth ~4 dB and confounded the earlier comparison. |
|
|
| Flow weighting multiplies the L1 term by optical-flow magnitude, spending capacity on |
| moving regions at the expense of static ones. **Overall PSNR cannot show this** — it |
| averages every pixel, so a pure redistribution nets out. The measurement that can |
| discriminate splits PSNR by the same motion magnitude the loss was weighted with |
| (`m̂ ≥ 0.5` moving, `≤ 0.1` static). |
|
|
| ## Result: it didn't work |
|
|
| Over **32 held-out val clips**, the flow-weighted adapter is **-0.28 dB** on moving regions — the regions it was built to improve — while static regions gain +0.20 dB. The intended trade ran backwards. |
|
|
| Caveat in the other direction: one training run per arm (seed 0), so a 0.28 dB gap does not rule out run-to-run variance. The honest claim is *no evidence flow weighting helps moving regions*, not proof that it hurts. Both adapters beat the frozen base by ~2.8 dB, so the LoRA finetune itself works; only the flow weighting is a wash. |
|
|
| ## Clips |
|
|
| - **CutTofu · tactile_left** (`no prefix`) — 77 frames @ 256×256, 6fps, whole clip |
| - **0510 ep000 · tactile_right** (`mb_`) — 117 frames @ 256×256, 6fps, first 20s — no `*_flow.npz` sibling, so whole-frame PSNR only |
| - **0510 ep003 · tactile_right** (`e3r_`) — 117 frames @ 256×256, 6fps, first 20s — no `*_flow.npz` sibling, so whole-frame PSNR only |
| - **0511 ep003 · tactile_left** (`l11_`) — 117 frames @ 256×256, 6fps, first 20s — no `*_flow.npz` sibling, so whole-frame PSNR only |
| - **0511 ep003 · tactile_right** (`r11_`) — 117 frames @ 256×256, 6fps, 210–230s — no `*_flow.npz` sibling, so whole-frame PSNR only |
| |
| Windows were picked to contain actual contact. The `motherboard_*` episodes are mostly a |
| static, defocused, no-contact gel view: a blank window still reconstructs at ~46 dB while |
| showing nothing, because there is no texture to lose. `0511_episode_003/tactile_right` is |
| blank for its first 20s, so it uses 210–230s, where the sensor is pressed against an |
| object. The clips with the most texture (CutTofu, and `0511 ep003 tactile_left`) have the |
| *lowest* base PSNR and gain the most from the LoRA — that is the signal to read. |
|
|
| Per clip: `original.mp4`, `recon_{base,fps6,flow}.mp4`, `diff_{base,fps6,flow}.mp4` |
| (|original − recon| amplified, hot colormap), `motion_weight.mp4` (the m̂ map, where flow |
| exists), `metrics.json`. Population-level numbers are in `val_split.json`. |
|
|
| Reconstruction is at **6fps** — the rate the adapters trained at. Evaluating them at the |
| sources' native 30fps would measure a domain mismatch instead of the adapter. |
|
|
