title: Wan2.2 VAE — Flow-Weighted LoRA vs Control
emoji: 🌊
colorFrom: indigo
colorTo: pink
sdk: static
pinned: false
Wan2.2 VAE — flow-weighted LoRA vs control
GelSight tactile clips reconstructed through the Wan2.2 causal 3D VAE three ways: the frozen base, a control LoRA, and a flow-weighted LoRA.
The two adapters' training configs differ in exactly one field — flow_weight (0 vs 1).
Everything else is identical: 6fps (stride 5 from 30fps sources), r=32 α=32, all conv
targets, 4000 steps, edge-gradient L1 + LPIPS 0.1. Holding fps fixed matters: fps alone is
worth ~4 dB and confounded the earlier comparison.
Flow weighting multiplies the L1 term by optical-flow magnitude, spending capacity on
moving regions at the expense of static ones. Overall PSNR cannot show this — it
averages every pixel, so a pure redistribution nets out. The measurement that can
discriminate splits PSNR by the same motion magnitude the loss was weighted with
(m̂ ≥ 0.5 moving, ≤ 0.1 static).
Result: it didn't work
Over 32 held-out val clips, the flow-weighted adapter is -0.28 dB on moving regions — the regions it was built to improve — while static regions gain +0.20 dB. The intended trade ran backwards.
Caveat in the other direction: one training run per arm (seed 0), so a 0.28 dB gap does not rule out run-to-run variance. The honest claim is no evidence flow weighting helps moving regions, not proof that it hurts. Both adapters beat the frozen base by ~2.8 dB, so the LoRA finetune itself works; only the flow weighting is a wash.
Clips
- CutTofu · tactile_left (
no prefix) — 77 frames @ 256×256, 6fps, whole clip - 0510 ep000 · tactile_right (
mb_) — 117 frames @ 256×256, 6fps, first 20s — no*_flow.npzsibling, so whole-frame PSNR only - 0510 ep003 · tactile_right (
e3r_) — 117 frames @ 256×256, 6fps, first 20s — no*_flow.npzsibling, so whole-frame PSNR only - 0511 ep003 · tactile_left (
l11_) — 117 frames @ 256×256, 6fps, first 20s — no*_flow.npzsibling, so whole-frame PSNR only - 0511 ep003 · tactile_right (
r11_) — 117 frames @ 256×256, 6fps, 210–230s — no*_flow.npzsibling, so whole-frame PSNR only
Windows were picked to contain actual contact. The motherboard_* episodes are mostly a
static, defocused, no-contact gel view: a blank window still reconstructs at ~46 dB while
showing nothing, because there is no texture to lose. 0511_episode_003/tactile_right is
blank for its first 20s, so it uses 210–230s, where the sensor is pressed against an
object. The clips with the most texture (CutTofu, and 0511 ep003 tactile_left) have the
lowest base PSNR and gain the most from the LoRA — that is the signal to read.
Per clip: original.mp4, recon_{base,fps6,flow}.mp4, diff_{base,fps6,flow}.mp4
(|original − recon| amplified, hot colormap), motion_weight.mp4 (the m̂ map, where flow
exists), metrics.json. Population-level numbers are in val_split.json.
Reconstruction is at 6fps — the rate the adapters trained at. Evaluating them at the sources' native 30fps would measure a domain mismatch instead of the adapter.