---
title: Wan2.2 VAE — Flow-Weighted LoRA vs Control
emoji: 🌊
colorFrom: indigo
colorTo: pink
sdk: static
pinned: false
---

# Wan2.2 VAE — flow-weighted LoRA vs control

GelSight tactile clips reconstructed through the Wan2.2 causal 3D VAE three ways: the
**frozen base**, a **control** LoRA, and a **flow-weighted** LoRA.

The two adapters' training configs differ in exactly one field — `flow_weight` (0 vs 1).
Everything else is identical: 6fps (stride 5 from 30fps sources), r=32 α=32, all conv
targets, 4000 steps, edge-gradient L1 + LPIPS 0.1. Holding fps fixed matters: fps alone is
worth ~4 dB and confounded the earlier comparison.

Flow weighting multiplies the L1 term by optical-flow magnitude, spending capacity on
moving regions at the expense of static ones. **Overall PSNR cannot show this** — it
averages every pixel, so a pure redistribution nets out. The measurement that can
discriminate splits PSNR by the same motion magnitude the loss was weighted with
(`m̂ ≥ 0.5` moving, `≤ 0.1` static).

## Result: it didn't work

Over **32 held-out val clips**, the flow-weighted adapter is **-0.28 dB** on moving regions — the regions it was built to improve — while static regions gain +0.20 dB. The intended trade ran backwards.

Caveat in the other direction: one training run per arm (seed 0), so a 0.28 dB gap does not rule out run-to-run variance. The honest claim is *no evidence flow weighting helps moving regions*, not proof that it hurts. Both adapters beat the frozen base by ~2.8 dB, so the LoRA finetune itself works; only the flow weighting is a wash.

## Clips

- **CutTofu · tactile_left** (`no prefix`) — 77 frames @ 256×256, 6fps, whole clip
- **0510 ep000 · tactile_right** (`mb_`) — 117 frames @ 256×256, 6fps, first 20s — no `*_flow.npz` sibling, so whole-frame PSNR only
- **0510 ep003 · tactile_right** (`e3r_`) — 117 frames @ 256×256, 6fps, first 20s — no `*_flow.npz` sibling, so whole-frame PSNR only
- **0511 ep003 · tactile_left** (`l11_`) — 117 frames @ 256×256, 6fps, first 20s — no `*_flow.npz` sibling, so whole-frame PSNR only
- **0511 ep003 · tactile_right** (`r11_`) — 117 frames @ 256×256, 6fps, 210–230s — no `*_flow.npz` sibling, so whole-frame PSNR only

Windows were picked to contain actual contact. The `motherboard_*` episodes are mostly a
static, defocused, no-contact gel view: a blank window still reconstructs at ~46 dB while
showing nothing, because there is no texture to lose. `0511_episode_003/tactile_right` is
blank for its first 20s, so it uses 210–230s, where the sensor is pressed against an
object. The clips with the most texture (CutTofu, and `0511 ep003 tactile_left`) have the
*lowest* base PSNR and gain the most from the LoRA — that is the signal to read.

Per clip: `original.mp4`, `recon_{base,fps6,flow}.mp4`, `diff_{base,fps6,flow}.mp4`
(|original − recon| amplified, hot colormap), `motion_weight.mp4` (the m̂ map, where flow
exists), `metrics.json`. Population-level numbers are in `val_split.json`.

Reconstruction is at **6fps** — the rate the adapters trained at. Evaluating them at the
sources' native 30fps would measure a domain mismatch instead of the adapter.