tmp/video-velocity-model/data_processing/README.md

Data Processing & Analysis

ucf101

bridgedatav2

UMAP visualization

data_processing/umap_alg.py runs UMAP on per-frame features and saves a 2D scatter plot. Points are colored by frame index, which makes it easy to inspect temporal trajectories. It can also mark GOP boundaries by detecting keyframes / I-frames from the original video bitstream.

Supported inputs:

• .npy: expects a 2D array with shape (T, D)
• .npz: expects a key named features by default

For the latent files produced by forward.py, features usually has shape (T, 768, 16, 16). umap_alg.py now supports these files directly and will project them to 2D frame features before UMAP:

• mean_pool (default): spatial average pooling, (T, 768, 16, 16) -> (T, 768)
• flatten: flatten each frame latent, (T, 768, 16, 16) -> (T, 196608)

PPL smoothness metric

data_processing/cal_metric.py computes a StyleGAN-inspired perceptual path length (PPL) on video latent interpolation paths. It is meant to compare the local smoothness of the dinov2, mae, siglip2, and flux2_ae feature spaces.

The script:

1. encodes raw video frames into the selected RAE latent space
2. samples two nearby points on each adjacent-frame latent path
3. decodes the two nearby latents back to images
4. computes LPIPS between the decoded images
5. divides by eps^2 and aggregates the result as PPL

Supported PPL interpolation geometries:

• linear: standard per-gap linear interpolation
• slerp: token-wise spherical interpolation
• smooth_continuous: the same cubic-Hermite path used by stage-2 TrajectoryPlan(sampling_mode="smooth_continuous")
  • frame velocities are estimated from the whole latent trajectory, so each adjacent-frame gap depends on its temporal neighbors
  • this mode is currently implemented in data_processing/cal_metric.py only; data_processing/forward_intrapolate.py still supports linear / slerp

It can also visualize the per-gap PPL distribution for a single video:

• x-axis: the left frame index of each adjacent-frame gap
• y-axis: the PPL value of that gap

When --samples-per-gap > 1, the plot uses the mean PPL of each gap, which helps show how local smoothness changes across the whole sequence.

Interpretation:

• lower PPL -> smoother local geometry / smoother interpolation path
• higher PPL -> sharper perceptual changes under small latent-space moves

Comparison protocols:

• default: use each backbone's runtime default interpolation
  • dinov2 -> linear
  • mae -> linear
  • siglip2 -> slerp
  • flux2_ae -> linear
• default can also be overridden explicitly with --interp-mode smooth_continuous
• controlled: force all backbones to use linear
• both: report both sets of numbers in one run

Important outputs per result row:

• ppl_mean_raw: mean over all collected PPL samples
• ppl_mean_trimmed: mean after percentile trimming
• ppl_median: median PPL
• ppl_std: standard deviation
• pairs_evaluated: number of adjacent latent-frame pairs
• samples_evaluated: total sampled (alpha, alpha + eps) comparisons

Trimmed mean uses the percentile window controlled by:

• --trim-percentile-low (default 1)
• --trim-percentile-high (default 99)

Weight / dependency notes:

• LPIPS uses the repo's cached vgg.pth weights under RAE/src/disc/.caches/
• torchvision also needs a VGG16 backbone checkpoint
• on this machine, a local VGG16 checkpoint is available at:

/home/i-zhaoshitian/models/vgg/vgg16-397923af.pth