README.md

metadata

title: A2A-Video
emoji: 🎬
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.6.0
app_file: app.py
pinned: false
license: apache-2.0

A2A-Video: Any-to-Any Video Modality Generation

Two tabs, both driven by the same trained multimodal video model:

• Any-to-Any Generation -- pick a curated example clip (or upload your own video, or type a caption), choose which modality to condition on, then build a generation chain one modality at a time. Predicts any subset of: RGB, depth, surface normals, optical flow, DINOv2 / V-JEPA-2 / SigLIP-2 feature maps (visualized via PCA-to-RGB), bounding boxes, a caption, and a transcription.
• Future Prediction -- give the model only the first frame (or first 5 frames) of one modality (RGB, depth, normal, optical flow, or bounding boxes) and it predicts the rest of that trajectory, optionally also fully conditioned on a caption/transcription to steer where it goes.

human_poses is intentionally left out of both tabs (it needs a TokHMR + SMPL + EGL rendering stack that isn't ported over here).

Setup before deploying

1. Checkpoints. Run scripts/upload_checkpoints_to_hub.py on the machine where the checkpoints actually live (this Space folder does not have them). It pushes to two separate Hub repos:

   • EPFL-VILAB/Video-4M-models -- the main A2A-Video checkpoint (changes often)
   • EPFL-VILAB/Video-4M-tokenizers -- the 7 tokenizer checkpoints (stable), one folder per modality: rgb/, surface-normals/, depth/, opticalflow/, v-jepa-2/, dinov2/, siglip-2/

2. Curated examples. Both tabs need example clips uploaded to EPFL-VILAB/Video-4M-examples (a HF dataset repo) -- the any-to-any tab reads a random/hand-picked set from examples.json, and the Future Prediction tab reads a separately curated set (clips with clear, steady motion) from future_examples.json. Follow scripts/README_EXAMPLES.md for the full 3-phase workflow (pick clips, render previews, upload) -- run it once for each manifest name.

3. Repo ids. The defaults above (EPFL-VILAB/Video-4M-models, EPFL-VILAB/Video-4M-tokenizers, EPFL-VILAB/Video-4M-examples) are baked into inference.py. If you used different repo ids, set these in the Space's Settings -> Variables and secrets instead:

   • FOURM_MODEL_REPO, FOURM_TOKENIZERS_REPO, FOURM_EXAMPLES_REPO
   • FOURM_FUTURE_EXAMPLES_MANIFEST (filename within the examples repo, default future_examples.json) if you picked a different manifest name for the Future Prediction set.
   • Do not set any of the FOURM_LOCAL_* variables on a real Space -- those are cluster-testing-only shortcuts that bypass the Hub entirely (see inference.py's _LOCAL_ENV_VARS / FOURM_LOCAL_EXAMPLES_DIR), and would need paths that don't exist inside the Space's container.

4. Hardware. Select ZeroGPU hardware for this Space.

5. First-request cold start. Both tabs' @spaces.GPU(duration=180) budget needs to cover loading the main checkpoint + all 7 tokenizers (the first generation call after a cold start) plus the actual generation time -- this combination hasn't been tested under ZeroGPU yet (only ever run on a persistent-GPU cluster server, where the model loads once and stays resident). If the first request times out or gets killed, raise duration (subject to your ZeroGPU plan's per-call cap).

This code is adapted from the research repo's cvpr_fvd_videos_no_poses.py (any-to-any) and cvpr_fvd_videos_no_poses_raw_rgb.py (future prediction, raw video tokenization) inference scripts.