---
title: A2A-Video
emoji: 🎬
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.6.0
app_file: app.py
pinned: false
license: apache-2.0
---

# A2A-Video: Any-to-Any Video Modality Generation

Two tabs, both driven by the same trained multimodal video model:

- **Any-to-Any Generation** -- pick a curated example clip (or upload your
  own video, or type a caption), choose which modality to condition on, then
  build a generation chain one modality at a time. Predicts any subset of:
  RGB, depth, surface normals, optical flow, DINOv2 / V-JEPA-2 / SigLIP-2
  feature maps (visualized via PCA-to-RGB), bounding boxes, a caption, and a
  transcription.
- **Future Prediction** -- give the model only the first frame (or first 5
  frames) of one modality (RGB, depth, normal, optical flow, or bounding
  boxes) and it predicts the rest of that trajectory, optionally also fully
  conditioned on a caption/transcription to steer where it goes.

`human_poses` is intentionally left out of both tabs (it needs a
TokHMR + SMPL + EGL rendering stack that isn't ported over here).

## Setup before deploying

1. **Checkpoints.** Run `scripts/upload_checkpoints_to_hub.py` **on the
   machine where the checkpoints actually live** (this Space folder does not
   have them). It pushes to two separate Hub repos:
   - `EPFL-VILAB/Video-4M-models` -- the main A2A-Video checkpoint (changes often)
   - `EPFL-VILAB/Video-4M-tokenizers` -- the 7 tokenizer checkpoints (stable),
     one folder per modality: `rgb/`, `surface-normals/`, `depth/`,
     `opticalflow/`, `v-jepa-2/`, `dinov2/`, `siglip-2/`

2. **Curated examples.** Both tabs need example clips uploaded to
   `EPFL-VILAB/Video-4M-examples` (a HF *dataset* repo) -- the any-to-any tab
   reads a random/hand-picked set from `examples.json`, and the Future
   Prediction tab reads a separately curated set (clips with clear, steady
   motion) from `future_examples.json`. Follow `scripts/README_EXAMPLES.md`
   for the full 3-phase workflow (pick clips, render previews, upload) --
   run it once for each manifest name.

3. **Repo ids.** The defaults above (`EPFL-VILAB/Video-4M-models`,
   `EPFL-VILAB/Video-4M-tokenizers`, `EPFL-VILAB/Video-4M-examples`) are
   baked into `inference.py`. If you used different repo ids, set these in
   the Space's Settings -> Variables and secrets instead:
   - `FOURM_MODEL_REPO`, `FOURM_TOKENIZERS_REPO`, `FOURM_EXAMPLES_REPO`
   - `FOURM_FUTURE_EXAMPLES_MANIFEST` (filename within the examples repo,
     default `future_examples.json`) if you picked a different manifest name
     for the Future Prediction set.
   - Do **not** set any of the `FOURM_LOCAL_*` variables on a real Space --
     those are cluster-testing-only shortcuts that bypass the Hub entirely
     (see `inference.py`'s `_LOCAL_ENV_VARS` / `FOURM_LOCAL_EXAMPLES_DIR`),
     and would need paths that don't exist inside the Space's container.

4. **Hardware.** Select **ZeroGPU** hardware for this Space.

5. **First-request cold start.** Both tabs' `@spaces.GPU(duration=180)`
   budget needs to cover loading the main checkpoint + all 7 tokenizers (the
   first generation call after a cold start) *plus* the actual generation
   time -- this combination hasn't been tested under ZeroGPU yet (only ever
   run on a persistent-GPU cluster server, where the model loads once and
   stays resident). If the first request times out or gets killed, raise
   `duration` (subject to your ZeroGPU plan's per-call cap).

This code is adapted from the research repo's `cvpr_fvd_videos_no_poses.py`
(any-to-any) and `cvpr_fvd_videos_no_poses_raw_rgb.py` (future prediction,
raw video tokenization) inference scripts.