scripts/README_EXAMPLES.md

Adding / refreshing curated examples for the Any-to-Any tab

Three phases, run in order, all from inside hf_space_demo/ on the cluster (where both the tokenized dataset and the checkpoints actually live).

Phase 1 -- pick clips + upload tokens

python scripts/upload_examples_to_hub.py \
    --source_dir /datasets/uzair/weights_from_clariden/test_cvpr_final_set_13_mod \
    --repo_id EPFL-VILAB/Video-4M-examples \
    --num_examples 6 --seed 0

This randomly picks --num_examples clips that have a file in every modality subfolder, uploads all of their tokenized data to the EPFL-VILAB/Video-4M-examples dataset repo, and saves the picked clip IDs locally to examples_manifest.json (in whatever directory you ran the command from -- keep this file, phases 2 and 3 both need it).

Phase 2 -- render human-viewable previews

cd ..   # repo root, since the script reads hf_space_demo/examples_manifest.json
python visualize_multimodal_pretraining_data_13_modalities.py \
    --config cfgs/default/4m/models/main/4m_L_video_all_modalities_final_config.yaml

This reads hf_space_demo/examples_manifest.json and renders a .mp4 preview per modality for exactly those clips (not a random sample), saved under args.output_dir_videos (default: /datasets/uzair/weights_from_clariden/cvpr_generations/GT_visualizations_post_neurips_opticalflow_fixed), in <domain>_detokenized/<clip_basename>.mp4 subfolders.

Phase 3 -- upload the previews

cd hf_space_demo
python scripts/upload_examples_to_hub.py \
    --repo_id EPFL-VILAB/Video-4M-examples \
    --detokenized_dir /datasets/uzair/weights_from_clariden/cvpr_generations/GT_visualizations_post_neurips_opticalflow_fixed \
    --previews_only

--previews_only skips re-uploading the (large, slow) tokens. Without --force_repick, this automatically reuses the exact clips already recorded in examples_manifest.json -- it will not pick a new random set.

Once this is done, the Any-to-Any tab's example/input-modality dropdowns will show a live preview before you even hit Generate.

A second curated set (e.g. for the Future Prediction tab)

The Future Prediction tab needs its own hand-picked clips (e.g. ones with clear, consistent motion) rather than a random sample, and shouldn't overwrite the any-to-any tab's set. Same repo, same 3 phases, just with --stems_file instead of --num_examples/--seed, and a distinct --manifest_out/--manifest_repo_filename so the two sets don't collide:

# Phase 1 -- explicit stems (one per line, e.g. "vol_12/clip_000123",
# matching the tok_video_rgb@128 subfolder layout)
python scripts/upload_examples_to_hub.py \
    --source_dir /datasets/uzair/weights_from_clariden/test_cvpr_final_set_13_mod \
    --repo_id EPFL-VILAB/Video-4M-examples \
    --stems_file my_future_pred_stems.txt \
    --manifest_out future_examples_manifest.json \
    --manifest_repo_filename future_examples.json

# Phase 2 -- point the visualization script at this manifest via env var
cd ..
EXAMPLES_MANIFEST_PATH=hf_space_demo/future_examples_manifest.json \
    python visualize_multimodal_pretraining_data_13_modalities.py \
    --config cfgs/default/4m/models/main/4m_L_video_all_modalities_final_config.yaml

# Phase 3 -- upload previews, same distinct manifest names as phase 1
cd hf_space_demo
python scripts/upload_examples_to_hub.py \
    --repo_id EPFL-VILAB/Video-4M-examples \
    --detokenized_dir /datasets/uzair/weights_from_clariden/cvpr_generations/GT_visualizations_post_neurips_opticalflow_fixed \
    --manifest_out future_examples_manifest.json \
    --manifest_repo_filename future_examples.json \
    --previews_only

Every stem in --stems_file must have a file in every modality subfolder -- the script errors out listing exactly which ones are missing, rather than silently dropping them (unlike phase 1's random-selection path, which just skips incomplete candidates).

Notes

• Lost your local examples_manifest.json? It was also uploaded to the repo as examples.json in phase 1 -- pull it back down instead of re-picking clips:

  python -c "
  from huggingface_hub import hf_hub_download
  import shutil
  path = hf_hub_download(repo_id='EPFL-VILAB/Video-4M-examples', filename='examples.json', repo_type='dataset')
  shutil.copy(path, 'examples_manifest.json')
  "
  

  (For the future-prediction set, substitute future_examples.json / future_examples_manifest.json.)
• Want a completely different set of clips? Re-run phase 1 with --force_repick (optionally a different --seed/--num_examples, or a different --stems_file), then repeat phases 2 and 3.
• Just fixing/re-rendering previews for the same clips? Skip phase 1, start at phase 2 -- the manifest already has the clip IDs.