| --- |
| license: apache-2.0 |
| --- |
| |
| # Video-4M Tokenizers |
|
|
| FSQ tokenizers used by Video-4M to convert each modality into discrete tokens |
| for training and generation. RGB, depth, surface normals, and optical flow |
| share one causal VidTok architecture (same config, different weights per |
| modality). The three self-supervised feature-map modalities (V-JEPA 2, |
| DINOv2, SigLIP 2) use FSQ tokenizers trained on top of the corresponding |
| frozen feature extractor. |
|
|
| Video-4M is trained on 4-second, 17-frame, 4 FPS clips at 128x128 |
| resolution. For the feature-map modalities, features are extracted at each |
| extractor's native resolution before tokenization: 224x224 for DINOv2 and |
| SigLIP 2, 256x256 for V-JEPA 2. |
|
|
| | Modality | Path in repo | Tokens per clip | Vocab size | |
| |---|---|---|---| |
| | RGB | `rgb/ckpt.ckpt` | 1280 | 32,768 | |
| | Depth | `depth/ckpt.ckpt` | 1280 | 32,768 | |
| | Surface normals | `surface-normals/ckpt.ckpt` | 1280 | 32,768 | |
| | Optical flow | `opticalflow/ckpt.ckpt` | 1280 | 32,768 | |
| | V-JEPA 2 | `v-jepa-2/ckpt.ckpt` | 1024 | 16,807 | |
| | DINOv2 | `dinov2/ckpt.ckpt` | 1280 | 16,807 | |
| | SigLIP 2 | `siglip-2/ckpt.ckpt` | 980 | 16,807 | |
|
|
| See also: [`EPFL-VILAB/Video-4M-models`](https://huggingface.co/EPFL-VILAB/Video-4M-models) |
| (the main any-to-any model these tokenizers feed into) and |
| [`EPFL-VILAB/Video-4M-examples`](https://huggingface.co/EPFL-VILAB/Video-4M-examples) |
| (example clips, pre-tokenized with these same tokenizers). |
|
|
