File size: 1,435 Bytes
e75c2c1 72c1c7d e75c2c1 72c1c7d e75c2c1 72c1c7d e75c2c1 38c3887 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | ---
license: apache-2.0
---
# Video-4M Tokenizers
FSQ tokenizers used by Video-4M to convert each modality into discrete tokens
for training and generation. RGB, depth, surface normals, and optical flow
share one causal VidTok architecture (same config, different weights per
modality). The three self-supervised feature-map modalities (V-JEPA 2,
DINOv2, SigLIP 2) use FSQ tokenizers trained on top of the corresponding
frozen feature extractor.
Video-4M is trained on 4-second, 17-frame, 4 FPS clips at 128x128
resolution. For the feature-map modalities, features are extracted at each
extractor's native resolution before tokenization: 224x224 for DINOv2 and
SigLIP 2, 256x256 for V-JEPA 2.
| Modality | Path in repo | Tokens per clip | Vocab size |
|---|---|---|---|
| RGB | `rgb/ckpt.ckpt` | 1280 | 32,768 |
| Depth | `depth/ckpt.ckpt` | 1280 | 32,768 |
| Surface normals | `surface-normals/ckpt.ckpt` | 1280 | 32,768 |
| Optical flow | `opticalflow/ckpt.ckpt` | 1280 | 32,768 |
| V-JEPA 2 | `v-jepa-2/ckpt.ckpt` | 1024 | 16,807 |
| DINOv2 | `dinov2/ckpt.ckpt` | 1280 | 16,807 |
| SigLIP 2 | `siglip-2/ckpt.ckpt` | 980 | 16,807 |
See also: [`EPFL-VILAB/Video-4M-models`](https://huggingface.co/EPFL-VILAB/Video-4M-models)
(the main any-to-any model these tokenizers feed into) and
[`EPFL-VILAB/Video-4M-examples`](https://huggingface.co/EPFL-VILAB/Video-4M-examples)
(example clips, pre-tokenized with these same tokenizers).
|