Video-4M Tokenizers
FSQ tokenizers used by Video-4M to convert each modality into discrete tokens
for training and generation. RGB, depth, surface normals, and optical flow
share one causal VidTok architecture (same config, different weights per
modality). The three self-supervised feature-map modalities (V-JEPA 2,
DINOv2, SigLIP 2) use FSQ tokenizers trained on top of the corresponding
frozen feature extractor.
Video-4M is trained on 4-second, 17-frame, 4 FPS clips at 128x128
resolution. For the feature-map modalities, features are extracted at each
extractor's native resolution before tokenization: 224x224 for DINOv2 and
SigLIP 2, 256x256 for V-JEPA 2.
| Modality | Path in repo | Tokens per clip | Vocab size |
|---|
| RGB | rgb/ckpt.ckpt | 1280 | 32,768 |
| Depth | depth/ckpt.ckpt | 1280 | 32,768 |
| Surface normals | surface-normals/ckpt.ckpt | 1280 | 32,768 |
| Optical flow | opticalflow/ckpt.ckpt | 1280 | 32,768 |
| V-JEPA 2 | v-jepa-2/ckpt.ckpt | 1024 | 16,807 |
| DINOv2 | dinov2/ckpt.ckpt | 1280 | 16,807 |
| SigLIP 2 | siglip-2/ckpt.ckpt | 980 | 16,807 |
See also:
EPFL-VILAB/Video-4M-models
(the main any-to-any model these tokenizers feed into) and
EPFL-VILAB/Video-4M-examples
(example clips, pre-tokenized with these same tokenizers).
