| --- |
| license: apache-2.0 |
| tags: |
| - video |
| - self-supervised-learning |
| - diffusion |
| pipeline_tag: other |
| --- |
| |
| # TT-VidT pretrained DiT decoders |
|
|
| Pretrained DiT decoders for [TT-VidT](https://github.com/KohakuBlueleaf/TTVidT) |
| (*Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining*, |
| NeurIPS 2026). [Paper](https://huggingface.co/papers/2609.33419) · |
| [Project page](https://kohakublueleaf.github.io/TTVidT/) |
|
|
| Each TT-VidT encoder pretraining run initialises its decoder from one of |
| these; the training code downloads them automatically through `DECODER_PRETRAINED`, |
| e.g. `"KBlueLeaf/TTVidT-decoders/pretrain_video_S_qknorm"`. |
|
|
| The decoders work in the 32x32x4 latent space of the frozen f8 frame VAE |
| [`KBlueLeaf/latentmaid-vae`](https://huggingface.co/KBlueLeaf/latentmaid-vae) and are |
| conditioned on DINOv3 ViT-B/16 features (768-d). |
|
|
| - **ImageNet** decoders reconstruct an ImageNet image from its DINOv3 class token. |
| - **Video** decoders reconstruct a later video frame from its class token plus, through |
| cross-attention, the spatial features of an earlier frame. |
|
|
| ## Files |
|
|
| Each decoder is `<name>.safetensors` plus `<name>.json`, which holds the decoder |
| config (`decoder_config`), the latent size and the latent statistics used in training. |
|
|
| `pretrain_{imgnet,video}_{S,B,L}_<variant>`: |
|
|
| | Size | Hidden | Layers | Heads | |
| |---|---:|---:|---:| |
| | S | 768 | 12 | 12 | |
| | B | 1024 | 16 | 16 | |
| | L | 1152 | 28 | 16 | |
|
|
| | Variant | Meaning | |
| |---|---| |
| | `qknorm` | QK-norm attention, diffusion head | |
| | `qknorm_nofinal` | as above, without the final norm | |
| | `qknorm_reg`, `qknorm_nofinal_reg` | latent-regression head instead of diffusion | |
| | `768d12l`, `1024d16l`, `1152d28l` (`_80k`) | earlier runs without QK-norm | |
|
|
| The ones used in the paper's configs: |
|
|
| | Decoder | Used by | |
| |---|---| |
| | `pretrain_imgnet_S_qknorm` | objective / encoder sweep | |
| | `pretrain_video_S_qknorm` | TT-VidT | |
| | `pretrain_imgnet_S_qknorm_nofinal` | DisMo-2D3D with dual augmentation | |
| | `pretrain_{imgnet,video}_B_qknorm_nofinal`, `pretrain_{imgnet,video}_L_qknorm` | decoder ablation | |
|
|
| Recipe: 100k steps (80k for `_80k`), global batch 256, AdamW lr 1e-4. See the |
| [code repository](https://github.com/KohakuBlueleaf/TTVidT) for training and usage. |
