| # 14B Asset Preparation |
|
|
| The evaluated model is the **three-encoder step-80 non-EMA generator**, trained on **8,000 reference videos**. Its asset identities are recorded in [`configs/train.yaml`](../configs/train.yaml). The former two-encoder / 8,852-video bank is a different experiment and cannot reproduce this model. |
|
|
| Prepare matching model and reference assets locally; this anonymous release does not embed hosted resource locations. Use the matching three-encoder checkpoint and reference assets; filenames or the step number alone do not identify the experiment. This source repository does not include model weights or reference tensors. A final distilled model is not interchangeable with the **causal ODE initialization** needed for training. |
|
|
| ## Directory layout |
|
|
| ```text |
| ASSET_ROOT/ |
| ├── models/ |
| │ ├── Wan2.1-T2V-14B/ # Base generator, T5, tokenizer, and VAE |
| │ ├── dinov3-src/ # Official DINOv3 source at the pinned commit |
| │ ├── dinov3_vitl16_lvd1689m.pth |
| │ ├── vjepa2-vitl-fpc64-256/ |
| │ └── VideoMAEv2-Large/ # Includes download_manifest.json |
| ├── checkpoints/ |
| │ └── causal-ode-init.safetensors |
| ├── teacher_bank/ |
| │ ├── manifest.json |
| │ └── ... # Preserve all shard-relative paths |
| ├── source/ |
| │ ├── matched.jsonl |
| │ └── matched.meta.json |
| ├── references/ |
| │ ├── dinov3.pt |
| │ ├── vjepa2.pt |
| │ └── videomae_v2.pt |
| └── videos/ # Feature extraction only; not decoded during bank-based training |
| ``` |
|
|
| Preserve original serialized manifests, reference packs, and shards. The three reference files may be renamed as above without changing their bytes. `prepare_config.py` reads DINOv3 and VideoMAEv2 metadata identities from the bank and keeps them separate from local encoder load paths. These local settings are excluded from Git. |
|
|
| ## Frozen encoders |
|
|
| Wan2.1-T2V-14B includes the shared T5 weights, VAE weights, and tokenizer. Place its files in the directory described in the model README. Before offline training, prepare the other encoders: |
|
|
| ```bash |
| git clone https://github.com/facebookresearch/dinov3.git "$ASSET_ROOT/models/dinov3-src" |
| git -C "$ASSET_ROOT/models/dinov3-src" checkout 6876159a11b4df116f30f667f8c9888617df0751 |
| # Obtain dinov3_vitl16_lvd1689m.pth through the official DINOv3 model access process. |
| # Place the matching model files at the local paths described below. |
| PYTHONPATH="$PWD/code" python code/tools/download_videomae_v2_large.py \ |
| --source-dir /path/to/local/VideoMAEv2-Large \ |
| --output-dir "$ASSET_ROOT/models/VideoMAEv2-Large" |
| ``` |
|
|
| The exact encoder hashes are in the training config; a newer checkpoint is not interchangeable with a frozen reference bank. VideoMAEv2 uses revision `9981a9c8f77118c421e5228e1b219468a4b0238d`. |
|
|
| | Encoder | Input and representation | Nyström landmarks | RBF bandwidth | |
| |---|---|---:|---:| |
| | DINOv3 ViT-L/16 | 20 sampled frames resized to 256²; final normalized patch features; mean each group of five frames; 1,024 × 1,024 matrix | 256 | 341.2794189453125 | |
| | V-JEPA 2 ViT-L | Frames 17–80; short side 292, center crop 256²; block index 15; 1,024 × 1,024 matrix | 1,024 | 3025.305908203125 | |
| | VideoMAEv2-Large | Frames 1–80 split into five 16-frame clips; short side/crop 224; block index 15; 256 tokens per clip, 1,280 × 1,024 per video | 256 | 3204.123291015625 | |
|
|
| VideoMAEv2 shares one frozen encoder across the five clips. All branches use ImageNet pixel normalization. The video-level kernel is `exp(-||X-Y||_F² / (2 sigma²))`, without dividing the distance by the number of tokens or channels. The fixed Nyström references use ridge `1e-6` and 20 k-means iterations. The loss normalization epsilon is separately `1e-4`. Full metadata is in [`reference-settings.json`](../provenance/reference-settings.json). |
|
|
| ## Rebuilding a reference bank |
|
|
| Rebuilding requires the original captioned videos and matched manifest. Re-extraction may change serialized hashes; another video collection defines a new experiment. The included three-encoder tools are: |
|
|
| ```bash |
| PYTHONPATH="$PWD/code" python code/tools/extract_triple_matched_teacher_bank.py --help |
| PYTHONPATH="$PWD/code" python code/tools/build_triple_nystrom_references.py --help |
| ``` |
|
|
| Use `--expected-count 8000 --shard-rows 32` for extraction. Pass all three pinned encoder locations and identities, the source manifest and CSV hashes, and the video root. For reference construction use `--expected-sample-count 8000 --expected-shard-size 32 --kmeans-iterations 20 --ridge 1e-6`, together with the resulting bank identities. The builder fixes the landmark counts to DINOv3 256, V-JEPA 2 1,024, and VideoMAEv2 256. |
|
|
| For a rebuilt bank, update a local copy of the training config with its new manifest/content hashes and all three reference filenames/hashes before preparing local paths. The optional `*_no_dinov3_*` tools are two-encoder ablation utilities and are not part of this recipe. |
|
|
| ## Local snapshot preparation |
|
|
| All model loaders require existing local directories. Hosted model repository URLs, repository IDs and automatic downloads are omitted from this source release. The legacy-named `download_videomae_v2_large.py` stages an existing local snapshot using `--source-dir` and an empty `--output-dir`; it does not download files. If the source includes a valid pinned `download_manifest.json`, that manifest is preserved for compatibility with frozen reference banks. A snapshot without a manifest gets a new local manifest; rebuild matching reference assets and update their configuration identities before training. `prepare_config.py` reads model identities from an available local feature bank into private local configuration files. |
|
|
