File size: 5,812 Bytes
2ea2835 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # 14B Asset Preparation
The evaluated model is the **three-encoder step-80 non-EMA generator**, trained on **8,000 reference videos**. Its asset identities are recorded in [`configs/train.yaml`](../configs/train.yaml). The former two-encoder / 8,852-video bank is a different experiment and cannot reproduce this model.
Prepare matching model and reference assets locally; this anonymous release does not embed hosted resource locations. Use the matching three-encoder checkpoint and reference assets; filenames or the step number alone do not identify the experiment. This source repository does not include model weights or reference tensors. A final distilled model is not interchangeable with the **causal ODE initialization** needed for training.
## Directory layout
```text
ASSET_ROOT/
├── models/
│ ├── Wan2.1-T2V-14B/ # Base generator, T5, tokenizer, and VAE
│ ├── dinov3-src/ # Official DINOv3 source at the pinned commit
│ ├── dinov3_vitl16_lvd1689m.pth
│ ├── vjepa2-vitl-fpc64-256/
│ └── VideoMAEv2-Large/ # Includes download_manifest.json
├── checkpoints/
│ └── causal-ode-init.safetensors
├── teacher_bank/
│ ├── manifest.json
│ └── ... # Preserve all shard-relative paths
├── source/
│ ├── matched.jsonl
│ └── matched.meta.json
├── references/
│ ├── dinov3.pt
│ ├── vjepa2.pt
│ └── videomae_v2.pt
└── videos/ # Feature extraction only; not decoded during bank-based training
```
Preserve original serialized manifests, reference packs, and shards. The three reference files may be renamed as above without changing their bytes. `prepare_config.py` reads DINOv3 and VideoMAEv2 metadata identities from the bank and keeps them separate from local encoder load paths. These local settings are excluded from Git.
## Frozen encoders
Wan2.1-T2V-14B includes the shared T5 weights, VAE weights, and tokenizer. Place its files in the directory described in the model README. Before offline training, prepare the other encoders:
```bash
git clone https://github.com/facebookresearch/dinov3.git "$ASSET_ROOT/models/dinov3-src"
git -C "$ASSET_ROOT/models/dinov3-src" checkout 6876159a11b4df116f30f667f8c9888617df0751
# Obtain dinov3_vitl16_lvd1689m.pth through the official DINOv3 model access process.
# Place the matching model files at the local paths described below.
PYTHONPATH="$PWD/code" python code/tools/download_videomae_v2_large.py \
--source-dir /path/to/local/VideoMAEv2-Large \
--output-dir "$ASSET_ROOT/models/VideoMAEv2-Large"
```
The exact encoder hashes are in the training config; a newer checkpoint is not interchangeable with a frozen reference bank. VideoMAEv2 uses revision `9981a9c8f77118c421e5228e1b219468a4b0238d`.
| Encoder | Input and representation | Nyström landmarks | RBF bandwidth |
|---|---|---:|---:|
| DINOv3 ViT-L/16 | 20 sampled frames resized to 256²; final normalized patch features; mean each group of five frames; 1,024 × 1,024 matrix | 256 | 341.2794189453125 |
| V-JEPA 2 ViT-L | Frames 17–80; short side 292, center crop 256²; block index 15; 1,024 × 1,024 matrix | 1,024 | 3025.305908203125 |
| VideoMAEv2-Large | Frames 1–80 split into five 16-frame clips; short side/crop 224; block index 15; 256 tokens per clip, 1,280 × 1,024 per video | 256 | 3204.123291015625 |
VideoMAEv2 shares one frozen encoder across the five clips. All branches use ImageNet pixel normalization. The video-level kernel is `exp(-||X-Y||_F² / (2 sigma²))`, without dividing the distance by the number of tokens or channels. The fixed Nyström references use ridge `1e-6` and 20 k-means iterations. The loss normalization epsilon is separately `1e-4`. Full metadata is in [`reference-settings.json`](../provenance/reference-settings.json).
## Rebuilding a reference bank
Rebuilding requires the original captioned videos and matched manifest. Re-extraction may change serialized hashes; another video collection defines a new experiment. The included three-encoder tools are:
```bash
PYTHONPATH="$PWD/code" python code/tools/extract_triple_matched_teacher_bank.py --help
PYTHONPATH="$PWD/code" python code/tools/build_triple_nystrom_references.py --help
```
Use `--expected-count 8000 --shard-rows 32` for extraction. Pass all three pinned encoder locations and identities, the source manifest and CSV hashes, and the video root. For reference construction use `--expected-sample-count 8000 --expected-shard-size 32 --kmeans-iterations 20 --ridge 1e-6`, together with the resulting bank identities. The builder fixes the landmark counts to DINOv3 256, V-JEPA 2 1,024, and VideoMAEv2 256.
For a rebuilt bank, update a local copy of the training config with its new manifest/content hashes and all three reference filenames/hashes before preparing local paths. The optional `*_no_dinov3_*` tools are two-encoder ablation utilities and are not part of this recipe.
## Local snapshot preparation
All model loaders require existing local directories. Hosted model repository URLs, repository IDs and automatic downloads are omitted from this source release. The legacy-named `download_videomae_v2_large.py` stages an existing local snapshot using `--source-dir` and an empty `--output-dir`; it does not download files. If the source includes a valid pinned `download_manifest.json`, that manifest is preserved for compatibility with frozen reference banks. A snapshot without a manifest gets a new local manifest; rebuild matching reference assets and update their configuration identities before training. `prepare_config.py` reads model identities from an available local feature bank into private local configuration files.
|