| --- |
| license: other |
| tags: |
| - video |
| - self-supervised |
| - video-understanding |
| - mamba |
| - dinov3 |
| --- |
| |
| # VECTRA — causal video encoder checkpoints |
|
|
| Frozen **DINOv3 ViT-L/16** dense patches → interleaved **A-A-M** temporal encoder |
| (2 spatial self-attention layers over the 16×16 patch grid + 1 Mamba-3 layer |
| running causally over time, *per patch track*) → 16×16 region-state grid per |
| frame. Pretrained self-supervised with a **region-transport appearance loss** and |
| a **belief (uncertainty) loss**; evaluated with frozen-encoder attentive probes. |
|
|
| Code: https://github.com/fw-ic/ARVR_videoencoder — `README.md` on `main` covers |
| the method, repository layout, environment setup, and how to run training and |
| evaluation. The full experimental record (per-run metrics, loss curves, ablation |
| ledgers, research notes, and the `EXPERIMENT_LOG.md` master index) is preserved |
| at the **`results-archive`** tag in that repository: |
|
|
| ```bash |
| git fetch origin tag results-archive && git checkout results-archive |
| ``` |
|
|
| --- |
|
|
| ## Checkpoints in this upload (2026-08-21) |
|
|
| All are **full** checkpoints: `model_state_dict` (student encoder + EMA teacher + |
| belief heads + frozen anchor pools) **and** `optimizer_state_dict`, so each can be |
| resumed or fine-tuned, not merely probed. Every run's exact hyperparameters are in |
| `checkpoint["train_args"]`. |
|
|
| | file | size | architecture | pretraining | SSV2 | K400 | |
| |---|---|---|---|---|---| |
| | `grandv2_24L_epoch001.pt` | 5.3 GB | 24L, 359M | 1 epoch of 788k mix (K400 200k + SSV2 200k + OpenVid 388k), belief v2 + A1 matchability | **51.42** | **75.99** | |
| | `depth42L_epoch001.pt` | 9.0 GB | **42L, 609M** | same data & recipe, 1 epoch | 39.31 ¹ | 74.48 | |
| | `paper_24L_1ep_transportonly.pt` | 5.4 GB | 24L, 359M | 1 epoch, **transport only (belief off)** | 52.12 ² | 77.58 ² | |
| | `july_24L_5ep_quantile.pt` | 5.0 GB | 24L, 359M | **5 epochs**, older quantile-evidence belief | 47.28 ² | 76.82 ² | |
| | `belief_champion_6L.pt` | 1.6 GB | 6L, 108M | 100k K400 × 3 epochs, belief v2 champion | 42.65 | — | |
|
|
| SSV2 / K400 numbers are top-1 on the **full test split** unless marked. |
| ¹ Probe-budget confounded — see "Known caveats". ² 5k-subsample monitor protocol. |
|
|
| **Evaluation protocol.** Frozen encoder; a 3-block bidirectional self-attention + |
| 1 cross-attention probe with factorized 3D RoPE is trained on the dataset's |
| *validation* split and evaluated on *test* (the encoder was pretrained on the |
| train splits, so both probe-training and evaluation data are unseen). |
|
|
| --- |
|
|
| ## Which one do you want? |
|
|
| - **Best encoder → `grandv2_24L_epoch001.pt`.** Everything current references it. |
| - **Reproducing the paper → `paper_24L_1ep_transportonly.pt`.** Note this is the |
| 1-epoch **transport-only** model; the belief loss is *off* in it. |
| - **Belief / uncertainty work → `belief_champion_6L.pt`.** Small, and it is the |
| checkpoint behind the calibration and split-conformal study. |
| - **Depth studies → `depth42L_epoch001.pt`.** |
|
|
| --- |
|
|
| ## Known caveats — please read before quoting numbers |
|
|
| **1. The paper checkpoint is transport-only.** `paper_24L_1ep_transportonly.pt` |
| (52.12 / 77.58) was trained for **1 epoch with the belief loss disabled**. The |
| 5-epoch belief run is a *different* checkpoint that scored **47.28** on SSV2. |
| Earlier internal notes misattributed 52.12 to the 5-epoch run. |
|
|
| **2. Depth did not help.** 42L (609M) lost to 24L (359M) on both benchmarks at |
| matched data and recipe. K400 is the clean comparison — the 42L probe there had |
| *more* optimizer steps than the 24L reference and still scored 1.5 lower. The |
| SSV2 gap of 12 points is **not** a clean measurement: that probe ran at global |
| batch 128 (3,880 optimizer steps) against the reference's global batch 8 (61,960). |
| Direction is trustworthy; magnitude is not. |
|
|
| **3. Most probe numbers are undertrained floors.** Probe schedule length alone |
| moved Diving48 from 85.53 (20 epochs) to 91.88 (50 epochs), converged at neither. |
| These encoders' probe curves are strongly back-loaded — treat every number here |
| as a lower bound and never compare across probe budgets. |
|
|
| **4. Optimization passes dominate architecture.** The same 6L model scores 42.65 |
| at 3 pretraining epochs and **51.10 at 20** — a larger gain than every loss-design |
| improvement in the project combined. Numbers here reflect very few epochs. |
|
|
| **5. Offline fixed-window encoder.** Despite the causal Mamba backbone, these are |
| trained and evaluated on fixed ~10 s windows at 5 fps with clip-normalized frame |
| positions, and the probes read bidirectionally. They are **not** streaming models. |
|
|
| --- |
|
|
| ## Loading |
|
|
| ```python |
| import torch |
| ckpt = torch.load("grandv2_24L_epoch001.pt", map_location="cpu", weights_only=False) |
| args = ckpt["train_args"] # every hyperparameter used |
| state = ckpt["model_state_dict"] # student_encoder.* / teacher_encoder.* / belief heads |
| ``` |
|
|
| Use `download/vjepa_single_view_probe/model.py:build_frozen_region_encoder` from |
| the code repo to rebuild the encoder from `train_args` and load the weights; it |
| reconstructs the exact architecture (layer count, region grid, residual mode, |
| register tokens) from the checkpoint rather than requiring you to specify it. |
|
|
| DINOv3 ViT-L/16 weights are **not** included — the encoder consumes frozen DINOv3 |
| features and expects them supplied at load time. |
|
|
| --- |
|
|
| ## Older files in this repo |
|
|
| Checkpoints uploaded before 2026-07-28 (under `mixed_openvid/` and |
| `k400_ssv2_400k_.../`) are from earlier generations, including a 12-layer |
| `d_state=256` era and the 5-epoch quantile run. They predate the belief-v2 recipe |
| and the region-grid-16 configuration; prefer the files listed above. |
|
|
