| --- |
| license: mit |
| library_name: transformers |
| pipeline_tag: feature-extraction |
| base_model: |
| - google/siglip2-so400m-patch14-384 |
| - Qwen/Qwen3-Embedding-0.6B |
| tags: |
| - video |
| - representation-learning |
| - view-invariant |
| - cross-view |
| - egocentric |
| - egoexo4d |
| - emnlp2026 |
| --- |
| |
| # PRISM |
|
|
| Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video |
| Representation Learning. EMNLP 2026, main conference. |
|
|
| [Paper][paper] · [Project page][project] · [Code][code] |
|
|
| PRISM separates what an action is from the viewpoint and scene it happens to be seen in. |
| Language is what makes that separation trainable: descriptions recombine freely, so the |
| action of one clip can be put together with the context of another into a composition that |
| never occurred in the training data, and the model is asked to predict the meaning of that |
| composition. Semantics that leaked between the two factors cannot survive the swap. What |
| this model returns is the view-invariant side of the split, so cosine similarity measures |
| what is happening rather than where the camera is. |
|
|
| This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated |
| by Gemini 3. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoImageProcessor |
| |
| model = AutoModel.from_pretrained( |
| "litcoderr/prism", trust_remote_code=True, dtype=torch.bfloat16 |
| ).eval().cuda() |
| proc = AutoImageProcessor.from_pretrained(model.config.vision_backbone_name) |
| |
| frames = [...] # list[PIL.Image], sampled at 4 fps |
| pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None] |
| pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384) |
| valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda") |
| |
| emb = model.encode(pixel_values, valid_mask) # (1, 512), L2-normalized |
| ``` |
|
|
| | | | |
| |:---|:---| |
| | `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding, mean-pooled `z_vi` over valid frames | |
| | `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame | |
|
|
| **Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to |
| `T = 128` (32 s), preprocessed by the SigLIP2 image processor. `valid_mask` is `(B, T)` |
| bool marking real frames in a padded batch. Both encode paths run under `torch.no_grad()` |
| and use the EMA target encoder θ̄. |
|
|
| For batched encoding of a clip manifest, see `scripts/encode.sh` in the [code repo][code]. |
|
|
| ## Results |
|
|
| **Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta: +10.4 on |
| Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill |
| Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82); |
| proficiency cues depend on execution style rather than action identity, so they land in the |
| view-variant stream. |
|
|
| | Method | <sub>**EgoExo4D**</sub><br>Retr. ego→exo | <br>exo→ego | <br>avg | <br>Recog. top-1 | <br>Skill | <sub>**EgoExoLearn**</sub><br>Assoc. avg | <br>Antic. avg | <br>Skill | |
| |:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| |
| | CLIP | 19.11 | 12.24 | 15.68 | 10.49 | 54.93 | 15.82 | 38.70 | 73.48 | |
| | SigLIP2 | 35.08 | 19.72 | 27.40 | 13.86 | 55.57 | 26.6 | 64.60 | **76.03** | |
| | LaViLa | 34.91 | 12.02 | 23.47 | 26.43 | 54.10 | 27.20 | 62.83 | 68.44 | |
| | SUM-L | 47.14 | 32.77 | 39.96 | 24.83 | 55.10 | 4.64 | 45.50 | 65.31 | |
| | ViewpointRosetta | 58.14 | 47.21 | 52.68 | 34.47 | **55.82** | 32.32 | 62.14 | 73.70 | |
| | **PRISM** | **75.89** | **50.27** | **63.08** | **41.93** | 55.28 | **43.86** | **69.46** | 68.53 | |
|
|
| **Fine-grained temporal modeling (AE2).** Best among out-of-domain models on all four |
| tasks, and ahead of the best in-domain model on phase ordering and phase progression. |
|
|
| | Method | AE2 videos | Frame retr.<br><sub>mAP@10</sub> | Phase order<br><sub>Kendall's τ</sub> | Phase class.<br><sub>F1</sub> | Phase prog.<br><sub>R²</sub> | |
| |:---|:--:|:--:|:--:|:--:|:--:| |
| | GTA | ✔ | 68.08 | 0.464 | 67.77 | 0.322 | |
| | AE2 | ✔ | 73.20 | 0.562 | 74.47 | 0.480 | |
| | SigLIP2 | ✘ | 45.56 | 0.020 | 43.91 | −1.322 | |
| | ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 | |
| | **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** | |
|
|
| **Robustness to background correlation (UNSCENE).** On videos whose action contradicts the |
| background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA, |
| against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2. |
|
|
| Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and |
| the [code repo][code]. |
|
|
| ## Architecture |
|
|
| | Component | | |
| |:---|:---| |
| | Vision backbone | `google/siglip2-so400m-patch14-384`, frozen and not stored here | |
| | Text backbone | `Qwen/Qwen3-Embedding-0.6B`, frozen and not stored here | |
| | Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream | |
| | Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` | |
| | Target encoder θ̄ | EMA of θ, decay 0.998 | |
| | Embedding dim | 512 | |
|
|
| This repo holds trained weights only: θ, φ, θ̄, and the logit scale. The two backbones are |
| re-downloaded from the Hub when the model is constructed, so nothing about them is |
| duplicated here. |
|
|
| ## Training |
|
|
| EgoExo4D, 6 epochs, batch 4 × 7 GPUs (DDP), constant-with-warmup lr 7e-5 (10% warmup), |
| weight decay 0.01, grad-clip 1.0, bf16, seed 42. Video sampled at 4 fps, clips capped at |
| 32 s / 128 frames, 384×384. Objective `L = 1.0 · L_decomp + 0.5 · L_temp`, InfoNCE |
| all-gathered across ranks, sliding-shift augmentation on. |
|
|
| Per-clip view-invariant / view-variant captions are a *provided input*; recomposed captions |
| are generated online by a local vLLM server running `Qwen/Qwen3-1.7B`. Full recipe in the |
| [code repo][code]. |
|
|
| ## Limitations |
|
|
| Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera |
| pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated |
| to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and |
| view-variant captions, which most datasets do not ship. |
|
|
| ## Citation |
|
|
| - [ ] TODO: Add citation once published. |
|
|
| MIT licensed. The frozen backbones keep their own licenses. |
|
|
| <!-- swap these when the links go live --> |
| [paper]: https://github.com/litcoderr/prism |
| [project]: https://github.com/litcoderr/prism |
| [code]: https://github.com/litcoderr/prism |
|
|
