--- license: mit library_name: transformers pipeline_tag: feature-extraction base_model: - google/siglip2-so400m-patch14-384 - Qwen/Qwen3-Embedding-0.6B tags: - video - representation-learning - view-invariant - cross-view - egocentric - egoexo4d - emnlp2026 --- # PRISM Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning. EMNLP 2026, main conference. [Paper][paper] · [Project page][project] · [Code][code] PRISM separates what an action is from the viewpoint and scene it happens to be seen in. Language is what makes that separation trainable: descriptions recombine freely, so the action of one clip can be put together with the context of another into a composition that never occurred in the training data, and the model is asked to predict the meaning of that composition. Semantics that leaked between the two factors cannot survive the swap. What this model returns is the view-invariant side of the split, so cosine similarity measures what is happening rather than where the camera is. This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated by Gemini 3. ## Usage ```python import torch from transformers import AutoModel, AutoImageProcessor model = AutoModel.from_pretrained( "litcoderr/prism", trust_remote_code=True, dtype=torch.bfloat16 ).eval().cuda() proc = AutoImageProcessor.from_pretrained(model.config.vision_backbone_name) frames = [...] # list[PIL.Image], sampled at 4 fps pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None] pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384) valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda") emb = model.encode(pixel_values, valid_mask) # (1, 512), L2-normalized ``` | | | |:---|:---| | `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding, mean-pooled `z_vi` over valid frames | | `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame | **Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to `T = 128` (32 s), preprocessed by the SigLIP2 image processor. `valid_mask` is `(B, T)` bool marking real frames in a padded batch. Both encode paths run under `torch.no_grad()` and use the EMA target encoder θ̄. For batched encoding of a clip manifest, see `scripts/encode.sh` in the [code repo][code]. ## Results **Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta: +10.4 on Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82); proficiency cues depend on execution style rather than action identity, so they land in the view-variant stream. | Method | **EgoExo4D**
Retr. ego→exo |
exo→ego |
avg |
Recog. top-1 |
Skill | **EgoExoLearn**
Assoc. avg |
Antic. avg |
Skill | |:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | CLIP | 19.11 | 12.24 | 15.68 | 10.49 | 54.93 | 15.82 | 38.70 | 73.48 | | SigLIP2 | 35.08 | 19.72 | 27.40 | 13.86 | 55.57 | 26.6 | 64.60 | **76.03** | | LaViLa | 34.91 | 12.02 | 23.47 | 26.43 | 54.10 | 27.20 | 62.83 | 68.44 | | SUM-L | 47.14 | 32.77 | 39.96 | 24.83 | 55.10 | 4.64 | 45.50 | 65.31 | | ViewpointRosetta | 58.14 | 47.21 | 52.68 | 34.47 | **55.82** | 32.32 | 62.14 | 73.70 | | **PRISM** | **75.89** | **50.27** | **63.08** | **41.93** | 55.28 | **43.86** | **69.46** | 68.53 | **Fine-grained temporal modeling (AE2).** Best among out-of-domain models on all four tasks, and ahead of the best in-domain model on phase ordering and phase progression. | Method | AE2 videos | Frame retr.
mAP@10 | Phase order
Kendall's τ | Phase class.
F1 | Phase prog.
R² | |:---|:--:|:--:|:--:|:--:|:--:| | GTA | ✔ | 68.08 | 0.464 | 67.77 | 0.322 | | AE2 | ✔ | 73.20 | 0.562 | 74.47 | 0.480 | | SigLIP2 | ✘ | 45.56 | 0.020 | 43.91 | −1.322 | | ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 | | **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** | **Robustness to background correlation (UNSCENE).** On videos whose action contradicts the background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA, against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2. Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and the [code repo][code]. ## Architecture | Component | | |:---|:---| | Vision backbone | `google/siglip2-so400m-patch14-384`, frozen and not stored here | | Text backbone | `Qwen/Qwen3-Embedding-0.6B`, frozen and not stored here | | Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream | | Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` | | Target encoder θ̄ | EMA of θ, decay 0.998 | | Embedding dim | 512 | This repo holds trained weights only: θ, φ, θ̄, and the logit scale. The two backbones are re-downloaded from the Hub when the model is constructed, so nothing about them is duplicated here. ## Training EgoExo4D, 6 epochs, batch 4 × 7 GPUs (DDP), constant-with-warmup lr 7e-5 (10% warmup), weight decay 0.01, grad-clip 1.0, bf16, seed 42. Video sampled at 4 fps, clips capped at 32 s / 128 frames, 384×384. Objective `L = 1.0 · L_decomp + 0.5 · L_temp`, InfoNCE all-gathered across ranks, sliding-shift augmentation on. Per-clip view-invariant / view-variant captions are a *provided input*; recomposed captions are generated online by a local vLLM server running `Qwen/Qwen3-1.7B`. Full recipe in the [code repo][code]. ## Limitations Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and view-variant captions, which most datasets do not ship. ## Citation - [ ] TODO: Add citation once published. MIT licensed. The frozen backbones keep their own licenses. [paper]: https://github.com/litcoderr/prism [project]: https://github.com/litcoderr/prism [code]: https://github.com/litcoderr/prism