--- license: mit library_name: transformers pipeline_tag: feature-extraction base_model: - google/siglip2-so400m-patch14-384 - Qwen/Qwen3-Embedding-0.6B tags: - video - representation-learning - view-invariant - cross-view - egocentric - egoexo4d - emnlp2026 --- # PRISM Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning. EMNLP 2026, main conference. [Paper][paper] · [Project page][project] · [Code][code] PRISM is a video encoder that captures viewpoint-invariant action semantics, matching the same action across egocentric and exocentric views. It decomposes each video into view-invariant and view-variant latents and recomposes them across videos under language supervision, which stays semantically valid beyond the co-occurrences observed in training. This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated by Gemini 3. ## Usage ```python import torch from transformers import AutoModel, AutoImageProcessor model = AutoModel.from_pretrained( "litcoderr/prism", trust_remote_code=True, dtype=torch.bfloat16 ).eval().cuda() proc = AutoImageProcessor.from_pretrained(model.config.vision_backbone_name) frames = [...] # list[PIL.Image], sampled at 4 fps pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None] pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384) valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda") emb = model.encode(pixel_values, valid_mask) # (1, 512), L2-normalized ``` | | | |:---|:---| | `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding, mean-pooled `z_vi` over valid frames | | `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame | **Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to `T = 128` (32 s), preprocessed by the SigLIP2 image processor. `valid_mask` is `(B, T)` bool marking real frames in a padded batch. Both encode paths run under `torch.no_grad()` and use the EMA target encoder θ̄. For batched encoding of a clip manifest, see `scripts/encode.sh` in the [code repo][code]. ## Results **Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta: +10.4 on Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82); proficiency cues depend on execution style rather than action identity, so they land in the view-variant stream. | Method | **EgoExo4D**
Retr. ego→exo |
exo→ego |
avg |
Recog. top-1 |
Skill | **EgoExoLearn**
Assoc. avg |
Antic. avg |
Skill | |:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | CLIP | 19.11 | 12.24 | 15.68 | 10.49 | 54.93 | 15.82 | 38.70 | 73.48 | | SigLIP2 | 35.08 | 19.72 | 27.40 | 13.86 | 55.57 | 26.6 | 64.60 | **76.03** | | LaViLa | 34.91 | 12.02 | 23.47 | 26.43 | 54.10 | 27.20 | 62.83 | 68.44 | | SUM-L | 47.14 | 32.77 | 39.96 | 24.83 | 55.10 | 4.64 | 45.50 | 65.31 | | ViewpointRosetta | 58.14 | 47.21 | 52.68 | 34.47 | **55.82** | 32.32 | 62.14 | 73.70 | | **PRISM** | **75.89** | **50.27** | **63.08** | **41.93** | 55.28 | **43.86** | **69.46** | 68.53 | **Fine-grained temporal modeling (AE2).** Best among out-of-domain models on all four tasks, and ahead of the best in-domain model on phase ordering and phase progression. | Method | AE2 videos | Frame retr.
mAP@10 | Phase order
Kendall's τ | Phase class.
F1 | Phase prog.
R² | |:---|:--:|:--:|:--:|:--:|:--:| | GTA | ✔ | 68.08 | 0.464 | 67.77 | 0.322 | | AE2 | ✔ | 73.20 | 0.562 | 74.47 | 0.480 | | SigLIP2 | ✘ | 45.56 | 0.020 | 43.91 | −1.322 | | ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 | | **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** | **Robustness to background correlation (UNSCENE).** On videos whose action contradicts the background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA, against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2. Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and the [code repo][code]. ## Architecture | Component | | |:---|:---| | Vision backbone | `google/siglip2-so400m-patch14-384`, frozen and not stored here | | Text backbone | `Qwen/Qwen3-Embedding-0.6B`, frozen and not stored here | | Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream | | Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` | | Target encoder θ̄ | EMA of θ, decay 0.998 | | Embedding dim | 512 | This repo holds trained weights only: θ, φ, θ̄, and the logit scale. The two backbones are re-downloaded from the Hub when the model is constructed, so nothing about them is duplicated here. ## Training EgoExo4D, 6 epochs, batch 4 × 7 GPUs (DDP), constant-with-warmup lr 7e-5 (10% warmup), weight decay 0.01, grad-clip 1.0, bf16, seed 42. Video sampled at 4 fps, clips capped at 32 s / 128 frames, 384×384. Objective `L = 1.0 · L_decomp + 0.5 · L_temp`, InfoNCE all-gathered across ranks, sliding-shift augmentation on. Per-clip view-invariant / view-variant captions are a *provided input*; recomposed captions are generated online by a local vLLM server running `Qwen/Qwen3-1.7B`. Full recipe in the [code repo][code]. ## Limitations Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and view-variant captions, which most datasets do not ship. ## Citation - [ ] TODO: Add citation once published. MIT licensed. The frozen backbones keep their own licenses. [paper]: https://github.com/litcoderr/prism [project]: https://github.com/litcoderr/prism [code]: https://github.com/litcoderr/prism