README.md

---
license: mit
library_name: transformers
pipeline_tag: feature-extraction
base_model:
  - google/siglip2-so400m-patch14-384
  - Qwen/Qwen3-Embedding-0.6B
tags:
  - video
  - representation-learning
  - view-invariant
  - cross-view
  - egocentric
  - egoexo4d
  - emnlp2026
---

# PRISM

Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video
Representation Learning. EMNLP 2026, main conference.

[Paper][paper] · [Project page][project] · [Code][code]

PRISM is a video encoder that captures viewpoint-invariant action semantics, matching the
same action across egocentric and exocentric views. It decomposes each video into
view-invariant and view-variant latents and recomposes them across videos under language
supervision, which stays semantically valid beyond the co-occurrences observed in training.

This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated
by Gemini 3.

## Usage

```python
import torch
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(
    "litcoderr/prism", trust_remote_code=True, dtype=torch.bfloat16
).eval().cuda()
proc = AutoImageProcessor.from_pretrained(model.config.vision_backbone_name)

frames = [...]                                              # list[PIL.Image], sampled at 4 fps
pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None]
pixel_values = pixel_values.to("cuda", torch.bfloat16)      # (1, T, 3, 384, 384)
valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")

emb = model.encode(pixel_values, valid_mask)                # (1, 512), L2-normalized
```

| | |
|:---|:---|
| `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding, mean-pooled `z_vi` over valid frames |
| `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame |

**Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to
`T = 128` (32 s), preprocessed by the SigLIP2 image processor. `valid_mask` is `(B, T)`
bool marking real frames in a padded batch. Both encode paths run under `torch.no_grad()`
and use the EMA target encoder θ̄.

For batched encoding of a clip manifest, see `scripts/encode.sh` in the [code repo][code].

## Results

**Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta: +10.4 on
Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill
Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82);
proficiency cues depend on execution style rather than action identity, so they land in the
view-variant stream.

| Method | <sub>**EgoExo4D**</sub><br>Retr. ego→exo | <br>exo→ego | <br>avg | <br>Recog. top-1 | <br>Skill | <sub>**EgoExoLearn**</sub><br>Assoc. avg | <br>Antic. avg | <br>Skill |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| CLIP | 19.11 | 12.24 | 15.68 | 10.49 | 54.93 | 15.82 | 38.70 | 73.48 |
| SigLIP2 | 35.08 | 19.72 | 27.40 | 13.86 | 55.57 | 26.6 | 64.60 | **76.03** |
| LaViLa | 34.91 | 12.02 | 23.47 | 26.43 | 54.10 | 27.20 | 62.83 | 68.44 |
| SUM-L | 47.14 | 32.77 | 39.96 | 24.83 | 55.10 | 4.64 | 45.50 | 65.31 |
| ViewpointRosetta | 58.14 | 47.21 | 52.68 | 34.47 | **55.82** | 32.32 | 62.14 | 73.70 |
| **PRISM** | **75.89** | **50.27** | **63.08** | **41.93** | 55.28 | **43.86** | **69.46** | 68.53 |

**Fine-grained temporal modeling (AE2).** Best among out-of-domain models on all four
tasks, and ahead of the best in-domain model on phase ordering and phase progression.

| Method | AE2 videos | Frame retr.<br><sub>mAP@10</sub> | Phase order<br><sub>Kendall's τ</sub> | Phase class.<br><sub>F1</sub> | Phase prog.<br><sub>R²</sub> |
|:---|:--:|:--:|:--:|:--:|:--:|
| GTA | ✔ | 68.08 | 0.464 | 67.77 | 0.322 |
| AE2 | ✔ | 73.20 | 0.562 | 74.47 | 0.480 |
| SigLIP2 | ✘ | 45.56 | 0.020 | 43.91 | −1.322 |
| ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 |
| **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** |

**Robustness to background correlation (UNSCENE).** On videos whose action contradicts the
background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA,
against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2.

Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and
the [code repo][code].

## Architecture

| Component | |
|:---|:---|
| Vision backbone | `google/siglip2-so400m-patch14-384`, frozen and not stored here |
| Text backbone | `Qwen/Qwen3-Embedding-0.6B`, frozen and not stored here |
| Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream |
| Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` |
| Target encoder θ̄ | EMA of θ, decay 0.998 |
| Embedding dim | 512 |

This repo holds trained weights only: θ, φ, θ̄, and the logit scale. The two backbones are
re-downloaded from the Hub when the model is constructed, so nothing about them is
duplicated here.

## Training

EgoExo4D, 6 epochs, batch 4 × 7 GPUs (DDP), constant-with-warmup lr 7e-5 (10% warmup),
weight decay 0.01, grad-clip 1.0, bf16, seed 42. Video sampled at 4 fps, clips capped at
32 s / 128 frames, 384×384. Objective `L = 1.0 · L_decomp + 0.5 · L_temp`, InfoNCE
all-gathered across ranks, sliding-shift augmentation on.

Per-clip view-invariant / view-variant captions are a *provided input*; recomposed captions
are generated online by a local vLLM server running `Qwen/Qwen3-1.7B`. Full recipe in the
[code repo][code].

## Limitations

Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera
pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated
to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and
view-variant captions, which most datasets do not ship.

## Citation

- [ ] TODO: Add citation once published.

MIT licensed. The frozen backbones keep their own licenses.

<!-- swap these when the links go live -->
[paper]:   https://github.com/litcoderr/prism
[project]: https://github.com/litcoderr/prism
[code]:    https://github.com/litcoderr/prism