Feature Extraction
Transformers
Safetensors
prism
video
representation-learning
view-invariant
cross-view
egocentric
egoexo4d
emnlp2026
custom_code
Instructions to use litcoderr/prism with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use litcoderr/prism with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="litcoderr/prism", trust_remote_code=True)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("litcoderr/prism", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Edit for tone; remove em dashes
Browse files
README.md
CHANGED
|
@@ -15,22 +15,22 @@ tags:
|
|
| 15 |
- emnlp2026
|
| 16 |
---
|
| 17 |
|
| 18 |
-
# PRISM
|
| 19 |
|
| 20 |
-
|
| 21 |
-
|
| 22 |
|
| 23 |
[Paper][paper] · [Project page][project] · [Code][code]
|
| 24 |
|
| 25 |
PRISM decomposes each video clip into a view-invariant latent `z_vi` and a view-variant
|
| 26 |
-
latent `z_vv`, then recomposes
|
| 27 |
-
`z_vv` of clip B is trained to match the embedding of the correspondingly recombined
|
| 28 |
-
caption.
|
| 29 |
-
between the
|
| 30 |
-
view-invariant stream,
|
| 31 |
-
camera is
|
| 32 |
-
|
| 33 |
-
This checkpoint is trained on
|
| 34 |
by Gemini 3.
|
| 35 |
|
| 36 |
## Usage
|
|
@@ -49,12 +49,12 @@ pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None]
|
|
| 49 |
pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384)
|
| 50 |
valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")
|
| 51 |
|
| 52 |
-
emb = model.encode(pixel_values, valid_mask) # (1, 512)
|
| 53 |
```
|
| 54 |
|
| 55 |
| | |
|
| 56 |
|:---|:---|
|
| 57 |
-
| `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding
|
| 58 |
| `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame |
|
| 59 |
|
| 60 |
**Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to
|
|
@@ -66,10 +66,11 @@ For batched encoding of a clip manifest, see `scripts/encode.sh` in the [code re
|
|
| 66 |
|
| 67 |
## Results
|
| 68 |
|
| 69 |
-
**Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta:
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
depend on execution style rather than action identity,
|
|
|
|
| 73 |
|
| 74 |
| Method | <sub>**EgoExo4D**</sub><br>Retr. ego→exo | <br>exo→ego | <br>avg | <br>Recog. top-1 | <br>Skill | <sub>**EgoExoLearn**</sub><br>Assoc. avg | <br>Antic. avg | <br>Skill |
|
| 75 |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|
|
@@ -91,8 +92,8 @@ tasks, and ahead of the best in-domain model on phase ordering and phase progres
|
|
| 91 |
| ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 |
|
| 92 |
| **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** |
|
| 93 |
|
| 94 |
-
**Robustness to background correlation (UNSCENE)**
|
| 95 |
-
background, averaged over three text encoders
|
| 96 |
against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2.
|
| 97 |
|
| 98 |
Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and
|
|
@@ -102,8 +103,8 @@ the [code repo][code].
|
|
| 102 |
|
| 103 |
| Component | |
|
| 104 |
|:---|:---|
|
| 105 |
-
| Vision backbone | `google/siglip2-so400m-patch14-384`
|
| 106 |
-
| Text backbone | `Qwen/Qwen3-Embedding-0.6B`
|
| 107 |
| Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream |
|
| 108 |
| Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` |
|
| 109 |
| Target encoder θ̄ | EMA of θ, decay 0.998 |
|
|
@@ -126,10 +127,10 @@ are generated online by a local vLLM server running `Qwen/Qwen3-1.7B`. Full reci
|
|
| 126 |
|
| 127 |
## Limitations
|
| 128 |
|
| 129 |
-
Trained on EgoExo4D
|
| 130 |
-
that distribution are untested. Clips longer than 32 s are truncated
|
| 131 |
-
sampled frames. The training pipeline
|
| 132 |
-
captions, which most datasets do not ship.
|
| 133 |
|
| 134 |
## Citation
|
| 135 |
|
|
|
|
| 15 |
- emnlp2026
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# PRISM
|
| 19 |
|
| 20 |
+
Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video
|
| 21 |
+
Representation Learning. EMNLP 2026, main conference.
|
| 22 |
|
| 23 |
[Paper][paper] · [Project page][project] · [Code][code]
|
| 24 |
|
| 25 |
PRISM decomposes each video clip into a view-invariant latent `z_vi` and a view-variant
|
| 26 |
+
latent `z_vv`, then recomposes the two under language supervision: `z_vi` of clip A paired
|
| 27 |
+
with `z_vv` of clip B is trained to match the embedding of the correspondingly recombined
|
| 28 |
+
caption. That composition never occurs in the training data, so semantics that leaked
|
| 29 |
+
between the streams cannot survive it. The clip embedding this model returns is the
|
| 30 |
+
view-invariant stream, which makes cosine similarity a measure of what is happening rather
|
| 31 |
+
than of where the camera is.
|
| 32 |
+
|
| 33 |
+
This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated
|
| 34 |
by Gemini 3.
|
| 35 |
|
| 36 |
## Usage
|
|
|
|
| 49 |
pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384)
|
| 50 |
valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")
|
| 51 |
|
| 52 |
+
emb = model.encode(pixel_values, valid_mask) # (1, 512), L2-normalized
|
| 53 |
```
|
| 54 |
|
| 55 |
| | |
|
| 56 |
|:---|:---|
|
| 57 |
+
| `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding, mean-pooled `z_vi` over valid frames |
|
| 58 |
| `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame |
|
| 59 |
|
| 60 |
**Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to
|
|
|
|
| 66 |
|
| 67 |
## Results
|
| 68 |
|
| 69 |
+
**Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta: +10.4 on
|
| 70 |
+
Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill
|
| 71 |
+
Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82);
|
| 72 |
+
proficiency cues depend on execution style rather than action identity, so they land in the
|
| 73 |
+
view-variant stream.
|
| 74 |
|
| 75 |
| Method | <sub>**EgoExo4D**</sub><br>Retr. ego→exo | <br>exo→ego | <br>avg | <br>Recog. top-1 | <br>Skill | <sub>**EgoExoLearn**</sub><br>Assoc. avg | <br>Antic. avg | <br>Skill |
|
| 76 |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|
|
|
|
| 92 |
| ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 |
|
| 93 |
| **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** |
|
| 94 |
|
| 95 |
+
**Robustness to background correlation (UNSCENE).** On videos whose action contradicts the
|
| 96 |
+
background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA,
|
| 97 |
against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2.
|
| 98 |
|
| 99 |
Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and
|
|
|
|
| 103 |
|
| 104 |
| Component | |
|
| 105 |
|:---|:---|
|
| 106 |
+
| Vision backbone | `google/siglip2-so400m-patch14-384`, frozen and not stored here |
|
| 107 |
+
| Text backbone | `Qwen/Qwen3-Embedding-0.6B`, frozen and not stored here |
|
| 108 |
| Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream |
|
| 109 |
| Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` |
|
| 110 |
| Target encoder θ̄ | EMA of θ, decay 0.998 |
|
|
|
|
| 127 |
|
| 128 |
## Limitations
|
| 129 |
|
| 130 |
+
Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera
|
| 131 |
+
pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated
|
| 132 |
+
to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and
|
| 133 |
+
view-variant captions, which most datasets do not ship.
|
| 134 |
|
| 135 |
## Citation
|
| 136 |
|