litcoderr commited on
Commit
a85f4eb
·
verified ·
1 Parent(s): da3fef1

Edit for tone; remove em dashes

Browse files
Files changed (1) hide show
  1. README.md +26 -25
README.md CHANGED
@@ -15,22 +15,22 @@ tags:
15
  - emnlp2026
16
  ---
17
 
18
- # PRISM — EgoExo4D
19
 
20
- **P**redictive **R**ecomposition v**I**a **S**emantic Latent Deco**M**position for
21
- View-invariant Video Representation Learning — **EMNLP 2026 (Main Conference)**.
22
 
23
  [Paper][paper] · [Project page][project] · [Code][code]
24
 
25
  PRISM decomposes each video clip into a view-invariant latent `z_vi` and a view-variant
26
- latent `z_vv`, then recomposes them under language supervision `z_vi` of clip A with
27
- `z_vv` of clip B is trained to match the embedding of the correspondingly recombined
28
- caption. Because that composition never occurs in the training data, semantics that leaked
29
- between the two streams cannot survive it. The clip embedding this model returns is the
30
- view-invariant stream, so cosine similarity measures *what is happening*, not *where the
31
- camera is*.
32
-
33
- This checkpoint is trained on **EgoExo4D**, with ego and exo views and captions generated
34
  by Gemini 3.
35
 
36
  ## Usage
@@ -49,12 +49,12 @@ pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None]
49
  pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384)
50
  valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")
51
 
52
- emb = model.encode(pixel_values, valid_mask) # (1, 512) L2-normalized
53
  ```
54
 
55
  | | |
56
  |:---|:---|
57
- | `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding mean-pooled `z_vi` over valid frames |
58
  | `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame |
59
 
60
  **Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to
@@ -66,10 +66,11 @@ For batched encoding of a clip manifest, see `scripts/encode.sh` in the [code re
66
 
67
  ## Results
68
 
69
- **Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta:
70
- **+10.4** Retrieval and **+11.5** Association, plus +7.46 Recognition and +7.32
71
- Anticipation. On Skill Assessment PRISM is comparable (55.28 vs 55.82) — proficiency cues
72
- depend on execution style rather than action identity, and land in the view-variant stream.
 
73
 
74
  | Method | <sub>**EgoExo4D**</sub><br>Retr. ego→exo | <br>exo→ego | <br>avg | <br>Recog. top-1 | <br>Skill | <sub>**EgoExoLearn**</sub><br>Assoc. avg | <br>Antic. avg | <br>Skill |
75
  |:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
@@ -91,8 +92,8 @@ tasks, and ahead of the best in-domain model on phase ordering and phase progres
91
  | ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 |
92
  | **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** |
93
 
94
- **Robustness to background correlation (UNSCENE)** videos whose action contradicts the
95
- background, averaged over three text encoders: PRISM reaches **14.9** R@10 / **0.181** RSA
96
  against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2.
97
 
98
  Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and
@@ -102,8 +103,8 @@ the [code repo][code].
102
 
103
  | Component | |
104
  |:---|:---|
105
- | Vision backbone | `google/siglip2-so400m-patch14-384` — **frozen**, not stored here |
106
- | Text backbone | `Qwen/Qwen3-Embedding-0.6B` — **frozen**, not stored here |
107
  | Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream |
108
  | Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` |
109
  | Target encoder θ̄ | EMA of θ, decay 0.998 |
@@ -126,10 +127,10 @@ are generated online by a local vLLM server running `Qwen/Qwen3-1.7B`. Full reci
126
 
127
  ## Limitations
128
 
129
- Trained on EgoExo4D skill-centric, mostly indoor, ego/exo camera pairs. Domains far from
130
- that distribution are untested. Clips longer than 32 s are truncated to the first 128
131
- sampled frames. The training pipeline depends on decoupled view-invariant / view-variant
132
- captions, which most datasets do not ship.
133
 
134
  ## Citation
135
 
 
15
  - emnlp2026
16
  ---
17
 
18
+ # PRISM
19
 
20
+ Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video
21
+ Representation Learning. EMNLP 2026, main conference.
22
 
23
  [Paper][paper] · [Project page][project] · [Code][code]
24
 
25
  PRISM decomposes each video clip into a view-invariant latent `z_vi` and a view-variant
26
+ latent `z_vv`, then recomposes the two under language supervision: `z_vi` of clip A paired
27
+ with `z_vv` of clip B is trained to match the embedding of the correspondingly recombined
28
+ caption. That composition never occurs in the training data, so semantics that leaked
29
+ between the streams cannot survive it. The clip embedding this model returns is the
30
+ view-invariant stream, which makes cosine similarity a measure of what is happening rather
31
+ than of where the camera is.
32
+
33
+ This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated
34
  by Gemini 3.
35
 
36
  ## Usage
 
49
  pixel_values = pixel_values.to("cuda", torch.bfloat16) # (1, T, 3, 384, 384)
50
  valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")
51
 
52
+ emb = model.encode(pixel_values, valid_mask) # (1, 512), L2-normalized
53
  ```
54
 
55
  | | |
56
  |:---|:---|
57
+ | `model.encode(pixel_values, valid_mask)` | `(B, 512)` L2-normalized clip embedding, mean-pooled `z_vi` over valid frames |
58
  | `model.encode_streams(pixel_values, valid_mask)` | `{"z_vi_seq", "z_vv_seq"}`, each `(B, T, 512)` per-frame |
59
 
60
  **Inputs.** `pixel_values` is `(B, T, 3, 384, 384)`, frames sampled at 4 fps, up to
 
66
 
67
  ## Results
68
 
69
+ **Cross-view semantic alignment.** Gains over the best baseline, ViewpointRosetta: +10.4 on
70
+ Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill
71
+ Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82);
72
+ proficiency cues depend on execution style rather than action identity, so they land in the
73
+ view-variant stream.
74
 
75
  | Method | <sub>**EgoExo4D**</sub><br>Retr. ego→exo | <br>exo→ego | <br>avg | <br>Recog. top-1 | <br>Skill | <sub>**EgoExoLearn**</sub><br>Assoc. avg | <br>Antic. avg | <br>Skill |
76
  |:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
 
92
  | ViewpointRosetta | ✘ | 54.17 | 0.047 | 46.93 | −0.150 |
93
  | **PRISM** | ✘ | **70.53** | **0.601** | **73.57** | **0.647** |
94
 
95
+ **Robustness to background correlation (UNSCENE).** On videos whose action contradicts the
96
+ background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA,
97
  against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2.
98
 
99
  Full tables, ablations, and the DEVIAS stream-probe analysis are in the [paper][paper] and
 
103
 
104
  | Component | |
105
  |:---|:---|
106
+ | Vision backbone | `google/siglip2-so400m-patch14-384`, frozen and not stored here |
107
+ | Text backbone | `Qwen/Qwen3-Embedding-0.6B`, frozen and not stored here |
108
  | Decompositional Encoder θ | 4-layer Q-Former (2 queries → `z_vi`, `z_vv`) + 12-layer causal temporal stack per stream |
109
  | Compositional Latent Predictor φ | 4-layer causal transformer over `concat(z_vv, z_vi)`, with `cls_head` / `vi_head` / `vv_head` |
110
  | Target encoder θ̄ | EMA of θ, decay 0.998 |
 
127
 
128
  ## Limitations
129
 
130
+ Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera
131
+ pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated
132
+ to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and
133
+ view-variant captions, which most datasets do not ship.
134
 
135
  ## Citation
136