README.md

metadata

license: mit
library_name: transformers
pipeline_tag: feature-extraction
base_model:
  - google/siglip2-so400m-patch14-384
  - Qwen/Qwen3-Embedding-0.6B
tags:
  - video
  - representation-learning
  - view-invariant
  - cross-view
  - egocentric
  - egoexo4d
  - emnlp2026

PRISM

Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning. EMNLP 2026, main conference.

Paper · Project page · Code

PRISM is a video encoder that captures viewpoint-invariant action semantics, matching the same action across egocentric and exocentric views. It decomposes each video into view-invariant and view-variant latents and recomposes them across videos under language supervision, which stays semantically valid beyond the co-occurrences observed in training.

This checkpoint is trained on EgoExo4D, using both ego and exo views and captions generated by Gemini 3.

Usage

import torch
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(
    "litcoderr/prism", trust_remote_code=True, dtype=torch.bfloat16
).eval().cuda()
proc = AutoImageProcessor.from_pretrained(model.config.vision_backbone_name)

frames = [...]                                              # list[PIL.Image], sampled at 4 fps
pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None]
pixel_values = pixel_values.to("cuda", torch.bfloat16)      # (1, T, 3, 384, 384)
valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")

emb = model.encode(pixel_values, valid_mask)                # (1, 512), L2-normalized

model.encode(pixel_values, valid_mask)	(B, 512) L2-normalized clip embedding, mean-pooled z_vi over valid frames
model.encode_streams(pixel_values, valid_mask)	{"z_vi_seq", "z_vv_seq"}, each (B, T, 512) per-frame

Inputs. pixel_values is (B, T, 3, 384, 384), frames sampled at 4 fps, up to T = 128 (32 s), preprocessed by the SigLIP2 image processor. valid_mask is (B, T) bool marking real frames in a padded batch. Both encode paths run under torch.no_grad() and use the EMA target encoder θ̄.

For batched encoding of a clip manifest, see scripts/encode.sh in the code repo.

Results

Cross-view semantic alignment. Gains over the best baseline, ViewpointRosetta: +10.4 on Retrieval, +11.5 on Association, +7.46 on Recognition, +7.32 on Anticipation. Skill Assessment is the exception, where PRISM (55.28) only matches ViewpointRosetta (55.82); proficiency cues depend on execution style rather than action identity, so they land in the view-variant stream.

Method	EgoExo4D Retr. ego→exo	exo→ego	avg	Recog. top-1	Skill	EgoExoLearn Assoc. avg	Antic. avg	Skill
CLIP	19.11	12.24	15.68	10.49	54.93	15.82	38.70	73.48
SigLIP2	35.08	19.72	27.40	13.86	55.57	26.6	64.60	76.03
LaViLa	34.91	12.02	23.47	26.43	54.10	27.20	62.83	68.44
SUM-L	47.14	32.77	39.96	24.83	55.10	4.64	45.50	65.31
ViewpointRosetta	58.14	47.21	52.68	34.47	55.82	32.32	62.14	73.70
PRISM	75.89	50.27	63.08	41.93	55.28	43.86	69.46	68.53

Fine-grained temporal modeling (AE2). Best among out-of-domain models on all four tasks, and ahead of the best in-domain model on phase ordering and phase progression.

Method	AE2 videos	Frame retr. mAP@10	Phase order Kendall's τ	Phase class. F1	Phase prog. R²
GTA	✔	68.08	0.464	67.77	0.322
AE2	✔	73.20	0.562	74.47	0.480
SigLIP2	✘	45.56	0.020	43.91	−1.322
ViewpointRosetta	✘	54.17	0.047	46.93	−0.150
PRISM	✘	70.53	0.601	73.57	0.647

Robustness to background correlation (UNSCENE). On videos whose action contradicts the background, averaged over three text encoders, PRISM reaches 14.9 R@10 and 0.181 RSA, against 7.5 / 0.098 for ViewpointRosetta and 14.1 / 0.146 for DINOv2.

Full tables, ablations, and the DEVIAS stream-probe analysis are in the paper and the code repo.

Architecture

Component
Vision backbone	google/siglip2-so400m-patch14-384, frozen and not stored here
Text backbone	Qwen/Qwen3-Embedding-0.6B, frozen and not stored here
Decompositional Encoder θ	4-layer Q-Former (2 queries → z_vi, z_vv) + 12-layer causal temporal stack per stream
Compositional Latent Predictor φ	4-layer causal transformer over concat(z_vv, z_vi), with cls_head / vi_head / vv_head
Target encoder θ̄	EMA of θ, decay 0.998
Embedding dim	512

This repo holds trained weights only: θ, φ, θ̄, and the logit scale. The two backbones are re-downloaded from the Hub when the model is constructed, so nothing about them is duplicated here.

Training

EgoExo4D, 6 epochs, batch 4 × 7 GPUs (DDP), constant-with-warmup lr 7e-5 (10% warmup), weight decay 0.01, grad-clip 1.0, bf16, seed 42. Video sampled at 4 fps, clips capped at 32 s / 128 frames, 384×384. Objective L = 1.0 · L_decomp + 0.5 · L_temp, InfoNCE all-gathered across ranks, sliding-shift augmentation on.

Per-clip view-invariant / view-variant captions are a provided input; recomposed captions are generated online by a local vLLM server running Qwen/Qwen3-1.7B. Full recipe in the code repo.

Limitations

Trained on EgoExo4D, which is skill-centric, mostly indoor, and recorded as ego/exo camera pairs. Domains far from that distribution are untested. Clips longer than 32 s are truncated to the first 128 sampled frames. The training pipeline needs decoupled view-invariant and view-variant captions, which most datasets do not ship.

Citation

• TODO: Add citation once published.

MIT licensed. The frozen backbones keep their own licenses.