---
license: mit
library_name: pytorch
base_model: facebook/vjepa2-vitl-fpc64-256
tags:
- world-model
- jepa
- v-jepa-2
- sam-3
- object-centric
- video-prediction
- physical-reasoning
- counterfactual-reasoning
- clevrer
---

# ObjectJEPA

**An object-centric world model that forecasts physical scenes in latent space, object by object, and answers "what if" and "what will happen next?" questions.**

Built for the Meta Global AI Developer Hackathon (2026), Indie Developer track, "Push the Frontier".
Code, evaluation and demos: [github.com/idrissa-gif/ObjectJEPA](https://github.com/idrissa-gif/ObjectJEPA)

A frozen V-JEPA 2 encoder turns each object (found by SAM 3) into a latent. The predictor here rolls those latents forward in time, without generating a single pixel. A learned collision head reads collisions out of the forecast, and a learned update turns "remove this object and roll out again" into the answer to a "what if" question. An LLM agent (Muse Spark) calls these as tools.

## Files

| folder | model | size | format |
|---|---|---|---|
| `predictor/` | **Object predictor.** Transformer over one latent per object (8 slots, 1024-d, from V-JEPA 2 ViT-L) and its position. 32 observed frames → the next 32, rolled out step by step. Width 512, depth 8, 60 epochs. | 26.3M params | `predictor_best.pt`: state dict, latent mean/std, config. Loads with `torch.load(..., weights_only=True)`. `history.json`: training curve. |
| `collision_head/` | **Collision head.** Temporal convolution over pairs of object latents plus geometry → a contact logit per step for each of the 28 object pairs. Threshold 1.5. | 0.47M params | `head_best.pt`, `history.json` |
| `whatif_update/` | **What-if update.** Gradient-boosted classifier over the head's scores on the real video, the factual rollout and the counterfactual rollout. | 300 trees | `update.pkl`: scikit-learn 1.6 `HistGradientBoostingClassifier`, pickled. Unpickling runs code: load it only from a source you trust. |

The encoder and the segmenter are not included: the predictor works on features of the frozen [`facebook/vjepa2-vitl-fpc64-256`](https://huggingface.co/facebook/vjepa2-vitl-fpc64-256), pooled inside SAM 3 masks (or the CLEVRER masks), both used as they are.

## Use

```bash
git clone https://github.com/idrissa-gif/ObjectJEPA && cd ObjectJEPA
hf download Idrissa242/ObjectJEPA --local-dir weights
```

```python
from whatif import load_model
from collision_head import load_head
from whatif_update import load_update

model, mu, sd, cfg = load_model("weights/predictor")
head, thr = load_head("weights/collision_head")
update = load_update("weights/whatif_update")
```

The agent and the demos take the folders directly: `--ckpt weights/predictor --head weights/collision_head --update weights/whatif_update` (`agent.py`, `demo_live.py`, `demo_web.py`, `visualize.py`). Per-object features come from `sam_masks.py` and `extract_object_features.py`; the GitHub README has the full pipeline.

## Training data

[CLEVRER](http://clevrer.csail.mit.edu/) (Yi, Gan, et al., ICLR 2020; CC0): 10,000 training videos of objects moving and colliding. 1,000 of them are held out **by video** for validation and never seen in training.

- Predictor: L1 loss in normalised latent space plus a position loss, on object features from the CLEVRER masks. It transfers to SAM 3 masks with almost no loss.
- Collision head: CLEVRER's annotated collisions, on both real and predicted sequences (8,000 videos; 1,000 other training videos pick the epoch and threshold).
- What-if update: the counterfactual questions of those 1,000 selection videos, never the validation videos. It sees only world-model signals, not shortcuts about the removed object's own contacts.

## Results

On the 1,000 held-out validation videos:

| task | ObjectJEPA | baseline |
|---|---|---|
| object forecast: latent L1 / position error | **0.581 / 4.6 px** | 0.683 / 16.3 px (mean / constant velocity) |
| same, on SAM 3 masks (200 videos) | **0.589 / 4.3 px** | |
| collision F1 on forecast steps | **0.898** | 0.762 (distance rule) |
| "What will happen next?" (714 questions, chance 0.5) | **0.973** | 0.933 (constant velocity) |
| "What if X is removed?" (1,858 questions), per option / per question | **0.844 / 0.564** | 0.830 / 0.522 (no change) |

Full agent with no ground truth at all (Muse Spark calling the tools, SAM 3 masks, 200 validation videos): **0.966** on "what will happen next?" (148 questions) and **0.811 / 0.517** on "what if" (373 questions).

## Limits

- Trained and tested on CLEVRER only: synthetic scenes, up to 8 objects, fixed camera. Not tested on real video.
- On "what if", most of the gain over the first version comes from reading the real video better. The counterfactual rollout itself adds about one point over answering "nothing changes".

## Built during the hackathon

These weights were trained during the event (Oct 26 – Nov 4, 2026), with code written during the event. Used as they are, not trained here: V-JEPA 2, SAM 3 and Muse Spark by Meta, and the CLEVRER dataset.

## License

MIT. V-JEPA 2, SAM 3 and CLEVRER keep their own licences.