| --- |
| license: mit |
| library_name: pytorch |
| base_model: facebook/vjepa2-vitl-fpc64-256 |
| tags: |
| - world-model |
| - jepa |
| - v-jepa-2 |
| - sam-3 |
| - object-centric |
| - video-prediction |
| - physical-reasoning |
| - counterfactual-reasoning |
| - clevrer |
| --- |
| |
| # ObjectJEPA |
|
|
| **An object-centric world model that forecasts physical scenes in latent space, object by object, and answers "what if" and "what will happen next?" questions.** |
|
|
| Built for the Meta Global AI Developer Hackathon (2026), Indie Developer track, "Push the Frontier". |
| Code, evaluation and demos: [github.com/idrissa-gif/ObjectJEPA](https://github.com/idrissa-gif/ObjectJEPA) |
|
|
| A frozen V-JEPA 2 encoder turns each object (found by SAM 3) into a latent. The predictor here rolls those latents forward in time, without generating a single pixel. A learned collision head reads collisions out of the forecast, and a learned update turns "remove this object and roll out again" into the answer to a "what if" question. An LLM agent (Muse Spark) calls these as tools. |
|
|
| ## Files |
|
|
| | folder | model | size | format | |
| |---|---|---|---| |
| | `predictor/` | **Object predictor.** Transformer over one latent per object (8 slots, 1024-d, from V-JEPA 2 ViT-L) and its position. 32 observed frames → the next 32, rolled out step by step. Width 512, depth 8, 60 epochs. | 26.3M params | `predictor_best.pt`: state dict, latent mean/std, config. Loads with `torch.load(..., weights_only=True)`. `history.json`: training curve. | |
| | `collision_head/` | **Collision head.** Temporal convolution over pairs of object latents plus geometry → a contact logit per step for each of the 28 object pairs. Threshold 1.5. | 0.47M params | `head_best.pt`, `history.json` | |
| | `whatif_update/` | **What-if update.** Gradient-boosted classifier over the head's scores on the real video, the factual rollout and the counterfactual rollout. | 300 trees | `update.pkl`: scikit-learn 1.6 `HistGradientBoostingClassifier`, pickled. Unpickling runs code: load it only from a source you trust. | |
|
|
| The encoder and the segmenter are not included: the predictor works on features of the frozen [`facebook/vjepa2-vitl-fpc64-256`](https://huggingface.co/facebook/vjepa2-vitl-fpc64-256), pooled inside SAM 3 masks (or the CLEVRER masks), both used as they are. |
|
|
| ## Use |
|
|
| ```bash |
| git clone https://github.com/idrissa-gif/ObjectJEPA && cd ObjectJEPA |
| hf download Idrissa242/ObjectJEPA --local-dir weights |
| ``` |
|
|
| ```python |
| from whatif import load_model |
| from collision_head import load_head |
| from whatif_update import load_update |
| |
| model, mu, sd, cfg = load_model("weights/predictor") |
| head, thr = load_head("weights/collision_head") |
| update = load_update("weights/whatif_update") |
| ``` |
|
|
| The agent and the demos take the folders directly: `--ckpt weights/predictor --head weights/collision_head --update weights/whatif_update` (`agent.py`, `demo_live.py`, `demo_web.py`, `visualize.py`). Per-object features come from `sam_masks.py` and `extract_object_features.py`; the GitHub README has the full pipeline. |
|
|
| ## Training data |
|
|
| [CLEVRER](http://clevrer.csail.mit.edu/) (Yi, Gan, et al., ICLR 2020; CC0): 10,000 training videos of objects moving and colliding. 1,000 of them are held out **by video** for validation and never seen in training. |
|
|
| - Predictor: L1 loss in normalised latent space plus a position loss, on object features from the CLEVRER masks. It transfers to SAM 3 masks with almost no loss. |
| - Collision head: CLEVRER's annotated collisions, on both real and predicted sequences (8,000 videos; 1,000 other training videos pick the epoch and threshold). |
| - What-if update: the counterfactual questions of those 1,000 selection videos, never the validation videos. It sees only world-model signals, not shortcuts about the removed object's own contacts. |
|
|
| ## Results |
|
|
| On the 1,000 held-out validation videos: |
|
|
| | task | ObjectJEPA | baseline | |
| |---|---|---| |
| | object forecast: latent L1 / position error | **0.581 / 4.6 px** | 0.683 / 16.3 px (mean / constant velocity) | |
| | same, on SAM 3 masks (200 videos) | **0.589 / 4.3 px** | | |
| | collision F1 on forecast steps | **0.898** | 0.762 (distance rule) | |
| | "What will happen next?" (714 questions, chance 0.5) | **0.973** | 0.933 (constant velocity) | |
| | "What if X is removed?" (1,858 questions), per option / per question | **0.844 / 0.564** | 0.830 / 0.522 (no change) | |
|
|
| Full agent with no ground truth at all (Muse Spark calling the tools, SAM 3 masks, 200 validation videos): **0.966** on "what will happen next?" (148 questions) and **0.811 / 0.517** on "what if" (373 questions). |
|
|
| ## Limits |
|
|
| - Trained and tested on CLEVRER only: synthetic scenes, up to 8 objects, fixed camera. Not tested on real video. |
| - On "what if", most of the gain over the first version comes from reading the real video better. The counterfactual rollout itself adds about one point over answering "nothing changes". |
|
|
| ## Built during the hackathon |
|
|
| These weights were trained during the event (Oct 26 – Nov 4, 2026), with code written during the event. Used as they are, not trained here: V-JEPA 2, SAM 3 and Muse Spark by Meta, and the CLEVRER dataset. |
|
|
| ## License |
|
|
| MIT. V-JEPA 2, SAM 3 and CLEVRER keep their own licences. |
|
|
