raivn/ObjectForesight-HOT3D-DiT

ObjectForesight-DiT (HOT3D)

📄 Paper (arXiv:2601.05237) · 🛠️ Code: RustinS/ObjectForesight · 🎥 Data: HOT3D (Meta)

The HOT3D variant of ObjectForesight, a 3D object-centric dynamics model that predicts H=8 future 6-DoF object poses from a single egocentric observation (scene point cloud + the object's recent pose context). This is the DiT (diffusion-transformer) model trained on HOT3D-Clips at frame-skip 4 (fs=4). It shares the exact architecture of the EPIC-KITCHENS model; only the training data differs.

Model

PoserV1 = PTv3 scene encoder (PointTransformer V3 / Sonata, 50.6M) + DiT diffusion temporal head (132.7M) → 183.25M params.

Encoder	PTv3, embed_dim=768, in_channels=6 (camera-xyz + object-centric-xyz), attn_obj pooling, voxel grid 0.005 m
Temporal head	DiT, 12 layers / 768-d / 12 heads, adaln_zero conditioning, cosine β-schedule, v-prediction, T=1000, 50 DDIM steps
Input	scene point cloud [N,3] (SpaTrackerV2 pinhole depth, voxel-downsampled to ~4096 pts) + context_len=3 frames of [t(3), rot6d(6)] + bbox + object-in-camera pose
Output	[H=8, 9] future poses: [t_x, t_y, t_z, rot6d(6)] per frame; 6D rotation to SO(3) via Gram-Schmidt
Training data	HOT3D-Clips (Aria egocentric RGB, motion-capture ground-truth object poses), frame_skips=4, stationary-window filtering
Checkpoint	epoch 194 / step 11k; raw (non-EMA) weights

Metrics

This is the fs=4 DiT model. See the paper for the HOT3D results table (ObjectForesight uses HOT3D as a second benchmark alongside EPIC-KITCHENS-100).

Files

File	Size	Description
model.safetensors	0.73 GB	Inference weights (raw, non-EMA), pickle-free. 183.25M params, fp32.
best.pt	0.73 GB	Same weights as a torch checkpoint (state_dict + training metrics) for the repo's loader.
config.yaml	—	Exact architecture + data recipe that defines this model.
architecture.png	—	Model diagram.

Weights are raw (non-EMA). An EMA shadow is tracked during training but is not used at evaluation/inference in the reference code (the loaders read the raw state_dict), so these raw weights are what reproduce the paper.

Usage

Weights-only release; the model definition and the HOT3D training/eval setup live in RustinS/ObjectForesight (see the "Training and evaluation on HOT3D" section of its README, and the hot3d config). The PTv3 encoder needs CUDA-compiled deps:

pip install spconv-cu124          # match your CUDA (e.g. 12.4)
pip install torch-scatter -f https://data.pyg.org/whl/torch-2.8.0+cu124.html
pip install flash-attn --no-build-isolation   # optional; falls back to SDPA if absent

Load the weights into PoserV1 (built from config.yaml):

import torch
from safetensors.torch import load_file
from src.models.poser_v1.builder import build_poser_v1   # from the objectforesight repo

model = build_poser_v1(**model_cfg)          # model_cfg from config.yaml (encoder + temporal)
sd = load_file("model.safetensors")          # raw inference weights
model.load_state_dict(sd, strict=False)      # only the tied `dit.*` alias is reported missing
model.eval().cuda()

with torch.no_grad():
    cond = model.condition_from_batch(batch)              # batch from the HOT3D dataset loader
    future = model.sample(cond["scene_pcd"], cond["context_vec"],
                          T_cam_anchor_obj=cond["T_cam_anchor_obj"],
                          steps=50, sampler="ddim", ctx_tokens_9d=ctx_9d)   # -> [B, 8, 9]

best.pt loads via the repo's own utilities with no missing/unexpected keys:

from src.models.poser_v1.utils.checkpoint import resolve_and_load_state_dict
sd, _ = resolve_and_load_state_dict("best.pt", map_location="cpu", prefer_ema=False)
model.load_state_dict(sd, strict=False)

Inputs come from HOT3DClipsDataset in the code repo, which produces the same batch contract as the EPIC loader (scene_pcd, context_init_9d, context_bbox_norm, context_T_cam_anchor_obj).

License & attribution

Released under CC BY-SA 4.0, matching the HOT3D components used to train it (HOT3D Aria/Quest sequence data and object-pose annotations are distributed by Meta under CC BY-SA). Attribution and ShareAlike apply: credit ObjectForesight and the HOT3D dataset, link the license, and license derivatives under CC BY-SA 4.0. See LICENSE.

This model does not use HOT3D "Hand data" (MANO/umetrack hand annotations), which Meta licenses under CC BY-NC-SA; the architecture has no hand-conditioning parameters. Comply with the HOT3D Dataset License Agreement for any HOT3D data you obtain. Do not use this model to identify or infer private information about individuals depicted in the source video.

Citation

@article{soraki2026objectforesight,
  title   = {ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos},
  author  = {Soraki, Rustin and Bharadhwaj, Homanga and Farhadi, Ali and Mottaghi, Roozbeh},
  journal = {arXiv preprint arXiv:2601.05237},
  year    = {2026}
}
@article{banerjee2024hot3d,
  title   = {HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos},
  author  = {Banerjee, Prithviraj and Shkodrani, Sindi and Moulon, Pierre and Hampali, Shreyas and
             Han, Shangchen and Zhang, Fan and Zhang, Linguang and Fountain, Jade and Miller, Edward and
             Basol, Selen and Newcombe, Richard and Wang, Robert and Engel, Jakob Julian and Hodan, Tomas},
  journal = {arXiv preprint arXiv:2411.19167},
  year    = {2024}
}

Built with (please also cite): PointTransformer V3 / Sonata · HOT3D (Meta). See the code repository for full references.