README.md

metadata

license: cc-by-nc-4.0
tags:
  - 3d-tracking
  - point-tracking
  - video
  - arxiv:2609.30222
pipeline_tag: keypoint-detection

TrackEverything

Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Project Page · Paper · Code

TrackEverything tracks dense 3D points across long video sequences. Given RGB frames along with depth and camera poses, it builds and maintains a persistent, de-duplicated 3D scene representation. Trajectories are refined in 3D across temporal windows, and points are classified as static or dynamic.

If depth and camera poses are not provided, they are estimated on the fly with feedforward 3D reconstruction models (VGGT-Ω or Pi3).

This repo holds the trained checkpoint (model.pth). Access is gated: request access above, and authenticate with hf auth login (older huggingface_hub: huggingface-cli login), or pass token= explicitly, before downloading.

Installation

Tested on Linux with Python 3.10, PyTorch 2.8, and CUDA 12.9.

git clone https://github.com/ayushjain1144/trackeverything.git
cd trackeverything

conda create -n trackeverything python=3.10 -y
conda activate trackeverything

pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu129
pip install torch_scatter -f https://data.pyg.org/whl/torch-2.8.0+cu129.html
pip install omegaconf hydra-core ipdb einops rich transformers wandb tqdm tensorboard iopath \
  numpy fsspec torchinfo h5py imageio imageio-ffmpeg matplotlib opencv-python viztracer pillow scipy \
  opt-einsum python-box line-profiler typed-argument-parser plyfile av rerun-sdk accelerate \
  huggingface_hub

# The repository directory is the package `trackeverything`; add its parent to PYTHONPATH:
export PYTHONPATH="$(dirname "$PWD"):$PYTHONPATH"

Inference needs nothing further; only training requires the deformable-attention CUDA extension (see the repository README for training setup).

Quick Start

from trackeverything.demo_hf import TrackEverything

# Load the checkpoint from the Hub (or pass a local .pth path)
model = TrackEverything.from_pretrained(
    "katefgroup/TrackEverything",
    device="cuda",
    geometry_backend="vggt_omega",  # default; pass "pi3" to switch
)

# Run on a video; depth and camera poses are estimated automatically
out = model.infer(video="path/to/video.mp4")

# Compact scene tracks: one track per distinct scene point, O(T) in sequence length
tracks = out.tracks()
print(tracks.coords.shape)        # (N, T, 3)   world-frame 3D positions
print(tracks.visibs.shape)        # (N, T)      visibility logits
print(tracks.dyn_logits.shape)    # (N, T)      static vs. dynamic logits
print(tracks.src_time.shape)      # (N,)        birth frame of each point
print(tracks.cell_to_token.shape) # (T, H', W') which point each grid cell belongs to

Entries before a point is born are set to -1000.

To expand back to per-pixel tracks (grows with T squared, so keep clips short):

dense_coords, dense_visibs, dense_dyn = tracks.to_dense()
# (1, T, H', W', T, 3), (1, T, H', W', T), (1, T, H', W', T)

Input Options

out = model.infer(video="path/to/video.mp4")        # video file
out = model.infer(video="path/to/frames/")          # folder of images
out = model.infer(video="https://example.com/v.mp4")# URL
out = model.infer(video=frames_array)               # (T, H, W, 3) uint8 or (T, 3, H, W) float [0,1]

# Temporal cropping / subsampling
out = model.infer(video="video.mp4", start_frame=10, frame_stride=3, max_frames=32)

Providing Your Own Geometry

Pass depths, intrinsics and extrinsics together to skip the geometry backend:

out = model.infer(
    video="path/to/video.mp4",
    depths=depths,          # (T, H, W) float32 numpy or torch
    intrinsics=intrinsics,  # (T, 3, 3)
    extrinsics=extrinsics,  # (T, 4, 4) world-to-camera
    conf_mask=conf_mask,    # optional (T, H, W) in [0, 1]
)

2D Query Points

Query points are given as [frame_index, x_pixel, y_pixel] and unprojected to 3D with the estimated or supplied geometry. They are never merged with other scene points, so they can retain higher resolution than the dense tracks.

out = model.infer(
    video="path/to/video.mp4",
    query_points_2d=[
        [0, 100, 200],   # frame 0, pixel (100, 200)
        [5, 150, 300],   # frame 5, pixel (150, 300)
    ],
)

print(out.query_coords.shape)  # (B, T, N, 3)
print(out.query_visibs.shape)  # (B, T, N)

Output

model.infer() returns a TrackEverythingOutput dataclass. Its native payload is prediction, a set of per-window track snapshots that grows linearly with T; the track views are built from it on demand.

Field	Shape	Description
prediction	—	Per-window snapshots the track views are built from (None in sparse mode)
query_coords	(B, T, N, 3)	Query point 3D positions (if query points were given)
query_visibs	(B, T, N)	Query point visibility (if query points were given)
norm_factor	float	Scene normalization factor applied to depths/extrinsics
depths	(T, H, W)	Preprocessed depth maps at inference resolution
conf_mask	(T, H, W)	Geometry confidence in [0, 1], or None
intrinsics	(T, 3, 3)	Camera intrinsics at inference resolution
extrinsics	(T, 4, 4)	Extrinsics after cam0-alignment and normalization
rgb	(T, 3, H, W)	Preprocessed RGB, float [0, 1]
dense_source_mask	(T_src, H', W')	Dense source validity mask
voxel_size, dynamic_threshold	float	Settings the model ran with

Derived views:

Accessor	Shape	Cost
out.tracks()	SceneTokens: coords (N, T, 3), visibs / dyn_logits (N, T), cell_to_token (T, H', W')	linear in T — preferred
out.coords	(B, T_src, H, W, T_tgt, 3)	built on first access, quadratic in T
out.visibs	(B, T_src, H, W, T_tgt)	built on first access, quadratic in T
out.dynamic_logits	(B, T_src, H, W, T_tgt)	built on first access, quadratic in T

License

Released under CC BY-NC 4.0: share and adapt for non-commercial purposes with attribution.

Citation

@misc{jain2026trackeverythinglonghorizondense,
      title={TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations},
      author={Ayush Jain and Sreeharsha Paruchuri and Ishita Gupta and Fan Zhang and Tanner Schmidt and Jakob Engel and Katerina Fragkiadaki and Adam W. Harley},
      year={2026},
      eprint={2609.30222},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.30222},
}