AllTracker (fastforks)

AllTracker (Harley et al., ICCV 2025): tracking every pixel of a query frame through a video offline, as long-range optical flow. A ConvNeXt (tiny: a residual CNN) encodes every frame at 1/8; 16-frame windows (stride 8) refine the flow and visibility of all anchor pixels to all window frames at once, with a correlation pyramid, 2D ConvNeXt blocks over space and 1D blocks over time, then an 8x convex upsampling. Point queries read the dense result.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/aharley/alltracker at e7553135e7b3.

Author weight repos: https://huggingface.co/aharley/alltracker.

Task: tracking. Tier: experimental. Outputs: tracks, visibility. Backends: portable, kernels.

Input: a video (its frames in order) and query points (t, x, y). Output: the position of every query in every frame and the probability that it is visible there.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
alltracker@1 alltracker.pth ffd9ebcfb6d2 MIT yes
alltracker-tiny@1 alltracker_tiny.pth 43991b4fcee5 MIT yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: MIT. Licence files: LICENSE.

Training data: Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), FlyingChairs (Freiburg), Spring (scene flow / optical flow / stereo, University of Stuttgart), HCI HD1K optical flow benchmark, KITTI stereo / optical flow 2012 / 2015, TartanAir, not stated by the authors (Upstream README: stage 1 on Kubric MOVi-F, stage 2 on a mix of point tracking data (Kubric, PointOdyssey, Dynamic Replica and point tracks made from FlyingThings3D, Monkaa, Driving, SPRING) and optical flow data (FlyingChairs, FlyingThings3D, Monkaa, Driving, AutoFlow, SPRING, VIPER, HD1K, KITTI, TartanAir). Kubric, PointOdyssey, Dynamic Replica, AutoFlow and VIPER have no dataset entry here yet ("unstated").).

Licence evidence: https://huggingface.co/aharley/alltracker

Input size

any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): alltracker 3000x4000 (5.5 GiB peak allocated, 5.9 reserved); alltracker-tiny 3000x4000 (5.5 GiB peak allocated, 5.9 reserved)

Speed and memory

Tracking, synthetic, 480x854

model time (ms): original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
alltracker 485 391 221 2.2x 1.8x 2100 / 2318 993 / 1238 1076 / 1288
alltracker-tiny 424 330 217 2.0x 1.5x 1995 / 2200 919 / 1178 978 / 1166

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
alltracker 1.68 0.22 0.98 1.31 0.85 0.45
alltracker-tiny 1.22 0.06 0.96 1.40 0.73 0.54

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
alltracker 480x854 391 ms 284 ms planned for 0.2.1 221 ms 105 s – 993 / 1238 1273 / 1358 1076 / 1288 –
alltracker-tiny 480x854 330 ms 229 ms planned for 0.2.1 217 ms 103 s – 919 / 1178 1198 / 1338 978 / 1166 –

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
alltracker TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % 67.45 / 78.66 / 90.82 67.41 / 78.65 / 90.69 67.42 / 78.71 / 90.70 67.38 / 78.63 / 90.65
alltracker TAP-Vid DAVIS (query first), AJ / delta_avg / OA % 64.17 / 77.00 / 89.49 64.14 / 77.04 / 89.44 64.15 / 77.04 / 89.44
alltracker-tiny TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % 65.33 / 76.69 / 88.14 65.36 / 76.66 / 88.14 65.34 / 76.66 / 88.10

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0019 and p99 <= 0.0099 (absolute error of the output tracks).

Limitations

  • Offline: the features of every frame are kept in memory (about 2.5 MB per frame in float16 at 384 x 512); each query frame costs a forward pass (and a backward pass from frame 0 if it is not the first).
  • Frames are resized to 384 x 512 (upstream's default evaluation size), whatever their size and aspect ratio. Upstream reports higher accuracy at larger sizes (TAP-Vid DAVIS AJ 65.9 at 448 x 768, 67.2 at 768 x 1024); the size is a card parameter.
  • Tracks are sampled bilinearly at the query position; upstream's evaluation rounds the query to the nearest pixel.
  • One video per call; all queries of a call share the video.

Usage

from fastforks import tracking

model = tracking.load("alltracker")  # mode="auto": the preferred backend from recorded measurements
result = model(frames, [[0, x, y]])  # frames: the video's images in order; queries (t, x, y)
result.tracks  # [T, Q, 2] float32: x, y in input pixels; result.visible [T, Q] bool
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: alltracker, alltracker-tiny.

Kernels backend

tracking.load("alltracker", mode="kernels") runs our CUDA kernels (fvm_conv). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for fastforks/alltracker

Finetuned
(1)
this model

Collection including fastforks/alltracker