cyun9286 nielsr HF Staff commited on
Commit
93ae344
·
1 Parent(s): 9d9331d

Add model card for Track4World (#1)

Browse files

- Add model card for Track4World (7db70c06e9f1c3efd13c740f777570c243b56e9a)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +88 -0
README.md ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ pipeline_tag: other
4
+ tags:
5
+ - 3d-tracking
6
+ - video-understanding
7
+ - 4d-reconstruction
8
+ - computer-vision
9
+ ---
10
+
11
+ # Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
12
+
13
+ Track4World is a feedforward model for efficient holistic 3D tracking of every pixel in a world-centric coordinate system from a monocular video. Built on a global 3D scene representation, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs.
14
+
15
+ * **Paper:** [Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels](https://huggingface.co/papers/2603.02573)
16
+ * **Project Page:** [jiah-cloud.github.io/Track4World](https://jiah-cloud.github.io/Track4World.github.io/)
17
+ * **Repository:** [GitHub Repository](https://github.com/TencentARC/Track4World)
18
+
19
+ ---
20
+
21
+ ### 🖼️ Framework
22
+
23
+ Track4World estimates dense 3D scene flow of every pixel between arbitrary frame pairs from a monocular video in a global feedforward manner, enabling efficient and dense 3D tracking of every pixel in the world-centric coordinate system.
24
+
25
+ ---
26
+
27
+ ## ⚙️ Setup and Installation
28
+
29
+ ```bash
30
+ # Clone the repository with submodules
31
+ git clone --recursive https://github.com/TencentARC/Track4World.git
32
+ cd Track4World
33
+
34
+ # Create and activate environment
35
+ conda create -n track4world python=3.11
36
+ conda activate track4world
37
+
38
+ # Install PyTorch
39
+ pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
40
+
41
+ # Install dependencies
42
+ pip install -r requirements.txt
43
+ ```
44
+
45
+ Please refer to the [official GitHub README](https://github.com/TencentARC/Track4World) for detailed instructions on installing third-party modules and downloading weights.
46
+
47
+ ---
48
+
49
+ ## 🚀 Sample Usage
50
+
51
+ You can perform tracking and reconstruction on the provided demo video using the following commands:
52
+
53
+ ### First Frame 3D Tracking (`3d_ff`)
54
+
55
+ ```bash
56
+ python demo.py \
57
+ --mp4_path demo_data/cat.mp4 \
58
+ --mode 3d_ff \
59
+ --Ts -1 \
60
+ --save_base_dir results/cat
61
+ ```
62
+
63
+ ### Dense Tracking: Every Pixel, Every Frame (`3d_efep`)
64
+
65
+ ```bash
66
+ python demo.py \
67
+ --mp4_path demo_data/cat.mp4 \
68
+ --coordinate world_depthanythingv3 \
69
+ --mode 3d_efep \
70
+ --Ts -1 \
71
+ --ckpt_init checkpoints/track4world_da3.pth \
72
+ --save_base_dir results/cat
73
+ ```
74
+
75
+ ---
76
+
77
+ ## Citation
78
+
79
+ If you find Track4World useful for your research, please cite:
80
+
81
+ ```bibtex
82
+ @article{lu2026track4world,
83
+ title = {Track4World: Feedforward World-Centric Dense 3D Tracking of All Pixels},
84
+ author = {Jiahao Lu and Jiayi Xu and Wenbo Hu and Ruijie Zhu and Chengfeng Zhao and Sai-Kit Yeung and Ying Shan and Yuan Liu},
85
+ journal = {arXiv preprint arXiv:2603.02573},
86
+ year = {2026}
87
+ }
88
+ ```