|
Download README.md from AIGeeksGroup/PINE: direct link, hf CLI and curl.
- Browser
- Download file 3.39 kB
-
https://huggingface.co/AIGeeksGroup/PINE/resolve/main/README.md
- Command line
-
hf download hf://AIGeeksGroup/PINE/README.md
-
curl -L -o README.md https://huggingface.co/AIGeeksGroup/PINE/resolve/main/README.md
3.39 kB
| license: cc-by-nc-4.0 | |
| pipeline_tag: depth-estimation | |
| tags: | |
| - monocular-geometry | |
| - point-map | |
| - surface-normals | |
| - instance-segmentation | |
| - diffusion-transformer | |
| # PINE: Point, Instance and Normal Estimation in One-Step Diffusion Model | |
| Zeyu Ren<sup>1</sup>, Jie Ni<sup>2✉︎</sup>, Rui Qian<sup>3✉︎</sup>, Jiahao Wei<sup>3</sup>, Hao Tang<sup>4</sup>, Yongtai Liu<sup>5</sup>, Zeyu Zhang<sup>4†</sup> | |
| <sup>1</sup>The University of Melbourne · <sup>2</sup>Southeast University · <sup>3</sup>Fudan University · <sup>4</sup>Peking University · <sup>5</sup>Hanyang University | |
| <sup>✉︎</sup>Corresponding authors · <sup>†</sup>Project lead | |
| [Code](https://github.com/AIGeeksGroup/PINE) | [Website](https://aigeeksgroup.github.io/PINE/) | |
| PINE is a one-step omni Diffusion Transformer that predicts the point map, the surface normals and the instance map of an | |
| image in one forward pass. Each map is a token stream with its own input and output layers; the streams share the | |
| transformer blocks, and a stream mask decides which streams exchange tokens. A camera-aware ray encoding positions | |
| every token by its camera ray, and grounding attention adds the affinity of frozen DINOv3 features to the attention | |
| logits. | |
| ## Models | |
| All models run at `512×512` with one sampling step. Rel<sup>p</sup> is the point map error on seven zero-shot | |
| datasets (DIODE, KITTI, NYUv2, ETH3D, HAMMER, iBims-1, Booster), with and without camera intrinsics. | |
| | Model | File | DINOv3 encoder | Parameters | Rel<sup>p</sup> with K | Rel<sup>p</sup> without K | | |
| | --- | --- | :---: | :---: | :---: | :---: | | |
| | PINE-B | `pine-b-512.pth` (0.6 GB) | ViT-B/16 | 0.24 B | 4.69 | 5.62 | | |
| | PINE-L | `pine-l-512.pth` (2.0 GB) | ViT-L/16 | 0.80 B | 3.76 | 4.70 | | |
| | PINE-H | `pine-h-512.pth` (4.0 GB) | ViT-H+/16 | 1.85 B | 3.33 | 4.16 | | |
| A file holds the EMA weights of the last training epoch. The frozen DINOv3 encoder is not part of the files: its | |
| weights are gated. Request access on the [DINOv3 repository](https://github.com/facebookresearch/dinov3) and place | |
| them in `pretrained/dinov3/`. | |
| ## Usage | |
| ```bash | |
| git clone https://github.com/AIGeeksGroup/PINE.git && cd PINE | |
| hf download AIGeeksGroup/PINE pine-l-512.pth --local-dir pretrained | |
| IMAGE_DIR=assets/demo bash scripts/demo_l_512.sh | |
| ``` | |
| ```python | |
| import torch | |
| from util.checkpoint import load_model | |
| model = load_model('pretrained/pine-l-512.pth') # one sampling step | |
| image = ... # (1, 3, H, W) in [-1, 1], H and W multiples of 16 | |
| with torch.amp.autocast('cuda', dtype=torch.bfloat16): | |
| maps = model.generate(image.cuda(), intrinsics=None) # (1, 9, H, W): point map, normals, instance map | |
| points, normals, instances = maps.float().chunk(3, dim=1) | |
| normals = torch.nn.functional.normalize(normals, dim=1) | |
| ``` | |
| See the [code repository](https://github.com/AIGeeksGroup/PINE) for installation, evaluation and training. | |
| ## License | |
| The weights are for non-commercial use only, under the | |
| [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. The code is released under the Apache | |
| License 2.0. | |
| ## Citation | |
| ```bibtex | |
| @article{ren2026pine, | |
| title={PINE: Point, Instance and Normal Estimation in One-Step Diffusion Model}, | |
| author={Ren, Zeyu and Ni, Jie and Qian, Rui and Wei, Jiahao and Tang, Hao and Liu, Yongtai and Zhang, Zeyu}, | |
| year={2026} | |
| } | |
| ``` | |