--- license: cc-by-nc-4.0 pipeline_tag: depth-estimation tags: - monocular-geometry - point-map - surface-normals - instance-segmentation - diffusion-transformer --- # PINE: Point, Instance and Normal Estimation in One-Step Diffusion Model Zeyu Ren1, Jie Ni2✉︎, Rui Qian3✉︎, Jiahao Wei3, Hao Tang4, Yongtai Liu5, Zeyu Zhang4† 1The University of Melbourne · 2Southeast University · 3Fudan University · 4Peking University · 5Hanyang University ✉︎Corresponding authors · †Project lead [Code](https://github.com/AIGeeksGroup/PINE) | [Website](https://aigeeksgroup.github.io/PINE/) PINE is a one-step omni Diffusion Transformer that predicts the point map, the surface normals and the instance map of an image in one forward pass. Each map is a token stream with its own input and output layers; the streams share the transformer blocks, and a stream mask decides which streams exchange tokens. A camera-aware ray encoding positions every token by its camera ray, and grounding attention adds the affinity of frozen DINOv3 features to the attention logits. ## Models All models run at `512×512` with one sampling step. Relp is the point map error on seven zero-shot datasets (DIODE, KITTI, NYUv2, ETH3D, HAMMER, iBims-1, Booster), with and without camera intrinsics. | Model | File | DINOv3 encoder | Parameters | Relp with K | Relp without K | | --- | --- | :---: | :---: | :---: | :---: | | PINE-B | `pine-b-512.pth` (0.6 GB) | ViT-B/16 | 0.24 B | 4.69 | 5.62 | | PINE-L | `pine-l-512.pth` (2.0 GB) | ViT-L/16 | 0.80 B | 3.76 | 4.70 | | PINE-H | `pine-h-512.pth` (4.0 GB) | ViT-H+/16 | 1.85 B | 3.33 | 4.16 | A file holds the EMA weights of the last training epoch. The frozen DINOv3 encoder is not part of the files: its weights are gated. Request access on the [DINOv3 repository](https://github.com/facebookresearch/dinov3) and place them in `pretrained/dinov3/`. ## Usage ```bash git clone https://github.com/AIGeeksGroup/PINE.git && cd PINE hf download AIGeeksGroup/PINE pine-l-512.pth --local-dir pretrained IMAGE_DIR=assets/demo bash scripts/demo_l_512.sh ``` ```python import torch from util.checkpoint import load_model model = load_model('pretrained/pine-l-512.pth') # one sampling step image = ... # (1, 3, H, W) in [-1, 1], H and W multiples of 16 with torch.amp.autocast('cuda', dtype=torch.bfloat16): maps = model.generate(image.cuda(), intrinsics=None) # (1, 9, H, W): point map, normals, instance map points, normals, instances = maps.float().chunk(3, dim=1) normals = torch.nn.functional.normalize(normals, dim=1) ``` See the [code repository](https://github.com/AIGeeksGroup/PINE) for installation, evaluation and training. ## License The weights are for non-commercial use only, under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. The code is released under the Apache License 2.0. ## Citation ```bibtex @article{ren2026pine, title={PINE: Point, Instance and Normal Estimation in One-Step Diffusion Model}, author={Ren, Zeyu and Ni, Jie and Qian, Rui and Wei, Jiahao and Tang, Hao and Liu, Yongtai and Zhang, Zeyu}, year={2026} } ```