File size: 3,390 Bytes
7633149
 
12f8395
 
 
 
 
 
 
7633149
12f8395
 
 
1a7cc0d
765503c
 
 
1a7cc0d
12f8395
 
 
765503c
12f8395
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
831483b
12f8395
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
765503c
12f8395
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
---
license: cc-by-nc-4.0
pipeline_tag: depth-estimation
tags:
  - monocular-geometry
  - point-map
  - surface-normals
  - instance-segmentation
  - diffusion-transformer
---

# PINE: Point, Instance and Normal Estimation in One-Step Diffusion Model

Zeyu Ren<sup>1</sup>, Jie Ni<sup>2✉︎</sup>, Rui Qian<sup>3✉︎</sup>, Jiahao Wei<sup>3</sup>, Hao Tang<sup>4</sup>, Yongtai Liu<sup>5</sup>, Zeyu Zhang<sup>4†</sup>

<sup>1</sup>The University of Melbourne · <sup>2</sup>Southeast University · <sup>3</sup>Fudan University · <sup>4</sup>Peking University · <sup>5</sup>Hanyang University

<sup>✉︎</sup>Corresponding authors · <sup>†</sup>Project lead

[Code](https://github.com/AIGeeksGroup/PINE) | [Website](https://aigeeksgroup.github.io/PINE/)

PINE is a one-step omni Diffusion Transformer that predicts the point map, the surface normals and the instance map of an
image in one forward pass. Each map is a token stream with its own input and output layers; the streams share the
transformer blocks, and a stream mask decides which streams exchange tokens. A camera-aware ray encoding positions
every token by its camera ray, and grounding attention adds the affinity of frozen DINOv3 features to the attention
logits.

## Models

All models run at `512×512` with one sampling step. Rel<sup>p</sup> is the point map error on seven zero-shot
datasets (DIODE, KITTI, NYUv2, ETH3D, HAMMER, iBims-1, Booster), with and without camera intrinsics.

| Model | File | DINOv3 encoder | Parameters | Rel<sup>p</sup> with K | Rel<sup>p</sup> without K |
| --- | --- | :---: | :---: | :---: | :---: |
| PINE-B | `pine-b-512.pth` (0.6 GB) | ViT-B/16 | 0.24 B | 4.69 | 5.62 |
| PINE-L | `pine-l-512.pth` (2.0 GB) | ViT-L/16 | 0.80 B | 3.76 | 4.70 |
| PINE-H | `pine-h-512.pth` (4.0 GB) | ViT-H+/16 | 1.85 B | 3.33 | 4.16 |

A file holds the EMA weights of the last training epoch. The frozen DINOv3 encoder is not part of the files: its
weights are gated. Request access on the [DINOv3 repository](https://github.com/facebookresearch/dinov3) and place
them in `pretrained/dinov3/`.

## Usage

```bash
git clone https://github.com/AIGeeksGroup/PINE.git && cd PINE
hf download AIGeeksGroup/PINE pine-l-512.pth --local-dir pretrained
IMAGE_DIR=assets/demo bash scripts/demo_l_512.sh
```

```python
import torch
from util.checkpoint import load_model

model = load_model('pretrained/pine-l-512.pth')            # one sampling step
image = ...                                                 # (1, 3, H, W) in [-1, 1], H and W multiples of 16
with torch.amp.autocast('cuda', dtype=torch.bfloat16):
    maps = model.generate(image.cuda(), intrinsics=None)   # (1, 9, H, W): point map, normals, instance map
points, normals, instances = maps.float().chunk(3, dim=1)
normals = torch.nn.functional.normalize(normals, dim=1)
```

See the [code repository](https://github.com/AIGeeksGroup/PINE) for installation, evaluation and training.

## License

The weights are for non-commercial use only, under the
[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. The code is released under the Apache
License 2.0.

## Citation

```bibtex
@article{ren2026pine,
  title={PINE: Point, Instance and Normal Estimation in One-Step Diffusion Model},
  author={Ren, Zeyu and Ni, Jie and Qian, Rui and Wei, Jiahao and Tang, Hao and Liu, Yongtai and Zhang, Zeyu},
  year={2026}
}
```