3ZadeSSG commited on
Commit
a949be7
·
1 Parent(s): 4cfaf7d

updated requirements

Browse files
Files changed (2) hide show
  1. README.md +2 -87
  2. requirements.txt +2 -2
README.md CHANGED
@@ -1,97 +1,12 @@
1
  <div align="center">
2
- <a href="#"><img src='https://img.shields.io/badge/-Paper-00629B?style=flat&logo=ieee&logoColor=white' alt='arXiv'></a>
3
  <a href='https://realistic3d-miun.github.io/PVSDNet/'><img src='https://img.shields.io/badge/Project_Page-Website-green?logo=googlechrome&logoColor=white' alt='Project Page'></a>
4
- <a href='https://huggingface.co/spaces/3ZadeSSG/PVSDNet'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo_(Coming_Soon)-blue'></a>
5
  </div>
6
 
7
  # PVSDNet: Joint Depth Prediction and View Synthesis via Shared Latent Spaces in Real-Time.
8
 
 
9
 
10
- ## Supplementary Video (Head to Project Page for more visual results)
11
- [![Watch the video](https://img.youtube.com/vi/49s2UPvRA6I/maxresdefault.jpg)](https://youtu.be/49s2UPvRA6I)
12
 
13
 
14
- # 1. PVSDNet - Joint Depth and View
15
- **Note:** Will be added soon.
16
 
17
- ## 1.A. Normal Inference (Recommended for minimal setup)
18
- **Note:** Will be added soon.
19
-
20
- ## 2.A. Faster Inference (For best possible FPS)
21
- **Note:** Will be added soon.
22
-
23
-
24
-
25
- # 2. PVSDNet Depth-Only Model
26
- This model is a variant of the original PVSDNet model, where we only predict depth and not the target views. The model core is similar except the rendering network and the positional encoding are removed.
27
-
28
- * Download the checkpoints from following table and place them in `checkpoint_onnx` directory.
29
-
30
- | Model | Size | Checkpoint |
31
- |-----------------|--------|----------------|
32
- | PVSDNet-Depth-Only | 1.11 GB| [Download](https://huggingface.co/3ZadeSSG/PVSDNet-Depth-Only/resolve/main/depth_only_model.pth) |
33
- | PVSDNet-Depth-Only-Lite | 279 MB | [Download](https://huggingface.co/3ZadeSSG/PVSDNet-Depth-Only/resolve/main/depth_only_lite_model.pth) |
34
-
35
- ## 2.A. Normal Inference (Recommended for minimal setup)
36
-
37
- ## 2.B. Faster Inference (For best possible FPS)
38
- You need to setup your own TRT Engine for this purpose.
39
-
40
- * Make sure you modify the `depth_only_parameters` to set resolution you need. By default we have kept it at `384x384`.
41
-
42
- * Run `export_onnx_depth.py` to conver the normal pytorch models located into into onnx
43
- ```
44
- python export_onnx_depth.py
45
- ```
46
- * Create TRT Engine directory
47
- ```
48
- mkdir TRT_Engine
49
- ```
50
- * Build the TRT engine based on created onnx files (which by default will be located in `checkpoint_onnx`)
51
- ```
52
- trtexec --onnx=./checkpoint_onnx/depth_only_model.onnx --saveEngine=./TRT_Engine/depth_only_model_fp16.engine --fp16
53
- ```
54
- ```
55
- trtexec --onnx=./checkpoint_onnx/depth_only_lite_model.onnx --saveEngine=./TRT_Engine/depth_only_lite_model_fp16.engine --fp16
56
- ```
57
-
58
-
59
- ## 2.C. Predicting on Depth Datasets using Multi-Resolution Fusion
60
-
61
- We run the scripts inside the `depth_dataset_predictor` directory. There are two sample images for each dataset to test the code.
62
- * First we build the TRT engine for each dataset as we use multi-resolution fusion.
63
- ```
64
- python depth_dataset_predictor/build_trt_<dataset_name>.py
65
- ```
66
- * Then we run the prediction script
67
- ```
68
- python depth_dataset_predictor/predict_<dataset_name>_TensorRT.py
69
- ```
70
-
71
- |Dataset|Setp 1|Step 2|
72
- |---|---|---|
73
- |ETH3D| ```python depth_dataset_predictor/build_trt_ETH3D.py``` | ```python depth_dataset_predictor/predict_ETH3D_TensorRT.py```|
74
- |Sintel| ```python depth_dataset_predictor/build_trt_Sintel.py``` | ```python depth_dataset_predictor/predict_Sintel_TensorRT.py```|
75
- |KITTI| ```python depth_dataset_predictor/build_trt_KITTI.py``` | ```python depth_dataset_predictor/predict_KITTI_TensorRT.py```|
76
- |DIODE| ```python depth_dataset_predictor/build_trt_DIODE.py``` | ```python depth_dataset_predictor/predict_DIODE_TensorRT.py```|
77
- |NYU| ```python depth_dataset_predictor/build_trt_NYU.py``` | ```python depth_dataset_predictor/predict_NYU_TensorRT.py```|
78
-
79
-
80
- ## 2.D. Predicting on 1080p In-The-Wild Images/Videos using Multi-Resolution Fusion
81
- Similar to dataset, we can use the mutli-resolution fusion to predict on 1080p In-The-Wild Images/Videos.
82
-
83
- * First we build the trt engine
84
- ```
85
- python depth_in_wild_predictor/build_trt_1080p.py
86
- ```
87
- * Then we run the prediction script for images
88
- ```
89
- python depth_in_wild_predictor/predict_1080p_TensorRT.py
90
- ```
91
- OR, run the prediction script for videos
92
- ```
93
- python depth_in_wild_predictor/predict_video_1080p_TensorRT.py
94
- ```
95
- #### Note
96
- * For any other resolutions, you can modify the resolutions in these above scripts to suit your needs. We have kept the default resolution as 1080p for this example.
97
- * We recommend 3-6 resolutions for best results, but you can use 1-2 smaller resolutions if working with low reoslution images/videos since receptive field of the network can handle that without any issues.
 
1
  <div align="center">
 
2
  <a href='https://realistic3d-miun.github.io/PVSDNet/'><img src='https://img.shields.io/badge/Project_Page-Website-green?logo=googlechrome&logoColor=white' alt='Project Page'></a>
 
3
  </div>
4
 
5
  # PVSDNet: Joint Depth Prediction and View Synthesis via Shared Latent Spaces in Real-Time.
6
 
7
+ Head to our [Project Page](https://realistic3d-miun.github.io/PVSDNet/) for more details about supplementary materials and full code.
8
 
9
+ * This space only contains the PVSDNet-Depth-Only Model.
 
10
 
11
 
 
 
12
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
requirements.txt CHANGED
@@ -1,6 +1,6 @@
1
  numpy
2
- torch==2.9.1+cu130
3
- torchvision==0.24.1+cu130
4
  pytorch-msssim==1.0.0
5
  pytorchvideo==0.1.5
6
  gradio==6.2.0
 
1
  numpy
2
+ torch==2.9.1
3
+ torchvision==0.24.1
4
  pytorch-msssim==1.0.0
5
  pytorchvideo==0.1.5
6
  gradio==6.2.0