Spaces:
Running
Running
updated requirements
Browse files- README.md +2 -87
- requirements.txt +2 -2
README.md
CHANGED
|
@@ -1,97 +1,12 @@
|
|
| 1 |
<div align="center">
|
| 2 |
-
<a href="#"><img src='https://img.shields.io/badge/-Paper-00629B?style=flat&logo=ieee&logoColor=white' alt='arXiv'></a>
|
| 3 |
<a href='https://realistic3d-miun.github.io/PVSDNet/'><img src='https://img.shields.io/badge/Project_Page-Website-green?logo=googlechrome&logoColor=white' alt='Project Page'></a>
|
| 4 |
-
<a href='https://huggingface.co/spaces/3ZadeSSG/PVSDNet'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo_(Coming_Soon)-blue'></a>
|
| 5 |
</div>
|
| 6 |
|
| 7 |
# PVSDNet: Joint Depth Prediction and View Synthesis via Shared Latent Spaces in Real-Time.
|
| 8 |
|
|
|
|
| 9 |
|
| 10 |
-
|
| 11 |
-
[](https://youtu.be/49s2UPvRA6I)
|
| 12 |
|
| 13 |
|
| 14 |
-
# 1. PVSDNet - Joint Depth and View
|
| 15 |
-
**Note:** Will be added soon.
|
| 16 |
|
| 17 |
-
## 1.A. Normal Inference (Recommended for minimal setup)
|
| 18 |
-
**Note:** Will be added soon.
|
| 19 |
-
|
| 20 |
-
## 2.A. Faster Inference (For best possible FPS)
|
| 21 |
-
**Note:** Will be added soon.
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
# 2. PVSDNet Depth-Only Model
|
| 26 |
-
This model is a variant of the original PVSDNet model, where we only predict depth and not the target views. The model core is similar except the rendering network and the positional encoding are removed.
|
| 27 |
-
|
| 28 |
-
* Download the checkpoints from following table and place them in `checkpoint_onnx` directory.
|
| 29 |
-
|
| 30 |
-
| Model | Size | Checkpoint |
|
| 31 |
-
|-----------------|--------|----------------|
|
| 32 |
-
| PVSDNet-Depth-Only | 1.11 GB| [Download](https://huggingface.co/3ZadeSSG/PVSDNet-Depth-Only/resolve/main/depth_only_model.pth) |
|
| 33 |
-
| PVSDNet-Depth-Only-Lite | 279 MB | [Download](https://huggingface.co/3ZadeSSG/PVSDNet-Depth-Only/resolve/main/depth_only_lite_model.pth) |
|
| 34 |
-
|
| 35 |
-
## 2.A. Normal Inference (Recommended for minimal setup)
|
| 36 |
-
|
| 37 |
-
## 2.B. Faster Inference (For best possible FPS)
|
| 38 |
-
You need to setup your own TRT Engine for this purpose.
|
| 39 |
-
|
| 40 |
-
* Make sure you modify the `depth_only_parameters` to set resolution you need. By default we have kept it at `384x384`.
|
| 41 |
-
|
| 42 |
-
* Run `export_onnx_depth.py` to conver the normal pytorch models located into into onnx
|
| 43 |
-
```
|
| 44 |
-
python export_onnx_depth.py
|
| 45 |
-
```
|
| 46 |
-
* Create TRT Engine directory
|
| 47 |
-
```
|
| 48 |
-
mkdir TRT_Engine
|
| 49 |
-
```
|
| 50 |
-
* Build the TRT engine based on created onnx files (which by default will be located in `checkpoint_onnx`)
|
| 51 |
-
```
|
| 52 |
-
trtexec --onnx=./checkpoint_onnx/depth_only_model.onnx --saveEngine=./TRT_Engine/depth_only_model_fp16.engine --fp16
|
| 53 |
-
```
|
| 54 |
-
```
|
| 55 |
-
trtexec --onnx=./checkpoint_onnx/depth_only_lite_model.onnx --saveEngine=./TRT_Engine/depth_only_lite_model_fp16.engine --fp16
|
| 56 |
-
```
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
## 2.C. Predicting on Depth Datasets using Multi-Resolution Fusion
|
| 60 |
-
|
| 61 |
-
We run the scripts inside the `depth_dataset_predictor` directory. There are two sample images for each dataset to test the code.
|
| 62 |
-
* First we build the TRT engine for each dataset as we use multi-resolution fusion.
|
| 63 |
-
```
|
| 64 |
-
python depth_dataset_predictor/build_trt_<dataset_name>.py
|
| 65 |
-
```
|
| 66 |
-
* Then we run the prediction script
|
| 67 |
-
```
|
| 68 |
-
python depth_dataset_predictor/predict_<dataset_name>_TensorRT.py
|
| 69 |
-
```
|
| 70 |
-
|
| 71 |
-
|Dataset|Setp 1|Step 2|
|
| 72 |
-
|---|---|---|
|
| 73 |
-
|ETH3D| ```python depth_dataset_predictor/build_trt_ETH3D.py``` | ```python depth_dataset_predictor/predict_ETH3D_TensorRT.py```|
|
| 74 |
-
|Sintel| ```python depth_dataset_predictor/build_trt_Sintel.py``` | ```python depth_dataset_predictor/predict_Sintel_TensorRT.py```|
|
| 75 |
-
|KITTI| ```python depth_dataset_predictor/build_trt_KITTI.py``` | ```python depth_dataset_predictor/predict_KITTI_TensorRT.py```|
|
| 76 |
-
|DIODE| ```python depth_dataset_predictor/build_trt_DIODE.py``` | ```python depth_dataset_predictor/predict_DIODE_TensorRT.py```|
|
| 77 |
-
|NYU| ```python depth_dataset_predictor/build_trt_NYU.py``` | ```python depth_dataset_predictor/predict_NYU_TensorRT.py```|
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
## 2.D. Predicting on 1080p In-The-Wild Images/Videos using Multi-Resolution Fusion
|
| 81 |
-
Similar to dataset, we can use the mutli-resolution fusion to predict on 1080p In-The-Wild Images/Videos.
|
| 82 |
-
|
| 83 |
-
* First we build the trt engine
|
| 84 |
-
```
|
| 85 |
-
python depth_in_wild_predictor/build_trt_1080p.py
|
| 86 |
-
```
|
| 87 |
-
* Then we run the prediction script for images
|
| 88 |
-
```
|
| 89 |
-
python depth_in_wild_predictor/predict_1080p_TensorRT.py
|
| 90 |
-
```
|
| 91 |
-
OR, run the prediction script for videos
|
| 92 |
-
```
|
| 93 |
-
python depth_in_wild_predictor/predict_video_1080p_TensorRT.py
|
| 94 |
-
```
|
| 95 |
-
#### Note
|
| 96 |
-
* For any other resolutions, you can modify the resolutions in these above scripts to suit your needs. We have kept the default resolution as 1080p for this example.
|
| 97 |
-
* We recommend 3-6 resolutions for best results, but you can use 1-2 smaller resolutions if working with low reoslution images/videos since receptive field of the network can handle that without any issues.
|
|
|
|
| 1 |
<div align="center">
|
|
|
|
| 2 |
<a href='https://realistic3d-miun.github.io/PVSDNet/'><img src='https://img.shields.io/badge/Project_Page-Website-green?logo=googlechrome&logoColor=white' alt='Project Page'></a>
|
|
|
|
| 3 |
</div>
|
| 4 |
|
| 5 |
# PVSDNet: Joint Depth Prediction and View Synthesis via Shared Latent Spaces in Real-Time.
|
| 6 |
|
| 7 |
+
Head to our [Project Page](https://realistic3d-miun.github.io/PVSDNet/) for more details about supplementary materials and full code.
|
| 8 |
|
| 9 |
+
* This space only contains the PVSDNet-Depth-Only Model.
|
|
|
|
| 10 |
|
| 11 |
|
|
|
|
|
|
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
requirements.txt
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
numpy
|
| 2 |
-
torch==2.9.1
|
| 3 |
-
torchvision==0.24.1
|
| 4 |
pytorch-msssim==1.0.0
|
| 5 |
pytorchvideo==0.1.5
|
| 6 |
gradio==6.2.0
|
|
|
|
| 1 |
numpy
|
| 2 |
+
torch==2.9.1
|
| 3 |
+
torchvision==0.24.1
|
| 4 |
pytorch-msssim==1.0.0
|
| 5 |
pytorchvideo==0.1.5
|
| 6 |
gradio==6.2.0
|