neuralangelo-main/DATA_PROCESSING.md

# Data Preparation

The following sections provide a step-by-step guide on how to convert a video to a json file that Neuralangelo parses.

## Prerequisites
Initialize the COLMAP submodule:
```bash
git submodule update --init --recursive
```

## Self-captured video sequence
To capture your own data, we recommend using a high-shutter speed to avoid motion blur (which is very common when using a phone camera). To follow the instructions below, you can download a toy example from the link: https://drive.google.com/file/d/1VJeWYNJEBK0MFIzHjI8xZzwf3_I3eLwH/view?usp=drive_link

### Preprocessing
You can run the following command to preprocess your data:

```bash
EXPERIMENT_NAME=toy_example
PATH_TO_VIDEO=toy_example.MOV
SKIP_FRAME_RATE=24
SCENE_TYPE=object  # {outdoor,indoor,object}
bash projects/neuralangelo/scripts/preprocess.sh ${EXPERIMENT_NAME} ${PATH_TO_VIDEO} ${SKIP_FRAME_RATE} ${SCENE_TYPE}
```

Alternatively, you can follow the steps below if you want more fine-grained control.

1. Convert video to images

    ```bash
    PATH_TO_VIDEO=toy_example.MOV
    SKIP_FRAME_RATE=30
    bash projects/neuralangelo/scripts/run_ffmpeg.sh ${PATH_TO_VIDEO} ${SKIP_FRAME_RATE}
    ```
    `PATH_TO_VIDEO`: path to video  
    `SKIP_FRAME_RATE`: downsampling rate (recommended 10 for 24 fps captured videos)

2. Run COLMAP

    ```bash
    PATH_TO_IMAGES=toy_example_skip30
    bash projects/neuralangelo/scripts/run_colmap.sh ${PATH_TO_IMAGES}
    ```
    `PATH_TO_IMAGES`: path to extracted images

    After COLMAP finishes, the folder structure will look like following:
    ```bash
    PATH_TO_IMAGES
    |__ database.db (COLMAP databse)
    |__ raw_images (raw input images)
    |__ dense
    |____ images (undistorted images)
    |____ sparse (COLMAP correspondences, intrinsics and sparse point cloud)
    |____ stereo (COLMAP files for MVS)
    ```
    `dense/images` will be the input for surface reconstruction.

3. Generate json file for data loading

    In this step, we define the bounding region for reconstruction and convert the COLMAP data to json format following Instant NGP. We strongly recommend you go to step 5 to validate the quality of the automatic bounding region extraction for improved performance.

    ```bash
    PATH_TO_IMAGES=toy_example_skip30
    SCENE_TYPE=object  # {outdoor,indoor,object}
    python3 projects/neuralangelo/scripts/convert_data_to_json.py --data_dir ${PATH_TO_IMAGES}/dense --scene_type ${SCENE_TYPE}
    ```
    `PATH_TO_IMAGES`: path to extracted images

4. Config files

    Use the following to configure and generate your config files
    ```bash
    EXPERIMENT_NAME=toy_example
    SCENE_TYPE=object  # {outdoor,indoor,object}
    python3 projects/neuralangelo/scripts/generate_config.py --experiment_name ${EXPERIMENT_NAME} --data_dir ${PATH_TO_IMAGES}/dense --scene_type ${SCENE_TYPE} --auto_exposure_wb
    ```
    The config file will be generated as `projects/neuralangelo/configs/custom/{EXPERIMENT_NAME}.yaml`.

    To find more arguments and how they work:
    ```bash
    python3 projects/neuralangelo/scripts/generate_config.py -h
    ```
    You can also manually adjust the parameters in the yaml file directly.

5. Inspect results in Blender (optional but recommended)

    For certain cases, the camera poses estimated by COLMAP could be wrong, and the bounding region estimation could be off.
    We offer some tools to to inspect the pre-processing results. Below are some options:

    - Blender: Download [Blender](https://www.blender.org/download/) and follow the instructions in our [add-on repo](https://github.com/mli0603/BlenderNeuralangelo).
    - This [Jupyter notebook](projects/neuralangelo/scripts/visualize_colmap.ipynb) (using K3D) can be helpful for visualizing the COLMAP results.

## DTU dataset
- Please use respecting the license terms of the dataset.

You can run the following command to download [the DTU dataset](https://roboimagedata.compute.dtu.dk/?page_id=36) that is preprocessed by NeuS authors and generate json files:
```bash
PATH_TO_DTU=datasets/dtu  # Modify this to be the DTU dataset root directory.
bash projects/neuralangelo/scripts/preprocess_dtu.sh ${PATH_TO_DTU}
```

## Tanks and Temples dataset
- Please use respecting the license terms of the dataset.

Download the data from [Tanks and Temples](https://tanksandtemples.org/download/) website.
You will also need to download additional [COLMAP/camera/alignment](https://drive.google.com/file/d/1jAr3IDvhVmmYeDWi0D_JfgiHcl70rzVE/view?resourcekey=) and the images of each scene.  
The file structure should look like (you may need to move around the downloaded images):
```
tanks_and_temples
├─ Barn
│  ├─ Barn_COLMAP_SfM.log (camera poses)
│  ├─ Barn.json (cropfiles)
│  ├─ Barn.ply (ground-truth point cloud)
│  ├─ Barn_trans.txt (colmap-to-ground-truth transformation)
│  └─ images (folder of images)
│     ├─ 000001.png
│     ├─ 000002.png
│     ...
├─ Caterpillar
│  ├─ ...
...
```
Run the following command to generate json files:
```bash
PATH_TO_TNT=datasets/tanks_and_temples  # Modify this to be the Tanks and Temples root directory.
bash projects/neuralangelo/scripts/preprocess_tnt.sh ${PATH_TO_TNT}
```