| <<<<<<< HEAD |
| # LASA |
| Repository of LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset accepted by CVPR 2024 |
|
|
| [<a href="https://arxiv.org/abs/2312.12418">paper</a>]/[<a href="https://gap-lab-cuhk-sz.github.io/LASA/">Project page</a>] |
|
|
| ## Demo Results |
|  |
|  |
|  |
|
|
| ## Dataset |
| Complete raw data will be released soon. |
|
|
| ## Download preprocessed data and processing |
| Download the preprocessed data from <a href="https://pan.baidu.com/s/10jabZCSoTP4Yu1Eu3_twFg?pwd=z64v"> |
| BaiduYun (code: z64v)<a/>. (Currently it only contains chair category, it will be fully released after the cleaning process) Put all the downloaded data under LASA, unzip the align_mat_all.zip mannually. |
| You can choose to use the script ./process_scripts/unzip_all_data to unzip all the data in occ_data and other_data by following commands: |
| ```angular2html |
| cd process_scripts |
| python unzip_all_data.py --unzip_occ --unzip_other |
| ``` |
| Run the following commands to generate augmented partial point cloud for synthetic dataset and LASA dataset |
| ```angular2html |
| cd process_scripts |
| python augment_arkit_partial_point.py --cat arkit_chair arkit_stool ... |
| python augment_synthetic_partial_point.py --cat 03001627 future_chair ABO_chair future_stool ... |
| ``` |
| Run the following command to extract image features |
| ```angular2html |
| cd process_scripts |
| bash dist_extract_vit.sh |
| ``` |
| Finally, run the following command to generate train/val splits, please check ./dataset/taxonomy for the sub-cateory definition: |
| ```angular2html |
| cd process_scripts |
| python generate_split_for_arkit --cat arkit_chair arkit_stool ... |
| python generate_split_for_synthetic_data.py --cat 03001627 future_chair ABO_chair future_stool ... |
| ``` |
| |
| ## Evaluation |
| Download the pretrained weight for chair from <a href="https://pan.baidu.com/s/10liUOaC4CXGn7bN6SQkZsw?pwd=hlf9"> chair_checkpoint.<a/> (code:hlf9). |
| Put these folder under LASA/output.<br> The ae folder stores the VAE weight, dm folder stores the diffusion model trained on synthetic data. |
| finetune_dm folder stores the diffusion model finetuned on LASA dataset. |
| Run the following commands to evaluate and extract the mesh: |
| ```angular2html |
| cd evaluation |
| bash dist_eval.sh |
| ``` |
| The category entries are the sub-category from arkit scenes, please see ./datasets/taxonomy.py about how they are defined. |
| For example, if you want to evaluate on LASA's chair, category should contain both arkit_chair and arkit_stool. |
| make sure the --ae-pth and --dm-pth entry points to the correct checkpoint path. If you are evaluating on LASA, |
| make sure the --dm-pth points to the finetuned weight in the ./output/finetune_dm folder. The result will be saved |
| under ./output_result. |
| |
| ## Training |
| Run the <strong>train_VAE.sh</strong> to train the VAE model. If you aims to train on one category, just specify one category from <strong> chair, |
| cabinet, table, sofa, bed, shelf</strong>. Inputting <strong>all</strong> will train on all categories. Makes sure to download and preprocess all |
| the required sub-category data. The sub-category arrangement can be found in ./datasets/taxonomy.py <br> |
| After finish training the VAE model, run the following commands to pre-extract the VAE features for every object: |
| ```angular2html |
| cd process_scripts |
| bash dist_export_triplane_features.sh |
| ``` |
| Then, we can start training the diffusion model on the synthetic dataset by running the <strong>train_diffusion.sh</strong>.<br> |
| Finally, finetune the diffusion model on LASA dataset by running <strong> finetune_diffusion.sh</strong>. <br><br> |
|
|
| Early stopping is used by mannualy stopping the training by 150 epochs and 500 epochs for training VAE model and diffusion model respetively. |
| All experiments in the paper are conducted on 8 A100 GPUs with batch size = 22. |
| ## TODO |
|
|
| - [ ] Object Detection Code |
| - [ ] Code for Demo on both arkitscene and in the wild data |
|
|
| ## Citation |
| ``` |
| @article{liu2023lasa, |
| title={LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset}, |
| author={Liu, Haolin and Ye, Chongjie and Nie, Yinyu and He, Yingfan and Han, Xiaoguang}, |
| journal={arXiv preprint arXiv:2312.12418}, |
| year={2023} |
| } |
| ``` |
| ======= |
| --- |
| title: LASA |
| emoji: 🏃 |
| colorFrom: indigo |
| colorTo: red |
| sdk: gradio |
| sdk_version: 4.32.2 |
| app_file: app.py |
| pinned: false |
| license: mit |
| --- |
|
|
| Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
| >>>>>>> hf/main |
|
|
